Measuring Adoption of Patient Priorities-Aligned Care Using Natural Language Processing of Electronic Health Records: Development and Validation of the Model.
NLP
decision support
geriatric decision support system
machine learning
natural language processing
pattern recognition
social work note
Journal
JMIR medical informatics
ISSN: 2291-9694
Titre abrégé: JMIR Med Inform
Pays: Canada
ID NLM: 101645109
Informations de publication
Date de publication:
19 Feb 2021
19 Feb 2021
Historique:
received:
16
03
2020
accepted:
17
12
2020
revised:
16
11
2020
entrez:
19
2
2021
pubmed:
20
2
2021
medline:
20
2
2021
Statut:
epublish
Résumé
Patient Priorities Care (PPC) is a model of care that aligns health care recommendations with priorities of older adults who have multiple chronic conditions. Following identification of patient priorities, this information is documented in the patient's electronic health record (EHR). Our goal is to develop and validate a natural language processing (NLP) model that reliably documents when clinicians identify patient priorities (ie, values, outcome goals, and care preferences) within the EHR as a measure of PPC adoption. This is a retrospective analysis of unstructured National Veteran Health Administration EHR free-text notes using an NLP model. The data were sourced from 778 patient notes of 658 patients from encounters with 144 social workers in the primary care setting. Each patient's free-text clinical note was reviewed by 2 independent reviewers for the presence of PPC language such as priorities, values, and goals. We developed an NLP model that utilized statistical machine learning approaches. The performance of the NLP model in training and validation with 10-fold cross-validation is reported via accuracy, recall, and precision in comparison to the chart review. Of 778 notes, 589 (75.7%) were identified as containing PPC language (kappa=0.82, P<.001). The NLP model in the training stage had an accuracy of 0.98 (95% CI 0.98-0.99), a recall of 0.98 (95% CI 0.98-0.99), and precision of 0.98 (95% CI 0.97-1.00). The NLP model in the validation stage had an accuracy of 0.92 (95% CI 0.90-0.94), recall of 0.84 (95% CI 0.79-0.89), and precision of 0.84 (95% CI 0.77-0.91). In contrast, an approach using simple search terms for PPC only had a precision of 0.757. An automated NLP model can reliably measure with high precision, recall, and accuracy when clinicians document patient priorities as a key step in the adoption of PPC.
Sections du résumé
BACKGROUND
BACKGROUND
Patient Priorities Care (PPC) is a model of care that aligns health care recommendations with priorities of older adults who have multiple chronic conditions. Following identification of patient priorities, this information is documented in the patient's electronic health record (EHR).
OBJECTIVE
OBJECTIVE
Our goal is to develop and validate a natural language processing (NLP) model that reliably documents when clinicians identify patient priorities (ie, values, outcome goals, and care preferences) within the EHR as a measure of PPC adoption.
METHODS
METHODS
This is a retrospective analysis of unstructured National Veteran Health Administration EHR free-text notes using an NLP model. The data were sourced from 778 patient notes of 658 patients from encounters with 144 social workers in the primary care setting. Each patient's free-text clinical note was reviewed by 2 independent reviewers for the presence of PPC language such as priorities, values, and goals. We developed an NLP model that utilized statistical machine learning approaches. The performance of the NLP model in training and validation with 10-fold cross-validation is reported via accuracy, recall, and precision in comparison to the chart review.
RESULTS
RESULTS
Of 778 notes, 589 (75.7%) were identified as containing PPC language (kappa=0.82, P<.001). The NLP model in the training stage had an accuracy of 0.98 (95% CI 0.98-0.99), a recall of 0.98 (95% CI 0.98-0.99), and precision of 0.98 (95% CI 0.97-1.00). The NLP model in the validation stage had an accuracy of 0.92 (95% CI 0.90-0.94), recall of 0.84 (95% CI 0.79-0.89), and precision of 0.84 (95% CI 0.77-0.91). In contrast, an approach using simple search terms for PPC only had a precision of 0.757.
CONCLUSIONS
CONCLUSIONS
An automated NLP model can reliably measure with high precision, recall, and accuracy when clinicians document patient priorities as a key step in the adoption of PPC.
Identifiants
pubmed: 33605893
pii: v9i2e18756
doi: 10.2196/18756
pmc: PMC7935648
doi:
Types de publication
Journal Article
Langues
eng
Pagination
e18756Subventions
Organisme : NHLBI NIH HHS
ID : K25 HL152006
Pays : United States
Informations de copyright
©Javad Razjouyan, Jennifer Freytag, Lilian Dindo, Lea Kiefer, Edward Odom, Jaime Halaszynski, Jennifer W Silva, Aanand D Naik. Originally published in JMIR Medical Informatics (http://medinform.jmir.org), 19.02.2021.
Références
J Am Med Inform Assoc. 2019 Apr 1;26(4):364-379
pubmed: 30726935
J Am Geriatr Soc. 2019 Jul;67(7):1379-1385
pubmed: 30844080
J Biomed Inform. 2006 Dec;39(6):589-99
pubmed: 16359928
J Aging Health. 2018 Jun;30(5):778-799
pubmed: 28553806
Clin Geriatr Med. 2016 May;32(2):261-75
pubmed: 27113145
J Am Geriatr Soc. 2020 Sep;68(9):2112-2116
pubmed: 32687218
BioData Min. 2017 Dec 8;10:35
pubmed: 29234465
Database (Oxford). 2019 Jan 1;2019:
pubmed: 31603193
J Am Geriatr Soc. 2016 Mar;64(3):625-31
pubmed: 27000335
JAMA Intern Med. 2019 Oct 7;:
pubmed: 31589281
Health Expect. 2016 Jun;19(3):679-90
pubmed: 25645124
J Am Geriatr Soc. 2018 Oct;66(10):2009-2016
pubmed: 30281777
Am J Epidemiol. 2014 Mar 15;179(6):749-58
pubmed: 24488511
J Biomed Inform. 2015 Dec;58 Suppl:S171-82
pubmed: 26375492
J Biomed Inform. 2018 Jul;83:73-86
pubmed: 29860093
J Am Geriatr Soc. 2018 Oct;66(10):1872-1879
pubmed: 30281794
BMC Med Inform Decis Mak. 2017 Dec 01;17(1):155
pubmed: 29191207
J Biomed Inform. 2018 Jan;77:34-49
pubmed: 29162496