The revised Approved Instructional Resources score: An improved quality evaluation tool for online educational resources.
Journal
AEM education and training
ISSN: 2472-5390
Titre abrégé: AEM Educ Train
Pays: United States
ID NLM: 101722142
Informations de publication
Date de publication:
Jul 2021
Jul 2021
Historique:
received:
28
01
2021
revised:
23
03
2021
accepted:
26
03
2021
entrez:
18
6
2021
pubmed:
19
6
2021
medline:
19
6
2021
Statut:
epublish
Résumé
Free Open-Access Medical education (FOAM) use among residents continues to rise. However, it often lacks quality assurance processes and residents receive little guidance on quality assessment. The Academic Life in Emergency Medicine Approved Instructional Resources tool (AAT) was created for FOAM appraisal by and for expert educators and has demonstrated validity in this context. It has yet to be evaluated in other populations. We assessed the AAT's usability in a diverse population of practicing emergency medicine (EM) physicians, residents, and medical students; solicited feedback; and developed a revised tool. As part of the Medical Education Translational Resources: Impact and Quality (METRIQ) study, we recruited medical students, EM residents, and EM attendings to evaluate five FOAM posts with the AAT and provide quantitative and qualitative feedback via an online survey. Two independent analysts performed a qualitative thematic analysis with discrepancies resolved through discussion and negotiated consensus. This analysis informed development of an initial revised AAT, which was then further refined after pilot testing among the author group. The final tool was reassessed for reliability. Of 330 recruited international participants, 309 completed all ratings. The Best Evidence in Emergency Medicine (BEEM) score was the component most frequently reported as difficult to use. Several themes emerged from the qualitative analysis: for ease of use-understandable, logically structured, concise, and aligned with educational value. Limitations include deviation from questionnaire best practices, validity concerns, and challenges assessing evidence-based medicine. Themes supporting its use include evaluative utility and usability. The author group pilot tested the initial revised AAT, revealing a total score average measure intraclass correlation coefficient (ICC) of moderate reliability (ICC = 0.68, 95% confidence interval [CI] = 0 to 0.962). The final AAT's average measure ICC was 0.88 (95% CI = 0.77 to 0.95). We developed the final revised AAT from usability feedback. The new score has significantly increased usability, but will need to be reassessed for reliability in a broad population.
Sections du résumé
BACKGROUND
BACKGROUND
Free Open-Access Medical education (FOAM) use among residents continues to rise. However, it often lacks quality assurance processes and residents receive little guidance on quality assessment. The Academic Life in Emergency Medicine Approved Instructional Resources tool (AAT) was created for FOAM appraisal by and for expert educators and has demonstrated validity in this context. It has yet to be evaluated in other populations.
OBJECTIVES
OBJECTIVE
We assessed the AAT's usability in a diverse population of practicing emergency medicine (EM) physicians, residents, and medical students; solicited feedback; and developed a revised tool.
METHODS
METHODS
As part of the Medical Education Translational Resources: Impact and Quality (METRIQ) study, we recruited medical students, EM residents, and EM attendings to evaluate five FOAM posts with the AAT and provide quantitative and qualitative feedback via an online survey. Two independent analysts performed a qualitative thematic analysis with discrepancies resolved through discussion and negotiated consensus. This analysis informed development of an initial revised AAT, which was then further refined after pilot testing among the author group. The final tool was reassessed for reliability.
RESULTS
RESULTS
Of 330 recruited international participants, 309 completed all ratings. The Best Evidence in Emergency Medicine (BEEM) score was the component most frequently reported as difficult to use. Several themes emerged from the qualitative analysis: for ease of use-understandable, logically structured, concise, and aligned with educational value. Limitations include deviation from questionnaire best practices, validity concerns, and challenges assessing evidence-based medicine. Themes supporting its use include evaluative utility and usability. The author group pilot tested the initial revised AAT, revealing a total score average measure intraclass correlation coefficient (ICC) of moderate reliability (ICC = 0.68, 95% confidence interval [CI] = 0 to 0.962). The final AAT's average measure ICC was 0.88 (95% CI = 0.77 to 0.95).
CONCLUSIONS
CONCLUSIONS
We developed the final revised AAT from usability feedback. The new score has significantly increased usability, but will need to be reassessed for reliability in a broad population.
Identifiants
pubmed: 34141997
doi: 10.1002/aet2.10601
pii: AET210601
pmc: PMC8194147
doi:
Types de publication
Journal Article
Langues
eng
Pagination
e10601Informations de copyright
© 2021 by the Society for Academic Emergency Medicine.
Déclaration de conflit d'intérêts
The authors have no potential conflicts to disclose. All authors had full access to all the study data and had final responsibility for the decision to submit for publication.
Références
J Grad Med Educ. 2015 Dec;7(4):549-54
pubmed: 26692965
CJEM. 2015 Mar;17(2):101-6
pubmed: 25927253
Ann Emerg Med. 2016 Dec;68(6):729-735
pubmed: 27033141
AEM Educ Train. 2019 Jul 30;3(4):387-392
pubmed: 31637356
Acad Med. 2014 Apr;89(4):598-601
pubmed: 24556776
West J Emerg Med. 2015 Nov;16(6):936-7
pubmed: 26594294
Emerg Med J. 2019 Aug;36(8):453-455
pubmed: 31292205
Postgrad Med J. 2015 Oct;91(1080):546-50
pubmed: 26275428
West J Emerg Med. 2016 Sep;17(5):574-84
pubmed: 27625722
Ann Emerg Med. 2018 Dec;72(6):696-702
pubmed: 29980461
Ann Emerg Med. 2015 Oct;66(4):396-402.e4
pubmed: 25840846
Acad Emerg Med. 2009 Dec;16 Suppl 2:S25-31
pubmed: 20053206
West J Emerg Med. 2015 Mar;16(2):242-9
pubmed: 25834664
J Grad Med Educ. 2016 May;8(2):219-25
pubmed: 27168891
Teach Learn Med. 2018 Jul-Sep;30(3):294-302
pubmed: 29381099
BMC Med Educ. 2006 Aug 15;6:41
pubmed: 16911779
AEM Educ Train. 2019 May 23;3(4):375-386
pubmed: 31637355
Emerg Med J. 2014 Oct;31(e1):e76-7
pubmed: 24554447
Acad Emerg Med. 2013 Oct;20(10):1004-12
pubmed: 24127703
AEM Educ Train. 2017 Mar 22;1(2):110-113
pubmed: 30051018
Perspect Med Educ. 2017 Apr;6(2):91-98
pubmed: 28243948