|
1
|
Wang H, Yeung WLK, Ng QX, Tung A, Tay JAM, Ryanputra D, Ong MEH, Feng M, Arulanandam S. A Weakly-Supervised Named Entity Recognition Machine Learning Approach for Emergency Medical Services Clinical Audit. INTERNATIONAL JOURNAL OF ENVIRONMENTAL RESEARCH AND PUBLIC HEALTH 2021; 18:ijerph18157776. [PMID: 34360065 PMCID: PMC8345494 DOI: 10.3390/ijerph18157776] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 07/01/2021] [Revised: 07/19/2021] [Accepted: 07/21/2021] [Indexed: 11/16/2022]
Abstract
Clinical performance audits are routinely performed in Emergency Medical Services (EMS) to ensure adherence to treatment protocols, to identify individual areas of weakness for remediation, and to discover systemic deficiencies to guide the development of the training syllabus. At present, these audits are performed by manual chart review, which is time-consuming and laborious. In this paper, we report a weakly-supervised machine learning approach to train a named entity recognition model that can be used for automatic EMS clinical audits. The dataset used in this study contained 58,898 unlabeled ambulance incidents encountered by the Singapore Civil Defence Force from 1st April 2019 to 30th June 2019. With only 5% labeled data, we successfully trained three different models to perform the NER task, achieving F1 scores of around 0.981 under entity type matching evaluation and around 0.976 under strict evaluation. The BiLSTM-CRF model was 1~2 orders of magnitude lighter and faster than our BERT-based models. Our proposed proof-of-concept approach may improve the efficiency of clinical audits and can also help with EMS database research. Further external validation of this approach is needed.
Collapse
Affiliation(s)
- Han Wang
- Saw Swee Hock School of Public Health, National University of Singapore, Singapore 117549, Singapore;
| | - Wesley Lok Kin Yeung
- Singapore Civil Defence Force, Singapore 408827, Singapore; (W.L.K.Y.); (Q.X.N.); (A.T.); (J.A.M.T.); (D.R.); (S.A.)
- National University Hospital, National University Health System, Singapore 119074, Singapore
| | - Qin Xiang Ng
- Singapore Civil Defence Force, Singapore 408827, Singapore; (W.L.K.Y.); (Q.X.N.); (A.T.); (J.A.M.T.); (D.R.); (S.A.)
| | - Angeline Tung
- Singapore Civil Defence Force, Singapore 408827, Singapore; (W.L.K.Y.); (Q.X.N.); (A.T.); (J.A.M.T.); (D.R.); (S.A.)
- Home Team Science & Technology Agency, Singapore 329560, Singapore
| | - Joey Ai Meng Tay
- Singapore Civil Defence Force, Singapore 408827, Singapore; (W.L.K.Y.); (Q.X.N.); (A.T.); (J.A.M.T.); (D.R.); (S.A.)
| | - Davin Ryanputra
- Singapore Civil Defence Force, Singapore 408827, Singapore; (W.L.K.Y.); (Q.X.N.); (A.T.); (J.A.M.T.); (D.R.); (S.A.)
- National University Hospital, National University Health System, Singapore 119074, Singapore
| | - Marcus Eng Hock Ong
- Health Services Research Centre, Singapore Health Services, Singapore 169856, Singapore;
- Health Services and Systems Research, Duke-NUS Medical School, National University of Singapore, Singapore 169857, Singapore
- Department of Emergency Medicine, Singapore General Hospital, Singapore 169608, Singapore
| | - Mengling Feng
- Saw Swee Hock School of Public Health, National University of Singapore, Singapore 117549, Singapore;
- Institute of Data Science, National University of Singapore, Singapore 117602, Singapore
- Correspondence: ; Tel.: +65-985-038-11
| | - Shalini Arulanandam
- Singapore Civil Defence Force, Singapore 408827, Singapore; (W.L.K.Y.); (Q.X.N.); (A.T.); (J.A.M.T.); (D.R.); (S.A.)
| |
Collapse
|