Improving Inpatient Surveys: Web-Based Computer Adaptive Testing Accessed via Mobile Phone QR Codes

Application of the Rasch measurement model in rehabilitation research and practice: early developments, current practice, and future challenges

The application of the Rasch measurement model in rehabilitation is now well established. Both its dichotomous and polytomous forms provide for transforming ordinal scales into interval-level measures, consistent with the requirements of fundamental measurement. The growth of applying the model in rehabilitation spans 30 years, during which both the protocol has steadily developed and several software packages have emerged that provide for analysis, together with the "R" language that has an increasing set of codes for applying the model. This article reviews that development and highlights current practice requirements, including those for providing the relevant information for the methods, and what is expected of the analysis. In addition, this provides a worked example and looks at the remaining issues and current developments of its application.

Survey Response Rates to a Self-Initiated Longitudinal Survey Accessed by a Quick Response Code in Six Different Regions of the United States

Background

A quick response (QR) code allows rapid access to an online survey via a smartphone and may improve response rates for web-based surveys. We report the response rates for a QR code-based, self-initiated, longitudinal survey of opioid use and pain scores following hospital discharge in pediatric surgical patients.

Methodology

All parents of pediatric patients who underwent surgery at one of six pediatric medical facilities were asked to participate in the study from October 5, 2020, until July 15, 2021. Those who chose to participate accessed the initial enrollment survey using a QR code on a handout provided. The next day they received an emailed link to a daily survey until their child was not requiring opioids and had pain scores of less than 4 for the previous 48 hours.

Results

A total of 1,759 families were asked to participate in the study. The parents of 44 patients completed the initial enrollment survey by accessing the QR code (response rate of 2.5%). Of those who completed the initial survey, 67% were lost to follow-up during the survey series.

Conclusions

We found an extremely low response rate for a self-initiated survey accessed by QR code. Additionally, we found a drop in the response rate with each successive daily email-based survey. At the end of the survey series, the majority of the initial participants had dropped out. We recommend using alternative modalities (informed consent, telephone call, weekly surveys) for initiating and delivering surveys to improve response rates for similarly designed studies.

An app for predicting nurse intention to quit the job using artificial neural networks (ANNs) in Microsoft Excel

Background:

Numerous studies have identified factors related to nurses’ intention to leave. However, none has successfully predicted the nurse’s intention to quit the job. Whether the intention to quit the job can be predicted is an interesting topic in healthcare settings. A model to predict the nurse’s intention to quit the job for novice nurses should be investigated. The aim of this study is to build a model to develop an app for the automatic prediction and classification of nurses’ intention to quit their jobs.

Methods:

We recruited 1104 novice nurses working in 6 medical centers in Taiwan to complete 100-item questionnaires related to the nurse’s intention to quit the job in October 2018. The k-mean was used to divide nurses into 2 classes based on 5 items regarding leave intention. Feature variables were selected from the 100-item survey. Two models, including an artificial neural network (ANN) and a convolutional neural network, were compared across 4 scenarios made up of 2 training sets (n = 1104 and n = 804 ≅ 70%) and their corresponding testing (n = 300 ≅ 30%) sets to verify the model accuracy. An app for predicting the nurse’s intention to quit the job was then developed as a website assessment.

Results:

We observed that 24 feature variables extracted from this study in the ANN model yielded a higher area under the ROC curve of 0.82 (95% CI 0.80-0.84) based on the 1104 cases, the ANN performed better than the convolutional neural network on the accuracy, and a ready and available app for predicting the nurse’s intention to quit the job was successfully developed in this study.

Conclusions:

A 24-item ANN model with 53 parameters estimated by the ANN was developed to improve the accuracy of nurses’ intention to quit their jobs. The app would help team leaders take care of nurses who intend to quit the job before their actions are taken.

Key Points

An application for classifying perceptions on my health bank in Taiwan using convolutional neural networks and web-based computerized adaptive testing: A development and usability study

BACKGROUND

The classification of a respondent's opinions online into positive and negative classes using a minimal number of questions is gradually changing and helps turn techniques into practices. A survey incorporating convolutional neural networks (CNNs) into web-based computerized adaptive testing (CAT) was used to collect perceptions on My Health Bank (MHB) from users in Taiwan. This study designed an online module to accurately and efficiently turn a respondent's perceptions into positive and negative classes using CNNs and web-based CAT.

METHODS

In all, 640 patients, family members, and caregivers with ages ranging from 20 to 70 years who were registered MHB users were invited to complete a 3-domain, 26-item, 5-category questionnaire asking about their perceptions on MHB (PMHB26) in 2019. The CNN algorithm and k-means clustering were used for dividing respondents into 2 classes of unsatisfied and satisfied classes and building a PMHB26 predictive model to estimate parameters. Exploratory factor analysis, the Rasch model, and descriptive statistics were used to examine the demographic characteristics and PMHB26 factors that were suitable for use in CNNs and Rasch multidimensional CAT (MCAT). An application was then designed to classify MHB perceptions.

RESULTS

We found that 3 construct factors were extracted from PMHB26. The reliability of PMHB26 for each subscale beyond 0.94 was evident based on internal consistency and stability in the data. We further found the following: the accuracy of PMHB26 with CNN yields a higher accuracy rate (0.98) with an area under the curve of 0.98 (95% confidence interval, 0.97-0.99) based on the 391 returned questionnaires; and for the efficiency, approximately one-third of the items were not necessary to answer in reducing the respondents' burdens using Rasch MCAT.

CONCLUSIONS

The PMHB26 CNN model, combined with the Rasch online MCAT, is recommended for improving the accuracy and efficiency of classifying patients' perceptions of MHB utility. An application developed for helping respondents self-assess the MHB cocreation of value can be applied to other surveys in the future.

An artificial neural network model to predict the mortality of COVID-19 patients using routine blood samples at the time of hospital admission: Development and validation study

Background:

In a pandemic situation (e.g., COVID-19), the most important issue is to select patients at risk of high mortality at an early stage and to provide appropriate treatments. However, a few studies applied the model to predict in-hospital mortality using routine blood samples at the time of hospital admission. This study aimed to develop an app, name predict the mortality of COVID-19 patients (PMCP) app, to predict the mortality of COVID-19 patients at hospital-admission time.

Methods:

We downloaded patient records from 2 studies, including 361 COVID-19 patients in Wuhan, China, and 106 COVID-19 patients in 3 Korean medical institutions. A total of 30 feature variables were retrieved, consisting of 28 blood biomarkers and 2 demographic variables (i.e., age and gender) of patients. Two models, namely, artificial neural network (ANN) and convolutional neural network (CNN), were compared with each other across 2 scenarios using

An app for predicting the mortality of COVID-19 patients was developed using the model's estimated parameters for the prediction and classification of PMCP at an earlier stage. Feature variables and prediction results were visualized using the forest plot and category probability curves shown on Google Maps.

Results:

We observed that

Conclusions:

Our new PMCP app with ANN model accurately predicts the mortality probability for COVID-19 patients. It is publicly available and aims to help health care providers fight COVID-19 and improve patients’ classifications against treatment risk.

Predicting the 14-Day Hospital Readmission of Patients with Pneumonia Using Artificial Neural Networks (ANN)

Unplanned patient readmission (UPRA) is frequent and costly in healthcare settings. No indicators during hospitalization have been suggested to clinicians as useful for identifying patients at high risk of UPRA. This study aimed to create a prediction model for the early detection of 14-day UPRA of patients with pneumonia. We downloaded the data of patients with pneumonia as the primary disease (e.g., ICD-10:J12*-J18*) at three hospitals in Taiwan from 2016 to 2018. A total of 21,892 cases (1208 (6%) for UPRA) were collected. Two models, namely, artificial neural network (ANN) and convolutional neural network (CNN), were compared using the training (n = 15,324; ≅70%) and test (n = 6568; ≅30%) sets to verify the model accuracy. An app was developed for the prediction and classification of UPRA. We observed that (i) the 17 feature variables extracted in this study yielded a high area under the receiver operating characteristic curve of 0.75 using the ANN model and that (ii) the ANN exhibited better AUC (0.73) than the CNN (0.50), and (iii) a ready and available app for predicting UHA was developed. The app could help clinicians predict UPRA of patients with pneumonia at an early stage and enable them to formulate preparedness plans near or after patient discharge from hospitalization.

Predicting Active NBA Players Most Likely to Be Inducted into the Basketball Hall of Famers Using Artificial Neural Networks in Microsoft Excel: Development and Usability Study

The prediction of whether active NBA players can be inducted into the Hall of Fame (HOF) is interesting and important. However, no such research have been published in the literature, particularly using the artificial neural network (ANN) technique. The aim of this study is to build an ANN model with an app for automatic prediction and classification of HOF for NBA players. We downloaded 4728 NBA players’ data of career stats and accolades from the website at basketball-reference.com. The training sample was collected from 85 HOF members and 113 retired Non-HOF players based on completed data and a longer career length (≥15 years). Featured variables were taken from the higher correlation coefficients (<0.1) with HOF and significant deviations apart from the two HOF/Non-HOF groups using logistical regression. Two models (i.e., ANN and convolutional neural network, CNN) were compared in model accuracy (e.g., sensitivity, specificity, area under the receiver operating characteristic curve, AUC). An app predicting HOF was then developed involving the model’s parameters. We observed that (1) 20 feature variables in the ANN model yielded a higher AUC of 0.93 (95% CI 0.93–0.97) based on the 198-case training sample, (2) the ANN performed better than CNN on the accuracy of AUC (= 0.91, 95% CI 0.87–0.95), and (3) an ready and available app for predicting HOF was successfully developed. The 20-variable ANN model with the 53 parameters estimated by the ANN for improving the accuracy of HOF has been developed. The app can help NBA fans to predict their players likely to be inducted into the HOF and is not just limited to the active NBA players.

The Usefulness of the QR Code in Orthotic Applications after Orthopedic Surgery

The purpose of this study is to evaluate the utility of QR (quick response) codes in explaining the proper method for orthotic use after orthopedic surgery. A questionnaire survey was adopted to evaluate patient satisfaction with education and training in orthotic applications after orthopedic surgery. The study periods were 1 April to 30 April 2017, and 1 October to 31 October 2017. The oral training involving the conventional orthoses was conducted in April, and the videos with the orthosis on the QR code were captured in October. The QR code containing the data was distributed and the education was conducted. A total of 68 patients (QR-code group: 33) participated in the questionnaire survey. After the QR code application, the number of retraining cases increased from 62.9 to 93.9% (p-value < 0.01). The mean scores of the four items measuring the comprehension increased from 10.97 to 14.39. The satisfaction level rose from 7.14 to 9.30, and the performance increased from 7.14 to 9.52 (p-value < 0.01). The QR code is expected to be a valuable method for explaining the orthotic application after orthopedic surgery, and especially when repeated explanations are needed for elderly patients.

An App for Classifying Personal Mental Illness at Workplace Using Fit Statistics and Convolutional Neural Networks: Survey-Based Quantitative Study

BACKGROUND

Mental illness (MI) is common among those who work in health care settings. Whether MI is related to employees' mental status at work is yet to be determined. An MI app is developed and proposed to help employees assess their mental status in the hope of detecting MI at an earlier stage.

OBJECTIVE

This study aims to build a model using convolutional neural networks (CNNs) and fit statistics based on 2 aspects of measures and outfit mean square errors for the automatic detection and classification of personal MI at the workplace using the emotional labor and mental health (ELMH) questionnaire, so as to equip the staff in assessing and understanding their own mental status with an app on their mobile device.

METHODS

We recruited 352 respiratory therapists (RTs) working in Taiwan medical centers and regional hospitals to fill out the 44-item ELMH questionnaire in March 2019. The exploratory factor analysis (EFA), Rasch analysis, and CNN were used as unsupervised and supervised learnings for (1) dividing RTs into 4 classes (ie, MI, false MI, health, and false health) and (2) building an ELMH predictive model to estimate 108 parameters of the CNN model. We calculated the prediction accuracy rate and created an app for classifying MI for RTs at the workplace as a web-based assessment.

RESULTS

We observed that (1) 8 domains in ELMH were retained by EFA, (2) 4 types of mental health (n=6, 63, 265, and 18 located in 4 quadrants) were classified using the Rasch analysis, (3) the 44-item model yields a higher accuracy rate (0.92), and (4) an MI app available for RTs predicting MI was successfully developed and demonstrated in this study.

CONCLUSIONS

The 44-item model with 108 parameters was estimated by using CNN to improve the accuracy of mental health for RTs. An MI app developed to help RTs self-detect work-related MI at an early stage should be made more available and viable in the future.

An App for Detecting Bullying of Nurses Using Convolutional Neural Networks and Web-Based Computerized Adaptive Testing: Development and Usability Study

Background

Workplace bullying has been measured in many studies to investigate its effects on mental health issues. However, none have used web-based computerized adaptive testing (CAT) with bully classifications and convolutional neural networks (CNN) for reporting the extent of individual bullying in the workplace.

Objective

This study aims to build a model using CNN to develop an app for automatic detection and classification of nurse bullying-levels, incorporated with online Rasch computerized adaptive testing, to help assess nurse bullying at an earlier stage.

Methods

We recruited 960 nurses working in a Taiwan Ch-Mei hospital group to fill out the 22-item Negative Acts Questionnaire-Revised (NAQ-R) in August 2012. The k-mean and the CNN were used as unsupervised and supervised learnings, respectively, for: (1) dividing nurses into three classes (n=918, 29, and 13 with suspicious mild, moderate, and severe extent of being bullied, respectively); and (2) building a bully prediction model to estimate 69 different parameters. Finally, data were separated into training and testing sets in a proportion of 70:30, where the former was used to predict the latter. We calculated the sensitivity, specificity, and receiver operating characteristic curve (area under the curve [AUC]), along with the accuracy across studies for comparison. An app predicting the respondent bullying-level was developed, involving the model’s 69 estimated parameters and the online Rasch CAT module as a website assessment.

Results

We observed that: (1) the 22-item model yields higher accuracy rates for three categories, with an accuracy of 94% for the total 960 cases, and accuracies of 99% (AUC 0.99; 95% CI 0.99-1.00) and 83% (AUC 0.94; 95% CI 0.82-0.99) for the lower and upper groups (cutoff points at 49 and 66 points) based on the 947 cases and 42 cases, respectively; and (2) the 700-case training set, with 95% accuracy, predicts the 260-case testing set reaching an accuracy of 97. Thus, a NAQ-R app for nurses that predicts bullying-level was successfully developed and demonstrated in this study.

Conclusions

The 22-item CNN model, combined with the Rasch online CAT, is recommended for improving the accuracy of the nurse NAQ-R assessment. An app developed for helping nurses self-assess workplace bullying at an early stage is required for application in the future.

An App Developed for Detecting Nurse Burnouts Using the Convolutional Neural Networks in Microsoft Excel: Population-Based Questionnaire Study

Background

Burnout (BO), a critical syndrome particularly for nurses in health care settings, substantially affects their physical and psychological status, the institute’s well-being, and indirectly, patient outcomes. However, objectively classifying BO levels has not been defined and noticed in the literature.

Objective

The aim of this study is to build a model using the convolutional neural network (CNN) to develop an app for automatic detection and classification of nurse BO using the Maslach Burnout Inventory–Human Services Survey (MBI-HSS) to help assess nurse BO at an earlier stage.

Methods

We recruited 1002 nurses working in a medical center in Taiwan to complete the Chinese version of the 20-item MBI-HSS in August 2016. The k-mean and CNN were used as unsupervised and supervised learnings for dividing nurses into two classes (n=531 and n=471 of suspicious BO+ and BO−, respectively) and building a BO predictive model to estimate 38 parameters. Data were separated into training and testing sets in a proportion 70%:30%, and the former was used to predict the latter. We calculated the sensitivity, specificity, and receiver operating characteristic curve (area under the curve) across studies for comparison. An app predicting respondent BO was developed involving the model’s 38 estimated parameters for a website assessment.

Results

We observed that (1) the 20-item model yields a higher accuracy rate (0.95) with an area under the curve of 0.97 (95% CI 0.94-0.95) based on the 1002 cases, (2) the scheme named matching personal response to adapt for the correct classification in model drives the prior model’s predictive accuracy at 100%, (3) the 700-case training set with 0.96 accuracy predicts the 302-case testing set reaching an accuracy of 0.91, and (4) an available MBI-HSS app for nurses predicting BO was successfully developed and demonstrated in this study.

Conclusions

The 20-item model with the 38 parameters estimated by using CNN for improving the accuracy of nurse BO has been particularly demonstrated in Excel (Microsoft Corp). An app developed for helping nurses to self-assess job BO at an early stage is required for application in the future.

An App for Identifying Children at Risk for Developmental Problems Using Multidimensional Computerized Adaptive Testing: Development and Usability Study

BACKGROUND

The use of multidomain developmental screening tools is a viable strategy for pediatric professionals to identify children at risk for developmental problems. However, a specialized multidimensional computer adaptive testing (MCAT) tool has not been developed to date.

OBJECTIVE

We developed an app using MCAT, combined with Multidimensional Screening in Child Development (MuSiC) for toddlers, to help patients and their family members or clinicians identify developmental problems at an earlier stage.

METHODS

We retrieved 75 item parameters from the MuSiC literature item bank for 1- to 3-year-old children, and simulated 1000 person measures from a normal standard distribution to compare the efficiency and precision of MCAT and nonadaptive testing (NAT) in five domains (ie, cognitive skills, language skills, gross motor skills, fine motor skills, and socioadaptive skills). The number of items saved and the cutoff points for the tool were determined and compared. We then developed an app for a Web-based assessment.

RESULTS

MCAT yielded significantly more precise measurements and was significantly more efficient than NAT, with 46.67% (=(75-40)/75) saving in item length when measurement differences less than 5% were allowed. Person-measure correlation coefficients were highly consistent among the five domains. Significantly fewer items were answered on MCAT than on NAT without compromising the precision of MCAT.

CONCLUSIONS

Developing an app as a tool for parents that can be implemented with their own computers, tablets, or mobile phones for the online screening and prediction of developmental delays in toddlers is useful and not difficult.

Calibration and validation of an item bank for measuring general physical function of patients in medical rehabilitation settings

Objective

The objective of this study was to report the item response theory (IRT) calibration of an 18-item bank to measure general physical function (GPF) in a wide range of conditions and evaluate the validity of the derived scores.

Methods

All 18 items were administered to a large sample of patients (n=2337) who responded to the items in the context of their outpatient rehabilitation care. The responses, collected 1997– 2000, were modeled using the graded response model, an IRT model appropriate for items with two or more response options. Inter-item consistency was evaluated based on Cronbach’s alpha and item to total correlations. Validity of scores was evaluated based on known-groups comparisons (age, number of health problems, symptom severity). The strength of a single, general factor was evaluated using a bi-factor model. Results were used to evaluate IRT assumption and as an indicator of construct validity. Local independence of item responses was also evaluated.

Results

Response data met the assumptions of unidimensionality and local independence. Explained common variance of a single general factor was 0.88 (omega hierarchical =0.86). Only two of the 153 pairs of item residuals were flagged for local dependence. Inter-item consistency was high (0.93) as were item to total correlations (mean =0.61). Substantial variation was found in both IRT location (difficulty) and discrimination parameters. All omnibus known-groups comparisons were statistically significant (p<0.001).

Conclusion

Item responses fit the IRT unidimensionality assumptions and were internally consistent. The usefulness of GPF scores in discriminating among patients with different levels of physical function was confirmed. Future studies should evaluate the validity of GPF scores based on an adaptive administration of items.

Development of a Microsoft Excel tool for one-parameter Rasch model of continuous items: an application to a safety attitude survey

Background

Many continuous item responses (CIRs) are encountered in healthcare settings, but no one uses item response theory’s (IRT) probabilistic modeling to present graphical presentations for interpreting CIR results. A computer module that is programmed to deal with CIRs is required.

To present a computer module, validate it, and verify its usefulness in dealing with CIR data, and then to apply the model to real healthcare data in order to show how the CIR that can be applied to healthcare settings with an example regarding a safety attitude survey.

Methods

Using Microsoft Excel VBA (Visual Basic for Applications), we designed a computer module that minimizes the residuals and calculates model’s expected scores according to person responses across items. Rasch models based on a Wright map and on KIDMAP were demonstrated to interpret results of the safety attitude survey.

Results

The author-made CIR module yielded OUTFIT mean square (MNSQ) and person measures equivalent to those yielded by professional Rasch Winsteps software. The probabilistic modeling of the CIR module provides messages that are much more valuable to users and show the CIR advantage over classic test theory.

Conclusions

Because of advances in computer technology, healthcare users who are familiar to MS Excel can easily apply the study CIR module to deal with continuous variables to benefit comparisons of data with a logistic distribution and model fit statistics.

Electronic supplementary material

The online version of this article (doi:10.1186/s12874-016-0276-2) contains supplementary material, which is available to authorized users.