Deep Learning-Based Computer-Aided Diagnosis of Rheumatoid Arthritis with Hand X-ray Images Conforming to Modified Total Sharp/van der Heijde Score

Advancement and independent validation of a deep learning-based tool for automated scoring of nail psoriasis severity using the modified nail psoriasis severity index

Objective

To improve and validate a convolutional neural network (CNN)-based model for the automated scoring of nail psoriasis severity using the modified Nail Psoriasis Severity Index (mNAPSI) with adequate accuracy across all severity classes and without dependency on standardized conditions.

Methods

Patients with psoriasis (PsO), psoriatic arthritis (PsA), and non-psoriatic controls including healthy individuals and patients with rheumatoid arthritis were included for training, while validation utilized an independent cohort of psoriatic patients. Nail photographs were pre-processed and segmented and mNAPSI scores were annotated by five expert readers. A CNN based on Bidirectional Encoder representation from Image Transformers (BEiT) architecture and pre-trained on ImageNet-22k was fine-tuned for mNAPSI classification. Model performance was compared with human annotations by using area under the receiver operating characteristic curve (AUROC) and other metrics. A reader study was performed to assess inter-rater variability.

Results

In total, 460 patients providing 4,400 nail photographs were included in the training dataset. The independent validation dataset included 118 further patients who provided 929 nail photographs. The CNN demonstrated high classification performance on the training dataset, achieving mean (SD) AUROC of 86% ± 7% across mNAPSI classes. Performance remained robust on the independent validation dataset, with a mean AUROC of 80% ± 9%, despite variability in imaging conditions. Compared with human annotation, the CNN achieved a Pearson correlation of 0.94 on a patient-level, which remained consistent in the validation dataset.

Conclusion

We developed and validated a CNN that enables the automated, objective scoring of nail psoriasis severity based on mNAPSI with high reliability and without need of image standardization. This approach has potential clinical utility for enabling a standardized time-efficient assessment of nail involvement in the psoriatic disease and possibly as a self-reporting tool.

Radiography in Inflammatory Arthritis: Current Roles and Updates in Automated Assessment

Plain radiography continues to be a crucial imaging modality in the field of rheumatology. It provides a comprehensive view of bone-related changes and highlights soft tissue abnormalities. The significance of radiography extends to early disease detection, aiding in differentiating various conditions and monitoring the effectiveness of treatment. It remains the preferred imaging technique for evaluating disease progression, offering insights into cumulative damage over time.In the early stages of arthritis, magnetic resonance imaging and ultrasound are the preferred methods because they can identify subtle disease activity, such as synovitis, tenosynovitis, and dactylitis, osteitis or bone edema, and enthesitis. But they have a lower specificity in distinguishing among various rheumatic conditions.We evaluate the use of radiography in inflammatory arthropathies, highlighting its role in differential diagnoses. Advances in automated radiographic assessment for arthritis are addressed. The discussion encompasses rheumatoid arthritis, juvenile idiopathic arthritis, other connective tissue diseases, and spondyloarthritis.

Automatic Joint Lesion Detection by enhancing local feature interaction

Recently, deep learning models have demonstrated impressive performance in Automatic Joint Lesion Detection (AJLD), yet balancing accuracy and efficiency remains a significant challenge. This paper focuses on achieving end-to-end lesion detection while improving accuracy to meet clinical requirements. To enhance the overall performance of AJLD, we propose novel modules: Local Attention Feature Fusion (LAFF) and Gaussian Positional Encoding (GPE). These modules are extensively integrated into YOLO, resulting in an improved YOLO model by enhancing Local Feature interaction, named YOLOlf for short. The LAFF module, based on pathological features presented by arthritis, strengthens the implicit connections between joints by acquiring local attention information. The GPE module enhances the connections between joints by encoding their local positional information. In this paper, we validate our approach using two arthritis datasets, including the largest AJLD dataset in the literature (960 X-ray images annotated by two arthritis specialists and one radiologist) and another arthritis dataset with 216 X-ray images, supplemented by the MURA dataset, a more general dataset for abnormality detection in musculoskeletal radiographs. In various series of YOLO models, the improved YOLOlf shows a significant increase in detection accuracy. Taking YOLOv8 as an example, the improved YOLOlfv8 increases mAP@50 from 0.765 to 0.785 and from 0.831 to 0.859 on two arthritis datasets, demonstrating the plug-and-play nature and clinical applicability of the proposed LAFF and GPE modules.

Deep learning enables automatic detection of joint damage progression in rheumatoid arthritis-model development and external validation

OBJECTIVES

Although deep learning has demonstrated substantial potential in automatic quantification of joint damage in RA, evidence for detecting longitudinal changes at an individual patient level is lacking. Here, we introduce and externally validate our automated RA scoring algorithm (AuRA), and demonstrate its utility for monitoring radiographic progression in a real-world setting.

METHODS

The algorithm, originally developed during the Rheumatoid Arthritis 2-Dialogue for Reverse Engineering Assessment and Methods (RA2-DREAM) challenge, was trained to predict expert-curated Sharp-van der Heijde total scores in hand and foot radiographs from two previous clinical studies (n = 367). We externally validated AuRA against data (n = 205) from Turku University Hospital and compared the performance against two top-performing RA2-DREAM solutions. Finally, for 54 patients, we extracted additional radiograph sets from another control visit to the clinic (average time interval of 4.6 years).

RESULTS

In the external validation cohort, with a root mean square error (RMSE) of 23.6, AuRA outperformed both top-performing RA2-DREAM algorithms (RMSEs 35.0 and 35.6). The improved performance was explained mostly by lower errors at higher expert-assessed scores. The longitudinal changes predicted by our algorithm were significantly correlated with changes in expert-assessed scores (Pearson's R = 0.74, P < 0.001).

CONCLUSION

AuRA had the best external validation performance and demonstrated potential for detecting longitudinal changes in joint damage. Available from https://hub.docker.com/r/elolab/aura, our algorithm can easily be applied for automatic detection of radiographic progression in the future, reducing the need for laborious manual scoring.

Multistage deep learning methods for automating radiographic sharp score prediction in rheumatoid arthritis

The Sharp-van der Heijde score (SvH) is crucial for assessing joint damage in rheumatoid arthritis (RA) through radiographic images. However, manual scoring is time-consuming and subject to variability. This study proposes a multistage deep learning model to predict the Overall Sharp Score (OSS) from hand X-ray images. The framework involves four stages: image preprocessing, hand segmentation with UNet, joint identification via YOLOv7, and OSS prediction utilizing a custom Vision Transformer (ViT). Evaluation metrics included Intersection over Union (IoU), Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), Huber loss, and Intraclass Correlation Coefficient (ICC). The model was trained using stratified group 3-fold cross-validation on a dataset of 679 patients and tested externally on 291 subjects. The joint identification model achieved 99% accuracy. The ViT model achieved the best OSS prediction for patients with Sharp scores < 50. It achieved a Huber loss of 4.9, an RMSE of 9.73, and an MAE of 5.35, demonstrating a strong correlation with expert scores (ICC = 0.702, P < 0.001). This study is the first to apply a ViT for OSS prediction in RA. It presents an efficient and automated alternative for overall damage assessment. This approach may reduce reliance on manual scoring.

Ammodaucus Leucotrichus Seed Extract as a Potential Therapy in Animal Models of Rheumatoid Arthritis Induced by Complete Freund Adjuvant and Chicken Cartilage Collagen

This study assessed the efficacy of an Ammodaucus leucotrichus seed extract to treat rheumatoid arthritis in rat models of this disease. Rheumatoid arthritis was induced in rats using two methods: immunization with 100 µL of Complete Freund Adjuvant (CFA) and immunization with 100 µL of a 3 mg/ml solution of type II collagen (CII) from chicken cartilage. The therapeutic potential of the extract was assessed at different doses (150, 300, and 600 mg/kg/day for 21 days in the CII-induced arthritis model and for 14 days in the CFA-induced arthritis model) and compared with methotrexate (MTX; 0.2 mg/kg for the same periods), a commonly used drug for rheumatoid arthritis treatment in humans. In both models (CII-induced arthritis and CFA-induced arthritis), walking distance, step length, intra-step distance and footprint area were improved following treatment with the A. leucotrichus seed extract (all concentrations) and MTX compared with untreated animals. Both treatments increased the serum concentration of glutathione and reduced that of complement C3, malondialdehyde and myeloperoxidase. Radiographic data and histological analysis indicated that cartilage destruction was reduced already with the lowest dose of the extract (100 mg/kg/dose) in both models. These results show the substantial antiarthritic potential of the A. leucotrichus seed extract, even at the lowest dose, suggesting that it may be a promising alternative therapy for rheumatoid arthritis and joint inflammation. They also emphasize its efficacy at various doses, providing impetus for more research on this extract as a potential therapeutic agent for arthritis.

Unveiling Artificial Intelligence's Power: Precision, Personalization, and Progress in Rheumatology

This review examines the increasing use of artificial intelligence (AI) in rheumatology, focusing on its potential impact in key areas. AI, including machine learning (ML) and deep learning (DL), is revolutionizing diagnosis, treatment personalization, and prognosis prediction in rheumatologic diseases. Specifically, AI models based on convolutional neural networks (CNNs) demonstrate significant efficacy in analyzing medical images for disease classification and severity assessment. Predictive AI models also have the ability to forecast disease trajectories and treatment responses, enabling more informed clinical decisions. The role of wearable devices and mobile applications in continuous disease monitoring is discussed, although their effectiveness varies across studies. Despite existing challenges, such as data privacy concerns and issues of model generalizability, the compelling results highlight the transformative potential of AI in rheumatologic disease management. As AI technologies continue to evolve, further research will be essential to address these challenges and fully harness the potential of AI to improve patient outcomes in rheumatology.

Can eye-tracking help to create a new method for X-ray analysis of rheumatoid arthritis patients, including joint segmentation and scoring methods?

Reading hand and foot X-rays in rheumatoid arthritis patients is difficult and time-consuming. In research, physicians use the modified Sharp van der Heijde Sharp (mvdH) score by reading of hand and foot radiographs. The aim of this study was to create a new method of determining the mvdH via eye tracking and to study its concordance with the mvdH score. We created a new method of quantifying the mvdH score based on reading time of a reader monitored via eye tracking (Tobii Pro Lab software) after training with the aid of a metronome. Radiographs were read twice by the trained eye-tracking reader and once by an experienced reference radiologist. A total of 440 joints were selected; 416 could be interpreted for erosion, and 396 could be interpreted for joint space narrowing (JSN) when read by eye tracking (eye tracking could not measure the time spent when two pathological joints were too close together). The agreement between eye tracking mvdH Sharp score and classical mvdH Sharp score yes (at least one erosion or JSN) versus no (no erosion or no JSN) was excellent for both erosions (kappa 0.97; 95% CI: 0.96-0.99) and JSN (kappa: 0.95; 95% CI: 0.93-0.097). This agreement by class (0 to 10) remained excellent for both erosions (kappa 0.82; 95% CI: 0.79-0.0.85) and JSN (kappa: 0.68; 95% CI: 0.65-0.0.71). To conclude, eye-tracking reading correlates strongly with classical mvdH-Sharp and is useful for assessing severity, segmenting joints and establishing a rapid score for lesions.

RA-XTNet: A Novel CNN Model to Predict Rheumatoid Arthritis from Hand Radiographs and Thermal Images: A Comparison with CNN Transformer and Quantum Computing

The aim and objective of the research are to develop an automated diagnosis system for the prediction of rheumatoid arthritis (RA) based on artificial intelligence (AI) and quantum computing for hand radiographs and thermal images. The hand radiographs and thermal images were segmented using a UNet++ model and color-based k-means clustering technique, respectively. The attributes from the segmented regions were generated using the Speeded-Up Robust Features (SURF) feature extractor and classification was performed using k-star and Hoeffding classifiers. For the ground truth and the predicted test image, the study utilizing UNet++ segmentation achieved a pixel-wise accuracy of 98.75%, an intersection over union (IoU) of 0.87, and a dice coefficient of 0.86, indicating a high level of similarity. The custom RA-X-ray thermal imaging (XTNet) surpassed all the models for the detection of RA with a classification accuracy of 90% and 93% for X-ray and thermal imaging modalities, respectively. Furthermore, the study employed quantum support vector machine (QSVM) as a quantum computing approach which yielded an accuracy of 93.75% and 87.5% for the detection of RA from hand X-ray and thermal images. In addition, vision transformer (ViT) was employed to classify RA which obtained an accuracy of 80% for hand X-rays and 90% for thermal images. Thus, depending on the performance measures, the RA-XTNet model can be used as an effective automated diagnostic method to diagnose RA accurately and rapidly in hand radiographs and thermal images.

A Stepwise Approach to Analyzing Musculoskeletal Imaging Data With Artificial Intelligence

The digitization of medical records and expanding electronic health records has created an era of "Big Data" with an abundance of available information ranging from clinical notes to imaging studies. In the field of rheumatology, medical imaging is used to guide both diagnosis and treatment of a wide variety of rheumatic conditions. Although there is an abundance of data to analyze, traditional methods of image analysis are human resource intensive. Fortunately, the growth of artificial intelligence (AI) may be a solution to handle large datasets. In particular, computer vision is a field within AI that analyzes images and extracts information. Computer vision has impressive capabilities and can be applied to rheumatologic conditions, necessitating a need to understand how computer vision works. In this article, we provide an overview of AI in rheumatology and conclude with a five step process to plan and conduct research in the field of computer vision. The five steps include (1) project definition, (2) data handling, (3) model development, (4) performance evaluation, and (5) deployment into clinical care.

Artificial intelligence in rheumatoid arthritis: potential applications and future implications

The widespread adoption of digital health records, coupled with the rise of advanced diagnostic testing, has resulted in an explosion of patient data, comparable in scope to genomic datasets. This vast information repository offers significant potential for improving patient outcomes and decision-making, provided one can extract meaningful insights from it. This is where artificial intelligence (AI) tools like machine learning (ML) and deep learning come into play, helping us leverage these enormous datasets to predict outcomes and make informed decisions. AI models can be trained to analyze and interpret patient data, including physician notes, laboratory testing, and imaging, to aid in the management of patients with rheumatic diseases. As one of the most common autoimmune diseases, rheumatoid arthritis (RA) has attracted considerable attention, particularly concerning the evolution of diagnostic techniques and therapeutic interventions. Our aim is to underscore those areas where AI, according to recent research, demonstrates promising potential to enhance the management of patients with RA.

Imaging in inflammatory arthritis: progress towards precision medicine

Imaging techniques such as ultrasonography and MRI have gained ground in the diagnosis and management of inflammatory arthritis, as these imaging modalities allow a sensitive assessment of musculoskeletal inflammation and damage. However, these techniques cannot discriminate between disease subsets and are currently unable to deliver an accurate prediction of disease progression and therapeutic response in individual patients. This major shortcoming of today's technology hinders a targeted and personalized patient management approach. Technological advances in the areas of high-resolution imaging (for example, high-resolution peripheral quantitative computed tomography and ultra-high field MRI), functional and molecular-based imaging (such as chemical exchange saturation transfer MRI, positron emission tomography, fluorescence optical imaging, optoacoustic imaging and contrast-enhanced ultrasonography) and artificial intelligence-based data analysis could help to tackle these challenges. These new imaging approaches offer detailed anatomical delineation and an in vivo and non-invasive evaluation of the immunometabolic status of inflammatory reactions, thereby facilitating an in-depth characterization of inflammation. By means of these developments, the aim of earlier diagnosis, enhanced monitoring and, ultimately, a personalized treatment strategy looms closer.

Computer model of IL-6-dependent rheumatoid arthritis in F759 mice

The interleukin-6 (IL-6) amplifier, which describes the simultaneous activation of signal transducer and activator of transcription 3 (STAT3) and NF-κb nuclear factor kappa B (NF-κB), in synovial fibroblasts causes the infiltration of immune cells into the joints of F759 mice. The result is a disease that resembles human rheumatoid arthritis. However, the kinetics and regulatory mechanisms of how augmented transcriptional activation by STAT3 and NF-κB leads to F759 arthritis is unknown. We here show that the STAT3-NF-κB complex is present in the cytoplasm and nucleus and accumulates around NF-κB binding sites of the IL-6 promoter region and established a computer model that shows IL-6 and IL-17 (interleukin 17) signaling promotes the formation of the STAT3-NF-κB complex followed by its binding on promoter regions of NF-κB target genes to accelerate inflammatory responses, including the production of IL-6, epiregulin, and C-C motif chemokine ligand 2 (CCL2), phenotypes consistent with in vitro experiments. The binding also promoted cell growth in the synovium and the recruitment of T helper 17 (Th17) cells and macrophages in the joints. Anti-IL-6 blocking antibody treatment inhibited inflammatory responses even at the late phase, but anti-IL-17 and anti-TNFα antibodies did not. However, anti-IL-17 antibody at the early phase showed inhibitory effects, suggesting that the IL-6 amplifier is dependent on IL-6 and IL-17 stimulation at the early phase, but only on IL-6 at the late phase. These findings demonstrate the molecular mechanism of F759 arthritis can be recapitulated in silico and identify a possible therapeutic strategy for IL-6 amplifier-dependent chronic inflammatory diseases.

Improving the Age Estimation Efficiency by Calculation of the Area Ratio Index Using Semi-Automatic Segmentation of Knee MRI Images

The knee is an anatomical structure that can provide a great deal of data for research on age estimation. The aim of this study was to evaluate and apply a method for semi-automatic measurements of the area under the growth plate closure of the femur distal epiphysis and the growth plate closure itself on the 2D coronary slices using T2 weighted images (T2WI) generated on magnetic resonance (MRI) devices of different technical and technological characteristics. After the semi-automatic segmentation of the femur distal epiphysis under the growth plate closure and the growth plate closure itself, the areas of the measured closures were calculated using MATLAB version: 9.12. (R2022a), MathWorks Inc., Natick, MA, USA, for each individual coronal slice. The area ratio index (ARI) was calculated as the ratio between the area under the growth plate closure of the femur distal epiphysis and the growth plate closure itself. The study sample consisted of 27 female and 23 male Caucasian participants aged 10 to 26 years. A total of 339 T2WI images were used for ARI calculations. There was a positive correlation between chronological age and the average ARI measured by three independent observers (r = 0.8280, p < 0.001). Multiple regression analysis did not show any significant impact of the technical and technological characteristics of the MRI devices on ARI. The results of this study showed that ARI could serve as a useful tool for age estimation using knee MRI as well as for the further development of artificial intelligence (AI) applications.

A survey of artificial intelligence in rheumatoid arthritis

The article offers a survey of currently notable artificial intelligence methods (released between 2019-2023), with a particular emphasis on the latest advancements in detecting rheumatoid arthritis (RA) at an early stage, providing early treatment, and managing the disease. We discussed challenges in these areas followed by specific artificial intelligence (AI) techniques and summarized advances, relevant strengths, and obstacles. Overall, the application of AI in the fields of RA has the potential to enable healthcare professionals to detect RA at an earlier stage, thereby facilitating timely intervention and better disease management. However, more research is required to confirm the precision and dependability of AI in RA, and several problems such as technological and ethical concerns related to these approaches must be resolved before their widespread adoption.

A Framework of Faster CRNN and VGG16-Enhanced Region Proposal Network for Detection and Grade Classification of Knee RA

We developed a framework to detect and grade knee RA using digital X-radiation images and used it to demonstrate the ability of deep learning approaches to detect knee RA using a consensus-based decision (CBD) grading system. The study aimed to evaluate the efficiency with which a deep learning approach based on artificial intelligence (AI) can find and determine the severity of knee RA in digital X-radiation images. The study comprised people over 50 years with RA symptoms, such as knee joint pain, stiffness, crepitus, and functional impairments. The digitized X-radiation images of the people were obtained from the BioGPS database repository. We used 3172 digital X-radiation images of the knee joint from an anterior-posterior perspective. The trained Faster-CRNN architecture was used to identify the knee joint space narrowing (JSN) area in digital X-radiation images and extract the features using ResNet-101 with domain adaptation. In addition, we employed another well-trained model (VGG16 with domain adaptation) for knee RA severity classification. Medical experts graded the X-radiation images of the knee joint using a consensus-based decision score. We trained the enhanced-region proposal network (ERPN) using this manually extracted knee area as the test dataset image. An X-radiation image was fed into the final model, and a consensus decision was used to grade the outcome. The presented model correctly identified the marginal knee JSN region with 98.97% of accuracy, with a total knee RA intensity classification accuracy of 99.10%, with a sensitivity of 97.3%, a specificity of 98.2%, a precision of 98.1%, and a dice score of 90.1% compared with other conventional models.

Effectiveness of Yishen Tongbi decoction versus methotrexate in patients with active rheumatoid arthritis: A double-blind, randomized, controlled, non-inferiority trial

BACKGROUND

Yishen Tongbi decoction (YSTB) which is an herbal formula, has been used for the treatment of rheumatoid arthritis (RA) for more than ten years with a better curative effect. Methotrexate (MTX) is an effective anchoring agent used to treat rheumatoid arthritis. There were, however, no head-to-head comparative randomized controlled trials comparing traditional Chinese medicine (TCM) to MTX, Therefore, we performed this double-blind, double-model, randomized controlled trial of the efficacy and safety of YSTB and MTX in the treatment of active RA for 24 weeks.

METHODS

Patients who met the enrollment criteria were randomly selected (1:1) to receive either YSTB therapy (YSTB 150 ml once daily + MTX placebo 7.5-15 mg once weekly) or MTX therapy (MTX 7.5-15 mg once weekly + YSTB placebo 150 ml once daily) in treatment cycles lasting 24 weeks. The percentage of patients who achieve a clinical disease activity index (CDAI) response at week 24 is the primary efficacy outcome. A 10% risk differential non-inferiority margin was previously defined. The Chinese Clinical Trials Registry has recorded this trial (ChiCTR-1,900,024,902, registered on August 3rd 2019, http://www.chictr.org.cn/index.aspx).

RESULTS

Out of 118 patients whose eligibility was determined from September 2019 to May 2022, 100 patients (n = 50 for each group) were enrolled in the research overall. The 24-week trial was completed by 82% (40/49) of the YSTB group's patients and 86% (42/49) of the MTX group's patients. In the intention-to-treat analysis, 67.4% (33/49) of patients in the YSTB group met the main outcome of CDAI response criteria at week 24, compared to 57.1% (28/49) in the MTX group. The risk difference was 0.102 (95% CI -0.089 to 0.293), which demonstrated the non-inferiority of YSTB to MTX. After further testing for superiority, the ratio of CDAI responses achieved by the YSTB and MTX groups was not statistically significant (p = 0.298). At the same time, in week 24, secondary outcomes such as the ACR 20/50/70 response, the European Alliance of Associations for Rheumatology good or moderate response, remission rate, simplified disease activity index response, and low disease activity rate all showed similar statistically significant patterns. There was statistically significant attainment of ACR20 (p = 0.008) and EULAR good or moderate response (p = 0.009) in two groups at week 4. The intention-to-treat analysis results and the per-protocol analysis results were in agreement. The incidence of drug-related adverse events was not statistically different between the two groups (p = 0.487).

CONCLUSIONS

Previous studies have used TCM as an adjunct to conventional therapy, and few of them have directly compared it with MTX. In order to lessen disease activity in RA patients, this trial demonstrated that YSTB compound monotherapy was non-inferior to MTX monotherapy and had superior efficacy following short-term treatment. This study provided evidence-based medicine in the treatment of RA with compound prescriptions of TCM and contributed to promoting phytomedicine use in RA patients.

Detecting hand joint ankylosis and subluxation in radiographic images using deep learning: A step in the development of an automatic radiographic scoring system for joint destruction

We propose a wrist joint subluxation/ankylosis classification model for an automatic radiographic scoring system for X-ray images. In managing rheumatoid arthritis, the evaluation of joint destruction is important. The modified total Sharp score (mTSS), which is conventionally used to evaluate joint destruction of the hands and feet, should ideally be automated because the required time depends on the skill of the evaluator, and there is variability between evaluators. Since joint subluxation and ankylosis are given a large score in mTSS, we aimed to estimate subluxation and ankylosis using a deep neural network as a first step in developing an automatic radiographic scoring system for joint destruction. We randomly extracted 216 hand X-ray images from an electronic medical record system for the learning experiments. These images were acquired from patients who visited the rheumatology department of Keio University Hospital in 2015. Using our newly developed annotation tool, well-trained rheumatologists and radiologists labeled the mTSS to the wrist, metacarpal phalangeal joints, and proximal interphalangeal joints included in the images. We identified 21 X-ray images containing one or more subluxation joints and 42 X-ray images with ankylosis. To predict subluxation/ankylosis, we conducted five-fold cross-validation with deep neural network models: AlexNet, ResNet, DenseNet, and Vision Transformer. The best performance on wrist subluxation/ankylosis classification was as follows: accuracy, precision, recall, F1 value, and AUC were 0.97±0.01/0.89±0.04, 0.92±0.12/0.77±0.15, 0.77±0.16/0.71±0.13, 0.82±0.11/0.72±0.09, and 0.92±0.08/0.85±0.07, respectively. The classification model based on a deep neural network was trained with a relatively small dataset; however, it showed good accuracy. In conclusion, we provided data collection and model training schemes for mTSS prediction and showed an important contribution to building an automated scoring system.

Adaptive IoU Thresholding for Improving Small Object Detection: A Proof-of-Concept Study of Hand Erosions Classification of Patients with Rheumatic Arthritis on X-ray Images

In recent years, much research evaluating the radiographic destruction of finger joints in patients with rheumatoid arthritis (RA) using deep learning models was conducted. Unfortunately, most previous models were not clinically applicable due to the small object regions as well as the close spatial relationship. In recent years, a new network structure called RetinaNets, in combination with the focal loss function, proved reliable for detecting even small objects. Therefore, the study aimed to increase the recognition performance to a clinically valuable level by proposing an innovative approach with adaptive changes in intersection over union (IoU) values during training of Retina Networks using the focal loss error function. To this end, the erosion score was determined using the Sharp van der Heijde (SvH) metric on 300 conventional radiographs from 119 patients with RA. Subsequently, a standard RetinaNet with different IoU values as well as adaptively modified IoU values were trained and compared in terms of accuracy, mean average accuracy (mAP), and IoU. With the proposed approach of adaptive IoU values during training, erosion detection accuracy could be improved to 94% and an mAP of 0.81 ± 0.18. In contrast Retina networks with static IoU values achieved only an accuracy of 80% and an mAP of 0.43 ± 0.24. Thus, adaptive adjustment of IoU values during training is a simple and effective method to increase the recognition accuracy of small objects such as finger and wrist joints.

RATING: Medical knowledge-guided rheumatoid arthritis assessment from multimodal ultrasound images via deep learning

Multimodal ultrasound has demonstrated its power in the clinical assessment of rheumatoid arthritis (RA). However, for radiologists, it requires strong experience. In this paper, we propose a rheumatoid arthritis knowledge guided (RATING) system that automatically scores the RA activity and generates interpretable features to assist radiologists' decision-making based on deep learning. RATING leverages the complementary advantages of multimodal ultrasound images and solves the limited training data problem with self-supervised pretraining. RATING outperforms all of the existing methods, achieving an accuracy of 86.1% on a prospective test dataset and 85.0% on an external test dataset. A reader study demonstrates that the RATING system improves the average accuracy of 10 radiologists from 41.4% to 64.0%. As an assistive tool, not only can RATING indicate the possible lesions and enhance the diagnostic performance with multimodal ultrasound but it can also enlighten the road to human-machine collaboration in healthcare.