DADP: Dynamic abnormality detection and progression for longitudinal knee magnetic resonance images from the Osteoarthritis Initiative

Multiscale-temporal Feature Extraction and boundary confusion alleviation for VAG-based fine-grained multi-grade osteoarthritis deterioration monitoring

BACKGROUND AND OBJECTIVE

Multi-grade osteoarthritis (OA) deterioration monitoring in the daily paradigm using Vibroarthrography (VAG) is very challenging due to two difficulties: (1) the composition of VAG signals is complex in the daily paradigm where friction is intensified because of weight-bearing movements. (2) VAG signal samples near the decision boundary of adjacent deterioration grades are easy to be misclassified. The majority of existing works only focus on the binary classification of OA, providing inadequate assistance in instructing physicians to develop treatment plans based on the presence or absence of OA. Thus, we propose a novel framework for fine-grained multi-grade OA deterioration monitoring in the daily paradigm.

METHODS

We propose an end-to-end deep learning framework termed Fine-grained Multi-grade OA Deterioration Monitor (FMOADM), which consists of Multiscale-temporal Feature Extraction (MTFE) and Confusion-Free Master-Slave (CF-MS) Classification. Specifically, MTFE is adopted to extract multiscale-temporal discriminative features from the complicated VAG signals. And center loss is introduced by CF-MS to alleviate confusion at the boundary of adjacent deterioration grades in the feature space. Meanwhile, a master-slave structure is proposed for further fine-grained classification, where the master classifier integrates a channel attention mechanism and the slave classifier is designed to update MTFE parameters. As a result, the proposed method ensures fine-grained multi-grade OA monitoring performance via multiscale-temporal discriminative features and boundary confusion alleviation.

RESULTS

Experimental results on the VAG-OA dataset demonstrate that our framework outperforms counterpart methods in the daily paradigm. The proposed framework achieved 78% in precision, obtaining an 8% improvement over the state-of-the-art method.

CONCLUSION

The proposed framework benefits efficient multi-grade OA deterioration monitoring, empowering physicians to develop treatment plans based on fine-grained monitoring results. It takes knee joint health monitoring in daily activities a step further toward feasible.

Enhancing the vision-language foundation model with key semantic knowledge-emphasized report refinement

Recently, vision-language representation learning has made remarkable advancements in building up medical foundation models, holding immense potential for transforming the landscape of clinical research and medical care. The underlying hypothesis is that the rich knowledge embedded in radiology reports can effectively assist and guide the learning process, reducing the need for additional labels. However, these reports tend to be complex and sometimes even consist of redundant descriptions that make the representation learning too challenging to capture the key semantic information. This paper develops a novel iterative vision-language representation learning framework by proposing a key semantic knowledge-emphasized report refinement method. Particularly, raw radiology reports are refined to highlight the key information according to a constructed clinical dictionary and two model-optimized knowledge-enhancement metrics. The iterative framework is designed to progressively learn, starting from gaining a general understanding of the patient's condition based on raw reports and gradually refines and extracts critical information essential to the fine-grained analysis tasks. The effectiveness of the proposed framework is validated on various downstream medical image analysis tasks, including disease classification, region-of-interest segmentation, and phrase grounding. Our framework surpasses seven state-of-the-art methods in both fine-tuning and zero-shot settings, demonstrating its encouraging potential for different clinical applications.

Learning co-plane attention across MRI sequences for diagnosing twelve types of knee abnormalities

Multi-sequence magnetic resonance imaging is crucial in accurately identifying knee abnormalities but requires substantial expertise from radiologists to interpret. Here, we introduce a deep learning model incorporating co-plane attention across image sequences to classify knee abnormalities. To assess the effectiveness of our model, we collected the largest multi-sequence knee magnetic resonance imaging dataset involving the most comprehensive range of abnormalities, comprising 1748 subjects and 12 types of abnormalities. Our model achieved an overall area under the receiver operating characteristic curve score of 0.812. It achieved an average accuracy of 0.78, outperforming junior radiologists (accuracy 0.65) and remains competitive with senior radiologists (accuracy 0.80). Notably, with the assistance of model output, the diagnosis accuracy of all radiologists was improved significantly (p < 0.001), elevating from 0.73 to 0.79 on average. The interpretability analysis demonstrated that the model decision-making process is consistent with the clinical knowledge, enhancing its credibility and reliability in clinical practice.

Investigating the relationship between radiographic joint space width loss and deep learning-derived magnetic resonance imaging-based cartilage thickness loss in the medial weight-bearing region of the tibiofemoral joint

Objective

To investigate the relationship between measures of radiographic joint space width (JSW) loss and magnetic resonance imaging (MRI)-based cartilage thickness loss in the medial weight-bearing region of the tibiofemoral joint over 12-24 months. To stratify this relationship by clinically meaningful subgroups (sex and pain status).

Design

We analyzed a subset of knees (n ​= ​256) from the Osteoarthritis Initiative (OAI) likely in early stage OA based on joint space narrowing (JSN) measurements. Natural logarithm transformation was used to approximate near normal distributions for JSW loss. Pearson Correlation coefficients described the relationship between ln-transformed JSW loss and several versions of deep learning-derived MRI-based cartilage thickness loss parameters (minimum, maximum, and mean) in subregions of the femoral condyle, tibial plateau, and combined femoral and tibial regions. Linear mixed-effects models evaluated the associations between the ln-transformed radiographic and MRI-derived measures including potential confounders.

Results

We found weak correlations between ln-transformed JSW loss and MRI-based cartilage thickness ranging from R ​= ​-0.13 (p ​= ​0.20) to R ​= ​0.26 (p ​< ​0.01). Correlations were higher (still poor) among females compared to males and painful compared to non-painful knees. Model results showed weak associations for nearly all MRI-based measures, ranging from no association to β (95% CI) ​= ​0.25 (0.11, 0.39). Associations were higher among females compared to males and minimal differences between painful and non-painful knees.

Conclusions

Despite its recommended use in disease-modifying OA drug clinical trials, results suggest that JSW loss is an ineffective proxy measure of cartilage thickness loss over 12-24 months and within a localized region of the tibiofemoral joint.

Modeling Longitudinal Optical Coherence Tomography Images for Monitoring and Analysis of Glaucoma Progression

Glaucoma causes progressive visual field deterioration and is the leading cause of blindness worldwide. Glaucomatous damage is irreversible and greatly impacts quality of life. Therefore, it is critically important to detect glaucoma early and closely monitor progression to preserve functional vision. Glaucoma is routinely monitored in the clinical setting using optical coherence tomography (OCT) for derived measures such as the thickness of important visual structures. There is not a consensus of what measures represent the most relevant biomarkers of glaucoma progression. Further, despite the increasing availability of longitudinal OCT data, a quantitative model of 3D structural change over time associated with glaucoma does not exist. In this paper we present an algorithm that will perform hierarchical geodesic modeling at the imaging level, considering 3D OCT images as observations of structural change over time. Hierarchical modeling includes subject-wise trajectories as geodesics in the space of diffeomorphisms and population level (glaucoma vs control) trajectories are also geodesics which explain subject-wise trajectories as deviations from the mean. Our preliminary experiments demonstrate a greater magnitude of structural change associated with glaucoma compared to normal aging. Our algorithm has the potential application in patient-specific monitoring and analysis of glaucoma progression as well as a statistical model of population trends and population variability.

MRIS: A Multi-modal Retrieval Approach for Image Synthesis on Diverse Modalities

Multiple imaging modalities are often used for disease diagnosis, prediction, or population-based analyses. However, not all modalities might be available due to cost, different study designs, or changes in imaging technology. If the differences between the types of imaging are small, data harmonization approaches can be used; for larger changes, direct image synthesis approaches have been explored. In this paper, we develop an approach based on multi-modal metric learning to synthesize images of diverse modalities. We use metric learning via multi-modal image retrieval, resulting in embeddings that can relate images of different modalities. Given a large image database, the learned image embeddings allow us to use k-nearest neighbor (k-NN) regression for image synthesis. Our driving medical problem is knee osteoarthritis (KOA), but our developed method is general after proper image alignment. We test our approach by synthesizing cartilage thickness maps obtained from 3D magnetic resonance (MR) images using 2D radiographs. Our experiments show that the proposed method outperforms direct image synthesis and that the synthesized thickness maps retain information relevant to downstream tasks such as progression prediction and Kellgren-Lawrence grading (KLG). Our results suggest that retrieval approaches can be used to obtain high-quality and meaningful image synthesis results given large image databases.

Charting Aging Trajectories of Knee Cartilage Thickness for Early Osteoarthritis Risk Prediction: An MRI Study from the Osteoarthritis Initiative Cohort

Knee osteoarthritis (OA), a prevalent joint disease in the U.S., poses challenges in terms of predicting of its early progression. Although high-resolution knee magnetic resonance imaging (MRI) facilitates more precise OA diagnosis, the heterogeneous and multifactorial aspects of OA pathology remain significant obstacles for prognosis. MRI-based scoring systems, while standardizing OA assessment, are both time-consuming and labor-intensive. Current AI technologies facilitate knee OA risk scoring and progression prediction, but these often focus on the symptomatic phase of OA, bypassing initial-stage OA prediction. Moreover, their reliance on complex algorithms can hinder clinical interpretation. To this end, we make this effort to construct a computationally efficient, easily-interpretable, and state-of-the-art approach aiding in the radiographic OA (rOA) auto-classification and prediction of the incidence and progression, by contrasting an individual's cartilage thickness with a similar demographic in the rOA-free cohort. To better visualize, we have developed the toolset for both prediction and local visualization. A movie demonstrating different subtypes of dynamic changes in local centile scores during rOA progression is available at https://tli3.github.io/KneeOA/. Specifically, we constructed age-BMI-dependent reference charts for knee OA cartilage thickness, based on MRI scans from 957 radiographic OA (rOA)-free individuals from the Osteoarthritis Initiative cohort. Then we extracted local and global centiles by contrasting an individual's cartilage thickness to the rOA-free cohort with a similar age and BMI. Using traditional boosting approaches with our centile-based features, we obtain rOA classification of KLG ≤ 1 versus KLG = 2 (AUC = 0.95, F1 = 0.89), KLG ≤ 1 versus KLG ≥ 2 (AUC = 0.90, F1 = 0.82) and prediction of KLG2 progression (AUC = 0.98, F1 = 0.94), rOA incidence (KLG increasing from < 2 to ≥ 2; AUC = 0.81, F1 = 0.69) and rOA initial transition (KLG from 0 to 1; AUC = 0.64, F1 = 0.65) within a future 48-month period. Such performance in classifying KLG ≥ 2 matches that of deep learning methods in recent literature. Furthermore, its clinical interpretation suggests that cartilage changes, such as thickening in lateral femoral and anterior femoral regions and thinning in lateral tibial regions, may serve as indicators for prediction of rOA incidence and early progression. Meanwhile, cartilage thickening in the posterior medial and posterior lateral femoral regions, coupled with a reduction in the central medial femoral region, may signify initial phases of rOA transition.

Artificial intelligence in knee osteoarthritis: A comprehensive review for 2022

Objective

The aim of this literature review is to yield a comprehensive and exhaustive overview of the existing evidence and up-to-date applications of artificial intelligence for knee osteoarthritis.

Methods

A literature review was performed by using PubMed, Google Scholar, and IEEE databases for articles published in peer-reviewed journals in 2022. The articles focusing on the use of artificial intelligence in diagnosis and prognosis of knee osteoarthritis and accelerating the image acquisition were selected. For each selected study, the code availability, considered number of patients and knees, imaging type, covariates, grading type of osteoarthritis, models, validation approaches, objectives, and results were reviewed.

Results

395 articles were screened, and 35 of them were reviewed. Eight articles were based on diagnosis, six on prognosis prediction, three on classification, three on accelerated image acquisition, and 15 on segmentation of knee osteoarthritis. 57% of the articles used MRI, 26% radiography, 6% MRI together with radiography, 6% ultrasonography, and 6% only clinical data. 23% of the articles made the computer codes available for their study, and 26% used clinical data. External validation and nested cross-validation were used in 17% and 14% of articles, respectively.

Conclusions

The use of artificial intelligence provided a promising potential to enhance the detection and management of knee osteoarthritis. Translating the developed models into clinics is still in the early stages of development. The translation of artificial intelligence models is expected to be further examined in prospective studies to support clinicians in improving routine healthcare practice.

Magnetic resonance imaging assessments for knee segmentation and their use in combination with machine/deep learning as predictors of early osteoarthritis diagnosis and prognosis

Knee osteoarthritis (OA) is a prevalent and disabling disease that can develop over decades. This disease is heterogeneous and involves structural changes in the whole joint, encompassing multiple tissue types. Detecting OA before the onset of irreversible changes is crucial for early management, and this could be achieved by allowing knee tissue visualization and quantifying their changes over time. Although some imaging modalities are available for knee structure assessment, magnetic resonance imaging (MRI) is preferred. This narrative review looks at existing literature, first on MRI-developed approaches for evaluating knee articular tissues, and second on prediction using machine/deep-learning-based methodologies and MRI as input or outcome for early OA diagnosis and prognosis. A substantial number of MRI methodologies have been developed to assess several knee tissues in a semi-quantitative and quantitative fashion using manual, semi-automated and fully automated systems. This dynamic field has grown substantially since the advent of machine/deep learning. Another active area is predictive modelling using machine/deep-learning methodologies enabling robust early OA diagnosis/prognosis. Moreover, incorporating MRI markers as input/outcome in such predictive models is important for a more accurate OA structural diagnosis/prognosis. The main limitation of their usage is the ability to move them in rheumatology practice. In conclusion, MRI knee tissue determination and quantification provide early indicators for individuals at high risk of developing this disease or for patient prognosis. Such assessment of knee tissues, combined with the development of models/tools from machine/deep learning using, in addition to other parameters, MRI markers for early diagnosis/prognosis, will maximize opportunities for individualized risk assessment for use in clinical practice permitting precision medicine. Future efforts should be made to integrate such prediction models into open access, allowing early disease management to prevent or delay the OA outcome.

Patterns of variation among baseline femoral and tibial cartilage thickness and clinical features: Data from the osteoarthritis initiative

Objective

To employ novel methodologies to identify phenotypes in knee OA based on variation among three baseline data blocks: 1) femoral cartilage thickness, 2) tibial cartilage thickness, and 3) participant characteristics and clinical features.

Methods

Baseline data were from 3321 Osteoarthritis Initiative (OAI) participants with available cartilage thickness maps (6265 knees) and 77 clinical features. Cartilage maps were obtained from 3D DESS MR images using a deep-learning based segmentation approach and an atlas-based analysis developed by our group. Angle-based Joint and Individual Variation Explained (AJIVE) was used to capture and quantify variation, both shared among multiple data blocks and individual to each block, and to determine statistical significance.

Results

Three major modes of variation were shared across the three data blocks. Mode 1 reflected overall thicker cartilage among men, those with higher education, and greater knee forces; Mode 2 showed associations between worsening Kellgren-Lawrence Grade, medial cartilage thinning, and worsening symptoms; and Mode 3 contrasted lateral and medial-predominant cartilage loss associated with BMI and malalignment. Each data block also demonstrated individual, independent modes of variation consistent with the known discordance between symptoms and structure in knee OA and reflecting the importance of features such as physical function, symptoms, and comorbid conditions independent of structural damage.

Conclusions

This exploratory analysis, combining the rich OAI dataset with novel methods for determining and visualizing cartilage thickness, reinforces known associations in knee OA while providing insights into the potential for data integration in knee OA phenotyping.

Discovering Knee Osteoarthritis Imaging Features for Diagnosis and Prognosis: Review of Manual Imaging Grading and Machine Learning Approaches

Knee osteoarthritis (OA) is a deliberating joint disorder characterized by cartilage loss that can be captured by imaging modalities and translated into imaging features. Observing imaging features is a well-known objective assessment for knee OA disorder. However, the variety of imaging features is rarely discussed. This study reviews knee OA imaging features with respect to different imaging modalities for traditional OA diagnosis and updates recent image-based machine learning approaches for knee OA diagnosis and prognosis. Although most studies recognized X-ray as standard imaging option for knee OA diagnosis, the imaging features are limited to bony changes and less sensitive to short-term OA changes. Researchers have recommended the usage of MRI to study the hidden OA-related radiomic features in soft tissues and bony structures. Furthermore, ultrasound imaging features should be explored to make it more feasible for point-of-care diagnosis. Traditional knee OA diagnosis mainly relies on manual interpretation of medical images based on the Kellgren-Lawrence (KL) grading scheme, but this approach is consistently prone to human resource and time constraints and less effective for OA prevention. Recent studies revealed the capability of machine learning approaches in automating knee OA diagnosis and prognosis, through three major tasks: knee joint localization (detection and segmentation), classification of OA severity, and prediction of disease progression. AI-aided diagnostic models improved the quality of knee OA diagnosis significantly in terms of time taken, reproducibility, and accuracy. Prognostic ability was demonstrated by several prediction models in terms of estimating possible OA onset, OA deterioration, progressive pain, progressive structural change, progressive structural change with pain, and time to total knee replacement (TKR) incidence. Despite research gaps, machine learning techniques still manifest huge potential to work on demanding tasks such as early knee OA detection and estimation of future disease events, as well as fundamental tasks such as discovering the new imaging features and establishment of novel OA status measure. Continuous machine learning model enhancement may favour the discovery of new OA treatment in future.