Light field otoscope design for 3D in vivo imaging of the middle ear

Development and Validation of an Automated Classifier to Diagnose Acute Otitis Media in Children

Importance

Acute otitis media (AOM) is a frequently diagnosed illness in children, yet the accuracy of diagnosis has been consistently low. Multiple neural networks have been developed to recognize the presence of AOM with limited clinical application.

Objective

To develop and internally validate an artificial intelligence decision-support tool to interpret videos of the tympanic membrane and enhance accuracy in the diagnosis of AOM.

Design, Setting, and Participants

This diagnostic study analyzed otoscopic videos of the tympanic membrane captured using a smartphone during outpatient clinic visits at 2 sites in Pennsylvania between 2018 and 2023. Eligible participants included children who presented for sick visits or wellness visits.

Exposure

Otoscopic examination.

Main Outcomes and Measures

Using the otoscopic videos that were annotated by validated otoscopists, a deep residual-recurrent neural network was trained to predict both features of the tympanic membrane and the diagnosis of AOM vs no AOM. The accuracy of this network was compared with a second network trained using a decision tree approach. A noise quality filter was also trained to prompt users that the video segment acquired may not be adequate for diagnostic purposes.

Results

Using 1151 videos from 635 children (majority younger than 3 years of age), the deep residual-recurrent neural network had almost identical diagnostic accuracy as the decision tree network. The finalized deep residual-recurrent neural network algorithm classified tympanic membrane videos into AOM vs no AOM categories with a sensitivity of 93.8% (95% CI, 92.6%-95.0%) and specificity of 93.5% (95% CI, 92.8%-94.3%) and the decision tree model had a sensitivity of 93.7% (95% CI, 92.4%-94.9%) and specificity of 93.3% (92.5%-94.1%). Of the tympanic membrane features outputted, bulging of the TM most closely aligned with the predicted diagnosis; bulging was present in 230 of 230 cases (100%) in which the diagnosis was predicted to be AOM in the test set.

Conclusions and Relevance

These findings suggest that given its high accuracy, the algorithm and medical-grade application that facilitates image acquisition and quality filtering could reasonably be used in primary care or acute care settings to aid with automated diagnosis of AOM and decisions regarding treatment.

Objective Quality Assessment Metrics for Light Field Image Based on Textural Features

Light Field (LF) imaging is a plenoptic data collection method enabling a wide variety of image post-processing such as 3D extraction, viewpoint change and digital refocusing. Moreover, LF provides the capability to capture rich information about a scene, e.g., texture, geometric information, etc. Therefore, a quality assessment model for LF images is needed and poses significant challenges. Many LF Image Quality Assessment (LF-IQA) metrics have been recently presented based on the unique characteristics of LF images. The state-of-the-art objective assessment metrics have taken into account the image content and human visual system such as SSIM and IW-SSIM. However, most of these metrics are designed for images and video with natural content. Additionally, other models based on the LF characteristics (e.g., depth information, angle information) trade high performance for high computational complexity, along with them possessing difficulties of implementation for LF applications due to the immense data requirements of LF images. Hence, this paper presents a novel content-adaptive LF-IQA metric to improve the conventional LF-IQA performance that is also low in computational complexity. The experimental results clearly show improved performance compared to conventional objective IQA metrics, and we also identify metrics that are well-suited for LF image assessment. In addition, we present a comprehensive content-based feature analysis to determine the most appropriate feature that influences human visual perception among the widely used conventional objective IQA metrics. Finally, a rich LF dataset is selected from the EPFL dataset, allowing for the study of light field quality by qualitative factors such as depth (wide and narrow), focus (background or foreground) and complexity (simple and complex).

Three-dimensional light-field microendoscopy with a GRIN lens array

Optical endoscopy has emerged as an indispensable clinical tool for modern minimally invasive surgery. Most systems primarily capture a 2D projection of the 3D surgical field. Currently available 3D endoscopes can restore stereoscopic vision directly by projecting laterally shifted views of the operating field to each eye through 3D glasses. These tools provide surgeons with informative 3D visualizations, but they do not enable quantitative volumetric rendering of tissue. Therefore, advanced tools are desired to quantify tissue tomography for high precision microsurgery or medical robotics. Light-field imaging suggests itself as a promising solution to the challenge. The approach can capture both the spatial and angular information of optical signals, permitting the computational synthesis of the 3D volume of an object. In this work, we present GRIN lens array microendoscopy (GLAM), a single-shot, full-color, and quantitative 3D microendoscopy system. GLAM contains integrated fiber optics for illumination and a GRIN lens array to capture the reflected light field. The system exhibits a 3D resolution of ∼100 µm over an imaging depth of ∼22 mm and field of view up to 1 cm². GLAM maintains a small form factor consistent with the clinically desirable design, making the system readily translatable to a clinical prototype.

New Approaches and Technologies to Improve Accuracy of Acute Otitis Media Diagnosis

Several studies have shown that in recent years incidence of acute otitis media (AOM) has declined worldwide. However, related medical, social, and economic problems for patients, their families, and society remain very high. Better knowledge of potential risk factors for AOM development and more effective preventive interventions, particularly in AOM-prone children, can further reduce disease incidence. However, a more accurate AOM diagnosis seems essential to achieve this goal. Diagnostic uncertainty is common, and to avoid risks related to a disease caused mainly by bacteria, several children without AOM are treated with antibiotics and followed as true AOM cases. The main objective of this manuscript is to discuss the most common difficulties that presently limit accurate AOM diagnosis and the new approaches and technologies that have been proposed to improve disease detection. We showed that misdiagnosis can be dangerous or lead to relevant therapeutic mistakes. The need to improve AOM diagnosis has allowed the identification of a long list of technologies to visualize and evaluate the tympanic membrane and to assess middle-ear effusion. Most of the new instruments, including light field otoscopy, optical coherence tomography, low-coherence interferometry, and Raman spectroscopy, are far from being introduced in clinical practice. Video-otoscopy can be effective, especially when it is used in association with telemedicine, parents' cooperation, and artificial intelligence. Introduction of otologic telemedicine and use of artificial intelligence among pediatricians and ENT specialists must be strongly promoted in order to reduce mistakes in AOM diagnosis.

PlenoptiCam v1.0: A Light-Field Imaging Framework

Light-field cameras play a vital role for rich 3D information retrieval in narrow range depth sensing applications. The key obstacle in composing light-fields from exposures taken by a plenoptic camera is to computationally calibrate, align and rearrange four-dimensional image data. Several attempts have been proposed to enhance the overall image quality by tailoring pipelines dedicated to particular plenoptic cameras and improving the consistency across viewpoints at the expense of high computational loads. The framework presented herein advances prior outcomes thanks to its novel micro image scale-space analysis for generic camera calibration independent of the lens specifications and its parallax-invariant, cost-effective viewpoint color equalization from optimal transport theory. Artifacts from the sensor and micro lens grid are compensated in an innovative way to enable superior quality in sub-aperture image extraction, computational refocusing and Scheimpflug rendering with sub-sampling capabilities. Benchmark comparisons using established image metrics suggest that our proposed pipeline outperforms state-of-the-art tool chains in the majority of cases. Results from a Wasserstein distance further show that our color transfer outdoes the existing transport methods. Our algorithms are released under an open-source license, offer cross-platform compatibility with few dependencies and different user interfaces. This makes the reproduction of results and experimentation with plenoptic camera technology convenient for peer researchers, developers, photographers, data scientists and others working in this field.

Developing an optical design pipeline for correcting lens aberrations and vignetting in light field cameras

Light field cameras have been employed in myriad applications thanks to their 3D imaging capability. By placing a microlens array in front of a conventional camera, one can measure both the spatial and angular information of incoming light rays and reconstruct a depth map. The unique optical architecture of light field cameras poses new challenges on controlling aberrations and vignetting in lens design process. The results of our study show that field curvature can be numerically corrected for by digital refocusing, and vignetting must be minimized because it reduces the depth reconstruction accuracy. To address this unmet need, we herein present an optical design pipeline for light field cameras and demonstrated its implementation in a light field endoscope.

Stepwise calibration of plenoptic cameras based on corner features of raw images

Plenoptic cameras are increasingly gaining attention in various fields due to their ability to capture both spatial and angular information of light rays. Accurate geometric calibration can lay a solid foundation for the applications that use the plenoptic camera. In this paper, to the best of our knowledge, we first introduce an accurate corner detection method based on a novel selection and refinement strategy. The detected-corner candidates on raw images are selected by a random sample consensus (RANSAC)-based algorithm and optimized by the photometric similarity, as well as the sub-pixel refinement. In addition, a robust and accurate stepwise calibration method is proposed based on separated intrinsic parameters, including parameters related to the pinhole model and those unique to the plenoptic camera. Experiments on both simulated and real data demonstrate that our method outperforms the state-of-the-art methods and is able to support a more accurate calibration of plenoptic cameras.

Panel 1: Biotechnology, biomedical engineering and new models of otitis media

OBJECTIVE

To summarize recently published key articles on the topics of biomedical engineering, biotechnology and new models in relation to otitis media (OM).

DATA SOURCES

Electronic databases: PubMed, Ovid Medline, Cochrane Library and Clinical Evidence (BMJ Publishing).

REVIEW METHODS

Articles on biomedical engineering, biotechnology, material science, mechanical and animal models in OM published between May 2015 and May 2019 were identified and subjected to review. A total of 132 articles were ultimately included.

RESULTS

New imaging technologies for the tympanic membrane (TM) and the middle ear cavity are being developed to assess TM thickness, identify biofilms and differentiate types of middle ear effusions. Artificial intelligence (AI) has been applied to train software programs to diagnose OM with a high degree of certainty. Genetically modified mice models for OM have further investigated what predisposes some individuals to OM and consequent hearing loss. New vaccine candidates protecting against major otopathogens are being explored and developed, especially combined vaccines, targeting more than one pathogen. Transcutaneous vaccination against non-typeable Haemophilus influenzae has been successfully tried in a chinchilla model. In terms of treatment, novel technologies for trans-tympanic drug delivery are entering the clinical domain. Various growth factors and grafting materials aimed at improving healing of TM perforations show promising results in animal models.

CONCLUSION

New technologies and AI applications to improve the diagnosis of OM have shown promise in pre-clinical models and are gradually entering the clinical domain. So are novel vaccines and drug delivery approaches that may allow local treatment of OM.

IMPLICATIONS FOR PRACTICE

New diagnostic methods, potential vaccine candidates and the novel trans-tympanic drug delivery show promising results, but are not yet adapted to clinical use.

High resolution, programmable aperture light field laparoscope for quantitative depth mapping

Recent applications have shown that light field imaging can be useful for developing uniaxial three-dimensional (3D) endoscopes. The immediate challenges in implementation are a tradeoff in lateral resolution and acquiring enough depth information in the physically limited environment of minimally invasive surgery. Here we propose using programmable aperture light field imaging in laparoscopy to capture 3D information without sacrificing the camera sensor's native, high spatial resolution. This hybrid design utilizes a programmable aperture to preserve the conventional laparoscope's functionality and, upon demand, to compute a depth map for surgical guidance. A working prototype is demonstrated.

Optical-aberrations-corrected light field re-projection for high-quality plenoptic imaging

The singlet plenoptic camera, which consists of a single lens, microlens array (MLA) and image sensor, possesses the superiority that the imaging system is compact and lightweight, which is beneficial to miniaturization. However, such plenoptic cameras suffer from severe optical aberrations and their imaging quality is inferior for post-capture processing. Therefore, this paper proposes an optical-aberrations-corrected light field re-projection method to obtain high-quality singlet plenoptic imaging. First, optical aberrations are modeled by Seidel polynomials and included into point spread function (PSF) modeling. The modeled PSF is subsequently used to reconstruct imaging object information. Finally, the reconstructed imaging object information is re-projected back to the plenoptic imaging plane to obtain high-quality plenoptic images without optical aberrations. PSF modeling is validated by a self-built singlet plenoptic camera and the utility of the proposed optical-aberrations-corrected light field re-projection method is verified by numerical simulations and real imaging experiments.

Functional assessment of moisture influenced cadaveric tympanic membrane using phase shift-resolved optical Doppler vibrography

An elevated relative moisture in the external ear canal and middle ear cavity may predispose to chronic otorrhea and related infections along with abnormal tympanic membrane (TM) vibration patterns. Therefore, phase shift-resolved optical Doppler vibrography (ODV) was used for vibration assessments of moisture influenced cadaveric TM. ODV was applied to generate time resolved cross-sectional and volumetric vibrographs of a cadaveric TM, driven acoustically at several frequencies. In order to analyze the effect of moisture on TM, homogenous moisture conditions were provided by soaking the cadaveric TM specimens in 1× phosphate buffer saline with a pH of 7.4. The TM specimen was exposed to a rapidly switchable frequency generator during the ODV image acquisition. The experiment was conducted for 3 hours and the cadaveric TM was exposed to each frequency with an interval of 30 minutes. Acquired phase shift-resolved ODV assessments revealed a depth dependent vibration tendency between the applied frequencies, along with a decline in the moisture level of the cadaveric TM specimen. Thus, the ODV method can aid our understanding of sound conduction in the middle ear, thus supporting the diagnosis of TM diseases.

Snapshot hyperspectral light field imaging using image mapping spectrometry

In this Letter, we present a snapshot hyperspectral light field imaging system using a single camera. By integrating an unfocused light field camera with a snapshot hyperspectral imager, the image mapping spectrometer, we captured a five-dimensional (5D) ($x,y,u,v,\lambda $x,y,u,v,λ) ($x,y,$x,y, spatial coordinates; $u,v,$u,v, emittance angles; $\lambda ,$λ, wavelength) datacube in a single camera exposure. The corresponding volumetric image ($x,y,z$x,y,z) at each wavelength is then computed through a scale-depth space transform. We demonstrated the snapshot advantage of our system by imaging the spectral-volumetric scenes in real time.

Automated classification platform for the identification of otitis media using optical coherence tomography

The diagnosis and treatment of otitis media (OM), a common childhood infection, is a significant burden on the healthcare system. Diagnosis relies on observer experience via otoscopy, although for non-specialists or inexperienced users, accurate diagnosis can be difficult. In past studies, optical coherence tomography (OCT) has been used to quantitatively characterize disease states of OM, although with the involvement of experts to interpret and correlate image-based indicators of infection with clinical information. In this paper, a flexible and comprehensive framework is presented that automatically extracts features from OCT images, classifies data, and presents clinically relevant results in a user-friendly platform suitable for point-of-care and primary care settings. This framework was used to test the discrimination between OCT images of normal controls, ears with biofilms, and ears with biofilms and middle ear fluid (effusion). Predicted future performance of this classification platform returned promising results (90%+ accuracy) in various initial tests. With integration into patient healthcare workflow, users of all levels of medical experience may be able to collect OCT data and accurately identify the presence of middle ear fluid and/or biofilms.

Complete plenoptic imaging using a single detector

Multi-dimensional imaging is a powerful technique for many applications, such as biological analysis, remote sensing, and object recognition. Most existing multi-dimensional imaging systems rely on scanning or camera array, which make the system bulky and unstable. To some extent, these problems can be mitigated by employing compressed sensing algorithms. However, they are computationally expensive and highly rely on the ill-posed assumption that the information is sparse in a given domain. Here, we propose a snapshot spectral-volumetric imaging (SSVI) system by introducing the paradigm of light-field imaging into Fourier transform imaging spectroscopy. We demonstrate that SSVI can reconstruct a complete plenoptic function, P(x,y,z,θ,φ,λ,t), of the incoming light rays using a single detector. Compared with other multidimensional imagers, SSVI features prominent advantages in compactness, robustness, and low cost.

Combined high-speed holographic shape and full-field displacement measurements of tympanic membrane

The conical shape of the tympanic membrane (TM or eardrum) plays an important role in its function, such that variations in shape alter the acoustically induced motions of the TM. We present a method that precisely determines both shape and acoustically induced transient response of the entire TM using the same optics and maintaining the same coordinate system, where the TM transient displacements due to a broadband acoustic click excitation (50-μs impulse) and the shape are consecutively measured within <200  ms. Interferograms gathered with continuous high-speed (>2  kHz) optical phase sampling during a single 100-ms wavelength tuning ramp allow precise and rapid reconstructions of the TM shape at varied resolutions (50 to 200  μm). This rapid acquisition of full-field displacements and shape is immune to slow disturbances introduced by breathing or heartbeat of live subjects. Knowledge of TM shape and displacements enables the estimation of surface normal displacements regardless of the orientation of the TM within the measurement system. The proposed method helps better define TM mechanics and provides TM structure and function information useful for the diagnosis of ear disease.

Glare-free retinal imaging using a portable light field fundus camera

We present the retinal plenoptoscope, a novel light field retinal imaging device designed to overcome many of the problems that limit the use of portable non-mydriatic fundus cameras, including image quality and lack of stereopsis. The design and prototype construction of this device is detailed and the ideal relationship between the eye pupil, system aperture stop and micro-image separation is investigated. A comparison of the theoretical entrance pupil size, multi-view baseline and depth resolution indicates that a higher degree of stereopsis is possible than with stereo fundus cameras. We also show that the effects of corneal backscatter on image quality can be removed through a novel method of glare identification and selective image rendering. This method is then extended to produce glare-free depth maps from densely estimated depth fields, creating representations of retinal topography from a single exposure. These methods are demonstrated on physical models and live human eyes using a prototype device based on a Lytro Illum consumer light field camera. The Retinal Plenoptoscope offers a viable, robust modality for non-mydriatic color and 3-D retinal imaging.

Optical Coherence Tomography of the Tympanic Membrane and Middle Ear: A Review

Objective To evaluate the recent developments in optical coherence tomography (OCT) for tympanic membrane (TM) and middle ear (ME) imaging and to identify what further development is required for the technology to be integrated into common clinical use. Data Sources PubMed, Embase, Google Scholar, Scopus, and Web of Science. Review Methods A comprehensive literature search was performed for English language articles published from January 1966 to January 2018 with the keywords "tympanic membrane or middle ear,""optical coherence tomography," and "imaging." Conclusion Conventional imaging techniques cannot adequately resolve the microscale features of TM and ME, sometimes necessitating diagnostic exploratory surgery in challenging otologic pathology. As a high-resolution noninvasive imaging technique, OCT offers promise as a diagnostic aid for otologic conditions, such as otitis media, cholesteatoma, and conductive hearing loss. Using OCT vibrometry to image the nanoscale vibrations of the TM and ME as they conduct acoustic waves may detect the location of ossicular chain dysfunction and differentiate between stapes fixation and incus-stapes discontinuity. The capacity of OCT to image depth and thickness at high resolution allows 3-dimensional volumetric reconstruction of the ME and has potential use for reconstructive tympanoplasty planning and the follow-up of ossicular prostheses. Implications for Practice To achieve common clinical use beyond these initial discoveries, future in vivo imaging devices must feature low-cost probe or endoscopic designs and faster imaging speeds and demonstrate superior diagnostic utility to computed tomography and magnetic resonance imaging. While such technology has been available for OCT, its translation requires focused development through a close collaboration between engineers and clinicians.

On the fundamental comparison between unfocused and focused light field cameras

Light field cameras have been extensively used in a variety of applications, thanks to their snapshot three-dimensional imaging capability. However, little is known regarding their pros and cons for a given application. Herein we report a fundamental comparison between two types of light field cameras-focused and unfocused. Our results indicate that the unfocused light field camera outperforms its focused counterpart in depth range and number of resolvable depth steps, while the focused light field camera has an edge in lateral resolution and reconstruction accuracy.

FIMic: design for ultimate 3D-integral microscopy of in-vivo biological samples

In this work, Fourier integral microscope (FIMic), an ultimate design of 3D-integral microscopy, is presented. By placing a multiplexing microlens array at the aperture stop of the microscope objective of the host microscope, FIMic shows extended depth of field and enhanced lateral resolution in comparison with regular integral microscopy. As FIMic directly produces a set of orthographic views of the 3D-micrometer-sized sample, it is suitable for real-time imaging. Following regular integral-imaging reconstruction algorithms, a 2.75-fold enhanced depth of field and [Formula: see text]-time better spatial resolution in comparison with conventional integral microscopy is reported. Our claims are supported by theoretical analysis and experimental images of a resolution test target, cotton fibers, and in-vivo 3D-imaging of biological specimens.