Development of image mappers for hyperspectral biomedical imaging applications

Fabrication of a multifaceted mapping mirror using two-photon polymerization for a snapshot image mapping spectrometer

A design and fabrication technique for making high-precision and large-format multifaceted mapping mirrors is presented. The method is based on two-photon polymerization, which allows more flexibility in the mapping mirror design. The mirror fabricated in this paper consists of 36 2D tilted square pixels, instead of the continuous facet design used in diamond cutting. The paper presents a detailed discussion of the fabrication parameters and optimization process, with particular emphasis on the optimization of stitching defects by compensating for the overall tilt angle and reducing the printing field of view. The fabricated mirrors were coated with a thin layer of aluminum (93 nm) using sputter coating to enhance the reflection rate over the target wave range. The mapping mirror was characterized using a white light interferometer and a scanning electron microscope, which demonstrates its optical quality surface (with a surface roughness of 12 nm) and high-precision tilt angles (with an average of 2.03% deviation). Finally, the incorporation of one of the 3D printed mapping mirrors into an image mapping spectrometer prototype allowed for the acquisition of high-quality images of the USAF resolution target and bovine pulmonary artery endothelial cells stained with three fluorescent dyes, demonstrating the potential of this technology for practical applications.

An ultra-small nine-color spectrometer with a two-layer biparted ten-dichroic-mirror array and an image sensor

An ultra-small (54 × 58 × 8.5 mm) and large aperture (1 × 7 mm) nine-color spectrometer-using an array of ten dichroic mirrors "biparted" as two layers-was developed and used for snapshot spectral imaging. Incident-light flux with a cross section smaller than the aperture size is split into nine color fluxes with 20-nm-width contiguous wavelength bands and central wavelengths of 530, 550, 570, 590, 610, 630, 650, 670, and 690 nm. Images of the nine color fluxes are simultaneously and efficiently measured by an image sensor. Unlike a conventional dichroic-mirror array, the developed dichroic-mirror array has a unique biparted configuration that not only increases the number of colors that can be measured simultaneously but also improves the image resolution of each color flux. The developed nine-color spectrometer was used for four-capillary-array electrophoresis. Eight dyes concurrently migrating in each capillary were simultaneously quantified by nine-color laser-induced fluorescence detection. Since the nine-color spectrometer is not only ultra-small and inexpensive but also has high light throughput and sufficient spectral resolution for most spectral-imaging applications, it has the potential to be widely used in various fields.

Hybrid non-sequential modeling of an image mapping spectrometer

An image mapping spectrometer (IMS) is a kind of snapshot imaging spectrometer characterized by containing several array components including the image mapper, prism array, and reimaging lens array. We propose a hybrid non-sequential modeling method of IMS and present the complete optical model of the system built in Zemax. This method utilizes the spatial periodicity of the array components and requires only a small number of input parameters. Moreover, we design a collimating lens of a large relative aperture, sufficient working distance, and low aberration to meet the requirements of an IMS with good optical performance and compact volume. The designed lens is quantitatively evaluated in the entire IMS model, and the results demonstrate that the lens has excellent optical performance. The evaluation on the collimating lens also demonstrates the capability of the proposed modeling method in the design and optimization of systems such as the IMS that contain multiple array components. The designed collimating lens is manufactured and assembled in the experimental setup of the IMS. The proposed modeling method is verified by experimental results.

Radiometric and design model for the tunable light-guide image processing snapshot spectrometer (TuLIPSS)

The tunable light-guide image processing snapshot spectrometer (TuLIPSS) is a novel remote sensing instrument that can capture a spectral image cube in a single snapshot. The optical modelling application for the absolute signal intensity on a single pixel of the sensor in TuLIPSS has been developed through a numerical simulation of the integral performance of each optical element in the TuLIPSS system. The absolute spectral intensity of TuLIPSS can be determined either from the absolute irradiance of the observed surface or from the tabulated spectral reflectance of various land covers and by the application of a global irradiance approach. The model is validated through direct comparison of the simulated results with observations. Based on tabulated spectral reflectance, the deviation between the simulated results and the measured observations is less than 5% of the spectral light flux across most of the detection bandwidth for a Lambertian-like surface such as concrete. Additionally, the deviation between the simulated results and the measured observations using global irradiance information is less than 10% of the spectral light flux across most of the detection bandwidth for all surfaces tested. This optical modelling application of TuLIPSS can be used to assist the optimal design of the instrument and explore potential applications. The influence of the optical components on the light throughput is discussed with the optimal design being a compromise among the light throughput, spectral resolution, and cube size required by the specific application under consideration. The TuLIPSS modelling predicts that, for the current optimal low-cost configuration, the signal to noise ratio can exceed 10 at 10 ms exposure time, even for land covers with weak reflectance such as asphalt and water. Overall, this paper describes the process by which the optimal design is achieved for particular applications and directly connects the parameters of the optical components to the TuLIPSS performance.

Combination of Structured Illumination Microscopy with Hyperspectral Imaging for Cell Analysis

Existing structured illumination microscopy (SIM) allows super-resolution live-cell imaging in few color channels that provide merely morphological information but cannot acquire the sample spectrum that is strongly relevant to the underlying physicochemical property. We develop hyperspectral SIM which enables high-speed spectral super-resolution imaging in SIM for the first time. Through optically mapping the three-dimensional (x, y, and λ) datacube of the sample to the detector plane, hyperspectral SIM allows snapshot spectral imaging of the SIM raw image, detecting the sample spectrum while retaining the high-speed and super-resolution characteristics of SIM. We demonstrate hyperspectral SIM imaging and reconstruct a datacube containing 31 super-resolution images of different wavelengths from only 9 exposures, achieving a 15 nm spectral resolution. We show time-lapse hyperspectral SIM imaging that achieves an imaging speed of 2.7 s per datacube-31-fold faster than the existing wavelength scanning strategy. To demonstrate the great prospects for further combining hyperspectral SIM with various spectral analysis methods, we also perform spectral unmixing of the hyperspectral SIM result while imaging the spectrally overlapped sample.

Joint artifact correction and super-resolution of image slicing and mapping system via a convolutional neural network

As the key component of the image mapping spectrometer, the image mapper introduces complex image degradation in the reconstructed images, including low spatial resolution and intensity artifacts. In this paper, we propose a novel image processing method based on the convolutional neural network to perform artifact correction and super-resolution (SR) simultaneously. The proposed joint network contains two branches to handle the artifact correction task and SR task in parallel. The artifact correction module is designed to remove the artifacts in the image and the SR module is used to improve the spatial resolution. An attention fusion module is constructed to combine the features extracted by the artifact correction and SR modules. The fused features are used to reconstruct an artifact-free high-resolution image. We present extensive simulation results to demonstrate that the proposed joint method outperforms state-of-the-art methods and can be generalized to other image mapper designs. We also provide experimental results to prove the efficiency of the joint network.

Snapshot spectroscopic microscopy with double spherical slicer mirrors

Snapshot hyperspectral microscopic imaging can obtain the morphological characteristics and chemical specificity of samples simultaneously and instantaneously. We demonstrate a double-slicer spectroscopic microscopy (DSSM) that uses two spherical slicer mirrors to magnify the target image and slice it. These slits are lined up and dispersed, then mapped onto an area-array detector. An anamorphosis unit optimizes the capacity of the limited pixels. With a single shot and image recombination, a data cube can be constructed for sample analysis, and a model of DSSM is simulated. The system covers the spectral range from 500 nm to 642.5 nm with 20 spectral channels. The spatial resolution is 417 nm, and the spectral resolution is 7.5 nm.

Spatial-frequency encoded imaging of multangular and multispectral images

Modern imaging techniques increasingly require signals to be collected from multiple viewpoints and spectral bands to realize multi-dimensional and multi-species detections. For this purpose, multiple cameras are commonly required. Each camera collects signals from one viewpoint or one spectral band, resulting in a considerable experimental cost. Based on frequency modulation, this work proposes an encoded-imaging technique that can record multangular and multispectral images in one acquisition. The signals recorded from different viewpoints and spectral bands are superimposed in the spatial domain, while being separate in the frequency domain. This allows us to extract individual images based on their respective frequency components. In this work, a proof-of-concept experiment was conducted. The high correlation coefficient between the superimposition of the extracted images and a normal superimposed image demonstrates the effectiveness of this technique. In addition, an improved mathematical formulation was proposed to describe the higher spatial-frequency components, which were considered merely to be residual lines in previous studies. The proposed encoded-imaging technique may have potential for multangular and multispectral imaging, which is especially useful for tomographic reconstructions.

Development of a fast calibration method for image mapping spectrometry

An image mapping spectrometer (IMS) is a snapshot hyperspectral imager that simultaneously captures both the spatial (x, y) and spectral (λ) information of incoming light. The IMS maps a three-dimensional (3D) datacube (x, y, λ) to a two-dimensional (2D) detector array (x, y) for parallel measurement. To reconstruct the original 3D datacube, one must construct a lookup table that connects voxels in the datacube and pixels in the raw image. Previous calibration methods suffer from either low speed or poor image quality. We herein present a slit-scan calibration method that can significantly reduce the calibration time while maintaining high accuracy. Moreover, we quantitatively analyzed the major artifact in the IMS, the striped image, and developed three numerical methods to correct for it.

Snapshot hyperspectral light field imaging using image mapping spectrometry

In this Letter, we present a snapshot hyperspectral light field imaging system using a single camera. By integrating an unfocused light field camera with a snapshot hyperspectral imager, the image mapping spectrometer, we captured a five-dimensional (5D) ($x,y,u,v,\lambda $x,y,u,v,λ) ($x,y,$x,y, spatial coordinates; $u,v,$u,v, emittance angles; $\lambda ,$λ, wavelength) datacube in a single camera exposure. The corresponding volumetric image ($x,y,z$x,y,z) at each wavelength is then computed through a scale-depth space transform. We demonstrated the snapshot advantage of our system by imaging the spectral-volumetric scenes in real time.

Accurate ray tracing model of an imaging system based on image mapper

The image mapper plays a key role in the imaging process of the image mapping spectrometer (IMS), which is a snapshot imaging spectrometer with superiority in light throughput, temporal resolution, and compactness. In this paper, an accurate ray tracing model of the imaging units of the IMS, especially the image mapper, is presented in the form of vector operation. Based on the proposed model, the behavior of light reflection on the image mapper is analyzed thoroughly, including the precise position of the reflection point and interaction between adjacent facets. Rigorous spatial correspondence between object points and pixels on the detector is determined by tracing the chief ray of an arbitrary point in the field. The shadowing effect, which is shadowing between adjacent facets and blocking caused by the facets' side walls, is analyzed based on our model. The experimental results verify the fidelity of the model and the existence of the shadowing effect. The research is meaningful for comprehending the imaging mechanism of the IMS and facilitates the design and analysis process in the future.

Hyperspectral imaging solutions for brain tissue metabolic and hemodynamic monitoring: past, current and future developments

Hyperspectral imaging (HSI) technologies have been used extensively in medical research, targeting various biological phenomena and multiple tissue types. Their high spectral resolution over a wide range of wavelengths enables acquisition of spatial information corresponding to different light-interacting biological compounds. This review focuses on the application of HSI to monitor brain tissue metabolism and hemodynamics in life sciences. Different approaches involving HSI have been investigated to assess and quantify cerebral activity, mainly focusing on: (1) mapping tissue oxygen delivery through measurement of changes in oxygenated (HbO2) and deoxygenated (HHb) hemoglobin; and (2) the assessment of the cerebral metabolic rate of oxygen (CMRO2) to estimate oxygen consumption by brain tissue. Finally, we introduce future perspectives of HSI of brain metabolism, including its potential use for imaging optical signals from molecules directly involved in cellular energy production. HSI solutions can provide remarkable insight in understanding cerebral tissue metabolism and oxygenation, aiding investigation on brain tissue physiological processes.

Snapshot Hyperspectral Light-Sheet Imaging of Signal Transduction in Live Pancreatic Islets

The observation of ionic signaling dynamics in intact pancreatic islets has contributed greatly to our understanding of both α- and β-cell function. Insulin secretion from β-cells depends on the firing of action potentials and consequent rises of intracellular calcium activity ([Ca(2+)]i). Zinc (Zn(2+)) is cosecreted with insulin, and has been postulated to play a role in cell-to-cell cross talk within an islet, in particular inhibiting glucagon secretion from α-cells. Thus, measuring [Ca(2+)]i and Zn(2+) dynamics from both α- and β-cells will elucidate mechanisms underlying islet hormone secretion. [Ca(2+)]i and intracellular Zn(2+) can be measured using fluorescent biosensors, but the most efficient sensors have overlapping spectra that complicate their discrimination. Hyperspectral imaging can be used to distinguish signals from multiple fluorophores, but available hyperspectral implementations are either too slow to measure the dynamics of ionic signals or not suitable for thick samples. We have developed a five-dimensional (x,y,z,t,λ) imaging system that leverages a snapshot hyperspectral imaging method, image mapping spectrometry, and light-sheet microscopy. This system provides subsecond temporal resolution from deep within multicellular structures. Using a single excitation wavelength (488 nm) we acquired images from triply labeled samples with two biosensors and a genetically expressing fluorescent protein (spectrally overlapping with one of the biosensors) with high temporal resolution. Measurements of [Ca(2+)]i and Zn(2+) within both α- and β-cells as a function of glucose concentration show heterogeneous uptake of Zn(2+) into α-cells that correlates to the known heterogeneities in [Ca(2+)]i. These differences in intracellular Zn(2+) among α-cells may contribute to the inhibition in glucagon secretion observed at elevated glucose levels.

A review of snapshot multidimensional optical imaging: measuring photon tags in parallel

Multidimensional optical imaging has seen remarkable growth in the past decade. Rather than measuring only the two-dimensional spatial distribution of light, as in conventional photography, multidimensional optical imaging captures light in up to nine dimensions, providing unprecedented information about incident photons' spatial coordinates, emittance angles, wavelength, time, and polarization. Multidimensional optical imaging can be accomplished either by scanning or parallel acquisition. Compared with scanning-based imagers, parallel acquisition-also dubbed snapshot imaging-has a prominent advantage in maximizing optical throughput, particularly when measuring a datacube of high dimensions. Here, we first categorize snapshot multidimensional imagers based on their acquisition and image reconstruction strategies, then highlight the snapshot advantage in the context of optical throughput, and finally we discuss their state-of-the-art implementations and applications.

Hyperspectral optical tomography of intrinsic signals in the rat cortex

We introduce a tomographic approach for three-dimensional imaging of evoked hemodynamic activity, using broadband illumination and diffuse optical tomography (DOT) image reconstruction. Changes in diffuse reflectance in the rat somatosensory cortex due to stimulation of a single whisker were imaged at a frame rate of 5 Hz using a hyperspectral image mapping spectrometer. In each frame, images in 38 wavelength bands from 484 to 652 nm were acquired simultaneously. For data analysis, we developed a hyperspectral DOT algorithm that used the Rytov approximation to quantify changes in tissue concentration of oxyhemoglobin ([Formula: see text]) and deoxyhemoglobin (ctHb) in three dimensions. Using this algorithm, the maximum changes in [Formula: see text] and ctHb were found to occur at [Formula: see text] and [Formula: see text] beneath the surface of the cortex, respectively. Rytov tomographic reconstructions revealed maximal spatially localized increases and decreases in [Formula: see text] and ctHb of [Formula: see text] and [Formula: see text], respectively, with these maximum changes occurring at [Formula: see text] poststimulus. The localized optical signals from the Rytov approximation were greater than those from modified Beer-Lambert, likely due in part to the inability of planar reflectance to account for partial volume effects.

Optical hyperspectral imaging in microscopy and spectroscopy - a review of data acquisition

Rather than simply acting as a photographic camera capturing two-dimensional (x, y) intensity images or a spectrometer acquiring spectra (λ), a hyperspectral imager measures entire three-dimensional (x, y, λ) datacubes for multivariate analysis, providing structural, molecular, and functional information about biological cells or tissue with unprecedented detail. Such data also gives clinical insights for disease diagnosis and treatment. We summarize the principles underpinning this technology, highlight its practical implementation, and discuss its recent applications at microscopic to macroscopic scales. Datacube acquisition strategies in hyperspectral imaging x, y, spatial coordinates; λ, wavelength.

Optical hyperspectral imaging in microscopy and spectroscopy - a review of data acquisition

Rather than simply acting as a photographic camera capturing two-dimensional (x, y) intensity images or a spectrometer acquiring spectra (λ), a hyperspectral imager measures entire three-dimensional (x, y, λ) datacubes for multivariate analysis, providing structural, molecular, and functional information about biological cells or tissue with unprecedented detail. Such data also gives clinical insights for disease diagnosis and treatment. We summarize the principles underpinning this technology, highlight its practical implementation, and discuss its recent applications at microscopic to macroscopic scales. Datacube acquisition strategies in hyperspectral imaging x, y, spatial coordinates; λ, wavelength.

Modern technologies for retinal scanning and imaging: an introduction for the biomedical engineer

This review article is meant to help biomedical engineers and nonphysical scientists better understand the principles of, and the main trends in modern scanning and imaging modalities used in ophthalmology. It is intended to ease the communication between physicists, medical doctors and engineers, and hopefully encourage “classical” biomedical engineers to generate new ideas and to initiate projects in an area which has traditionally been dominated by optical physics. Most of the methods involved are applicable to other areas of biomedical optics and optoelectronics, such as microscopic imaging, spectroscopy, spectral imaging, opto-acoustic tomography, fluorescence imaging etc., all of which are with potential biomedical application. Although all described methods are novel and important, the emphasis of this review has been placed on three technologies introduced in the 1990’s and still undergoing vigorous development: Confocal Scanning Laser Ophthalmoscopy, Optical Coherence Tomography, and polarization-sensitive retinal scanning.

Snapshot 3D optical coherence tomography system using image mapping spectrometry

A snapshot 3-Dimensional Optical Coherence Tomography system was developed using Image Mapping Spectrometry. This system can give depth information (Z) at different spatial positions (XY) within one camera integration time to potentially reduce motion artifact and enhance throughput. The current (x,y,λ) datacube of (85×356×117) provides a 3D visualization of sample with 400 μm depth and 13.4 μm in transverse resolution. Axial resolution of 16.0 μm can also be achieved in this proof-of-concept system. We present an analysis of the theoretical constraints which will guide development of future systems with increased imaging depth and improved axial and lateral resolutions.

Image mapping spectrometry: calibration and characterization

Image mapping spectrometry (IMS) is a hyperspectral imaging technique that simultaneously captures spatial and spectral information about an object in real-time. We present a new calibration procedure for the IMS as well as the first detailed evaluation of system performance. We correlate optical components and device calibration to performance metrics such as light throughput, scattered light, distortion, spectral image coregistration, and spatial/spectral resolution. Spectral sensitivity and motion artifacts are also evaluated with a dynamic biological experiment. The presented methodology of evaluation is useful in assessment of a variety of hyperspectral and multi-spectral modalities. Results are important to any potential users/developers of an IMS instrument and to anyone who may wish to compare the IMS to other imaging spectrometers.

Correction of vignetting and distortion errors induced by two-axis light beam steering

A mirror facet's angle correction approach is presented for eliminating pupil plane distortions and sub-field image vignetting in the image mapping spectrometry (IMS). The two-axis light reflection problem on the image mapper is solved and a rigorous analytical solution is provided. The cellular fluorescence imaging experiment demonstrates that, with an angle-corrected image mapper, the acquired image quality of spectral channels has been significantly improved compared to previous IMS images. The proposed mathematical model can also be used in solving general two-axis beam steering problems for instruments with active optical mirrors.

Snapshot hyperspectral retinal camera with the Image Mapping Spectrometer (IMS)

We present a snapshot hyperspectral retinal camera with the Image Mapping Spectrometer (IMS) for eye imaging applications. The resulting system is capable of simultaneously acquiring 48 spectral channel images in the range 470 nm-650 nm with frame rate at 5.2 fps. The spatial sampling of each measured spectral scene is 350 × 350 pixels. The advantages of this snapshot device are elimination of the eye motion artifacts and pixel misregistration problems in traditional scanning-based hyperspectral retinal cameras, and real-time imaging of oxygen saturation dynamics with sub-second temporal resolution. The spectral imaging performance is demonstrated in a human retinal imaging experiment in vivo. The absorption spectral signatures of oxy-hemoglobin and macular pigments were successfully acquired by using this device.

Finite conjugate embedded relay lens hyperspectral imaging system (ERL-HIS)

We present a novel embedded relay lens hyperspectral imaging system (ERL-HIS) with high spectral resolution (nominal spectral resolution of 2.8 nm) and spatial resolution (30 μm×8 μm) that transfers the scanning plane to an additional imaging plane through the internal relay lens so as to alleviate all outside moving parts for the scanning mechanism used in the traditional HIS, where image scanning is achieved by the relative movement between the object and hyperspectrometer. The ERL-HIS also enables high-speed scanning and can attach to a variety of optical modules for versatile applications. Here, we also demonstrate an application of the proposed ERL-HIS attached to a microscopic system for observing autofluorescent images of sliced cancer tissue samples.

Compound prism design principles, I

Prisms have been needlessly neglected as components used in modern optical design. In optical throughput, stray light, flexibility, and in their ability to be used in direct-view geometry, they excel over gratings. Here we show that even their well-known weak dispersion relative to gratings has been overrated by designing doublet and double Amici direct-vision compound prisms that have 14° and 23° of dispersion across the visible spectrum, equivalent to 800 and 1300 lines/mm gratings. By taking advantage of the multiple degrees of freedom available in a compound prism design, we also show prisms whose angular dispersion shows improved linearity in wavelength. In order to achieve these designs, we exploit the well-behaved nature of prism design space to write customized algorithms that optimize directly in the nonlinear design space. Using these algorithms, we showcase a number of prism designs that illustrate a performance and flexibility that goes beyond what has often been considered possible with prisms.

Real-time snapshot hyperspectral imaging endoscope

Hyperspectral imaging has tremendous potential to detect important molecular biomarkers of early cancer based on their unique spectral signatures. Several drawbacks have limited its use for in vivo screening applications: most notably the poor temporal and spatial resolution, high expense, and low optical throughput of existing hyperspectral imagers. We present the development of a new real-time hyperspectral endoscope (called the image mapping spectroscopy endoscope) based on an image mapping technique capable of addressing these challenges. The parallel high throughput nature of this technique enables the device to operate at frame rates of 5.2 frames per second while collecting a (x, y, λ) datacube of 350 × 350 × 48. We have successfully imaged tissue in vivo, resolving a vasculature pattern of the lower lip while simultaneously detecting oxy-hemoglobin.

High-throughput detection of immobilized plasmonic nanoparticles by a hyperspectral imaging system based on Fourier transform spectrometry

To facilitate the application of plasmonic nanoparticles (PNPs) in high-throughput detection, we develop a hyperspectral imaging system (HSIS) combining dark-filed microscopy and imaging Fourier transform spectrometry to measure scattering spectra from immobilized PNPs. The current setup has acquisition time of 5 seconds and spectral resolution of 21.4 nm at 532.1 nm. We demonstrate the applicability of the HSIS in conjunction with spectral data analysis to quantify multiple types of PNPs and detect small changes in localized surface plasmon resonance wavelengths of PNPs due to changes in the environmental refractive index.

Snapshot Image Mapping Spectrometer (IMS) with high sampling density for hyperspectral microscopy

A snapshot Image Mapping Spectrometer (IMS) with high sampling density is developed for hyperspectral microscopy, measuring a datacube of dimensions 285 x 285 x 60 (x, y, lambda). The spatial resolution is approximately 0.45 microm with a FOV of 100 x 100 microm(2). The measured spectrum is from 450 nm to 650 nm and is sampled by 60 spectral channels with average sampling interval approximately 3.3 nm. The channel's spectral resolution is approximately 8nm. The spectral imaging results demonstrate the potential of the IMS for real-time cellular fluorescence imaging.