P-272 Automatic Tracking of the ICSI procedure using Deep Learning

Study question

Can deep learning be used to detect and track spermatozoa and the different parts of an ICSI procedure?

Summary answer

Deep learning can be used as a tool to assist and organize the contents of an ICSI procedure.

What is known already

Sperm tracking has been a topic of research and practice for many years, especially in the context of computer-aided sperm analysis (CASA). Recent studies have proposed using deep learning algorithms to track spermatozoa for spermatozoon selection in human and animal samples. One critical part of performing ICSI involves the selection of the “best” spermatozoon for injection, but other parts of the procedure may also be of importance. However, as far as we know, tracking using deep learning has not been applied to the ICSI procedure, where detecting instruments and the oocyte could also be helpful in post-analysis and training.

Study design, size, duration

The study was performed using three anonymized videos of the ICSI procedure. The frames of the videos were manually annotated by data scientists and verified by an embryologist. The annotations were bounding boxes around specific parts of the ICSI procedure, including sperm, pipettes, and the oocyte. We trained a YOLOv5 model on the collected data, where two videos were used for training and one video for validation.

Participants/materials, setting, methods

The videos of the ICSI procedure were captured at 200x magnification with a DeltaPix camera at Fertilitetssenteret in Oslo, Norway. ICSI was performed using a Nikon ECLIPSE TE2000-S microscope connected with Eppendorf TransferMan 4m micromanipulators. The spermatozoa were immobilised in 5 µl Polyvinylpyrrolidone (PVP; CooperSurgical). The videos had a resolution of 1920x1080 and were resized to 640x640 before being processed by the YOLOv5 model. The data will be made public in a later study.

Main results and the role of chance

Mean average precision (mAP) with the threshold of 0.5 (mAP@.5) is the main quantitative parameter measured in the YOLOv5 model. All the experiments were performed using three-fold cross-validation, where we present the average metrics calculated over the three folds. Overall, the method showed an average mAP@.5 of 0.50 across all predicted classes, which means that the method can track the different components with good accuracy. Looking closer at the individual classes, we see that instruments like the holding pipette and ICSI pipette are detected with high accuracy with a mAP@.5 of 0.87 and 0.94, respectively. The oocyte is also easily tracked with a mAP@.5 of 0.92. The first polar body is well detected with a mAP@.5 of 0.65. The model has issues detecting and tracking individual sperm (both outside and within the pipette), where the method achieved a mAP@.5 of 0.46 for tracking sperm outside the pipette and 0.03 for the sperm inside the pipette. The low score of detecting the sperm in the pipette can be explained by the often unclear visibility of the sperm through the pipette and the low number of training samples.

Limitations, reasons for caution

The limited sample size makes the generalizability of the method difficult to determine. A more extensive evaluation is necessary. Moreover, as the currency study focuses on tracking, patient information and clinical outcome were not included in the analysis.

Wider implications of the findings

Deep learning has the potential to aid embryologists to perform successful ICSI through tracking and detection of spermatozoa, pipettes, and the oocyte. This could potentially lead to better internal quality control and teaching possibilities, and hopefully better results.

Trial registration number

not applicable

P-108 Real-time deep learning based multi object tracking of spermatozoa in fresh samples

Study question

Can real-time deep learning model track hundreds of spermatozoa simultaneously?

Summary answer

The state-of-the-art deep learning detection model YOLOv5 shows possibilities of multi-sperm tracking with high sensitivity and precision.

What is known already

Computer-aided sperm analysis (CASA) systems can be used for the evaluation of sperm motility by applying tracking algorithms. However, the presence of cellular debris and/or cell aggregations in human semen samples makes the tracking of spermatozoa difficult, resulting in unreliable motility assessment. Thus, there is a need for an improved methodology. There are several studies on detecting spermatozoa in real-time, such as DeepSperm, which is trained and tested on bull semen samples. However, differences in characteristics between human and animal spermatozoa imply the requirement of a multi-sperm tracking system adapted to human spermatozoa.

Study design, size, duration

We used the open-access VISEM dataset consisting of video recordings of human semen samples from 85 participants. We selected three videos with low sperm counts to perform manual annotations (bounding boxes around sperms) and create a training dataset. Only the first 30 seconds of each video were extracted for annotations. Then, all the spermatozoa of the three videos were annotated with bounding boxes using LabelBox. The annotations will be made public in a future study.

Participants/materials, setting, methods

We used two different object detection models, YOLOv5 Nano and YOLOv5 XLarge to detect spermatozoa and performed transfer learning without layer freezing. Each model was trained for a maximum of 300 epochs, stopping if the validation loss did not improve for the previous 100 epochs. Leave-one-out cross-validation was performed to obtain the presented results. Precision, sensitivity, and mean average precision (mAP) are the quantitative metrics measured to evaluate our detection models.

Main results and the role of chance

Our models are able to detect a large number of human spermatozoa simultaneously in a given video in real-time. The model YOLOv5 Nano took around 45 min and the YOLOv5 XLarge model took around 270 min to train on one fold. In terms of training and detection, the Nano model is faster because it has a lower number of trainable parameters compared to the XLarge model, but at the expense of precision. YOLOv5 Nano has capability of predicting 200 frames per second (fps), and YOLOv5 XLarge can predict 56 fps. Both models show high prediction rates which are greater than the real-time prediction 25 fps threshold. Performance-wise, the Nano model shows an average recall value of 0.8545 which is better than the average recall value of 0.7632 of XLarge model. In contrast, the XLarge model shows a higher precision value of 0.8020 compared to the low precision of 0.7149 of the Nano model. Furthermore, the XLarge model has an average mAP (mAP_0.5) of 0.7632 which is larger than the mAP value of the Nano model of 0.7027. The current prediction errors in this stage might be a result of the small training dataset.

Limitations, reasons for caution

The low amount of data makes validating the models difficult, especially evaluating how well they generalize to unseen samples. Furthermore, the training data consisted of samples containing low sperm counts, making the performance on samples with high sperm concentration uncertain.

Wider implications of the findings

Sperm tracking is integral to achieving accurate and less subjective motility assessment. The detections can be used to analyse the individual spermatozoa, leading to better performance. Deep feature extraction, trajectory prediction, and trajectory extraction can be used for future studies like generating synthetic spermatozoa for training generalizable machine learning models.

Trial registration number

not applicable

P-243 Automating tracking of cell division for human embryo development in time lapse videos

Study question

Can tracking cell division and predicting human embryo cleavage stages be automated in time-lapse videos (TLV) using AI object detection methods?

Summary answer

We developed software predicting blastomere count and tracking cell cleavages up until 4-5 stage. The software employs object detection technique called YOLOv5 to detect cells.

What is known already

Embryo morphology plays an important part in determining viability. Parameters such as number of cells present following fertilization, abnormal cell division (reverse/direct) and evaluating cleavage stages have correlation with pregnancy rates. However, continuous manual evaluation can be time-consuming, and automation will assist in embryo viability assessment. YOLOv5 has proven to accurately detect objects in videos. YOLOv5 uses mean average precision (mAP) as a metric to quantify the portions of frames in videos having the correct count of the objects.

Study design, size, duration

We have developed a software that uses YOLOv5 to detect cells present in frames of TLV, then marks each cell boundary with different colored circular overlays using OpenCV. We trained YOLOv5 to detect objects: cell, morula and blastocyst using 150 images of different cell-stages, morula, blastocyst. For object cell mAP was 0.65. Annotated location of objects in images and YOLOv5 predictions were reviewed by embryologists. We evaluated the software on TLV from 11 patients.

Participants/materials, setting, methods

After YOLOv5 detects cells in frames of TLV, our software computes cell count and assigns each cell a different color which is maintained until cell division into daughter cells. Later, daughter cells were also assigned different colors. If the frame has a preceding frame, software calculates detected cells' proximity with each cell in the preceding frame and copies color scheme provided proximity is within some threshold. The software provides TLV with colored overlays as output.

Main results and the role of chance

In starting frames of TLV with single cell, software accurately detected 1-cell (high precision=0.99, high recall=0.83, high F1-score=0.90). We observed some misclassification between 1-cell and morula. The reason could be that compacted morula looks like 1-cell. Best performance is observed for 2-cells (high precision=0.91, high recall=0.98, high F1-score=0.95). 4-cells were sometimes misclassified with 3 or 5-cells (high precision=0.88, low recall=0.59, high F1-score=0.71). One reason for the misclassification can be that overlapping between cells increases with number of cells. 3-cell and 5-cell are confused with other stages, still cleavage stage detection is better than random: 3-cell (average precision=0.43, high recall=0.83, average F1-score=0.49), 5-cell (average precision=0.44, average recall=0.40, average F1-score=0.40). For cell-stages>5, YOLOv5 detects less cells than actual count and software predicts cleavage later than actual by 9-10 frames on average. The proximity threshold used was 0.10 for cell-count<4 and 0.05 for count>4.

In 5 TLV, overlay color for cells changes abruptly between frames, possibly because once YOLOv5 detected a stage, in consecutive frames less cell-number was recorded, and then again reported correct count. Sometimes, software selected the wrong parent for daughter cells (incorrect colored overlay). 2 TLV had direct and reverse cleavages and software could detect these two patterns.

Limitations, reasons for caution

Overall, our software can precisely detect cells, cell divisions and cleavage stages up to 4-cell stages. We hypothesize that training YOLOv5 on a bigger dataset and including several focal plane information will enable our software to detect overlapping cells and cleavage stages > =5.

Wider implications of the findings

Object detection proved to be pragmatic for ART and tracking cell division using our software will reduce time consumed in manual annotations, easier prediction of abnormal cleavages and more objective assessments. Qualitative evaluation by embryologists resulted in the overall verdict that this is useful and promising for further development.

Trial registration number

not applicable

P–029 Identification of spermatozoa by unsupervised learning from video data

Study question

Can artificial intelligence (AI) algorithms identify spermatozoa in a semen sample without using training data annotated by professionals?

Summary answer

Unsupervised AI methods can discriminate the spermatozoon from other cells and debris. These unsupervised methods may have a potential for several applications in reproductive medicine.

What is known already

Identification of individual sperm is essential to assess a given sperm sample’s motility behaviour. Existing computer-aided systems need training data based on annotations by professionals, which is resource demanding. On the other hand, data analysed by unsupervised machine learning algorithms can improve supervised algorithms that are more stable for clinical applications. Therefore, unsupervised sperm identification can improve computer-aided sperm analysis systems predicting different aspects of sperm samples. Other possible applications are assessing kinematics and counting of spermatozoa.

Study design, size, duration

Three sperm-like paint images were manipulated using a graphic design tool and used to train our AI system. Two paintings have an ash colour background and randomly distributed white colour circles, and one painting has a predefined pattern of circles. Selected semen sample videos from a public dataset with videos obtained from 85 participants were used to test our AI system.

Participants/materials, setting, methods

Generative adversarial networks (GANs) have become common AI methods to process data in an unsupervised way. Based on single image frames extracted from videos, a GAN (SinGAN) can be trained to determine and track locations of sperms by translating the real images into localization paintings. The resulting model showed the potential of identifying the presence of sperms without any prior knowledge about data.

Main results and the role of chance

Visual comparisons of localization paintings to real sperm images show that inverse training of SinGANs can track sperms. Converting colour frames into grayscale frames and using grayscale synthetic sperm-like frames showed the best visual quality of generated localization paintings of sperm frames. Feeding real sperm video frames to the SinGAN at different scaling factors, which is defining the resolution of the input image, showed different quality levels of generated sperm localization paintings. A sperm frame given to the algorithm with a scaling factor of one leads to random sperm tracking, while the scales two to four result in more accurate localization maps than scaling levels five to eight. In contrast, scales from six to eight result in an output close to the input frame. The proposed method is robust in terms of the number of spermatozoa, meaning that the detection works well for samples with a low or high sperm count. For visual comparisons, visit our Github page: https://vlbthambawita.github.io/singan-sperm/. The sperm tracking speed of our SinGAN using an NVIDIA 1080 graphic processing unit, is around 17 frames per second, which can be improved by using parallel video processing capabilities. This shows the capability of using this method for real-time analysis.

Limitations, reasons for caution

Unsupervised methods are hard to train, and the results need human verification. The proposed method will need quality control and must be standardized. Unsupervised sperm tracking SinGAN may identify blurry bright spots as non-existing sperm heads which may restrict the use of SinGAN sperm tracking for sperm counting.

Wider implications of the findings: Assessment of semen samples according to the WHO guidelines is subjective and resource-demanding. This unsupervised model might be used to develop new systems for less time-consuming and more accurate evaluation of semen samples. It may also be used for real-time analysis of prepared spermatozoa for use in assisted reproduction technology.

Trial registration number

N/A