Tracking Fish Abundance by Underwater Image Recognition

The re-use of offshore platforms as ecological observatories

The high number of offshore platforms at the end of their productive phase offers the opportunity of their re-use and the development of effective management solutions, such as the possibility of utilizing them as ecological observatories for monitoring marine ecosystems and their biological resources. Here, through a multiparametric observatory deployed at an unproductive offshore platform, located in the Central Adriatic Sea (Mediterranean Sea), we collected data for 13 months on benthopelagic fish assemblage and habitat conditions. A total of 155.5 h of high-frequency (30 min) video-monitoring, recorded higher fish abundances during spring-summer periods during daytime, while fish diversity was highest in autumn. Some environmental variables contributed significantly to explain the overall community variance. Our results suggest that offshore platforms can be re-converted into ecological observatories, to collect relevant amounts of information that can be difficulty obtained with alternative approaches, contributing to our understanding of changes occurring in open water ecosystems.

Underwater Image Enhancement Based on Luminance Reconstruction by Multi-Resolution Fusion of RGB Channels

Underwater image enhancement technology is crucial for the human exploration and exploitation of marine resources. The visibility of underwater images is affected by visible light attenuation. This paper proposes an image reconstruction method based on the decomposition-fusion of multi-channel luminance data to enhance the visibility of underwater images. The proposed method is a single-image approach to cope with the condition that underwater paired images are difficult to obtain. The original image is first divided into its three RGB channels. To reduce artifacts and inconsistencies in the fused images, a multi-resolution fusion process based on the Laplace-Gaussian pyramid guided by a weight map is employed. Image saliency analysis and mask sharpening methods are also introduced to color-correct the fused images. The results indicate that the method presented in this paper effectively enhances the visibility of dark regions in the original image and globally improves its color, contrast, and sharpness compared to current state-of-the-art methods. Our method can enhance underwater images in engineering practice, laying the foundation for in-depth research on underwater images.

DiffusionFR: Species Recognition of Fish in Blurry Scenarios via Diffusion and Attention

Blurry scenarios, such as light reflections and water ripples, often affect the clarity and signal-to-noise ratio of fish images, posing significant challenges for traditional deep learning models in accurately recognizing fish species. Firstly, deep learning models rely on a large amount of labeled data. However, it is often difficult to label data in blurry scenarios. Secondly, existing deep learning models need to be more effective for the processing of bad, blurry, and otherwise inadequate images, which is an essential reason for their low recognition rate. A method based on the diffusion model and attention mechanism for fish image recognition in blurry scenarios, DiffusionFR, is proposed to solve these problems and improve the performance of species recognition of fish images in blurry scenarios. This paper presents the selection and application of this correcting technique. In the method, DiffusionFR, a two-stage diffusion network model, TSD, is designed to deblur bad, blurry, and otherwise inadequate fish scene pictures to restore clarity, and a learnable attention module, LAM, is intended to improve the accuracy of fish recognition. In addition, a new dataset of fish images in blurry scenarios, BlurryFish, was constructed and used to validate the effectiveness of DiffusionFR, combining bad, blurry, and otherwise inadequate images from the publicly available dataset Fish4Knowledge. The experimental results demonstrate that DiffusionFR achieves outstanding performance on various datasets. On the original dataset, DiffusionFR achieved the highest training accuracy of 97.55%, as well as a Top-1 accuracy test score of 92.02% and a Top-5 accuracy test score of 95.17%. Furthermore, on nine datasets with light reflection noise, the mean values of training accuracy reached a peak at 96.50%, while the mean values of the Top-1 accuracy test and Top-5 accuracy test were at their highest at 90.96% and 94.12%, respectively. Similarly, on three datasets with water ripple noise, the mean values of training accuracy reached a peak at 95.00%, while the mean values of the Top-1 accuracy test and Top-5 accuracy test were at their highest at 89.54% and 92.73%, respectively. These results demonstrate that the method showcases superior accuracy and enhanced robustness in handling original datasets and datasets with light reflection and water ripple noise.

Residual toxins on aquatic animals in the Pacific areas: Current findings and potential health effects

The Pacific Ocean is among the five largest and deepest oceans in the world. The area of the Pacific Ocean covers about 28 % of the Earth's surface. This is the habitat of many marine species, and its diversity is recognized as a fundamental element of Pacific culture and heritage. The ecosystems of aquatic animals are highly affected by climate change and by other factors. Residual toxins on aquatic animals can be categorized into two types based on origin: toxins of marine origin and toxins associated with human activity. Residual toxins have emerged as a global concern in recent years due to their frequent presence in aquatic environments. Furthermore, residual toxins in organisms living in the marine environment in the Pacific Ocean region also seriously affect food safety, food security, and especially human health. In this review we discuss important issues about residual toxins on aquatic animals in the Pacific areas specifically about the types of toxins that exist in marine animals, their contamination pathways in the Asia, Pacific region and the potential health effects for humans, the application of information technology and artificial intelligence in residual toxins on aquatic animal.

A labeled data set of underwater images of fish and crab species from five mesohabitats in Puget Sound WA USA

The sustainable management of fisheries and aquaculture requires an understanding of how these activities interact with natural fish populations. GoPro cameras were used to collect an underwater video data set on and around shellfish aquaculture farms in an estuary in the NE Pacific from June to August 2017 and June to August 2018 to better understand habitat use by the local fish and crab communities. Images extracted from these videos were labeled to produce a data set that is suitable for use in training computer vision models. The labeled data set contains 77,739 images sampled from the collected video; 67,990 objects (fishes and crustaceans) have been annotated in 30,384 images (the remainder have been annotated as "empty"). The metadata of the data set also indicates whether a physical magenta filter was used during video collection to counteract reduced visibility. These data have the potential to help researchers address system-level and in-depth regional shellfish aquaculture questions related to ecosystem services and shellfish aquaculture interactions.

Environmental DNA metabarcoding serves as a promising method for aquatic species monitoring and management: A review focused on its workflow, applications, challenges and prospects

Marine and freshwater biodiversity is under threat from both natural and manmade causes. Biological monitoring is currently a top priority for biodiversity protection. Given present limitations, traditional biological monitoring methods may not achieve the proposed monitoring aims. Environmental DNA metabarcoding technology reflects species information by capturing and extracting DNA from environmental samples, using molecular biology techniques to sequence and analyze the DNA, and comparing the obtained information with existing reference libraries to obtain species identification. However, its practical application has highlighted several limitations. This paper summarizes the main steps in the environmental application of eDNA metabarcoding technology in aquatic ecosystems, including the discovery of unknown species, the detection of invasive species, and evaluations of biodiversity. At present, with the rapid development of big data and artificial intelligence, certain advanced technologies and devices can be combined with environmental DNA metabarcoding technology to promote further development of aquatic species monitoring and management.

An automated image-based workflow for detecting megabenthic fauna in optical images with examples from the Clarion-Clipperton Zone

Recent advances in optical underwater imaging technologies enable the acquisition of huge numbers of high-resolution seafloor images during scientific expeditions. While these images contain valuable information for non-invasive monitoring of megabenthic fauna, flora and the marine ecosystem, traditional labor-intensive manual approaches for analyzing them are neither feasible nor scalable. Therefore, machine learning has been proposed as a solution, but training the respective models still requires substantial manual annotation. Here, we present an automated image-based workflow for Megabenthic Fauna Detection with Faster R-CNN (FaunD-Fast). The workflow significantly reduces the required annotation effort by automating the detection of anomalous superpixels, which are regions in underwater images that have unusual properties relative to the background seafloor. The bounding box coordinates of the detected anomalous superpixels are proposed as a set of weak annotations, which are then assigned semantic morphotype labels and used to train a Faster R-CNN object detection model. We applied this workflow to example underwater images recorded during cruise SO268 to the German and Belgian contract areas for Manganese-nodule exploration, within the Clarion-Clipperton Zone (CCZ). A performance assessment of our FaunD-Fast model showed a mean average precision of 78.1% at an intersection-over-union threshold of 0.5, which is on a par with competing models that use costly-to-acquire annotations. In more detail, the analysis of the megafauna detection results revealed that ophiuroids and xenophyophores were among the most abundant morphotypes, accounting for 62% of all the detections within the surveyed area. Investigating the regional differences between the two contract areas further revealed that both megafaunal abundance and diversity was higher in the shallower German area, which might be explainable by the higher food availability in form of sinking organic material that decreases from east-to-west across the CCZ. Since these findings are consistent with studies based on conventional image-based methods, we conclude that our automated workflow significantly reduces the required human effort, while still providing accurate estimates of megafaunal abundance and their spatial distribution. The workflow is thus useful for a quick but objective generation of baseline information to enable monitoring of remote benthic ecosystems.

Context-Driven Detection of Invertebrate Species in Deep-Sea Video

Each year, underwater remotely operated vehicles (ROVs) collect thousands of hours of video of unexplored ocean habitats revealing a plethora of information regarding biodiversity on Earth. However, fully utilizing this information remains a challenge as proper annotations and analysis require trained scientists’ time, which is both limited and costly. To this end, we present a Dataset for Underwater Substrate and Invertebrate Analysis (DUSIA), a benchmark suite and growing large-scale dataset to train, validate, and test methods for temporally localizing four underwater substrates as well as temporally and spatially localizing 59 underwater invertebrate species. DUSIA currently includes over ten hours of footage across 25 videos captured in 1080p at 30 fps by an ROV following pre-planned transects across the ocean floor near the Channel Islands of California. Each video includes annotations indicating the start and end times of substrates across the video in addition to counts of species of interest. Some frames are annotated with precise bounding box locations for invertebrate species of interest, as seen in Fig. 1. To our knowledge, DUSIA is the first dataset of its kind for deep sea exploration, with video from a moving camera, that includes substrate annotations and invertebrate species that are present at significant depths where sunlight does not penetrate. Additionally, we present the novel context-driven object detector (CDD) where we use explicit substrate classification to influence an object detection network to simultaneously predict a substrate and species class influenced by that substrate. We also present a method for improving training on partially annotated bounding box frames. Finally, we offer a baseline method for automating the counting of invertebrate species of interest.

Altering Fish Behavior by Sensing Swarm Patterns of Fish in an Artificial Aquatic Environment Using an Interactive Robotic Fish

Numerous studies have been conducted to prove the calming and stress-reducing effects on humans of visiting aquatic environments. As a result, many institutions have utilized fish to provide entertainment and treat patients. The most common issue in this approach is controlling the movement of fish to facilitate human interaction. This study proposed an interactive robot, a robotic fish, to alter fish swarm behaviors by performing an effective, unobstructed, yet necessary, defined set of actions to enhance human interaction. The approach incorporated a minimalistic but futuristic physical design of the robotic fish with cameras and infrared (IR) sensors, and developed a fish-detecting and swarm pattern-recognizing algorithm. The fish-detecting algorithm was implemented using background subtraction and moving average algorithms with an accuracy of 78%, while the swarm pattern detection implemented with a Convolutional Neural Network (CNN) resulted in a 77.32% accuracy rate. By effectively controlling the behavior and swimming patterns of fish through the smooth movements of the robotic fish, we evaluated the success through repeated trials. Feedback from a randomly selected unbiased group of subjects revealed that the robotic fish improved human interaction with fish by using the proposed set of maneuvers and behavior.

Image dataset for benchmarking automated fish detection and classification algorithms

Multiparametric video-cabled marine observatories are becoming strategic to monitor remotely and in real-time the marine ecosystem. Those platforms can achieve continuous, high-frequency and long-lasting image data sets that require automation in order to extract biological time series. The OBSEA, located at 4 km from Vilanova i la Geltrú at 20 m depth, was used to produce coastal fish time series continuously over the 24-h during 2013-2014. The image content of the photos was extracted via tagging, resulting in 69917 fish tags of 30 taxa identified. We also provided a meteorological and oceanographic dataset filtered by a quality control procedure to define real-world conditions affecting image quality. The tagged fish dataset can be of great importance to develop Artificial Intelligence routines for the automated identification and classification of fishes in extensive time-lapse image sets.

Implementation of an automated workflow for image-based seafloor classification with examples from manganese-nodule covered seabed areas in the Central Pacific Ocean

Mapping and monitoring of seafloor habitats are key tasks for fully understanding ocean ecosystems and resilience, which contributes towards sustainable use of ocean resources. Habitat mapping relies on seafloor classification typically based on acoustic methods, and ground truthing through direct sampling and optical imaging. With the increasing capabilities to record high-resolution underwater images, manual approaches for analyzing these images to create seafloor classifications are no longer feasible. Automated workflows have been proposed as a solution, in which algorithms assign pre-defined seafloor categories to each image. However, in order to provide consistent and repeatable analysis, these automated workflows need to address e.g., underwater illumination artefacts, variances in resolution and class-imbalances, which could bias the classification. Here, we present a generic implementation of an Automated and Integrated Seafloor Classification Workflow (AI-SCW). The workflow aims to classify the seafloor into habitat categories based on automated analysis of optical underwater images with only minimal amount of human annotations. AI-SCW incorporates laser point detection for scale determination and color normalization. It further includes semi-automatic generation of the training data set for fitting the seafloor classifier. As a case study, we applied the workflow to an example seafloor image dataset from the Belgian and German contract areas for Manganese-nodule exploration in the Pacific Ocean. Based on this, we provide seafloor classifications along the camera deployment tracks, and discuss results in the context of seafloor multibeam bathymetry. Our results show that the seafloor in the Belgian area predominantly comprises densely distributed nodules, which are intermingled with qualitatively larger-sized nodules at local elevations and within depressions. On the other hand, the German area primarily comprises nodules that only partly cover the seabed, and these occur alongside turned-over sediment (artificial seafloor) that were caused by the settling plume following a dredging experiment conducted in the area.

An Embedding Skeleton for Fish Detection and Marine Organisms Recognition

The marine economy has become a new growth point of the national economy, and many countries have started to implement the marine ranch project and made the project a new strategic industry to support vigorously. In fact, with the continuous improvement of people’s living standards, the market demand for precious seafood such as fish, sea cucumbers, and sea urchins increases. Shallow sea aquaculture has extensively promoted the vigorous development of marine fisheries. However, traditional diving monitoring and fishing are not only time consuming but also labor intensive; moreover, the personal injury is significant and the risk factor is high. In recent years, underwater robots’ development has matured and has been applied in other technologies. Marine aquaculture energy and chemical construction is a new opportunity for growth. The detection of marine organisms is an essential part of the intelligent strategy in marine ranch, which requires an underwater robot to detect the marine organism quickly and accurately in the complex ocean environment. This paper proposes a method called YOLOv4-embedding, based on one-stage deep learning arithmetic to detect marine organisms, construct a real-time target detection system for marine organisms, extract the in-depth features, and improve the backbone’s architecture and the neck connection. Compared with other object detection arithmetics, the YOLOv4-embedding object detection arithmetic was better at detection accuracy—with higher detection confidence and higher detection ratio than other one-stage object detection arithmetics, such as EfficientDet-D3. The results show that the suggested method could quickly detect different varieties in marine organisms. Furthermore, compared to the original YOLOv4, the mAP75 of the proposed YOLOv4-embedding improves 2.92% for the marine organism dataset at a real-time speed of 51 FPS on an RTX 3090.

Investigating Shape Variation Using Generalized Procrustes Analysis and Machine Learning

The biological investigation of a population’s shape diversity using digital images is typically reliant on geometrical morphometrics, which is an approach based on user-defined landmarks. In contrast to this traditional approach, the progress in deep learning has led to numerous applications ranging from specimen identification to object detection. Typically, these models tend to become black boxes, which limits the usage of recent deep learning models for biological applications. However, the progress in explainable artificial intelligence tries to overcome this limitation. This study compares the explanatory power of unsupervised machine learning models to traditional landmark-based approaches for population structure investigation. We apply convolutional autoencoders as well as Gaussian process latent variable models to two Nile tilapia datasets to investigate the latent structure using consensus clustering. The explanatory factors of the machine learning models were extracted and compared to generalized Procrustes analysis. Hypotheses based on the Bayes factor are formulated to test the unambiguity of population diversity unveiled by the machine learning models. The findings show that it is possible to obtain biologically meaningful results relying on unsupervised machine learning. Furthermore we show that the machine learning models unveil latent structures close to the true population clusters. We found that 80% of the true population clusters relying on the convolutional autoencoder are significantly different to the remaining clusters. Similarly, 60% of the true population clusters relying on the Gaussian process latent variable model are significantly different. We conclude that the machine learning models outperform generalized Procrustes analysis, where 16% of the population cluster was found to be significantly different. However, the applied machine learning models still have limited biological explainability. We recommend further in-depth investigations to unveil the explanatory factors in the used model.

A low-cost, long-term underwater camera trap network coupled with deep residual learning image analysis

Understanding long-term trends in marine ecosystems requires accurate and repeatable counts of fishes and other aquatic organisms on spatial and temporal scales that are difficult or impossible to achieve with diver-based surveys. Long-term, spatially distributed cameras, like those used in terrestrial camera trapping, have not been successfully applied in marine systems due to limitations of the aquatic environment. Here, we develop methodology for a system of low-cost, long-term camera traps (Dispersed Environment Aquatic Cameras), deployable over large spatial scales in remote marine environments. We use machine learning to classify the large volume of images collected by the cameras. We present a case study of these combined techniques’ use by addressing fish movement and feeding behavior related to halos, a well-documented benthic pattern in shallow tropical reefscapes. Cameras proved able to function continuously underwater at deployed depths (up to 7 m, with later versions deployed to 40 m) with no maintenance or monitoring for over five months and collected a total of over 100,000 images in time-lapse mode (by 15 minutes) during daylight hours. Our ResNet-50-based deep learning model achieved 92.5% overall accuracy in sorting images with and without fishes, and diver surveys revealed that the camera images accurately represented local fish communities. The cameras and machine learning classification represent the first successful method for broad-scale underwater camera trap deployment, and our case study demonstrates the cameras’ potential for addressing questions of marine animal behavior, distributions, and large-scale spatial patterns.

Implementation of Whale Optimization for Budding Healthiness of Fishes with Preprocessing Approach

This article examines distinctive techniques for monitoring the condition of fishes in underwater and also provides tranquil procedures after catching the fishes. Once the fishes are hooked, two different techniques that are explicitly designed for smoking and drying are implemented for saving the time of fish suppliers. Existing methods do not focus on the optimization algorithms for solving this issue. When considering the optimization problem, the solution is adequate for any number of inputs at time t. For this combined new flanged technique, a precise system model has been designed and incorporated with a set of rules using contention protocols. In addition, the designed system is also instigated with a whale optimization algorithm that is having sufficient capability to test the different parameters of assimilated sensing devices using different sensors. Further to test the effectiveness of the proposed method, an online monitoring system has been presented that can monitor and in turn provides the consequences using a simulation model for better understanding. Moreover, after examining the simulation results under three different scenarios, it has been observed that the proposed method provides an enhancement in real-time monitoring systems for an average of 78%.

A Systematic Review on Recent Trends, Challenges, Privacy and Security Issues of Underwater Internet of Things

Owing to the hasty growth of communication technologies in the Underwater Internet of Things (UIoT), many researchers and industries focus on enhancing the existing technologies of UIoT systems for developing numerous applications such as oceanography, diver networks monitoring, deep-sea exploration and early warning systems. In a constrained UIoT environment, communication media such as acoustic, infrared (IR), visible light, radiofrequency (RF) and magnet induction (MI) are generally used to transmit information via digitally linked underwater devices. However, each medium has its technical limitations: for example, the acoustic medium has challenges such as narrow-channel bandwidth, low data rate, high cost, etc., and optical medium has challenges such as high absorption, scattering, long-distance data transmission, etc. Moreover, the malicious node can steal the underwater data by employing blackhole attacks, routing attacks, Sybil attacks, etc. Furthermore, due to heavyweight, the existing privacy and security mechanism of the terrestrial internet of things (IoT) cannot be applied directly to UIoT environment. Hence, this paper aims to provide a systematic review of recent trends, applications, communication technologies, challenges, security threats and privacy issues of UIoT system. Additionally, this paper highlights the methods of preventing the technical challenges and security attacks of the UIoT environment. Finally, this systematic review contributes much to the profit of researchers to analyze and improve the performance of services in UIoT applications.

Machine Learning Applications of Convolutional Neural Networks and Unet Architecture to Predict and Classify Demosponge Behavior

Biological data sets are increasingly becoming information-dense, making it effective to use a computer science-based analysis. We used convolution neural networks (CNN) and the specific CNN architecture Unet to study sponge behavior over time. We analyzed a large time series of hourly high-resolution still images of a marine sponge, Suberites concinnus (Demospongiae, Suberitidae) captured between 2012 and 2015 using the NEPTUNE seafloor cabled observatory, off the west coast of Vancouver Island, Canada. We applied semantic segmentation with the Unet architecture with some modifications, including adapting parts of the architecture to be more applicable to three-channel images (RGB). Some alterations that made this model successful were the use of a dice-loss coefficient, Adam optimizer and a dropout function after each convolutional layer which provided losses, accuracies and dice scores of up to 0.03, 0.98 and 0.97, respectively. The model was tested with five-fold cross-validation. This study is a first step towards analyzing trends in the behavior of a demosponge in an environment that experiences severe seasonal and inter-annual changes in climate. The end objective is to correlate changes in sponge size (activity) over seasons and years with environmental variables collected from the same observatory platform. Our work provides a roadmap for others who seek to cross the interdisciplinary boundaries between biology and computer science.

Cephalopod species identification using integrated analysis of machine learning and deep learning approaches

BACKGROUND

Despite the high commercial fisheries value and ecological importance as prey item for higher marine predators, very limited taxonomic work has been done on cephalopods in Malaysia. Due to the soft-bodied nature of cephalopods, the identification of cephalopod species based on the beak hard parts can be more reliable and useful than conventional body morphology. Since the traditional method for species classification was time-consuming, this study aimed to develop an automated identification model that can identify cephalopod species based on beak images.

METHODS

A total of 174 samples of seven cephalopod species were collected from the west coast of Peninsular Malaysia. Both upper and lower beaks were extracted from the samples and the left lateral views of upper and lower beak images were acquired. Three types of traditional morphometric features were extracted namely grey histogram of oriented gradient (HOG), colour HOG, and morphological shape descriptor (MSD). In addition, deep features were extracted by using three pre-trained convolutional neural networks (CNN) models which are VGG19, InceptionV3, and Resnet50. Eight machine learning approaches were used in the classification step and compared for model performance.

RESULTS

The results showed that the Artificial Neural Network (ANN) model achieved the best testing accuracy of 91.14%, using the deep features extracted from the VGG19 model from lower beak images. The results indicated that the deep features were more accurate than the traditional features in highlighting morphometric differences from the beak images of cephalopod species. In addition, the use of lower beaks of cephalopod species provided better results compared to the upper beaks, suggesting that the lower beaks possess more significant morphological differences between the studied cephalopod species. Future works should include more cephalopod species and sample size to enhance the identification accuracy and comprehensiveness of the developed model.

Don't catch me if you can - Using cabled observatories as multidisciplinary platforms for marine fish community monitoring: An in situ case study combining Underwater Video and environmental DNA data

Cabled observatories are marine infrastructures equipped with biogeochemical and oceanographic sensors as well as High-Definition video and audio equipment, hence providing unprecedented opportunities to study marine biotic and abiotic components. Additionally, non-invasive monitoring approaches such as environmental DNA (eDNA) metabarcoding have further enhanced the ability to characterize marine life. Although the use of non-invasive tools beholds great potential for the sustainable monitoring of biodiversity and declining natural resources, such techniques are rarely used in parallel and understanding their limitations is challenging. Thus, this study combined Underwater Video (UV) with eDNA metabarcoding data to produce marine fish community profiles over a 2 months period in situ at a cabled observatory in the northeast Atlantic (SmartBay Ireland). By combining both approaches, an increased number of fish could be identified to the species level (total of 22 species), including ecologically and economically important species such as Atlantic cod, whiting, mackerel and monkfish. The eDNA approach alone successfully identified a higher number of species (59%) compared to the UV approach (18%), whereby 23% of species were detected by both methods. The parallel implementation of point collection eDNA and time series UV data not only confirmed expectations of the corroborative effect of using multiple disciplines in fish community composition, but also enabled the assessment of limitations intrinsic to each technique including the identification of false-negative detections in one sampling technology relative to the other. This work showcased the usefulness of cabled observatories as key platforms for in situ empirical assessment of both challenges and prospects of novel technologies in aid to future monitoring of marine life.

Jinks E, Buelow CA, Brown CJ, Wang D, Kusy B, M. Ditria E, Connolly RM. Automatic detection of fish and tracking of movement for ecology

Animal movement studies are conducted to monitor ecosystem health, understand ecological dynamics, and address management and conservation questions. In marine environments, traditional sampling and monitoring methods to measure animal movement are invasive, labor intensive, costly, and limited in the number of individuals that can be feasibly tracked. Automated detection and tracking of small-scale movements of many animals through cameras are possible but are largely untested in field conditions, hampering applications to ecological questions.Here, we aimed to test the ability of an automated object detection and object tracking pipeline to track small-scale movement of many individuals in videos. We applied the pipeline to track fish movement in the field and characterize movement behavior. We automated the detection of a common fisheries species (yellowfin bream, Acanthopagrus australis) along a known movement passageway from underwater videos. We then tracked fish movement with three types of tracking algorithms (MOSSE, Seq-NMS, and SiamMask) and evaluated their accuracy at characterizing movement.We successfully detected yellowfin bream in a multispecies assemblage (F1 score =91%). At least 120 of the 169 individual bream present in videos were correctly identified and tracked. The accuracies among the three tracking architectures varied, with MOSSE and SiamMask achieving an accuracy of 78% and Seq-NMS 84%.By employing this integrated object detection and tracking pipeline, we demonstrated a noninvasive and reliable approach to studying fish behavior by tracking their movement under field conditions. These cost-effective technologies provide a means for future studies to scale-up the analysis of movement across many visual monitoring systems.

Smart Fishery: A Systematic Review and Research Agenda for Sustainable Fisheries in the Age of AI

Applications of artificial intelligence (AI) technologies for improving the sustainability of the smart fishery have become widespread. While sustainability is often claimed to be the desired outcome of AI applications, there is as yet little evidence on how AI contributes to the sustainable fishery. The purpose of this paper is to perform a systematic review of the literature on the smart fishery and to identify upcoming themes for future research on the sustainable fishery in the Age of AI. The findings of the review reveal that scholarly attention in AI-inspired fishery literature focuses mostly on automation of fishery resources monitoring, mainly detection, identification, and classification. Some papers list marine health and primary production which are vital dimensions for Large Marine Ecosystems to recycle nutrients to sustain anticipated production levels. Very few reviewed articles refer to assessing individual needs, particularly fishers, from AI deployment in fisheries and policy response from governments. We call for future AI for sustainable fishery studies on how fishers perceive AI needs, and how governments possess a tangible strategy or depth of understanding on the regulation of AI concerning smart fishery systems and research on resilience-enhancing policies to promote the value and potentials of the AI-inspired smart fishery in different locations.

Identifying geographically differentiated features of Ethopian Nile tilapia (Oreochromis niloticus) morphology with machine learning

Visual characteristics are among the most important features for characterizing the phenotype of biological organisms. Color and geometric properties define population phenotype and allow assessing diversity and adaptation to environmental conditions. To analyze geometric properties classical morphometrics relies on biologically relevant landmarks which are manually assigned to digital images. Assigning landmarks is tedious and error prone. Predefined landmarks may in addition miss out on information which is not obvious to the human eye. The machine learning (ML) community has recently proposed new data analysis methods which by uncovering subtle features in images obtain excellent predictive accuracy. Scientific credibility demands however that results are interpretable and hence to mitigate the black-box nature of ML methods. To overcome the black-box nature of ML we apply complementary methods and investigate internal representations with saliency maps to reliably identify location specific characteristics in images of Nile tilapia populations. Analyzing fish images which were sampled from six Ethiopian lakes reveals that deep learning improves on a conventional morphometric analysis in predictive performance. A critical assessment of established saliency maps with a novel significance test reveals however that the improvement is aided by artifacts which have no biological interpretation. More interpretable results are obtained by a Bayesian approach which allows us to identify genuine Nile tilapia body features which differ in dependence of the animals habitat. We find that automatically inferred Nile tilapia body features corroborate and expand the results of a landmark based analysis that the anterior dorsum, the fish belly, the posterior dorsal region and the caudal fin show signs of adaptation to the fish habitat. We may thus conclude that Nile tilapia show habitat specific morphotypes and that a ML analysis allows inferring novel biological knowledge in a reproducible manner.

An Automated Pipeline for Image Processing and Data Treatment to Track Activity Rhythms of Paragorgia arborea in Relation to Hydrographic Conditions

Imaging technologies are being deployed on cabled observatory networks worldwide. They allow for the monitoring of the biological activity of deep-sea organisms on temporal scales that were never attained before. In this paper, we customized Convolutional Neural Network image processing to track behavioral activities in an iconic conservation deep-sea species—the bubblegum coral Paragorgia arborea—in response to ambient oceanographic conditions at the Lofoten-Vesterålen observatory. Images and concomitant oceanographic data were taken hourly from February to June 2018. We considered coral activity in terms of bloated, semi-bloated and non-bloated surfaces, as proxy for polyp filtering, retraction and transient activity, respectively. A test accuracy of 90.47% was obtained. Chronobiology-oriented statistics and advanced Artificial Neural Network (ANN) multivariate regression modeling proved that a daily coral filtering rhythm occurs within one major dusk phase, being independent from tides. Polyp activity, in particular extrusion, increased from March to June, and was able to cope with an increase in chlorophyll concentration, indicating the existence of seasonality. Our study shows that it is possible to establish a model for the development of automated pipelines that are able to extract biological information from times series of images. These are helpful to obtain multidisciplinary information from cabled observatory infrastructures.

Deep-learning-based information mining from ocean remote-sensing imagery

With the continuous development of space and sensor technologies during the last 40 years, ocean remote sensing has entered into the big-data era with typical five-V (volume, variety, value, velocity and veracity) characteristics. Ocean remote-sensing data archives reach several tens of petabytes and massive satellite data are acquired worldwide daily. To precisely, efficiently and intelligently mine the useful information submerged in such ocean remote-sensing data sets is a big challenge. Deep learning-a powerful technology recently emerging in the machine-learning field-has demonstrated its more significant superiority over traditional physical- or statistical-based algorithms for image-information extraction in many industrial-field applications and starts to draw interest in ocean remote-sensing applications. In this review paper, we first systematically reviewed two deep-learning frameworks that carry out ocean remote-sensing-image classifications and then presented eight typical applications in ocean internal-wave/eddy/oil-spill/coastal-inundation/sea-ice/green-algae/ship/coral-reef mapping from different types of ocean remote-sensing imagery to show how effective these deep-learning frameworks are. Researchers can also readily modify these existing frameworks for information mining of other kinds of remote-sensing imagery.

ENDURUNS: An Integrated and Flexible Approach for Seabed Survey Through Autonomous Mobile Vehicles

The oceans cover more than two-thirds of the planet, representing the vastest part of natural resources. Nevertheless, only a fraction of the ocean depths has been explored. Within this context, this article presents the H2020 ENDURUNS project that describes a novel scientific and technological approach for prolonged underwater autonomous operations of seabed survey activities, either in the deep ocean or in coastal areas. The proposed approach combines a hybrid Autonomous Underwater Vehicle capable of moving using either thrusters or as a sea glider, combined with an Unmanned Surface Vehicle equipped with satellite communication facilities for interaction with a land station. Both vehicles are equipped with energy packs that combine hydrogen fuel cells and Li-ion batteries to provide extended duration of the survey operations. The Unmanned Surface Vehicle employs photovoltaic panels to increase the autonomy of the vehicle. Since these missions generate a large amount of data, both vehicles are equipped with onboard Central Processing units capable of executing data analysis and compression algorithms for the semantic classification and transmission of the acquired data.

Exo-Ocean Exploration with Deep-Sea Sensor and Platform Technologies

One of Saturn's largest moons, Enceladus, possesses a vast extraterrestrial ocean (i.e., exo-ocean) that is increasingly becoming the hotspot of future research initiatives dedicated to the exploration of putative life. Here, a new bio-exploration concept design for Enceladus' exo-ocean is proposed, focusing on the potential presence of organisms across a wide range of sizes (i.e., from uni- to multicellular and animal-like), according to state-of-the-art sensor and robotic platform technologies used in terrestrial deep-sea research. In particular, we focus on combined direct and indirect life-detection capabilities, based on optoacoustic imaging and passive acoustics, as well as molecular approaches. Such biologically oriented sampling can be accompanied by concomitant geochemical and oceanographic measurements to provide data relevant to exo-ocean exploration and understanding. Finally, we describe how this multidisciplinary monitoring approach is currently enabled in terrestrial oceans through cabled (fixed) observatories and their related mobile multiparametric platforms (i.e., Autonomous Underwater and Remotely Operated Vehicles, as well as crawlers, rovers, and biomimetic robots) and how their modified design can be used for exo-ocean exploration.

Species-specific schooling behaviour of fish in the freshwater pelagic habitat: an observational study

Social living of animals is a broadly occurring phenomenon, although poorly studied in freshwater systems, fish schooling behaviour is an excellent example. The composition of fish schools, species-specific schooling tendencies and preferences of adult fish were studied in the pelagic habitat of the Římov Reservoir, Czech Republic. Video recordings captured over a total of 34 days (16 h per day) in the clear water period of three seasons were analysed. From four species identified as school-forming species - bream, bleak, roach and perch, 40% of the individuals observed formed schools of 3-36 individuals. Although conspecific schools prevailed, 20% of individuals formed heterospecific schools, except bleak that schooled strictly with conspecifics. Schools were composed of individuals of similar body size and life strategy. Heterospecific schools were significantly larger than conspecific schools and showed uneven proportion among species, that is, one species being more abundant when the school dimension increased. Probability of encounter in bleak was lowest and proved highest inclination for schooling. Gregarianism levels depended on species morphology and body size, with larger and morphologically advanced fish tending less to sociability. This indicates that the antipredator function of schooling behaviour is intensified with increasing vulnerability of the species.

Towards Naples Ecological REsearch for Augmented Observatories (NEREA): The NEREA-Fix Module, a Stand-Alone Platform for Long-Term Deep-Sea Ecosystem Monitoring

Deep-sea ecological monitoring is increasingly recognized as indispensable for the comprehension of the largest biome on Earth, but at the same time it is subjected to growing human impacts for the exploitation of biotic and abiotic resources. Here, we present the Naples Ecological REsearch (NEREA) stand-alone observatory concept (NEREA-fix), an integrated observatory with a modular, adaptive structure, characterized by a multiparametric video-platform to be deployed in the Dohrn canyon (Gulf of Naples, Tyrrhenian Sea) at ca. 650 m depth. The observatory integrates a seabed platform with optoacoustic and oceanographic/geochemical sensors connected to a surface transmission buoy, plus a mooring line (also equipped with depth-staged environmental sensors). This reinforced high-frequency and long-lasting ecological monitoring will integrate the historical data conducted over 40 years for the Long-Term Ecological Research (LTER) at the station “Mare Chiara”, and ongoing vessel-assisted plankton (and future environmental DNA-eDNA) sampling. NEREA aims at expanding the observational capacity in a key area of the Mediterranean Sea, representing a first step towards the establishment of a bentho-pelagic network to enforce an end-to-end transdisciplinary approach for the monitoring of marine ecosystems across a wide range of animal sizes (from bacteria to megafauna).

The Hierarchic Treatment of Marine Ecological Information from Spatial Networks of Benthic Platforms

Measuring biodiversity simultaneously in different locations, at different temporal scales, and over wide spatial scales is of strategic importance for the improvement of our understanding of the functioning of marine ecosystems and for the conservation of their biodiversity. Monitoring networks of cabled observatories, along with other docked autonomous systems (e.g., Remotely Operated Vehicles [ROVs], Autonomous Underwater Vehicles [AUVs], and crawlers), are being conceived and established at a spatial scale capable of tracking energy fluxes across benthic and pelagic compartments, as well as across geographic ecotones. At the same time, optoacoustic imaging is sustaining an unprecedented expansion in marine ecological monitoring, enabling the acquisition of new biological and environmental data at an appropriate spatiotemporal scale. At this stage, one of the main problems for an effective application of these technologies is the processing, storage, and treatment of the acquired complex ecological information. Here, we provide a conceptual overview on the technological developments in the multiparametric generation, storage, and automated hierarchic treatment of biological and environmental information required to capture the spatiotemporal complexity of a marine ecosystem. In doing so, we present a pipeline of ecological data acquisition and processing in different steps and prone to automation. We also give an example of population biomass, community richness and biodiversity data computation (as indicators for ecosystem functionality) with an Internet Operated Vehicle (a mobile crawler). Finally, we discuss the software requirements for that automated data processing at the level of cyber-infrastructures with sensor calibration and control, data banking, and ingestion into large data portals.

A Flexible Autonomous Robotic Observatory Infrastructure for Bentho-Pelagic Monitoring

This paper presents the technological developments and the policy contexts for the project “Autonomous Robotic Sea-Floor Infrastructure for Bentho-Pelagic Monitoring” (ARIM). The development is based on the national experience with robotic component technologies that are combined and merged into a new product for autonomous and integrated ecological deep-sea monitoring. Traditional monitoring is often vessel-based and thus resource demanding. It is economically unviable to fulfill the current policy for ecosystem monitoring with traditional approaches. Thus, this project developed platforms for bentho-pelagic monitoring using an arrangement of crawler and stationary platforms at the Lofoten-Vesterålen (LoVe) observatory network (Norway). Visual and acoustic imaging along with standard oceanographic sensors have been combined to support advanced and continuous spatial-temporal monitoring near cold water coral mounds. Just as important is the automatic processing techniques under development that have been implemented to allow species (or categories of species) quantification (i.e., tracking and classification). At the same time, real-time outboard processed three-dimensional (3D) laser scanning has been implemented to increase mission autonomy capability, delivering quantifiable information on habitat features (i.e., for seascape approaches). The first version of platform autonomy has already been tested under controlled conditions with a tethered crawler exploring the vicinity of a cabled stationary instrumented garage. Our vision is that elimination of the tether in combination with inductive battery recharge trough fuel cell technology will facilitate self-sustained long-term autonomous operations over large areas, serving not only the needs of science, but also sub-sea industries like subsea oil and gas, and mining.

Potential environmental impact resulting from biased fish sampling in intensive aquaculture operations

Aquaculture contributes to global food security, producing over 70 million tons of fish and aquatic products annually. Protein rich fish feeds, together with labor costs are the most expensive component costs in aquaculture. Feed application is given as percent of fish weight and therefore, reliable biomass assessment is essential for profitable and environmentally sound aquaculture. Fish biomass estimates are typically based on sampling <2% of the fish population. The goals of this research were to estimate potential biases associated with fish sampling in recirculating aquaculture systems (RAS), and the potential economic and environmental implications of such biased estimations. The size of the biased sampling-based estimates of fish biomass in two cultured species was shown to be larger than what the confidence interval suggests, even after >20% of the population was sampled. Such biases, if indeed common, will most likely result in over/underfeeding, both entailing negative economic and environmental consequences. We advocate conducting similar studies with major cultured fish to generate "bias correction tables" for adjusting fish feeding rate to bias-corrected biomass. These will help reduce the potential economic losses and negative environmental impacts of aquaculture practice.

Video Image Enhancement and Machine Learning Pipeline for Underwater Animal Detection and Classification at Cabled Observatories

An understanding of marine ecosystems and their biodiversity is relevant to sustainable use of the goods and services they offer. Since marine areas host complex ecosystems, it is important to develop spatially widespread monitoring networks capable of providing large amounts of multiparametric information, encompassing both biotic and abiotic variables, and describing the ecological dynamics of the observed species. In this context, imaging devices are valuable tools that complement other biological and oceanographic monitoring devices. Nevertheless, large amounts of images or movies cannot all be manually processed, and autonomous routines for recognizing the relevant content, classification, and tagging are urgently needed. In this work, we propose a pipeline for the analysis of visual data that integrates video/image annotation tools for defining, training, and validation of datasets with video/image enhancement and machine and deep learning approaches. Such a pipeline is required to achieve good performance in the recognition and classification tasks of mobile and sessile megafauna, in order to obtain integrated information on spatial distribution and temporal dynamics. A prototype implementation of the analysis pipeline is provided in the context of deep-sea videos taken by one of the fixed cameras at the LoVe Ocean Observatory network of Lofoten Islands (Norway) at 260 m depth, in the Barents Sea, which has shown good classification results on an independent test dataset with an accuracy value of 76.18% and an area under the curve (AUC) value of 87.59%.

Fish-Pak: Fish species dataset from Pakistan for visual features based classification

Fishes are most diverse group of vertebrates with more than 33000 species. These are identified based on several visual characters including their shape, color and head. It is difficult for the common people to directly identify the fish species found in the market. Classifying fish species from images based on visual characteristics using computer vision and machine learning techniques is an interesting problem for the researchers. However, the classifier's performance depends upon quality of image dataset on which it has been trained. An imagery dataset is needed to examine the classification and recognition algorithms. This article exhibits Fish-Pak: an image dataset of 6 different fish species, captured by a single camera from different pools located nearby the Head Qadirabad, Chenab River in Punjab, Pakistan. The dataset Fish-Pak are quite useful to compare various factors of classifiers such as learning rate, momentum and their impact on the overall performance. Convolutional Neural Network (CNN) is one of the most widely used architectures for image classification based on visual features. Six data classes i.e. Ctenopharyngodon idella (Grass carp), Cyprinus carpio (Common carp), Cirrhinus mrigala (Mori), Labeo rohita (Rohu), Hypophthalmichthys molitrix (Silver carp), and Catla (Thala), with a different number of images, have been included in the dataset. Fish species are captured by one camera to ensure the fair environment to all data. Fish-Pak is hosted by the Zoology Lab under the mutual affiliation of the Department of Computer Science and the Department of Zoology, University of Gujrat, Gujrat, Pakistan.

Annual rhythms of temporal niche partitioning in the Sparidae family are correlated to different environmental variables

The seasonal timing of recurring biological processes is essential for organisms living in temperate regions. While ample knowledge of these processes exists for terrestrial environments, seasonal timing in the marine environment is relatively understudied. Here, we characterized the annual rhythm of habitat use in six fish species belonging to the Sparidae family, highlighting the main environmental variables that correlate to such rhythms. The study was conducted at a coastal artificial reef through a cabled observatory system, which allowed gathering underwater time-lapse images every 30 minutes consecutively over 3 years. Rhythms of fish counts had a significant annual periodicity in four out of the six studied species. Species-specific temporal patterns were found, demonstrating a clear annual temporal niche partitioning within the studied family. Temperature was the most important environmental variable correlated with fish counts in the proximity of the artificial reef, while daily photoperiod and salinity were not important. In a scenario of human-induced rapid environmental change, tracking phenological shifts may provide key indications about the effects of climate change at both species and ecosystem level. Our study reinforces the efficacy of underwater cabled video-observatories as a reliable tool for long-term monitoring of phenological events.