RESEARCH ARTICLE

Abnormal behaviour in rock bream (Oplegnathus fasciatus) detected using deep learning-based image analysis

Jun-Chul Jang1https://orcid.org/0000-0002-4296-2860, Yeo-Reum Kim1https://orcid.org/0000-0003-0318-3319, SuHo Bak2https://orcid.org/0000-0003-2158-9566, Seon-Woong Jang2https://orcid.org/0000-0002-5873-5064, Jong-Myoung Kim1,*https://orcid.org/0000-0003-1005-0952
Author Information & Copyright
1Department of Fisheries Biology, Pukyong National University, Busan 48513, Korea
2IREMTECH. Co., Ltd, Busan 46079, Korea
*Corresponding author: Jong-Myoung Kim, Department of Fisheries Biology, Pukyong National University, Busan 48513, Korea. Tel: +82-51-629-5919, Fax: +82-51-629-5908, E-mail:jongkim@pknu.ac.kr

Copyright © 2022 The Korean Society of Fisheries and Aquatic Science. This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0/) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

Received: Nov 15, 2021; Revised: Jan 13, 2022; Accepted: Jan 17, 2022

Published Online: Mar 31, 2022

Abstract

Various approaches have been applied to transform aquaculture from a manual, labour-intensive industry to one dependent on automation technologies in the era of the fourth industrial revolution. Technologies associated with the monitoring of physical condition have successfully been applied in most aquafarm facilities; however, real-time biological monitoring systems that can observe fish condition and behaviour are still required. In this study, we used a video recorder placed on top of a fish tank to observe the swimming patterns of rock bream (Oplegnathus fasciatus), first one fish alone and then a group of five fish. Rock bream in the video samples were successfully identified using the you-only-look-once v3 algorithm, which is based on the Darknet-53 convolutional neural network. In addition to recordings of swimming behaviour under normal conditions, the swimming patterns of fish under abnormal conditions were recorded on adding an anaesthetic or lowering the salinity. The abnormal conditions led to changes in the velocity of movement (3.8 ± 0.6 cm/s) involving an initial rapid increase in speed (up to 16.5 ± 3.0 cm/s, upon 2-phenoxyethanol treatment) before the fish stopped moving, as well as changing from swimming upright to dying lying on their sides. Machine learning was applied to datasets consisting of normal or abnormal behaviour patterns, to evaluate the fish behaviour. The proposed algorithm showed a high accuracy (98.1%) in discriminating normal and abnormal rock bream behaviour. We conclude that artificial intelligence-based detection of abnormal behaviour can be applied to develop an automatic bio-management system for use in the aquaculture industry.

Keywords: Abnormal behaviour; Deep learning; Object detection; Rock bream; Smart aquafarm

Introduction

The global fishing industry has experienced decreased production mainly due to depleted fishery resources and the strengthening of international fishing standards. In response, fish production by the aquaculture industry has steadily increased to provide alternative fish resources. Despite these increases, the Korean aquaculture industry has shown stagnant growth due to its manual labour-intensive farming techniques. To overcome this problem and create a more systematic aquaculture management system that is compatible with the oncoming fourth industrial evolution, it is important to automate some aspects of the aquaculture industry through innovations such as smart fisheries and automatic sensors for water temperature and dissolved chemical detection, as well as automatic feeding systems and real-time underwater monitoring. Although many technologies have already been adapted to monitor physical tank and cage conditions, those for automatic monitoring of fish behaviour are still needed for real-time fish condition assessment.

Fish have been used as model animals in behaviour monitoring studies, particularly those exploring image analysis of behaviour data collected by underwater acoustic cameras, with and without external stimulation. Underwater acoustic cameras have been applied to collect fish echo data to analyse the distribution of fishery resources and its effects on fish movement (Jang et al., 2016). Video tracking has also been used to trace moving organisms, on land and in air and water (Maggio & Cavallaro, 2011; Noldus et al., 2001). However, to date, no studies have explored the adaptation of automatic real-time monitoring systems that use artificial intelligence to analyse movement patterns that may indicate the health status or abnormal conditions of fish in aquaculture systems.

Deep learning is part of a broader family of machine learning methods based on artificial neural networks (Bengio et al., 2013; LeCun et al., 2015; Schmidhuber, 2015). With recent improvements in computational speed, various algorithms have been exploited to overcome the limitations of deep learning for real-time applications of video or image data, which are otherwise complex and time-consuming. Convolutional neural network (CNN) is a deep-learning technique that provides rapid analysis of calculations performed using a graphics processing unit (GPU); specialised CNNs have been developed for region extraction (e.g., R-CNN; Girshick et al., 2015) and object detection (e.g., you only look once [YOLO]; Redmon & Farhadi, 2018). The YOLO algorithm simultaneously integrates and processes object detection and recognition patterns, and therefore is suitable for rapid, high-accuracy real-time applications.

Rock bream (Oplegnathus fasciatus) inhabits rocky marine areas in shallow coastal regions of Korea, Japan, Taiwan, and Hawaii (Choi et al., 2002); its economic value is increasing in the Korean aquaculture industry. Despite technological advancements in artificial rock bream seeding (Fukusho, 1979; Kumai, 1984), its aquaculture productivity requires further improvement. Studies have examined the photo-response of rock bream (Jang et al., 2019). Artificial intelligence algorithms, including a deep-learning neural network, have been developed for identifying fish species (Allken et al., 2019). However, to solve problems associated with automated fish health monitoring systems in aquaculture, it is still necessary to develop an algorithm that can detect abnormal fish behaviours, which may require training and validation datasets. This study recorded the swimming patterns of healthy rock bream and those exposed to unhealthy conditions. A deep learning-based algorithm was applied to evaluate rock bream behaviour based on recorded fish movement data.

Materials and Methods

Materials

Rock bream O. fasciatus (body weight, ~8.8 ± 1.8 g; body length, 74.0 ± 6.0 mm) were obtained from Ji-Yeon Fisheries (Geoje, Korea). The animal tracking system EthoVision XT10 was from Noldus (Wageningen, the Netherlands). We purchased the anaesthetic 2-phenoxyethanol from Sigma-Aldrich (St. Louis, MO, USA) and obtained a charge-coupled device (CCD) camera from Samsung Electronics (Suwon, Korea). The YOLO v3 algorithm, which is based on a Darknet-53 CNN backbone, was used to analyse fish behaviour.

Fish rearing and video recording

Rock bream were maintained in a 2-ton circulation tank prior to behaviour monitoring. The breeding water was partially replaced with sand-filtered, aerated seawater (salinity, 33 ± 0.5 psu; pH 7.4 ± 0.7). Rock bream were acclimated for 1 week in a rectangular water tank (1 m × 0.45 m, natural seawater, 18°C–20°C). The water salinity experiment was conducted in a 1 m × 0.6 m tank with an inlet and outlet for exchanging water. After acclimatisation, fish movement was recorded by a CCD camera positioned at the top of the tank (1 m above the water surface), first for a single fish and then for a group of five fish. We induced abnormal rock bream behaviour using two methods: adding an anaesthetic (2-phenoxyethanol; 0.05% v/v) to the water tank and changing the tank water from seawater to fresh water. A complete change from seawater to fresh water was achieved in 20 min using the tank inlet and outlet. In each experiment, we recorded fish behaviour for 15–30 min after inducing the abnormal behaviour. The housing and maintenance of the fish conformed to the regulations of The Institutional Animal Care and Use Committee of Pukyong National University (Busan, Korea).

Dataset configuration

The collected data included 10,110 rock bream images obtained from the video recording. Among these, we randomly selected 210 images, and divided these into a training dataset containing 168 images and a test dataset containing 42 images. No images were included in both datasets.

Object detection using deep learning

In pre-processing, the rock bream images were adjusted to a standard size of 416 × 416 × 3 for analysis by the YOLO algorithm. Images of rock bream swimming upright and lying on their side were considered to exhibit normal and abnormal swimming, respectively. The input images were entered into the pretrained CNN, which predicted bounding box information including the location and size of the detected object in the input image for target detection. Within a total learning time of 2 h, class-specific bounding boxes were produced for all input images (Fig. 1).

fas-25-3-151-g1
Fig. 1. Architecture of the deep-learning algorithm used in this study to detect abnormal behaviour in rock bream. A video recording was uploaded to the detection program, and input frames were randomly sampled at a defined interval and divided into training and test datasets. A convolutional neural network based on the Darknet-53 framework analysed the images to detect abnormal swimming behaviour.
Download Original Figure
Algorithm evaluation

The ability of the algorithm to detect rock bream displaying abnormal behaviour was evaluated based on the training data, based on the rates of true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN) in a confusion matrix. The pixel accuracy, precision, and recall were calculated as follows:

pixel accuracy = (TP+TN)/(TP+TN+FP+FN) precision =TP/(TP+FP) Recall =TP/(TP+FN) F1 = 2 × precision × Recall/(precision+Recall)

Results and Discussion

Collecting data on abnormal rock bream behaviour

Behaviour analysis is important for confirming the health of target fish in aquaculture. In this study, images of swimming rock bream, one of the main aquaculture species in Korea, were used to explore the application of an artificial intelligence algorithm to analysing images for species identification and detecting abnormal swimming behaviour. This used a camera located in a fixed position on top of a tank containing one rock bream at a time. Most of the fish swam upright (Fig. 2). In addition to image analysis for species identification using artificial intelligence (see below), the speed of the rock bream was calculated by tracing their movements before and after adding an anaesthetic or abruptly changing the water salinity. Healthy rock bream moved at an average speed of 3.8 ± 0.6 cm/s, but after 2-phenoxyethanol (2-PE) treatment their speed initially increased to 16.5 ± 3.0 cm/s, and then they gradually slowed and finally stopped and lay on their sides (Fig. 3). The slightly faster speed (Fig. 2b) observed in the water salinity experiment might result from the larger tank used for this experiment. The condition-dependent change was also observed in images of rock bream taken from a fixed position, which involved a transition from a narrower top view of the fish under normal conditions, to a wider lateral view of the rock bream following the 2-PE or low salinity treatments (Fig. 3 insets). Since these behaviours mimic those of rock bream in poor condition in aquafarms, these images can be used as training data for evaluating the suitability of the deep-learning algorithm for detecting unhealthy aquafarm conditions.

fas-25-3-151-g2
Fig. 2. Photographs of rock bream taken from tanks containing a single fish (a), a group of five fish (b), or hundreds of rock bream (c). Objects identified as a rock bream using YOLO v3 are in rectangles. YOLO, you-only-look-once.
Download Original Figure
fas-25-3-151-g3
Fig. 3. Speed of rock bream measured on treatment with 0.05% (v/v) 2-phenoxyethanol (a) and water salinity change (b) and images taken from above the tank under the corresponding conditions. The velocity of rock bream was calculated from the distance moved within a time interval. Red arrows indicate the times when 0.05% (v/v) 2-phenoxyethanol was added or the salinity changed. Representative images of a rock bream with normal (left) and abnormal (right) behaviour induced by each treatment are shown.
Download Original Figure
Performance evaluation

Artificial intelligence has been applied to many industries, including aquaculture. In this study, images of a swimming rock bream were entered into a pretrained CNN for target detection. Fig. 2 shows that rock bream can be clearly identified from images of one or many individuals in a tank using the CNN-based YOLO algorithm. The application of artificial intelligence was extended to detect a disastrous situation in an aquaculture facility by analysing the swimming patterns of rock bream. This used images of rock bream swimming upright or lying on their sides as training data for normal and abnormal situations, respectively. In each iteration of the training process, class-specific bounding boxes were produced for each image, and the count information was updated. Within a total learning time of 2 h, the algorithm successfully learned to discriminate normal and abnormal swimming behaviour (Fig. 4). We also evaluated the training process of the YOLO algorithm through loss-of-function analysis. Loss function is the current learning indicator of deep-learning models and its value should be minimised as the number of lessons increases. Loss was minimised within 40 learning iterations (Fig. 5) with its value converging on zero. The result confirmed the suitability of the algorithm for detecting unhealthy aquafarm conditions. We further evaluated the performance of the YOLO algorithm for detecting abnormal rock bream behaviour using a confusion matrix based on the relationship between actual measurements and predicted results with data described as TP, TN, FP, and FN. Based on the learned data, the confusion matrix of abnormal rock bream detected showed TP = 78, FP = 0, FN = 2, and TN = 25 for the normal conditions and TP = 25, FP = 2, FN = 0, and TN = 78 for the abnormal conditions (Table 1). From these data, we calculated the accuracy (0.981), precision (0.963), recall (0.988), and F1 score (0.974) of the algorithm. Together, these results indicate excellent algorithm performance.

Table 1. Confusion matrix used to evaluate the ability of the deep-learning algorithm to detect abnormal rock bream swimming behaviour
Abnormal class Actual behaviour
Abnormal Normal
Predicted behaviour Abnormal 25 (TP) 2 (FP)
Normal 0 (FN) 78 (TN)

The detection of abnormal fish was described as true positive (TP), false positive (FP), true negative (TN), and false negative (FN) depending on the relationship between the actual measurements and predicted results.

Download Excel Table
fas-25-3-151-g4
Fig. 4. Deep learning-based detection of rock bream exhibiting abnormal behaviour. A photograph of five fish in a tank taken 15 min (inlet) after adding 0.05% (v/v) 2-phenoxyethanol. Yellow and green boxes indicate normal and abnormal swimming behaviours, respectively, as detected by the “you only look once” algorithm.
Download Original Figure
fas-25-3-151-g5
Fig. 5. Learning curve for abnormal object detection of rock bream. The plot shows changes in loss function (the current learning indicator) with learning iterations.
Download Original Figure
Artificial intelligence-based monitoring in aquaculture

The development of artificial intelligence-based technologies suitable for application in aquaculture has the potential to transform this manual labour-intensive industry to rely on more efficient automated systems (Yang et al., 2020). Various deep-learning algorithms have been applied in marine biology for the development of smart fisheries, including automated fish detection based on cascade classification of Haar-like features in underwater images captured by a remotely operated vehicle in an ocean survey (Cutter et al., 2015) and deep learning-based fish detection (Levy et al., 2018). Recent deep learning-based studies have focused on detecting fish species using CNNs (Han et al., 2020), classifying goldfish species using pruned neural networks (Ayob et al., 2021), and detecting sea cucumber using a single-shot multibox detector (Ma et al., 2019). This study also proved that rock bream can be identified using an artificial intelligence-based algorithm.

In aquaculture farms, real-time fish monitoring would allow machines to automatically detect deteriorating fish condition and trigger an alarm before serious damage could occur. To develop an artificial intelligence-based algorithm for identifying abnormal situations in aquaculture, it is critical to secure learning data corresponding to each situation. This study obtained behaviour datasets of rock bream under abnormal situations using anaesthetic or low-salinity treatments. Abnormal behaviors of fish under harsh conditions include an irregular movement, reduced food intake, increased surfacing behaviors, changes in the frequency of gill opening, sluggish swimming, delayed response, loss of vertical equilibrium, departure from crowds, and scrubbing of the tank floor. Changes in swimming speed is another criterion used to define abnormal behavior in fish. This was shown the phenomenon that, immediately after administration of the anesthetic, the speed of the fish increased rapidly and then decreased and eventually stops. Among these behavioral characteristics, the flipping phenomenon of the fish, one of the simplest indicators, was used to identify the abnormal behavior in this study. The study results indicate that a deep-learning algorithm can be used to detect abnormal swimming under aquaculture conditions after learning representative patterns of normal and abnormal swimming behaviour. The accuracy of the proposed algorithm for detecting abnormal swimming by anaesthetic-treated rock bream was 98.1%. Our findings suggest that deep learning has the potential to improve aquafarm efficiency and prevent economic losses through automated monitoring of fish.

Competing interests

No potential conflict of interest relevant to this article was reported.

Funding sources

This research was funded by the Ministry of Oceans and Fisheries, Republic of Korea, under Contract No. 20180373 as a part of the project “Establishing a Foundation for the Year-Round Production of Flatfish Eggs and Improving Productivity. YR Kim was also supported by ‘LED-Marine Technology Convergence R&D Center” funded by the Ministry of Ocean and Fisheries, Korea.

Acknowledgements

Not applicable.

Availability of data and materials

Upon reasonable request, the datasets of this study can be available from the corresponding author.

Ethics approval and consent to participate

This article does not require IRB/IACUC approval because there are no human and animal participants.

References

1.

Allken V, Handegard NO, Rosen S, Schreyeck T, Mahiout T, Malde K. Fish species identification using a convolutional neural network trained on synthetic data. ICES J Mar Sci. 2019; 76:342-9

2.

Ayob AF, Khairuddin K, Mustafah YM, Salisa AR, Kadir K. Analysis of pruned neural networks (MobileNetV2-YOLO v2) for underwater object detection.In: Proceedings of the 11th National Technical Seminar on Unmanned System Technology. 2019 Gambang, Malaysia.

3.

Bengio Y, Courville A, Vincent P. Representation learning: a review and new perspectives. IEEE Trans Pattern Anal Mach Intell. 2013; 35:1798-828

4.

Choi Y, Kim JH, Park JY. Marine fishes of Korea. Seoul, Korea: KyoHak. 2002.

5.

Cutter G, Stierhoff K, Zeng J. Automated detection of rockfish in unconstrained underwater videos using Haar cascades and a new image dataset: labeled fishes in the wild. Paper presented at: IEEE Winter Conference on Applications of Computer Vision Workshops. 2015 Waikoloa, HI.

6.

Fukusho K. Studies on fry production of Japanese striped knife jaw Oplegnathus fasciatus, with special reference to feeding ecology and mass culture of food organisms. Spec Rep Nagasaki Pre Ins Fish. 1979; 430:173.

7.

Girshick R. Fast R-CNN. Paper presented at: 2015 IEEE International Conference on Computer Vision (ICCV). 2015; Santiago, Chile. pp p. 1440-8

8.

Han F, Yao J, Zhu H, Wang C. Marine organism detection and classification from underwater vision based on the deep CNN method. Math Probl Eng. 2020; :3937580

9.

Jang JC, Choi MJ, Yang YS, Lee HB, Yu YM, Kim JM. Dim-light photoreceptor of chub mackerel Scomber japonicus and the photoresponse upon illumination with LEDs of different wavelengths. Fish Physiol Biochem. 2016; 42:1015-25

10.

Jang JC, Noh GE, Kim YR, Yu YM, Kim JM. Spectral sensitivity and photoresponse in the rock bream Oplegnathus fasciatus and their relationships with the absorption maximum of the photoreceptor. Fish Physiol Biochem. 2019; 45:1759-69

11.

Kumai H. Biological studies on culture of the Japanese parrot fish Oplegnathus fasciatus (Temmincket Schlegel). Bull Fish Lab Kinki Univ. 1984; 2:127.

12.

LeCun Y, Bengio Y, Hinton G. Deep learning. Nature. 2015; 521:436-44

13.

Levy D, Belfer Y, Osherov E, Bigal E, Scheinin AP, Nativ H, et al. Automated analysis of marine video with limited data. Papaer presented at: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). 2018 Salt Lake City, UT.

14.

Ma K, Huang B, Yin H. Underwater sea cucumbers detection based on improved SSD. Paper presented at: 2019 IEEE International Conference on Power, Intelligent Computing and Systems (ICPICS). 2019 Shenyang, China.

15.

Maggio E, Cavallaro A. Video tracking: theory and practice. Hoboken, NJ: John Wiley & Sons. 2011

16.

Noldus LPJJ, Spink AJ, Tegelenbosch RAJ. EthoVision: a versatile video tracking system for automation of behavioral experiments. Behav Res Methods Instrum Comput. 2001; 33:398-414

17.

Redmon J, Farhadi A. Yolov3: an incremental improvement. 2018https://arxiv.org/abs/1804.02767

18.

Ren S, He K, Girshick R, Sun J. Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell. 2017; 39:1137-1149

19.

Schmidhuber J. Deep learning in neural networks: an overview. Neural Netw. 2015; 61:85-117

20.

Sun M, Yang X, Xie Y. Deep learning in aquaculture: a review. J Comput. 2020; 31:294-319.

21.

Yang X, Zhang S, Liu J, Gao Q, Dong S, Zhou C. Deep learning for smart fish farming: applications, opportunities and challenges. Rev Aquacult. 2020; 13:66-90