An Overview of Horizon Detection Methods in Maritime Video Surveilance

e-mail: mpetkovic@pfst.hr The interest in video surveillance has been increasing in the fields of maritime industry in the past decade. Maritime transportation system is a vital part of the world’s economy and the extent of global ship traffic is increasing. This trend encourages the development of intelligent surveillance systems in the maritime zone. The development of intelligent surveillance systems includes sensor and data fusion, which incorporates multispectral and multisensory data to replace the traditional approach with radars only. Video cameras are widely used since they capture images of greater resolution than most sensor systems. Also, combined with video analytics they provide sensors with high capability, complex pattern recognition analytics, and multiple variables for the decision making process. In this paper, an overview of a small part of the system is presented – horizon detection. An Overview of Horizon Detection Methods in Maritime Video Surveilance


INTRODUCTION
Since radar tracking is sensitive to shape, size and material of the targets, it has to be enriched with other types of sensors for better situational awareness, collision avoidance, and navigation. Video cameras are widely used since they capture images of greater resolution than most sensor systems. Also, combined with video analytics they provide sensors with high capability, complex pattern recognition analytics, and multiple variables for the decision-making process.
Maritime video surveillance is important for a wide range of applications. For example, video surveillance systems are employed to increase the security of ports and ships, to control maritime traffic in ports or a maritime zone, and it is one of the vital systems in autonomous ships. Maritime video surveillance may take place from a small surface-vehicle (manned or unmanned)mounted camera or buoy in line with the water to static landbased cameras or aerial surveillance from drones. According to (Gladstone et al., 2016;Bloisi et al., 2017;Vujović and Kuzmanić, 2018), maritime surveillance systems have to overcome a set of challenges: • Wide monitored domain • Weather issues (rain, snow, fog …) • Ever-changing nature of the sea (waves, white foam, sun reflections) • Inconsistent size of tracked objects • Multiple tracked objects with possible occlusions. As described in Prasad et al. (2017), basic maritime video surveillance system is composed of five main components: the initial detector, image processor, classifier, tracker, and behaviour analyser if necessary. The basic components of maritime video surveillance are illustrated in Figure 1   Generalised flowchart of maritime surveillance data processing for OOI detection (Prasad et al., 2017).
The initial detector detects pixel motion or an object based on a classifier set. The obtained information is managed by the image processor for noise elimination. Also, relevant regions of interest (RoI) are determined. RoIs in the frame are evaluated by the classifier to determine presence of an object that is object of interest (OOI). The role of the tracker is to locate the OOI in a RoI at each frame and determine its position. The OOI's course and speed are sent to behaviour analyser. These components are not part of every maritime video surveillance system. This paper presents an overview of horizon detection methods. The second section describes the main topic, using math and examples. It is divided into several subsections dealing with projection-based, region-based, hybrid and ANN (Artificial Neural Networks) methods. Finally, the conclusions are given.

2.HORIZON DETECTION IN MARITIME ZONE VIDEO SURVEILLANCE
Horizon detection is one of the essential tasks in maritime video analysis since its results affect the performance of the surveillance system. This importance can be seen in Figure 2 as it shows a generalized flowchart of data processing in maritime surveillance for object detection.
Horizon information is used in some object detection approaches and for the reduction of false positives for a given object detection rate (Jeong et al., 2018a). Also, it is used for distance prediction of another object to the camera (Gladstone et al., 2016) or for maritime target detection and tracking in infrared images (Jian and Wen, 2019). We can distinguish three main approaches for horizon detection: projection-based, region-based and hybrid. We also noted a few methods based on artificial neural networks (ANN).

Projection-Based Methods
Projection-based methods use edge detectors (Moreira et al., 2014), such as Canny detector (Gershikov et.al., 2013), to compute the edge map of the image. To identify line features more easily, the edge map is projected to another space. For such projections Hugh and Radon transforms are often used.

R (θ,ρ) = ∫∫ I(x,y) δ (x cosθ+y sinθ-ρ) dxdy
where θ is the angle between x-axis and normal to the line, and ρ is the distance from the origin of the coordinate system to the line. The coordinates (x, y) of an edge pixel are transformed as a curve into the Hough space (θ, ρ) (Ginkel et al., 2004): where δ is the Dirac delta function while I(x, y) represents the edge map. (θ, ρ) cells in the histogram, corresponding to the largest values of H(θ, ρ) represents the line parameters (Prasad et al., 2017). The Radon transformation is formulated as (Ginkel et al., 2004): where the cells in (θ,ρ) containing the highest number of entries in R(θ, ρ) define line parameters. The simplicity of projective approaches makes them popular, but they are sensitive to preprocessing (Prasad et al., 2017). Also, when the dominant line is not the horizon line, they frequently fail to detect the correct line (Liang and Liang, 2019).

Region Based Methods
By estimating the RoI in an image containing the horizon line, Mou et al. (2016) reduced the processing area of the original image, thus reducing the computing expense. Also, the authors used the random sample consensus-based method (RANSAC) hierarchically. This approach proved to be fast, but it also renders errors on scenarios with much noise, e.g. sea-shore scenarios, objects on the horizon, etc. Some region-based methods rely on the fact that in the horizon region, intensity variations are higher than in the sky or sea regions alone. Such intensity variations can be used for horizon detection. Also (Gershikov, 2014) evaluated usage of colour variations in horizon detection. In the paper by Jeong et al. (2018b), the authors investigated the use of RoI method for horizon detection. In the first step, the input image is resized and divided into N horizontal regions with 50 % overlapping, and mean vectors and covariance matrices of the colour distribution are calculated for each region. To calculate the difference between two successive regions, Bhattacharyya distance is used: where µ and ∑ are the mean vectors and covariance matrices of the colour distribution of the regions.
The region with the highest distance is chosen as the RoI for horizon detection, confirming that average region colour suddenly changes near the horizon. This method detects edges by applying a median filter (with three different scales) as smoothing filter of various sizes. Then, Canny edge detector is N represents the number of median filters w s is the weight of scale s E s is the edge maps of the scale s. For the horizon line estimation, Hough transform and a least square method are used sequentially. This approach showed reliable performance, but when the horizon edges could not be detected because of blur occurring due to the moving camera problem, its performance degraded. To solve this problem, the authors suggested using sophisticated filtering methods to overcome the motion of vessels.
In Sun and Fu (2018), for unmanned surface vehicle (USV) application, the authors used line segment detection algorithm with fast computational speed (von Gioi et al., 2010). By applying gradient features to extract all line segments for building a pool of candidate lines. Since the pool will probably contain many false detection results, hybrid feature filtering is used to select segments of the horizon line from candidate pool. Morphology features and colour features of the horizon line are used to filter out the false results and, similar to Jeong et al. (2018b), calculate the distance between the regions. Line segments are stitched using RANSAC to obtain the whole horizon line. The average computing time for this method was 94 ms (Sun and Fu, 2018), and it proved to be fast and robust. Also, when the horizon lines are very blurry as shown in Fig. 3 b), its performance degraded. applied on the multi scale images independently (Jeong et al., 2018) to obtain the weighted edge map, as follows: Fefilatyev et al. (2012) used a hybrid approach to generate the line estimation by projection method and then applied statistical analysis. Regions below and above the estimated line are considered as potential sky and sea regions, then their statistical distributions were computed. Then Bhattacharyya distance between the two distributions was calculated as follows:

Hybrid Methods
where λ and ∑ represent the mean vector and the covariance matrix of distribution, and the estimated line with the maximum value is chosen as horizon. In Lipschutz et al. (2013) a similar method is proposed, where at the pre-processing stage they used morphological filter, and histograms of sky and sea regions were calculated to produce the colour distribution of each region.
The method proposed by Liang et al. (2015) consists of three parts, as shown in Fig. 4.  The first part locates the horizon region by using grey level concurrence matrix from the texture feature. This feature helps in the reduction of interference caused by waves and light. The second structure consists of OTSU algorithm used for obtaining a set of estimated points of the horizon line. The third structure is the horizon line detection, which eliminates unwanted points caused by ships and waves appearing on the horizon. For this purpose, the authors designed a simple clustering algorithm with a low computation cost. The proposed method proved to be very capable for maritime images under complex background, e.g. clouds, sea waves or too much light. The authors noted that by narrowing the search window this approach can perform faster to achieve a greater processing speed without affecting the accuracy of horizon detection.
In their paper, Prasad et al. (2016a) proposed a multiscale cross modal linear feature (MSCM-LiFe) method, where multi-scale approach for edge detection is adopted. Multi-scale images I s are computed with vertical median filter of scale s. For estimation of the horizon line, Hough transform (Eq. 1) is used, and top 10 candidates with the largest values of H(θ,ρ) are selected and their Hough score H n is stored . Also, mean multi-scale image ( Ĩ s ) is computed, and intensity variation is calculated for each column of pixels defined by pixel x, so the point (x,y'(x)) is determined as follows: Then, a line is fitted on all points of maximum intensity variation and is used as IVA candidate. For each scale s, the mean value of intensity gradients of all columns is determined to obtain IVA score S s : Each Hough candidate n and IVA candidate s are used to compute the goodness score of the pair and their geometric proximity. Goodness score is defined as:

G(n,s) = H n S s (11)
A(n,s) = G(n,s) P(n,s) (12) x cos (θ)+ y sin (θ) < ρ (13) x cos (θ)+ y sin (θ) ≥ ρ  while geometric proximity P(n,s) is defined as: where max(y) is the number of pixels along the y-axis, (Y n -Y s )/max(y) is the relative vertical distance between horizon candidates (n, s), the term (α n -α s ) represents angular difference (Prasad et al., 2016a). By analysing goodness score and geometric According to Sun and Fu (2018), MSCM-LiFe has high average computation time of 231 ms. Also, this method scored excellent results when horizon line was blurry, as shown in Fig.  5 b), but when horizon line was partially occluded by ships or objects, it failed to detect the line accurately Fig. 5 a).
proximity of each estimated pair, the final horizon line is obtained by selecting the Hugh candidate with highest affirm score A(n, s): Also, in the paper by Prasad et al. (2016b) multi-scale consistence of weighted edge Radon transform (MuSCoWERT) method was proposed. First, this method generates multi-scale images by applying edge preserving filter, with different sizes. It helps in smoothing intensity variations not related to the edges, coming from dynamic sea and sky noise. Then, by analysing the length of the edges the authors generated weighted edge map. Radon transform is applied for each weighted edge map to approximate parameter of the estimated line. Then, by observing each estimated line parameter, final horizon line is selected by voting. Despite its excellent performance, this method can fail in certain scenarios where the horizon line is occluded by various objects or ships.
Most hybrid methods require statistical analysis for the horizon detection since the number of estimated horizon lines is large. Hence, they have high computation time.

ANN-Based Methods
Machine learning approach was proposed by Fefilatyev et al. (2006) where they manually drew the horizon line as a ground truth θ and ρ parameters of line on each image used for classifier training. All pixels that satisfied: were labelled as sky-pixel, and all pixels that satisfied: were labelled as ground-pixel. The authors defined 21 attributes for each pixel, texture measurements for each of the three colour channels (described in Fefilatyev et al., 2006) for a histogram of 10 x 10 region centred on each pixel. The output of the classifier is a black and white image (representing ground and sky) in which they obtained the line that separated the white and black regions declaring it as the horizon line. The results of this approach largely depend on the amount and variety of data used for classifier training, and have lower accuracy under changing lighting conditions.
In Kristan et al. (2016), the authors used image segmentation with weak priors for obstacle detection on USV. The role of the semantic segmentation is to assign every pixel its appropriate class label. The authors observed that each image can be split into three semantic regions where the bottom region represents the sea, top region represents the sky while middle region can represent the land or horizon region. Their approach estimates per-pixel class probabilities and optimizes segmentation within a single online framework, avoiding the need for a good horizon detection estimation. However, this probabilistic approach is general enough to include horizon detection if needed. Semantic segmentation approach is evaluated by Ahmad et al. (2017) on land-sky images under various weather and illumination conditions. Fully convolutional network performed the best on said images, but further post processing is required to improve the segmentation. Cane and Ferryman (2018) evaluated semantic segmentation networks (SSN) for object detection system in the maritime environment. The authors proposed a simple system which takes RGB images on input. Then, images are processed using SSN to generate the probability distribution of a class for each pixel. A binary map is created for each class by selecting pixels with the maximum probability for that class. Next, by marking the connected components and computing bounding boxes, estimated regions are extracted from the binary maps. For network training, they used subset of the ADE20k dataset because it was the only dataset available at the time which covers the appropriate classes with pixel level ground truth. The authors found that their approach made the horizon line detection easy by extracting it from segmentation map.
A novel approach for horizon detection was proposed by Jeong et al. (2018a). The authors segmented each pixel into semantic categories using a pyramid scene parsing network (PSPnet). To extract sea line, in each column of the segmented image PSPnet searches for the maximal vertical location corresponding to the sea. Unnecessary edges are excluded by using brightness variation analysis. For situations where the horizon is occluded by objects such as ships or buoys, the authors implemented a robust line fitting method to complement the PSPnet. To estimate candidate line, the least squares method is applied to the boundary image. The residual, between the estimated line and the boundary pixels is calculated. The pixels with distances larger than the median residual are ignored. Repeating this process until convergence of the horizon parameters improves the accuracy and robustness of the horizon line detection of this method.
The use of back propagation neural network (NN) was evaluated in the paper by Kumeechai and Jiriwibhakorn (2019) and tested versus Hough transform, least squares and RANSAC. Their focus was on the accuracy rate and efficiency of the horizon detection. Back propagation NN gave the best results in general, but with high computation time. Therefore, it is not suited for embedded applications. Praczyk et al. (2019) applied AutoEncoder NN for horizon line detection in maritime images taken in the open sea. Hough transform was applied for line extraction and was represented by a feature vector containing the average brightness of the image fragment below and above the line. Then, authors trained an AutoEncoder on the representations of only true lines, while neglecting the remaining lines. The aim of this approach is to obtain the network that would be able to accurately reconstruct true lines on the output, while the other lines should be reconstructed with greater error than the true lines. This method proved highly effective for horizon detection, but is highly computation demanding.

CONCLUSION
Research in the field of maritime video surveillance is increasing every year, but there are not many papers in the horizon detection niche, as can been observed from this overview. The projection-based, region-based, hybrid and ANN-based methods for horizon detection are discussed in the paper. Simplicity is the main advantage of projection-based methods, but they often fail in the horizon line detection when horizon is not the dominant line in the frame. On the other hand, the region-based methods have proved to be reliable in the horizon line detection with low computation time, which is ideal for static land-based maritime surveillance. However, its performance degraded when used on buoy as system due to blur occurring from the moving camera problem. Hybrid method combines projection and/or regionbased methods with statistical analysis with excellent results. The usage of statistical analysis greatly increases the computation time, but this problem can be reduced by narrowing the search window. The researches of ANN methods in the maritime surveillance increased in the past couple of years. They have proved to be very effective in the maritime object segmentation and made the horizon line detection easy, accurate, and robust. ANN methods will improve even more if the number of datasets with pixel-level ground truth for semantic segmentation network training increases.

ACKNOWLEDGMENT
This research has been performed as a part of the scientific project "Establishment of reference database for studying the influence of weather conditions on marine video surveillance" at the Faculty of Maritime Studies, University of Split, and the project "Functional integration of the University of Split, Faculty of Maritime Studies, Faculty of Chemistry and Technology, and Faculty of Science through Development of Scientific and Research Infrastructure in the Building of 3 Faculties, KK.01.1.1.02.0018", financed by the EU. It is conducted by the research group "New technologies in maritime" at the Faculty of Maritime Studies in Split.