Concept and First Results of Optical Navigational System

e-mail: s.hozyn@amw.gdynia.pl The paper presents a concept of the optical coastal navigational system. The task of the system is to provide information about object position in coastal areas based on optical information. The system is intended for Autonomous Underwater Vehicles that operate in GPS denied environments. In addition to the concept itself, the paper also outlines the first results achieved during the research on the system. Concept and First Results of Optical Navigational System


INTRODUCTION
The accuracy of underwater navigation depends on the quality of sensors and navigational devices.For small and medium size autonomous underwater vehicles equipped with sensors and devices of imperfect accuracy, the underwater navigation has to be often supported by other systems with the purpose to reduce position error produced by dead reckoning navigation.For this purpose, satellite systems are typically applied.However, the problem arises in GPS denied environments where access to navigational information from GPS is limited or completely impossible.In this case, visual information can be used.An underwater vehicle may surface and determine its own position based on the visual features of its location.
Vision systems are widely applied in navigation of air and land robots (indoor navigation).The former ones use databases with geo-information or with vision information derived from satellite recordings.The latter systems are based on SLAM technology, which assumes continuous access to the vision information about environment surrounding the robot.There are also maritime solutions including the underwater ones, which are based on SLAM technique like their land counterparts.
However, the problem arises if the vehicle does not have continuous access to the vision information; this information is rare and is accessible from time to time when the vehicle appears on the surface for correction of its position.In this case, SLAM algorithm is useless, and a different technique has to be applied.It should be able to improve indications of underwater dead reckoning system by determining approximate location of the vehicle based on rare visual scans of the external world.In order to design such a technique, Polish Naval Academy lunched a project called "Optical Coastal Marine Navigational System" (OCMNS), and the current paper is a short report on the first stage of the project.Generally, the OCMNS is supposed to operate in different distances from the land, i.e. it should be able to work on the vision information of different characteristics.According to initial assumptions, the system is to work in three different modes: mode No. 1 -large distance to land, no land details are visible, only outline of the coast is visible; mode No. 2 -average distance to land, texture visible, e.g.beach, urban area, forest; mode No. 3 -small distance to land, details visible, e.g.trees, buildings, ships.The task of the first step of the project, which is reported in the paper, was to focus on the mode No. 1.The example results of this step are given below.
To determine the vehicle position, the OCMNS needs simplified navigational chart with contextual information, i.e. with the information that makes it possible to determine the nature of each coastal object visible in the chart.It is assumed that the chart is to be constructed based on freely available vision data derived e.g. from Google Map service.In order to build the chart, a database was implemented that is prepared to store Google Map images combined with the contextual information mentioned above.Moreover, a computer application was implemented which makes it possible to categorize selected fragments of the images.Both software components are shortly outlined below.
The other critical element of the OCMNS and the entire project is a set of example vision data (images) along with the contextual information (geographical coordinates, Euler angles at the point of image recording) that are necessary to prepare the system to work.Many image processing solutions that are intended for use in the OCMNS require training data, i.e. example images that can be recorded by underwater vehicle during its operation on the surface.In order to acquire the above mentioned data for different weather conditions, at daytime, at night, in winter, in summer, etc., a small remotely operated surface vehicle was constructed and equipped with two digital cameras, an inertial unit, and GPS receiver.The construction of the vehicle is also outlined in the paper.
The rest of the paper is organized as follows: Section II introduces the small surface vehicle mentioned above, Section III outlines design and implementation of both the database and the computer application for constructing and storing the chart with the contextual information, Section IV reports the first results of the OCMNS, and the final section summarizes the paper.

SURFACE VEHICLE
The primary ability of the OCMNS is image processing oriented to the estimation of vehicle position based on visual clues.The components of the system responsible for image processing can implement different algorithms.All of them require tuning and testing, and some of them also require training.To this end, example images are necessary, combined with exact position and orientation of the camera at the point of image recording.To record the images, a small remotely operated surface vehicle was designed and equipped with two cameras (one is mounted on the mast and the other is integrated within the vehicle hull), an inertial unit, and GPS receiver.Moreover, it was equipped with Wi-Fi communication system, batteries as an independent energy source, memory card for storing images, and Raspberry Pi 3 computer as a control system.Software of the vehicle enables an external operator to control motion of the vehicle and to record images along with the contextual information on memory card.The camera mounted on the mast can be raised up to a certain height.All key components of the vehicle are illustrated in Figure 1.

SOFTWARE TOOLS FOR CONSTRUCTION VEHICLE CHART
The primary assumption of the OCMNS is the ability to employ two types of information.The first one is visual information which, depending on the distance to land, takes different forms.The second type of information needed by the system is map/terrain information which, according to a next assumption, is acquired from freely accessible map sources like Google Maps service.
In order to determine the vehicle position based on visual information, extra information is necessary which is used to confront and to adjust the recorded image or sequence of images to the appearance of the coast from an assumed point of observation.In the best case, the extra information should take the form of example pattern images of the coast that should be recorded in advance (before the mission).Since the case when the system is in possession of example coast images is very rare, the decision was made to rely on publicly available data derived e.g. from above mentioned Google Maps service.The service provides images of almost any area in the form of a simple topographic map or a terrain map (satellite images).The images enable the system to determine the distance to land and the type of an observed object for a given vehicle position and a line of sight.
The above information can be used away from the land when no details are identifiable in the images -distances to land estimated for a number of lines of sight, based on visual information, are confronted with the distances determined with the help of Google Maps images.In this case a position is sought which fits the estimations best.If some details are noticeable in images, e.g.beaches or forests, then their location in the images can also be matched with the map to find best position estimation.If "point" objects like trees or buildings are detected, they are first placed on the map (if the system knows its own position, the line of sight for which the object is detected and distance to land from the position and for the line of sight, then it can estimate the position of the object), and after that, they can be further applied as landmarks to fix the vehicle position in successive points in time.In the latter case, Google Maps images are unnecessary.
To sum up, in further distances to land, the system should rely on the map and terrain information derived from Google Maps, and simultaneously it should try to extract isolated objects like trees and buildings from the recorded images.In turn, in closer distances to land, the vehicle position should be estimated based on the objects extracted earlier.
In order for the OCMNS to be able to take advantage of the information derived from Google Map, two software components were implemented.The first one is on-board system, and its task is to provide information about the distance to land for a given observation point and line of sight.Moreover, its responsibility is to supply the OCMNS with the information about the type of object that is observed from the specified point and line of sight.A next software component is applied to prepare the database of Google Maps images and to link pixels of the images to a number of categories, e.g.forest, beach, urban area, cliff.Since it is difficult to determine with one hundred percent certainty what the vehicle can see looking from a given point in a certain direction and using exclusively satellite images, each fragment of the coast in the images can have a number of categories assigned.Each assignment is also described by probability of a given category at a given place.Figure 2 illustrates example GUIs of the software outlined above.

EXTRACTION OF LAND FROM MARITIME IMAGES
In the first stage of the project, the objective was to design an algorithm for estimating the position of the vehicle far away from the land.To this end, a solution can be applied which, for a number of lines of sight, determines where the land is closer and where it is farther.The system equipped with such information can search the map for a point which best fits the observed situation.Of course, the point should be sought in proximity of position fixed by a dead reckoning navigational system.In order to determine the relative distance to land for different lines of sight, a method can be applied which extracts land from the images and compares the size of land in each image.If land occupies only a fragment of the image, left or right, the tip of the land can be used by the system as an extra useful landmark.
In order to extract the land from the images, three approaches can be applied, i.e. traditional segmentation, separating the land from the rest of the image by means of sky-land and land-sea lines, and Deep Learning semantic segmentation.In the first stage of the project, the first two approaches were considered.The example results of the research in this field are given below.

Traditional Segmentation
For the purpose of land extraction, three traditional image segmentation methods were used: Floodfill (Bradski & Kaehler, 2008), Watershed (Hlavac, 2011), and Graph Cut (Szeliski, 2011).All the tests were performed on 150 example images of Gdynia harbour that were recorded at a different distance to land.In order to facilitate the task of Floodfill and Watershed, the land extraction was performed as a two-stage process of sky extraction (upper part of the image), and sea extraction (bottom part of image).In the case of Graph Cut, segmentation was preceded by a rough marking of land segment.The example results are presented in the Figure 3  Generally, the experiments with traditional segmentation revealed that, regardless of the applied method, the traditional approach is insufficient to effectively extract land from marine images.The least problem was sky extraction, and Floodfill method often coped with this task quite effectively.In the case of sea extraction, the results were definitely worse in general.Additionally, the applicability of Graph Cut to the OCMNS appeared to be reduced due to the necessity of manually indicating rough land segment before segmentation.

Land-sea Line Extraction
Other method for land extraction is to separate it from the sky and the sea.To determine the line separating the sea from the land (land-sea line), Hough transform can be applied (Duda & Hart, 1972;Gershikov, Libe & Kosolapov, 2013).The transform is able to detect all the straight lines visible in the image.The set of lines determined by the transform can include both segments belonging to the land-sea line and other lines.The problem is, however, how to select the land-sea line from all the set.The simplest method is to select the longest line.The tests revealed, however, that this method is not effective.Another method is to describe each detected line with a feature vector and then to use identification techniques to find the right line.In the experiments made on marine images recorded at the open sea (without land visible), AutoEncoder neural networks were applied as tools for identification of the lines separating the sea and the remaining parts of the image.Each line detected by Hough transform was represented by a feature vector describing the average brightness of the image fragment below and above the line.Then, an AutoEncoder was trained on the representations of only true lines, while the remaining lines were neglected in the training process.The task of the training was to obtain the network that would be able to reconstruct an input signal on an output layer of the network.The objective of this approach is to sensitize the network to only true lines, which, after a training, should be properly reconstructed.Other lines which differ from those applied in the training process should be reconstructed with a noticeably greater error than the true lines.The results of the experiments with the Hough transform combined with AutoEncoders revealed a high effectiveness of this method.Example lines are presented in Figure 6.Unfortunately, the same experiments also showed that this method is computationally very demanding and, therefore, it is not suitable for on-thefly calculations on board the underwater vehicle.A useful alternative for the Hough-transform-based method is the QHLD algorithm (Quick Horizon Line Detection) proposed in (Praczyk, 2017).In this case, most calculations are performed on a small resized counterpart of an original image with the effect that the algorithm is very fast and, therefore, suitable for the task of land extraction and position estimation.The example results achieved by the QHLD algorithm are illustrated in Figure 8.

Sky-land Line Extraction
In order to separate land from the other elements of the image, in addition to the land-sea line, the sky-land line which separates the land from upper part of the image is necessary.To this end, an algorithm called GELD (Gradual Edge Level Decrease) was proposed (Praczyk, 2018).It generates contours in the image and tries to find the path from the left to the right edge of the image, or in the opposite direction.The path consists of contour pixels separated by a small predefined distance from each other.First, the path is sought for very strong contours (Figure 7a).In case of a failure, weaker contours are generated (Figure 7b-e) and the process of searching the path is repeated.Weakening of the contours and searching the path is continued until it is found.To verify the effectiveness of the combined application of the GELD and the QHLD, they were tested on 150 images of Gdynia harbour and its neighbourhood.The tests revealed a high effectiveness of both algorithms in land extraction.What is more, the algorithms also showed that they are very quick, which makes them a perfect tool for land extraction in the OCMNS.The example application of the GELD and the QHLD is illustrated in Figure 8.
In addition to the GELD and the QHLD, an algorithm was proposed which is their more complex and more computationally demanding variant.At the moment of preparing the paper, the algorithm is still under construction and tuning process.However, the first tests performed on a few selected images showed its almost perfect effectiveness in land extraction -see Figure 9.

SUMMARY
The paper presents the concept of the Optical Coastal Marine Navigational System which is intended for autonomous underwater vehicles that operate in GPS denied environments.The task of the system is to reduce the position error of dead reckoning navigational system when the vehicle is on the surface.To this end, the system extracts visual clues from images recorded by means of a single camera and uses the clues along with the map information to determine vehicle position.
Apart from the concept of the OCMNS, the paper shortly presents the first results of the project focused on the system.The following three artefacts are described: (i) remotely operated small surface vehicle for recording visual data that are necessary for further research, (ii) software for building the database of map images along with contextual information, (iii) algorithms for land extraction from marine images along with results of the verification tests.

Figure 1 .
Figure 1.Key components of the surface vehicle, and the vehicle itself.
Figure 2. Example user interfaces of the software for building the database of Google Maps images.

Figure 3 .
Figure 3. Example application of Floodfill: (a) original image, (b) image after extraction of the sea (black pixels), and (c) image after extraction of the sky.
Figure 4. Example application of Watershed: (a) original image, and (b) image after extraction of the sky.

Figure 5 .
Figure 5. Example application of Graph Cut: (a) original image, and (b) image after land extraction.

Figure 6 .
Figure 6.Example lines determined by Hough transform and AutoEncoders.

Figure 8 .
Figure 8. Example application of the GELD and the QHLD (Figure taken from (Praczyk,

Figure 9 .
Figure 9. Example performance of newly designed algorithm.