AUV-Based Multi-Sensor Dataset: Forward-Looking Camera (FLC) and Forward-Looking Sonar (FLS) Observations in the Red Sea

Context This dataset is the first part of a dataset collection comprised of forward-looking sonar (FLS) and forward-looking camera (FLC) underwater images. The entire data was collected during the years 2021-2023 using 2 underwater vehicles in both the Red Sea and the Mediterranean along the Israeli shoreline, depicting both man-made and natural underwater environments. The data is part of a research project aimed at developing fusion models for improved obstacle detection and navigation in autonomous underwater vehicles. Content This dataset consists of FLC and FLS images and their metadata, collected by the ALICE-AUV. Both sensors were installed in the front payload section in a configuration having aligned fields of view to achieve matching pairs of data. The data was collected to train and evaluate a complete perception and obstacle avoidance framework. A series of diving sessions were performed in the Red Sea, off the coast of Eilat, Israel. The experiments focused on two main sites: A "Sunboat" shipwreck and the Eilat-Ashkelon Pipeline Company (EAPC) pier pillars. The "Sunboat" shipwreck is a 40-meter long vessel resting at a depth of approximately 12 meters, with the surrounding seabed at a depth of 18-24 meters. This dataset contains approximately 8,000 FLC-FLS sample pairs from the first session conducted at the "Sun boat" shipwreck site on September 3, 2023. The data was recorded at depths ranging from 10 to 15 meters. The dataset is organized into separate sessions, each representing a specific dive or experiment. Within each session, the data is further categorized into modalities: camera (FLC images), sonar (FLS images), and navigation (dead reckoning data). The navigation data is derived from a combination of GPS, DVL, and IMU sensors, providing estimated positions when GPS is unavailable. Inside each modality directory, you will find the corresponding data files in PNG format for images and CSV format for navigation data. The file names follow a sequential numbering scheme (e.g., 00001.png, 00002.png, etc.). Each modality directory also contains a CSV file (e.g., camera.csv) that maps each data file to its respective timestamp. Additionally, the samples.json file documents the relationship between uni-modal and multi-modal samples, allowing for easy association of data from different modalities. By providing synchronized and aligned camera and sonar imagery, along with corresponding navigation data, this dataset enables researchers to explore novel algorithms and techniques for multi-modal sensor fusion in the context of autonomous underwater vehicles. Technical Details Sonar: Blueprint Oculus M1200d Operating frequency: 1.2 MHz (low frequency mode) Maximum range: 40 m (set to 20 m for this dataset) Horizontal aperture: 130° Vertical aperture: 20° Number of beams: 512 Angular resolution: 0.6° Beam separation: 0.25° Image resolution: 902x497 pixels Coordinate system: Polar Camera: Allied-Vision Manta G-917 Image dimensions: 3384x2710 pixels (downscaled to 1692x1355 for this dataset) Sensor type: CCD Progressive Sensor bit depth: 12-bit Captured bit depth: 8-bit Camera model: Pinhole with Plumb Bob (Brown–Conrady) distortion coefficients Focal length (fx, fy): (1638.36157, 1641.95202) Principal point (cx, cy): (1705.03529, 1380.27954) Radial distortion coefficients (k1, k2, k3): (-0.124823, 0.048851, 0.000000) Tangential distortion coefficients (p1, p2): (0.000259, -0.002945) Navigation: Data format: CSV Contains fused dead reckoning data based on GPS, DVL, and IMU sensors Columns: timestamp: Unix timestamp (seconds) latitude: Latitude (degrees) longitude: Longitude (degrees) altitude: Altitude (meters) yaw: Yaw angle (degrees) pitch: Pitch angle (degrees) roll: Roll angle (degrees) velocity_x: Velocity along the x-axis (meters per second) velocity_y: Velocity along the y-axis (meters per second) velocity_z: Velocity along the z-axis (meters per second) depth: Depth (meters) Frame rate: 2 Hz for both sonar and camera More datasets from this collection will be uploaded in the future, and a link to access them will be provided on this page. Acknowledgements The data in this repository is part of the DeeperSense project that received funding from the European Commission, Program H2020-ICT-2020-2 ICT-47-2020, Project Number: 101016958.