Crowds & Machines Next level: Meditteranean wheat classification labels from gamified crowd-sourcing

Machine learning (and especially deep learning) algorithms need lots of training and validation datasets, which are often unavailable. Creating on-ground datasets is costly and time consuming. Within the European Space Agency funded project ‘Crowds & Machine – Next Level’ (by Blackshore B.V., 52impact B.V. and The Hague Centre for Strategic Studies), we aimed to solve this issue by generating labelled data effectively using an innovative gamified crowdsourced-based method. The objective of the project ‘Crowds & Machines Next Level’ was to generate labelled data for the training and validation of machine learning algorithms to classify the crop wheat. We make those labelled datasets freely available as open data to organisations that use machine learning for their activities, mainly companies and knowledge institutes. As part of the project we developed example scripts (Jupyter notebooks) that enable organisations to use the crowdsourced generated data smoothly for their own machine learning systems. BlackShore has developed the online platform Cerberus to enable large scale generation of labelled datasets, which is deployed on twenty locations around the Mediterranean Sea to generate labelled datasets of wheat and other land cover classes (see table). Those different locations encompass a diversity of climate regions, harvest cultures and crop calendars, posing a challenge to the training of machine learning algorithms. Gamers click on hexagons plotted on top of very high resolution satellite imagery (captured during the harvest period in 2021), and by combining 3 different hexagon grids those clicks are converted into triangles. Each triangle has a number of clicks (by different users) per land cover category, which provides a measure of accuracy to the label. 52impact developed example tutorials to use the data to train pixel-based (Random Forest) and segmentation-based (U-Net) machine learning models, using Sentinel-2 imagery (provided in the data folder), which can be forked here: bitbucket.org.
Overview of locations ID location_id Country Region Shape Harvest period VHR image date S-2 pre-harvest S-2 harvest S-2 post-harvest 01 portugalAlentejo Portugal Alentejo 01_Portugal_Alentejo_SELECTION 10 Jul - 1 Aug 07/07/2021 14/05/2021 13/07/2021 22/08/2022 02 spainAndalusia Spain Andalusia 02_Spain_Andalusia_SELECTION 10 Jul - 1 Aug 02/07/2021 16/05/2021 15/07/2021 03/09/2021 03 spainAragon Spain Aragon 03_Spain_Aragon_SELECTION 10 Jul - 1 Aug 26/10/2021 20/05/2021 19/07/2021 05/09/2021 04 franceAude France Aude 04_France_Aude_SELECTION 1 Jul - 1 Oct 22/09/2021 12/05/2021 10/08/2021 18/11/2021 05 franceCamargue France Camargue 05_France_Camargue_SELECTION 1 Jul - 1 Oct 07/10/2021 12/05/2021 10/08/2021 18/11/2021 06 franceProvence France Provence 06_France_Provence_SELECTION 1 Jul - 1 Oct 26/10/2021 19/05/2021 17/08/2021 20/11/2021 07_08 italyMarche Italy Marche (East and West) 07_08_Italy_Marche_SELECTION 1 Jul - 1 Sept 09/08/2021 26/05/2021 25/07/2021 20/11/2021 09 italySardinia Italy Sardinia 09_Italy_Sardinia_SELECTION 1 Jul - 1 Sept 31/08/2021 26/05/2021 22/07/2021 10/10/2021 10 italySicily Italy Sicily 10_Italy_Sicily_SELECTION 1 Jul - 1 Sept 19/09/2021 22/05/2021 26/07/2021 10/10/2021 11 italyPugliaNorth Italy Puglia (North) 11_Italy_PugliaNorth_SELECTION 1 Jul - 1 Sept 06/10/2021 11/06/2021 31/07/2021 04/10/2021 12 italyPuglia Italy Puglia 12_Italy_Puglia_SELECTION 1 Jul - 1 Sept 19/08/2021 03/06/2021 02/08/2021 21/10/2021 13 greeceWest Greece West 13_Greece_West_SELECTION 1 Sept - 1 Nov 02/09/2021 27/07/2021 05/10/2021 14/12/2021 14 greeceThessaly Greece Thessaly 14_Greece_Thessaly_SELECTION 1 Sept - 1 Nov 14/07/2021 27/07/2021 25/09/2021 19/12/2021 15 greeceMacedoniaCentral Greece Macedonia (Central) 15_Greece_MacedoniaCentral_SELECTION 1 Jun - 1 Aug 22/07/2021 13/05/2021 22/07/2021 15/09/2021 16 greeceMacedoniaEast Greece Macedonia (East) 16_Greece_MacedoniaEast_SELECTION 1 Jun - 1 Aug 05/08/2021 25/05/2021 29/07/2021 27/10/2021 17 greeceRhodes Greece Rhodes 17_Greece_Rhodes_SELECTION 15 May - 1 Jul 09/05/2021 25/03/2021 24/05/2021 22/08/2021 18 cyprusLarnaca Cyprus Larnaca 18_Cyprus_Larnaca_SELECTION 15 May - 1 Jul 05/06/2021 19/03/2021 07/06/2021 21/08/2021 19 turkeyCyprus Cyprus (T) Farmagusta 19_Turkey_Cyprus_SELECTION 15 May - 1 Jul 05/06/2021 29/03/2021 17/06/2021 26/08/2021 20 egyptBehera Egypt Behera 20_Egypt_Behera_SELECTION 1 Apr - 1 Jul 06/03/2021 26/01/2021 07/03/2021 19/08/2021 The following data is provided: Triangulated_data.zip: contains per region and per category a geopackage (gpkg) file containing triangular polygons with the number of clicks per polygon. The filename of the polygon files depends on the location and category. For example, a file that contains the triangles corresponding to Cattle in Alentejo, Portugal, is called: 01_Portugal_Alentejo_Cattle.gpkg Data.zip: all data necessary to run the Jupyter notebooks, i.e., location data, cropped Sentinel-2 satellite imagery (for training location IDs 01, 02, 12 and 15, and validation locations near IDs 02 and 15) and also the triangulated polygons. Models.zip: pre-trained random forest and U-Net models based on the data, which can be generated by the Jupyter notebooks.