Tracking detailed tree level dynamics of tropical forest landscapes using deep convolutional neural networks and multitemporal, multimodal remote sensing data sources

International audience Tropical forests are an essential part of the global carbon cycle and home to two-thirds of terrestrial species. Improved monitoring of the growth, mortality and phenology of individual trees is essential to evaluating the resilience of tropical forests to climate change and other anthropogenic pressures. Identifying individual trees in tropical forest canopies from remote sensing data is challenging due to the diversity of species, the complex structure and spectral impurities from lianas and epiphytes. Modern convolutional neural network approaches can novel generate insights from aerial remote sensing imagery. Here we describe a new machine learning method which uses the Mask R-CNN algorithm to delineate tree crowns in tropical forest landscapes from RGB imagery. We deployed it to delineate 20,000 tree crowns in aerial images collected in Amazonian French Guiana and compared its accuracy to the AMS3D segmentation approach on LiDAR data. To assess its ability to identify tree species we compared its species identification of the delineated crowns to a machine learning approach (linear discriminant analysis with support vector machine) applied to hyperspectral imagery. To demonstrate an application of the methods, we tracked the foliar phenology of the species and individuals over an 18 month period. The skill of our automatic method in delineating the canopy tree crowns was high (F1 score = 0.74) compared to the LiDAR clustering approach (F1 score = 0.60). Its skill was further be improved when several time steps (12 dates) of imagery were used to locate the crowns (F1 = 0.82). The hyperspectral machine learning was able to label the crowns of the 80 most common species with an average F1 score of (F1 score = 0.83). Our approach demonstrates that modern computer vision methods can automatically process abundant RGB imagery to characterise the dynamics of tropical forests. The spectral richness of hyperspectral data allows for identifying more species with greater accuracy than the RGB approach but further development of methods may close the gap. The methods could be scaled and applied to satellite based Earth observation data for global analyses of forest dynamics and resilience.