alexgleith/sinergise-element84-sentinel-2-qa

A scratch space to explore the inventories of S-2 data between Sinergise and E84 for Africa. # Workspace for comparing differences between Sinergise and Element 84 Sentinel-2 buckets for DE Africa ## Enviroment notes * Sinergise bucket: `sentinel-s2-l2a` * Sinergise inventory bucket: `sentinel-inventory` * Element 84 bucket: `sentinel-cogs` * Element 84 inventory bucket: `sentinel-cogs-inventory` ## Process steps 1. Run the 0-parse-inventory-sinergise.py and 0-parse-inventory-element84.py scripts, which will dump out inventories from the Sinergise and Element 84 buckets. 2. Use the notebook 1-sinergise-element84-differences.ipynb to load the inventories and run a quick comparison of the two lists. This will result in data/sinergise-only.txt and data/element84-only.txt files. (The E84 one should only have the UTM 0 and 60 zoned scenes that _are_ in the Sinergise bucket, but which I'm not listing.) 3. The notebook 2-do-qa-inventories.ipynb will check for known flaws and check metadata and such from the Sinergise API. The result is data/sinergise-qa-output.json and data/sinergise-qa-todo.json. ## Brief QA Findings As of July 1, 2020, in our African region of interest there are `1,589,247` scenes in the Sinergise listing and `1,452,822` in the Element84 listing. There are `125` scenes in the Element 84 list that are not in the Sinergise list. This due to some scenes that cover the 0 and 60 UTM zones slipping in to the list due to issues relating to the antimeridian and reprojection to WGS84. There are `136,838` scenes in the Sinergise list that are not in the Element 84 list. Reason for these missing scenes are explored below. Of these, `56,240` have known reasons for not being in the Element 84 listing as follows: * `5,663` have no `tileDataGeometry` field. * `31,242` have `0.0` values for cloud in both level-2 and level-1 metadata * `20,777` have `0.0` values for cloud in level-2 and have no level-1 metadata. This leaves ~`80,000` scenes that don't have a clear reason for not being included. ### Known issues Scenes with known issues are listed in …