ODE CLAY

We have chosen seven tasks intended to highlight the operational value of foundation models and the type of signal and function needed, as proposed here.

Type of signal anchor: Spatial, Spectral, and Temporal
Type of functions: Object Detection, Change Detection, Classification, Regression, Generative

• Landcover classification [Spectral, Classification]: Classify an AoI on yearly land cover into the following 9 classes: No Data, Water, Trees, Flooded vegetation, Crops, Built area, Bare ground, Snow/ice, Clouds, Rangeland. The scoring and time reference will be Impact Observatory 10m landcover.

• Aquaculture detection [Spatial, Object Detection]: Identify the presence and location of aquaculture facilities on a given year. Sourcing and time reference is the open databases of aquaculture from Canada, California, and Norway. This will be assessed with a simulation of human-in-the-loop learning, where 1) first 10 similarity search results are returned 2) positive and negative results are labeled, and 3) search is performed once again, using these results for the evaluation metric.

• Carbon above ground stock [Spectral, Regression]: Estimate the above-ground carbon stock, with HGB at 300m resolution estimates as scoring and time reference.

• Crop yield estimation [Temporal, Regression]: Estimate crop yields at 1km2 resolution of maize and wheat using this open dataset of China as scoring and time reference.

• Disaster response floods [Spectral, Change detection]: Map the extent of a flood, with NRT as scoring and time reference. [Note this dataset license is NC-SA]

• Disaster fire severity [Temporal, Change detection ]: Map the severity of a fire, with NTBS as scoring and time reference.

• Cloud Gap Imputation [Temporal, Generative]: Fill in cloudy pixels in a time series (3 scenes) of Sentinel-2 imagery. The time difference between the scenes varies from a couple of days, to a couple of months. The scoring and time reference will be this HLS multi-temporal cloud gap imputation dataset.

Participants are required to develop one Apache-licensed Jupyter Notebook in Python that can perform the following tasks using:

• Only one foundation model across all tasks. [See Foundation Model Training Requirements below]

• For inference: Only fully open data (including commercial use) that is also openly available (available on public access on AWS, PC or Google, e.g., Sentinel, Landsat).

• You can use finetune, LoRA, embeddings or any other technique as you see fit to augment the foundation model’s performance.

• Code should scale to produce results over a given region smaller than 2000 km².

• Your whole notebook must run in less than 2h on a `g5.xlarge` AWS EC2 instance.

To submit your entry, participants can choose from the following options:

• Prize eligible: The model must be fully open-source and trained only on fully open data. You must release the code as Apache license or equivalent, and the training can only use fully open data (including commercial use) for training. You can use the scoring referenced where their license allows. You must specify sources. You are eligible for the award.

• Ranking eligible: If your model uses any commercial or proprietary datasets, we will still rank your score, but you won’t be eligible for the compute prize. You don’t need to specify sources, but we encourage it, especially if you use Clay as a base model.

The notebook should take as inputs a geojson and a year.

We will only adjust the region and time. If the code fails to run or takes more than 10 hours to complete, the team will be given 24 hours to submit a version with the corrected code.

The notebook will be scored on a location and year disclosed 2 days after the submission deadline.

The evaluation metric will be the F1 score for classification and object detection tasks, RMSE for regression tasks, and MAE for the generative task. Overall ranking will be determined by the average of the entrant's rank on each task.

On the test set: We do not hold up hidden test sets for global historical datasets. We consider “overfitting” to the whole Earth across all tasks into a single model a valid strategy.

Note: We are considering expanding the criteria to a “few shot challenge” where we allow the model to iterate with (automated) feedback on the results, to mimic how a model would be used operationally.

Participants must submit their notebooks as a pull request to the designated open repository. The submitted code will be run using a continuous integration (CI) system that anyone can access at any time.