SUSTAINBENCH: Benchmarks for Monitoring the Sustainable Development Goals with Machine Learning

The SDGs span social, economic, and environmental spheres, ranging from ending poverty to achieving gender equality to combating climate change. Progress toward SDGs is traditionally monitored through statistics collected by civil registrations, population-based surveys, and censuses. However, such data collection is expensive and requires adequate statistical capacity, and many countries go decades between making ground measurements on key SDG indicators. A lack of data on key environmental and socioeconomic indicators has hindered progress toward the SDGs. Recent advances in machine learning have made it possible to utilize abundant, frequently updated, and globally available data, such as from satellites or social media, to provide insights into progress toward SDGs. Despite promising early results, approaches to using such data for SDG measurement have largely evaluated on different datasets or used inconsistent evaluation metrics, making it hard to understand whether performance is improving and where additional research would be most fruitful. To address these challenges, the SUSTAINBENCH dataset was created to provide a standardized collection of benchmark tasks across 7 SDGs. The dataset includes 15 tasks, with data for 11 of the tasks being released publicly for the first time. The tasks cover a range of SDG-related outcomes, including poverty prediction, land cover classification, and crop yield prediction. The dataset is designed to facilitate methodological progress in machine learning for SDG monitoring, with the goal of improving sustainability measurements and offering tasks for machine learning challenges. The SUSTAINBENCH dataset includes a range of data types, including satellite imagery, social media posts, and mobile phone activity. The dataset is curated to provide high-quality domain-specific datasets in development economics and environmental science, and to provide benchmarks to standardize evaluation on tasks related to SDG monitoring. The dataset is also designed to encourage the machine learning community to evaluate and develop novel methods on problems of global significance where improved model performance facilitates progress toward SDGs. The SUSTAINBENCH dataset includes tasks related to No Poverty (SDG 1), Zero Hunger (SDG 2), Good Health and Well-being (SDG 3), Quality Education (SDG 4), Clean Water and Sanitation (SDG 6), Climate Action (SDG 13), and Life on Land (SDG 15). The dataset provides baseline models for each task and a public leaderboard, allowing researchers to compare their results and track progress. The SUSTAINBENCH dataset is a valuable resource for improving sustainability measurements and offers tasks for machine learning challenges, allowing for the development of self-supervised learning, meta-learning, and multi-modal/multi-task learning methods on real-world datasets. The SUSTAINBENCH dataset is compared to existing datasets that pertain to SDGs, are publicly available, provide ML-friendly inputs/outputs, and specify standardized evaluation metrics. The dataset is found to be a comprehensive benchmark for distribution shifts in real-world applications, and is designed to facilitate methodological progress in machine learning for SDG monitoring. The dataset is a valuable resource for researchers and practitioners working on SDG-related tasks, and offers a range of opportunities for advancing machine learning methods for sustainability measurements.