Learning to Interpret Satellite Images in Global Scale Using Wikipedia

This dataset is created by pairing geo-referenced Wikipedia articles with satellite imagery of their corresponding locations. The resulting dataset contains 888,696 article-image pairs, making it the largest satellite image dataset to date. The pairing of Wikipedia articles with satellite images is based on the idea that the article's text provides a detailed and comprehensive representation of the satellite image, often containing structured data in the form of tables and raw text. This information-rich label can be used to extract knowledge about the physical state and features of the entity, such as elevation, age, climate, and population. To acquire matching satellite imagery, high-resolution images from Digital-Globe satellites are used, with a ground sampling distance (GSD) of 0.3-0.5m. These images are among the highest resolution images available commercially and were also used in the recently released functional map of the world (fMoW) dataset. The size of the acquired images is 1000×1000 pixels, covering approximately an area of 900m2, and RGB images are prioritized over grayscale images. The resulting WikiSatNet multi-modal dataset is a set of tuples, where each tuple represents a location, corresponding DigitalGlobe image, and Wikipedia article text. This dataset is highly scalable and fully automated, allowing for the generation of even larger datasets by considering other Wikipedia languages and sensors. Two novel methods are proposed to pre-train a convolutional neural network (CNN) to extract information about images using information from Wikipedia articles. The first approach involves weakly labeling satellite images with curated summarization tags extracted from the article via an automated process. The second approach proposes a novel joint architecture where a textual embedding of each article is obtained using document summarization techniques from NLP, and then a deep convolutional network is trained to produce an embedding for each image that is "similar" to the textual one. The pre-trained networks are then evaluated on a downstream hand-labeled dataset, where a 4.5% higher accuracy is obtained compared to networks pre-trained on ImageNet, the standard approach for computer vision tasks. This demonstrates the effectiveness of pairing Wikipedia articles to satellite images for pre-training CNNs for satellite image recognition.