CSP: Self-Supervised Contrastive Spatial Pre-Training for Geospatial-Visual Representations

Gengchen Mai1,2,3 *, Ni Lao4*, Yutong He2,5, Jiaming Song2, Stefano Ermon2,
1Spatially Explicit Artificial Intelligence Lab, Department of Geography, University of Georgia, USA
2Department of Computer Science, Stanford University, USA
3School of Computing, University of Georgia, USA
4Google Inc, USA
5Machine Learning Department, Carnegie Mellon University, USA
*Equal contribution
Paper Openreview ArXiv ResearchGate Code Model Data

Conference Information

To be appeared at the Fortieth International Conference on Machine Learning (ICML 2023), Jul 23 - 29, 2023, Honolulu, Hawaii, USA.

Abstract

Geo-tagged images are publicly available in large quantities, whereas labels such as object classes are rather scarce and expensive to collect. Meanwhile, contrastive learning has achieved tremendous success in various natural image and language tasks with limited labeled data. However, existing methods fail to fully leverage geospatial information, which can be paramount to distinguishing objects that are visually similar.

To directly leverage the abundant geospatial information associated with images in pre-training, fine-tuning, and inference stages, we present Contrastive Spatial Pre-Training (CSP), a self-supervised learning framework for geo-tagged images. We use a dual-encoder to separately encode the images and their corresponding geo-locations, and use contrastive objectives to learn effective location representations from images, which can be transferred to downstream supervised tasks such as image classification.

Experiments show that CSP can improve model performance on both iNat2018 and fMoW dataset. Especially, on iNat2018, CSP significantly boosts the model performance with 10-34% relative improvement with various labeled training data sampling ratios.

Our Model Overview

model

Three Constrastive Spatial Pre-Training Objectives

model

BibTeX

@inproceedings{mai2023csp,
  title={CSP: Self-Supervised Contrastive Spatial Pre-Training for Geospatial-Visual Representations},
  author={Mai, Gengchen and Lao, Ni and He, Yutong and Song, Jiaming and Ermon, Stefano},
  booktitle={International Conference on Machine Learning},
  year={2023},
  organization={PMLR}
}