TubeDETR: Spatio-Temporal Video Grounding with Transformers
- Resource Type
- Conference
- Authors
- Yang, Antoine; Miech, Antoine; Sivic, Josef; Laptev, Ivan; Schmid, Cordelia
- Source
- 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) CVPR Computer Vision and Pattern Recognition (CVPR), 2022 IEEE/CVF Conference on. :16421-16432 Jun, 2022
- Subject
- Computing and Processing
Location awareness
Grounding
Natural languages
Computer architecture
Object detection
Benchmark testing
Transformers
Vision + language
- Language
- ISSN
- 2575-7075
We consider the problem of localizing a spatio-temporal tube in a video corresponding to a given text query. This is a challenging task that requires the joint and efficient modeling of temporal, spatial and multi-modal interactions. To address this task, we propose TubeDETR, a transformer-based architecture inspired by the recent success of such models for text-conditioned object detection. Our model notably includes: (i) an efficient video and text encoder that models spatial multi-modal interactions over sparsely sampled frames and (ii) a space-time decoder that jointly performs spatio-temporal localization. We demonstrate the advantage of our proposed components through an extensive ablation study. We also evaluate our full approach on the spatio-temporal video grounding task and demonstrate improvements over the state of the art on the challenging VidSTG and HC-STVG benchmarks.