Exploiting Long-Term Dependencies for Generating Dynamic Scene Graphs
- Resource Type
- Conference
- Authors
- Feng, Shengyu; Mostafa, Hesham; Nassar, Marcel; Majumdar, Somdeb; Tripathi, Subarna
- Source
- 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) WACV Applications of Computer Vision (WACV), 2023 IEEE/CVF Winter Conference on. :5119-5128 Jan, 2023
- Subject
- Computing and Processing
Engineering Profession
Computer vision
Fluctuations
Source coding
Genomics
Benchmark testing
Transformers
Bioinformatics
Algorithms: Video recognition and understanding (tracking
action recognition
etc.)
Image recognition and understanding (object detection
categorization
segmentation
scene modeling
visual reasoning)
- Language
- ISSN
- 2642-9381
Dynamic scene graph generation from a video is challenging due to the temporal dynamics of the scene and the inherent temporal fluctuations of predictions. We hypothesize that capturing long-term temporal dependencies is the key to effective generation of dynamic scene graphs. We propose to learn the long-term dependencies in a video by capturing the object-level consistency and inter-object relationship dynamics over object-level long-term tracklets using transformers. Experimental results demonstrate that our Dynamic Scene Graph Detection Transformer (DSG- DETR) outperforms state-of-the-art methods by a significant margin on the benchmark dataset Action Genome. Our ablation studies validate the effectiveness of each component of the proposed approach. The source code is available at https://github.com/Shengyu-Feng/DS G-DETR.