The Animation Transformer: Visual Correspondence via Segment Matching
- Resource Type
- Conference
- Authors
- Casey, Evan; Perez, Victor; Li, Zhuoru
- Source
- 2021 IEEE/CVF International Conference on Computer Vision (ICCV) ICCV Computer Vision (ICCV), 2021 IEEE/CVF International Conference on. :11303-11312 Oct, 2021
- Subject
- Computing and Processing
Visualization
Image segmentation
Computer vision
Image color analysis
Production
Animation
Transformers
Vision applications and systems
Segmentation
grouping and shape
- Language
- ISSN
- 2380-7504
Visual correspondence is a fundamental building block on the way to building assistive tools for hand-drawn animation. However, while a large body of work has focused on learning visual correspondences at the pixel-level, few approaches have emerged to learn correspondence at the level of line enclosures (segments) that naturally occur in hand-drawn animation. Exploiting this structure in animation has numerous benefits: it avoids the memory complexity of pixel attention over high resolution images and enables the use of real-world animation datasets that contain correspondence information at the level of per-segment colors. To that end, we propose the Animation Transformer (AnT) which uses a Transformer-based architecture to learn the spatial and visual relationships between segments across a sequence of images. By leveraging a forward match loss and a cycle consistency loss our approach attains excellent results compared to state-of-the-art pixel approaches on challenging datasets from real animation productions that lack ground-truth correspondence labels.