Query-centric Audio-Visual Cognition Network for Moment Retrieval, Segmentation and Step-Captioning [Code]

Yunbin Tu, Liang Li, Li Su, Qingming Huang

• QUAG addresses moment retrieval, moment segmentation, and step captioning in a unified framework.
• QUAG learns a query-centric audio-visual representation based on the shallow-to-deep principle.

Distractors-Immune Representation Learning with Cross-modal Contrastive Regularization for Change Captioning [Code] [Supp.]

Yunbin Tu, Liang Li, Li Su, Chenggang Yan, Qingming Huang

• DIRL attains a pair of stable image representations by correlating the corresponding their channels and decorrelating different ones.
• CCR regularizes the cross-modal alignment by maximizing the contrastive alignment between the attended difference features and generated words.

Context-aware Difference Distilling for Multi-change Captioning [Code]

Yunbin Tu, Liang Li, Li Su, Zheng-Jun Zha, Chenggang Yan, Qingming Huang

• CARD addresses multi-change captioning with an unknown number of changes.
• CARD decouples common/difference context features to guide the learning of features of all genuine changes for generating corresponding sentences.

Yunbin Tu, Liang Li, Li Su, Zheng-Jun Zha, Qingming Huang

• SMART uses a multi-aspect relation learning network to learn effective change features for caption generation.
• SMART uses a POS-based visual switch to dynamically use visual information during different word generation.

Self-supervised Cross-view Representation Reconstruction for Change Captioning [Code] [Supp.]

Yunbin Tu, Liang Li, Li Su, Zheng-Jun Zha, Chenggang Yan, Qingming Huang

• SCORER learns two view-invariant image representations by by maximizing cross-view contrastive alignment of two similar images.
• CBR reversely models a “hallucination” representation with the caption and “before” image, and matchs it with the real “after” image.

Viewpoint-Adaptive Representation Disentanglement Network for Change Captioning [Code]

Yunbin Tu, Liang Li, Li Su, Junping Du, Ke Lu, Qingming Huang

• VARD helps the model adapt to viewpoint changes via mining the intrinsic properties of two images and modeling their position information.
• VARD uses an unchanged representation disentanglement to distinguish the unchanged features from changed features.

Neighborhood Contrastive Transformer for Change Captioning [Code]

Yunbin Tu, Liang Li, Li Su, Ke Lu, Qingming Huang

• NCT learns the contrast features between two images via a neighboring feature aggregating and a common feature distilling.
• Explicit dependencies between words is used to help the decoder to better understand complex syntax structure during training.

R³Net: Relation-embedded Representation Reconstruction Network for Change Captioning [Code] [Video]

Yunbin Tu, Liang Li, Chenggang Yan, Shengxiang Gao, Zhengtao Yu

• R³Net learns relations within each image and between two images to reconstruct unchanged and changed features.
• R³Net uses syntactic skeleton predictor to enhance semantic interaction between change localization and caption generation.

Semantic Relation-aware Difference Representation Learning for Change Captioning [Code] [Video]

Yunbin Tu, Tingting Yao, Liang Li, Jiedong Lou, Shengxiang Gao, Zhengtao Yu, Chenggang Yan

• SRDRL models self-semantic relation to explore the underlying changes, and measures cross-semantic relation to localize the real change.
• SRDRL relys on the POS of words, and uses an attention-based visual switch to dynamically use visual information for caption generation.

Relation-aware Attention for Video Captioning via Graph Learning

Yunbin Tu, Chang Zhou, Junjun Guo, Huafeng Li, Shengxiang Gao, Zhengtao Yu

• HTG aims to enhance the inter-relation of attention by reversely modeling the relation of each word with respect to every attended visual feature.
• HMG aims to enhance the intra-relation of attention by capturing the relations among all of the attended visual features.

I²Transformer: Intra- and Inter-relation Embedding Transformer for TV Show Captioning [Code]

Yunbin Tu, Li Su, Shengxiang Gao, Chenggang Yan, Zheng-Jun Zha, Zhengtao Yu, Qingming Huang

• TV show captioning aims to generate a linguistic sentence based on the video and its associated subtitle.
• I²Transformer uses an intra-relation embedding to capture intra-relation in each modality and an inter-relation embedding to produce the omni-representation of all modalities.

Enhancing the alignment between target words and corresponding frames for video captioning [Code]

Yunbin Tu, Chang Zhou, Junjun Guo, Shengxiang Gao, Zhengtao Yu

• TTA uses pre-detected visual tags from the video to bridge the gap between vision and language.
• TTA exactly align the target words with corresponding frames guided by the visual tags.

Chenggang Yan, Yunbin Tu, Xingzheng Wang, Yongbing Zhang, Xinhong Hao, Yongdong Zhang, Qionghai Dai

• STAT uses both object-level visual and label features to address the problem of detail missing.
• STAT successfully takes into account both the spatial and temporal structures in a video.

Video Description with Spatial-Temporal Attention [Code]

Yunbin Tu, Xishan Zhang, Bingtao Liu, Chenggang Yan

• STAT uses object-level local features to address the problem of detail missing.
• STAT selects relevant local features by spatial attention and then attend to important frames by temporal attention to recognize related semantics.