Cross-Modal Relation-Aware Networks for Audio-Visual Event Localization

This repo holds the code for the work presented on ACM Multimedia 2020 [Paper]

Usage Guide

Prerequisites

We provide the implementation in PyTorch for the ease of use.

Install the requirements by runing the following command:

pip install -r requirements.txt

Code and Data Preparation

We highly appreciate @YapengTian for the shared features and code.

Download Features

Two kinds of features (i.e., Visual features and Audio features) are required for experiments.

• Visual Features: You can download the VGG visual features from here.

• Audio Features: You can download the VGG-like audio features from here.

• Additional Features: You can download the features of background videos here, which are required for the experiments of the weakly-supervised setting.

After downloading the features, please place them into the data folder. The structure of the data folder is shown as follows:

data
├── audio_feature.h5
├── audio_feature_noisy.h5
├── labels.h5
├── labels_noisy.h5
├── mil_labels.h5
├── test_order.h5
├── train_order.h5
├── val_order.h5
├── visual_feature.h5
└── visual_feature_noisy.h5

Download Datasets (Optional)

You can download the AVE dataset from the repo here.

Training and testing CMRAN in a fully-supervised setting

You can run the following command for training and testing the model. We evaluate the model on the test set every epoch (set by the arg "eval_freq" in the configs/default_config.yaml file) when training.

bash supv_train.sh
# The argument "--snapshot_pref" denotes the path for saving checkpoints and code.

Evaluating

bash supv_test.sh

After training, there will be a checkpoint file whose name contains the accuracy on the test set and the number of epoch.

Training and testing CMRAN in a Weakly-supervised setting

Similar to training the model in a fully-supervised setting, you can run training and testing using the following commands:

Training

bash weak_train.sh

Evaluating

bash weak_test.sh

Citation

Please cite the following paper if you feel this repo useful to your research

@inproceedings{CMRAN2020Xu,
  author    = {Haoming Xu and
               Runhao Zeng and
               Qingyao Wu and
               Mingkui Tan and
               Chuang Gan},
  title     = {Cross-Modal Relation-Aware Networks for Audio-Visual Event Localization},
  booktitle   = {{ACM} International Conference on Multimedia},
  year      = {2020},
}