MORE

This repository maintains the code and data for "MORE: A Multimodal Object-Entity Relation Extraction Dataset with a Benchmark Evaluation". (He, et al. ACM MM 2023).

Data

Data can be downloaded from here.

Before releasing the dataset publicly, we spent a lot of time rechecking all the data, removing most of the sensitive data, correcting obvious labeling errors, and adding some new data. As a result, we updated all the experimental results in the paper on the new dataset, and the updated version of the paper can be found here.

In the revised version, MORE consists of 21 distinct relation types and contains 20,264 multimodal relational facts annotated on 3,559 pairs of textual titles and corresponding images. The dataset includes 13,520 visual objects (3.8 objects per image on average). We split the dataset into training, development, and testing sets consisting of 15,486, 1,742 and 3,036 facts respectively.

The overall result:

Model	Accuracy	precision	Recall	F1-Ccore
BERT+SG	61.79	29.61	41.27	34.48
BERT+SG+Att	63.74	31.10	39.28	34.71
MEGA	65.97	33.30	38.53	35.72
IFAformer	79.28	55.13	54.24	54.68
MKGformer	80.17	55.76	53.74	54.73
VisualBERT	82.84	58.18	61.22	59.66
ViLBERT	83.50	62.53	59.73	61.10
MOREformer	83.50	62.18	63.34	62.75

The performance on multiple entities/objects:

	Ent = 1, Obj = 1	Ent = 1, Obj > 1	Ent > 1, Obj = 1	Ent > 1, Obj > 1
none Ratio	5%	74.10%	33.11%	80.90%
Accuracy	80.00	84.57	67.55	84.43
Precision	82.98	63.18	66.36	53.63
Recall	82.11	66.31	72.28	52.47
F1-Score	82.54	64.71	69.19	53.04

Macro-F1 on MORE dataset:

Model Name	Precision	Recall	Macro F1-Score
MKGformer	48.50	42.60	43.57
ViLBERT	49.13	49.37	48.66
MOREformer	51.02	50.66	50.02

Multi-object disambiguation analysis:

Model Name	Precision	Recall	F1-Score
MKGformer	70.12/66.08	67.58/62.29	68.83/64.13
ViLBERT	75.31/72.30	75.31/70.41	75.31/71.34
MOREformer	72.28/71.43	79.75/76.61	75.84/73.93

Checkpoint can be downloaded from here

Requirements

• transformers==4.11.3
• tensorboardX==2.4
• pytorch-crf==0.7.2
• torchvision==0.8.0
• scikit-learn==1.0
• numpy>=1.21
• tokenizers==0.10.3
• torch==1.7.0
• tqdm==4.49.0
• opencv-python==4.7.0.68
• ftfy==6.1.1

Usage

conda create -n moreformer python==3.7
conda activate moreformer
pip install -r requirements.txt
sh run.sh

Citation

If you used the datasets or code, please cite our paper.

@inproceedings{he2023more,
  title={MORE: A Multimodal Object-Entity Relation Extraction Dataset with a Benchmark Evaluation},
  author={He, Liang and Wang, Hongke and Cao, Yongchang and Wu, Zhen and Zhang, Jianbing and Dai, Xinyu},
  booktitle={Proceedings of the 31st ACM International Conference on Multimedia},
  pages={4564--4573},
  year={2023}
}