AgriBench: A Hierarchical Agriculture Benchmark for Multimodal Large Language Models

Contents

• News
• Overview
• The 5 Levels Evaluation Strategy
• MM-LUCAS Dataset
• Citation

To Do

• • Public MM-LUCAS dataset and update initial repo.
• • Add quantitative evaluation results.
• • Add Leaderboard.

News

• [08/2024] AgriBench is accepted by 9th Computer Vision in Plant Phenotyping and Agriculture (CVPPA 2024) @ECCV 2024.

Overview

AgriBench: the first agriculture benchmark designed to evaluate MultiModal Large Language Models (MM-LLMs) for agriculture applications.

MM-LUCAS: includes 1,784 landscape images, segmentation masks, depth maps, and detailed annotations (geographical location, country, date, land cover and land use taxonomic details, quality scores, aesthetic scores), based on the Land Use/Cover Area Frame Survey (LUCAS).

The 5 Levels Evaluation Strategy

MM-LUCAS Dataset

Content	Size	Files	Format	Details
🍀MM-LUCAS	-	7,141		Main Folder
├  1_images	877 MB	1,784	JPG	Scenery images (1600×1200 pixels) <Ref>
├  2_seg	18.0 MB	1,784	PNG	Segmentation masks <Ref>
├  2_seg_color	24.0 MB	1,784	PNG	Color-coded segmentation masks
├  3_depth	619 MB	1,784	PNG	Depth images <Ref-Depth Anything V2-Large>
├  4_mm_lucas	348 KB	1	CSV	Microdata (File name, Quality score <Ref-Q-Align>, Aesthetic score <Ref-Q-Align>, Geographical location, Country, Date, Land cover, Land use, Classes.)
├  5_aesthetics_score	264 KB	1	JSON	Single question-answering: {"images:", "questions:", "answer:"}
├  5_land_cover	559 KB	1	JSON	Multi-choice question-answering: {"images:", "questions:", "options:", "answer:"}
├  5_land_use	615 KB	1	JSON	Multi-choice question-answering: {"images:", "questions:", "options:", "answer:"}
├  5_quality_score	267 KB	1	JSON	Single question-answering: {"images:", "questions:", "answer:"}

Citation

If you find this paper and dataset helpful for your research, please consider citing as below:

@article{zhou2024agribench,
  title={AgriBench: A Hierarchical Agriculture Benchmark for Multimodal Large Language Models},
  author={Zhou, Yutong and Ryo, Masahiro},
  journal={arXiv preprint arXiv:2412.00465},
  year={2024}
}

@article{martinez2024semantic,
  title={Semantic segmentation dataset of Land Use/Cover Area frame Survey (LUCAS) rural landscape Street View Images},
  author={Martinez-Sanchez, Laura and Hufkens, Koen and Kearsley, Elizabeth and Naydenov, Dimitar and Cz{\'u}cz, B{\'a}lint and van de Velde, Marijn},
  journal={Data in Brief},
  volume={54},
  pages={110394},
  year={2024},
  publisher={Elsevier}
}

@article{d2020harmonised,
  title={Harmonised LUCAS in-situ land cover and use database for field surveys from 2006 to 2018 in the European Union},
  author={d’Andrimont, Rapha{\"e}l and Yordanov, Momchil and Martinez-Sanchez, Laura and Eiselt, Beatrice and Palmieri, Alessandra and Dominici, Paolo and Gallego, Javier and Reuter, Hannes Isaak and Joebges, Christian and Lemoine, Guido and others},
  journal={Scientific data},
  volume={7},
  number={1},
  pages={352},
  year={2020},
  publisher={Nature Publishing Group UK London}
}