HeyHomees Lab Notes · No. 3 · VCLab · Published October 16, 2026
By HeyHomees Newsroom ·
In 30 seconds
- OmniCompass includes 820 videos and 2,460 instances for vision-language evaluation and a 240-video audio-visual subset with 720 instances.
- It tests three tasks—temporal grounding, spatial grounding, and question answering—under text-only and text+image specifications.
- Evaluations of 50 multimodal LLMs show open-source models lag behind proprietary ones, especially on audio-visual understanding.
What is it?
OmniCompass is a benchmark designed to assess omni-modal video grounding and understanding. It defines two tracks: OmniCompass-VL for vision-language models and OmniCompass-AV for omni-modal models that use audio-visual evidence. Each track contains three core tasks—temporal grounding, spatial grounding, and question answering—covering when, where, and what happens in videos.
How does it work?
The benchmark provides paired target specifications (Text and Text+Image) for the same video content, allowing evaluation of consistency across modalities. All instances are manually annotated with multi-round quality verification. Evaluation pipelines are supplied via a GitHub repository, with scripts for self-hosted VLMs and API models, requiring Python 3.12+ on Linux, macOS, or WSL2.
How well does it do?
The authors evaluated 50 multimodal large language models on both tracks and tasks. Results indicate that open-source models still lag behind proprietary counterparts, particularly in audio-visual understanding, and that precise temporal and spatial grounding remains challenging even for leading models.
What does it need to run?
Running the benchmark requires installing the provided dependencies and using Python 3.12+ on a Linux, macOS, or WSL2 Windows environment. The repository includes an evaluation skill (omnicompass-eval) for integration with coding agents or direct script execution.
By the numbers
- Vision-language videos: 820
- Audio-visual videos: 240
- Human annotation hours: approximately 1,800
- Models evaluated: 50
The honest read
The benchmark reports that open-source models underperform proprietary ones, especially on audio-visual tasks, but these results are self-reported by the authors and reflect only the 50 models they tested. The figures for video counts and annotation effort describe dataset size, not model performance, and the evaluation focuses on paired text specifications rather than broader real-world usage. The sources note that precise temporal and spatial grounding remains challenging, indicating that current scores do not capture fine-grained understanding capabilities.
What we don't know yet
- Exact hardware requirements (GPU type, memory) are not specified.
- The benchmark’s download size is not provided.
- Licensing terms for the dataset and code are not stated.
Sources
- Paper (arXiv abstract) (primary)
- Code, GitHub
Researched and drafted with AI from the sources above; every figure is checked against them by code before it is published, and a line that fails is dropped.