HeyHomees Lab Notes · No. 8 · Published October 23, 2026
By HeyHomees Newsroom ·
In 30 seconds
- 200 multi-step web search tasks test agents' deep-search abilities.
- The highest model accuracy is 39.0% while humans achieve 21.1% per attempt.
- Evaluation runs under a one-hour, 100-tool-call budget on Python 3.10–3.13.
What is it?
OpenDS-World is a benchmark for deep search in real-world web environments, containing 200 challenging tasks that require agents to retrieve, access, and integrate multimodal evidence to answer factual questions.
How does it work?
Agents run in a shared tool environment with a search budget of one hour and up to 100 tool calls per task; they must autonomously browse, search, and process evidence to produce a final answer, which is then judged for correctness.
How well does it do?
Across 12 evaluated models, the best accuracy is 39.0%; a human study of 1,000 attempts by 50 participants achieved 21.1% accuracy per attempt, with 42.5% of tasks unsolved by any participant.
What does it need to run?
The code requires Python 3.10–3.13 and an Apache-2.0-licensed environment; evaluation uses a provided harness or a custom model harness, and results are saved as CSV and JSON files.
By the numbers
- Number of benchmark tasks: 200
- Best model accuracy: 39.0%
- Human accuracy per attempt: 21.1%
- Human study participants: 50
- Attempts per task by humans: 5
- Tasks unsolved by any human participant: 42.5%
- Search budget per task: one hour
- Tool call limit per task: 100
The honest read
The 39.0% figure is self-reported by the authors for the top evaluated model and reflects performance under a fixed one-hour, 100-tool-call budget on 200 tasks; it does not compare to other external baselines. Human accuracy of 21.1% comes from a study of 50 participants making five attempts per task, so it measures average per-attempt success rather than per-task success. The benchmark size (200 tasks) and the search limits constrain the difficulty and may not capture all real-world search scenarios.
What we don't know yet
- Exact hardware specifications required for running the models
- Download size of the dataset and evaluation harness
- Whether the benchmark includes any proprietary web content
Sources
- Paper (arXiv abstract) (primary)
- Code, GitHub
Researched and drafted with AI from the sources above; every figure is checked against them by code before it is published, and a line that fails is dropped.