HeyHomees Lab Notes · No. 5 · Qwen (Alibaba) · Published October 20, 2026
By HeyHomees Newsroom ·
In 30 seconds
- 223 tasks span products, architecture, environments, characters, animation, and scientific visualization.
- Each task is judged by 2,651 binary criteria and eight visual-quality dimensions.
- Running the benchmark requires Blender 4.2.10, Node.js 22.17.0, and Claude Code 2.1.220 CLI.
What is it?
BlenderWorld is a benchmark for visual agents that generate editable 3D scenes in Blender from real-world references. It provides 223 tasks across nine categories and 22 subcategories, each with a brief, reference media, and optional specification files, and requires agents to submit a ZIP containing a Blender project and exports.
How does it work?
Agents receive a workspace with query.txt, input.json, reference media, and any attachments, then produce final.blend and related assets packaged as a ZIP. Evaluation runs Blender 4.2.10 to render eight adaptive views at 1024 × 1024 with 64 samples, applying a weighted binary rubric and eight holistic visual ratings, combining scene-inspection and VLM judgments.
What does it need to run?
The benchmark requires Python 3.10+, Linux, Blender 4.2.10, Node.js 22.17.0, ffmpeg/ffprobe, and the Claude Code 2.1.220 CLI for rendering and scene inspection. The bundled coding-agent environment also pins Codex CLI 0.147.0 and Claude Code 2.1.220.
By the numbers
- Total tasks: 223
- Binary criteria: 2,651
- Image + text tasks: 130
- Video + text tasks: 15
- Tasks with specifications: 59
- Renderer resolution: 1024 × 1024
The honest read
The benchmark reports the number of tasks (223) and binary criteria (2,651) as its primary scale, but these figures do not indicate model performance; they are only dataset sizes. Evaluation scores are generated by a pipeline that combines scene-inspection by an Agent-as-Judge and visual ratings by a VLM, using weights and importance scores defined by the authors, so any reported results are self-reported by the benchmark creators and not independently verified. The requirement for fresh renders and full rubric completion means missing submissions receive zero, which can affect aggregate scores. The benchmark relies on specific software versions (Blender 4.2.10, Node.js 22.17.0, Claude Code 2.1.220) and a vision-capable Chat Completions endpoint; performance may vary with different hardware or model back-ends, and the source does not provide external comparisons or baseline numbers.
What we don't know yet
- Exact download size of the dataset or models.
- Baseline performance numbers for existing agents.
- Support for non-Linux operating systems.
Sources
- README, GitHub (primary)
Researched and drafted with AI from the sources above; every figure is checked against them by code before it is published, and a line that fails is dropped.