HeyHomees Lab Notes · No. 9 · alibaba · Published October 26, 2026
By HeyHomees Newsroom ·
In 30 seconds
- Contains 64,729 real user sessions from 11,010 shoppers across 566,102 items and 13 action types.
- Offers an interactive environment with offline validation against a production A/B test.
- Released under a Non-Commercial Research License with code on GitHub.
What is it?
UserArena is a benchmark that bundles 64,729 session trajectories from 11,010 users, covering 566,102 items and 13 distinct action types, together with an interactive simulation environment and offline validation against a controlled production A/B test.
How does it work?
The benchmark provides recorded trajectories and a simulator that can roll out student models; it also includes the On-Policy Trajectory Distillation (OPTD) method, which uses trajectory-level intent as dense teacher targets to improve simulator fidelity.
What does it need to run?
The code requires Python and the ability to load the provided SFT checkpoint for both student and frozen teacher models; evaluation scripts expect an empty output directory and use the held-out set of 535 sessions for scoring.
What's new versus the last version?
UserArena introduces a long-horizon, fine-grained real-user dataset and the OPTD training objective, which the authors report achieves the strongest overall fidelity among evaluated simulators under their protocol.
By the numbers
- session trajectories: 64,729
- unique users: 11,010
- items covered: 566,102
- action types: 13
- held-out evaluation sessions: 535
The honest read
The figures describe the scale of the dataset (64,729 sessions, 11,010 users, 566,102 items, 13 actions) and the size of the held-out set (535 sessions). These numbers reflect the authors' collection and do not indicate performance of any particular model; the reported fidelity advantage of OPTD is based on the authors' own evaluation protocol. No external benchmarks or comparative scores are provided beyond the authors' claim of "strongest overall fidelity" within their own tests.
What we don't know yet
- Exact download size of the dataset
- Hardware specifications required for optimal performance
- Whether the benchmark includes any pre-trained baseline agents
Sources
- Paper (arXiv abstract) (primary)
- Code, GitHub
Researched and drafted with AI from the sources above; every figure is checked against them by code before it is published, and a line that fails is dropped.