HeyHomees Lab Notes · No. 12 · Tsinghua Machine Learning Group · Published October 29, 2026
By HeyHomees Newsroom ·
In 30 seconds
- Falcon extracts a target speaker from video/audio streams in a single encoder-decoder pass.
- It uses 0.5M trainable parameters and reduces parameters by 91.9% versus AV-TFGridNet.
- The model cuts computation by 20.2× and GPU latency by 11.1×, enabling real-time use on laptops.
What is it?
Falcon is a causal audio-visual target speaker extraction (AVTSE) model that separates speech in a single encoder–decoder pass. It is designed for latency-sensitive applications such as calls and meetings and runs live on a laptop.
How does it work?
The model’s AudioBlock performs multi-scale time-frequency separation in one pass, and training uses an encoder-space speaker contrastive loss that suppresses near-field leakage at zero inference cost and transfers across backbones.
How well does it do?
Falcon achieves state-of-the-art extraction quality on the RealSSA, LRS2-2Mix, LRS3-2Mix and VoxCeleb2-2Mix benchmarks, according to the authors. Compared with the causal AV-TFGridNet baseline it cuts parameters by 91.9 %, computation by 20.2× and GPU latency by 11.1×.
What does it need to run?
The released code includes training, evaluation and inference scripts and a real-time demo that can run on a laptop GPU; no specific hardware requirements are listed beyond a GPU for the demo.
By the numbers
- trainable parameters: 0.5 M
- parameter reduction vs baseline: 91.9 %
- computation reduction: 20.2×
- GPU latency reduction: 11.1×
- RealSSA-Sim training examples: 100,826
- near-field interfering speakers per scene: 1--5
- far-field interfering speakers per scene: 10--15
The honest read
The reported extraction quality and efficiency gains are self-reported by the authors; no external benchmarks or independent evaluations are provided. The parameter, computation and latency reductions are measured relative to the causal AV-TFGridNet baseline, which the paper cites as the comparison point. The RealSSA benchmark includes simulated and real recordings, but the paper does not give absolute performance numbers (e.g., SI-SDR) for Falcon versus other methods.
What we don't know yet
- Exact inference speed (e.g., real-time factor) on specific hardware
- Details of the training data download size
- Whether the model is released under an open-source licence
Sources
- Paper (arXiv abstract) (primary)
- Code, GitHub
Researched and drafted with AI from the sources above; every figure is checked against them by code before it is published, and a line that fails is dropped.