Google's on-device dictation model 'Eloquent' turned out to be nearly impossible to benchmark
Google released a new on-device dictation model called Eloquent that converts speech to text directly on your device. When one developer tried to measure how good it actually is, they hit a wall — the model's closed design made independent testing basically impossible.
Eloquent is Google's speech-to-text model built to run entirely on your phone or computer, without sending your voice to a remote server. That means no internet required and better privacy — which sounds appealing for local AI setups.
However, a developer in the LocalLLaMA community attempted to run standard benchmarks on Eloquent and found it could not be properly tested from the outside. The model is locked into Google's own ecosystem in a way that blocks independent evaluation tools. This means users have no choice but to trust Google's own performance claims, with no way for the broader community to verify them. For anyone building local AI agents or trying to reduce cloud API costs, this highlights a key limitation: a model you can't independently evaluate is hard to rely on with confidence.
Key points
- Eloquent is Google's speech-to-text model that runs locally on your device without internet
- Local execution means better privacy and no cloud API costs for voice input
- An independent developer was unable to run standard performance tests on it
- Google's closed design prevents third-party verification of its accuracy claims
- If you need a benchmarkable on-device speech model, open-source options like Whisper remain more transparent
Quick term guide
- LocalLLaMA
- A Reddit community about AI models that people can often run on their own computers.
- benchmarks
- Benchmarks are standard tests used to compare performance.
- benchmark
- A test used to compare speed, quality, or cost.
- valuation
- The amount investors think a company is worth.
- AI agents
- AI agents are AI tools that can carry out steps toward a goal, not just answer once.
- API costs
- Fees paid when software calls an online service programmatically.
- local execution
- Running an AI model directly on your own computer instead of using a remote server.
- open-source
- Software whose code is shared publicly so others can inspect, use, or change it.