Published model metrics are useful. They are not acceptance tests. This note is the evaluation protocol we use when deciding whether an edge vision skill is ready for a real camera environment.
1. Freeze the operating definition
Write down, in one page:
- the event an operator must act on
- what counts as a true positive
- what counts as a miss
- which false positives are intolerable
- the maximum latency from frame to usable signal
If this page is vague, every later metric will be political.
2. Build a scene-faithful holdout
Collect clips from the target environment across:
- day and night
- weather and glare
- occlusion and crowd density
- camera angles that match production mounts
- hard negatives that already confuse operators
Do not evaluate only on clean textbook frames.
3. Measure by scene, not only in aggregate
Report precision, recall, and latency by scene group. A single blended accuracy number can hide a failing entrance camera behind a strong parking lot.
4. Validate the exported runtime
Train-framework accuracy is not deployment accuracy. Re-run the holdout on the intended ONNX/CPU/edge path at the intended resolution and concurrency. Record memory and sustained FPS, not only peak demo FPS.
5. Test the alert workflow
A box on a frame is not an incident. Confirm:
- threshold policy per scene
- who receives the alert
- what context is attached
- how events are closed and audited
- when human confirmation remains mandatory
6. Gate go-live with regression clips
Keep a short regression pack that must pass before any threshold or weight change ships. This is how edge systems stay trustworthy after the first week.
How public Innomium models fit this protocol
Sentinel, Vantage, and Ember can accelerate step zero — you can inspect behavior quickly via Hugging Face Spaces. They do not skip steps 2–6. Metrics such as 92%/93%/90% are Innomium protocol snapshots on defined splits.
If you want help implementing the protocol or adapting a public model, start at [Contact](/contact) or browse [Computer Vision services](/services/computer-vision).