Reliable systems are ones that are reproducible.
When working in medical informatics its imperative to understand that terms like "sensitivity", "confidence intervals", "F1 score" are some that make or break deployment of an ML model. Last year I watched a quantized model pass every metric we threw at it, then produce a different QC verdict on different NVIDIA Jetson.
Nobody told me this could happen and it concerns the broader community that deploy ML into production, wherever that setting may be. For me, that experience broke something in how I thought about edge deployment. I went looking for the tool, the framework, the checklist (whatever it was) that was supposed to catch this kind of thing before you ship. It's not really a straight answer or one specific tool to address this. It prompts a larger question: "Why is on-device AI not considered as a primary deployment setting?". It might as well be unreliability or just that use cases for Edge AI are narrow minded.
Benchmarking measures speed. It does not measure sameness.
We've reviewed our conversations and literature to craft our whitepaper titled Benchmarking Is Not Verification, and the argument is exactly what the title says. When you benchmark a compressed model on a target device, you learn how fast it runs or how "well" your model responds to your prompts (For the LLM geeks out there!). You do not learn whether it runs the same way on a different device, a different runtime, or after hours of continuous operation. Those are different questions, and right now the Edge AI stack does not have a good way to answer them.
I am not the first person to notice this. Schlögl, Hofer, and Böhme ran the same model under the same framework on 75 different x86 machines and found 26 distinct outputs. Not 26 slightly different floating-point approximations but 26 categorically different output vectors, some of which flipped classification decisions on inputs near the decision boundary. That paper landed at NeurIPS in 2023 and it has been sitting in my head ever since.
Figure 1. The qualification layer sits between compression pipelines and deployment targets, combining static verification, cross-runtime profiling, and drift monitoring across heterogeneous runtimes and hardware backends.
What I tried to do with this whitepaper is take that finding seriously and ask: what would it actually take to close the gap? To systematically across runtimes, across hardware backends, across time. The paper traces the problem through six different research areas: compression theory, formal verification, runtime heterogeneity, benchmarking, drift monitoring, MLOps. The honest conclusion is that each community has a piece of the answer but nobody has put the pieces together. The intersection is empty.
Making AI as ubiquitous as the smartphone.
At Capicú, we've coined the term qualification layer for what should go there. A system that sits between your compression pipeline and your deployment targets and does three things: verifies behavioral properties before you ship, characterizes how outputs vary across the hardware matrix, and monitors for drift after deployment. The paper formalizes this as two algorithms and identifies five open math problems that would need to be solved to make it real.
This is not a product announcement. At Capicú, this research informs how we qualify and deploy AI models into biological instrumentation, but the paper is a research argument, not a feature list.
If you deploy models AI to different production environments and you have ever had that sinking feeling that your benchmark numbers might not mean what you think they mean, this paper is for you.
S.