Most computer-vision conversations start with accuracy. The more useful question on a live video system is: how many milliseconds do you have per frame? That single number — your latency budget — quietly constrains the model, the hardware, and the architecture before you've written a line of code.
Where the budget comes from
Start at the camera. A 25-fps feed gives you 40 ms per frame. If you're running eight cameras through one GPU, that budget is now shared. Subtract decode time, pre-processing, tracking, post-processing, and the overhead of moving tensors on and off the device, and the slice left for the model itself is often far smaller than people expect.
So the budget isn't "how fast is the model." It's "how fast is the whole loop, across every camera, with headroom to spare."
What the budget decides for you
- The model. A heavier detector that's 3% more accurate is worthless if it blows the budget and starts dropping frames. The right model is the most accurate one that fits.
- The hardware. TensorRT, FP16, and batching exist precisely to buy back milliseconds. Whether you need them is a function of the budget, not taste.
- The architecture. Tracking between detections, running detection every Nth frame, decoding on the GPU instead of the CPU — these are all budget moves.
A simple way to plan it
Write the budget down as a table before building: frame interval, number of cameras, and a line item for each stage of the pipeline. If the stages sum to more than the interval, the system can't keep up — and no amount of accuracy fixes that. It's a boring spreadsheet, and it will save you a rebuild.
The short version
Decide the latency budget first. Then choose the model that fits inside it. A system that's a little less accurate but never falls behind beats a brilliant one that can't keep up with the cameras — every shift.
If you're sizing up a real-time video problem, book a call and we'll work the budget together.