Intersection over Union (IoU) is the evaluation metric that decides whether a detection is right or wrong. It measures how much a model’s box overlaps the box a human annotator drew, returning a single number between 0 and 1.
Almost every headline figure in object detection, from a research paper’s benchmark score to a production model’s acceptance test, resolves back to this one ratio. Understanding how to calculate IoU, where its thresholds come from, and where the metric quietly misleads you is the difference between trusting a benchmark and knowing what it actually proves.
What Is Intersection Over Union (IoU)?
Intersection over Union, also known as the Jaccard index, is the ratio of the area shared by two bounding boxes to the total area they cover between them. One box is the ground truth, meaning the annotated location of a real object. The other is the predicted bounding box that the model produced. The IoU meaning is therefore straightforward: it quantifies agreement between prediction and reality using nothing but geometry.
IoU = Area of Intersection ÷ Area of Union. A score of 0 means the boxes do not touch. A score of 1 means they are identical. Everything useful happens in between.
IoU Formula: The Intersection over Union (IoU) equals the Area of Intersection, divided by the Area of Union
Self-building AI vision
Bring a new AI vision application to life.
Turn ideas into real-world computer vision systems.
Why the Jaccard Index Matters Here
The Jaccard index predates computer vision by decades. It was built to compare sets, and a bounding box is simply a set of pixels. Because the measure is scale-invariant, a box that is 20 pixels wide and a box that is 2,000 pixels wide are judged by the same standard. That property is what allows one number to compare model performance across images, datasets, and hardware without renormalization.
Ground Truth and Prediction: The Two Inputs
Every IoU calculation begins with two rectangles that describe the same object.
Ground truth box: the reference rectangle produced during image annotation. It defines where the object genuinely sits.
Predicted box: the rectangle a neural network emits at inference time, usually with a confidence score attached.
Overlap: the region the two share. More overlap means better localization.
Both are expressed as bounding box coordinates. The most common convention stores each rectangle as (x1, y1, x2, y2), where (x1, y1) marks the top left corner and (x2, y2) marks the bottom right. Some pipelines instead store center coordinates with width and height, which is why mismatched formats are a frequent source of silently broken evaluation code.
Ground-truth Bounding Box, Predicted Bounding Box, and Overlap Region in IoU
How to Calculate IoU Step by Step
Working the arithmetic by hand once makes the metric’s behavior far easier to reason about. Take a ground truth box at (50, 100, 200, 300) and a predicted box at (80, 120, 220, 310).
Step 1: Find the Intersection Rectangle
The coordinates of the intersection come from taking the maximum of the two left edges, the maximum of the two top edges, the minimum of the two right edges, and the minimum of the two bottom edges.
x1 = max(50, 80) = 80
y1 = max(100, 120) = 120
x2 = min(200, 220) = 200
y2 = min(300, 310) = 300
The intersection rectangle spans (80, 120, 200, 300). If either resulting width or height comes out negative, the boxes do not overlap at all and the intersection area is clamped to zero.
Step 2: Compute the Two Areas
Intersection area is (200 – 80) × (300 – 120) = 120 × 180 = 21,600 pixels. Union is the sum of both box areas minus the intersection, so 30,000 + 26,600 – 21,600 = 35,000 pixels. Subtracting the intersection is essential, since the shared region would otherwise be counted twice.
Step 3: Divide and Interpret
Dividing gives 21,600 ÷ 35,000 = 0.617. That is a moderately good localization, comfortably above the conventional acceptance line but well short of a tight fit.
Worked Example for an IoU Calculation. In industrial deployments, a single global threshold rarely holds. Detecting a forklift at thirty meters and detecting a hand near a pinch point ask very different overlap tolerances of the same object detection models.
IoU Thresholds, Precision, and Recall
On its own, an IoU score describes one box pair. Turning it into a verdict requires a threshold, which converts a continuous measurement into a binary decision.
True Positive (TP): a predicted box whose IoU against a ground truth box clears the threshold.
False Positive (FP): a predicted box that fails the threshold against every ground truth box, or a duplicate detection of an object already matched.
False Negative (FN): a ground truth box that no prediction matched, meaning a real object was missed.
From these counts come the two metrics that most teams actually report. Precision and recall answer different questions: precision is TP ÷ (TP + FP) and asks how many of the model’s detections were correct, while recall is TP ÷ (TP + FN) and asks how many of the real objects the model managed to find. Plotting both against each other produces the curve that underpins nearly all detection benchmarking, and a confusion matrix makes the same relationships legible at a glance.
Raising the threshold tightens the standard, so precision climbs and recall falls. Lowering it does the reverse.
IoU Threshold
Effect
Typical Use
0.3 to 0.4
High recall, lower precision
Screening passes where a missed object is costlier than a false alarm
0.5
Balanced, and the historical default
General benchmarking and most published comparisons
0.75 to 0.9
High precision, lower recall
Measurement, robotic grasping, and dimensional inspection
The curve shows the relationship between precision and recall at different IoU thresholds – source.
From IoU to Average Precision
Benchmarks do not rank models on raw overlap. They aggregate precision and recall into mean average precision (mAP), and the IoU intersection over union threshold is the gate that decides which detections feed that calculation.
Pascal VOC established the classic protocol, computing mAP at a fixed threshold of 0.5. The COCO dataset made the standard considerably harder. As the GIoU authors note, IoU is the dominant evaluation metric across these benchmarks, and COCO averages average precision across ten thresholds from 0.50 to 0.95 in steps of 0.05.
A detector that scores well under that protocol has to localize tightly, not merely find the right neighborhood. Cityscapes applies the same logic at the pixel level, where mean IoU grades semantic and instance segmentation masks instead of rectangles.
Small objects punish this metric hardest. A five-pixel drift on a distant hard hat can halve the score, which is why safety-critical systems are usually evaluated across several thresholds rather than one.
IoU as a Loss Function for Bounding Box Regression
For years, models were trained with L1 or L2 losses on coordinates while being judged on overlap, an obvious mismatch between the training objective and the evaluation metric. Using the intersection over union IoU score directly as a loss closes that gap, but plain IoU has a flaw: when two boxes do not overlap at all, the gradient vanishes, and the neural network receives no signal about which direction to move.
Generalized Intersection over Union solves this. GIoU adds a penalty based on the smallest enclosing box around both rectangles, so non-overlapping predictions still produce a usable gradient for bounding box regression. Distance-IoU and Complete-IoU extend the idea further by incorporating center point distance and aspect ratio, which speeds convergence and improves final localization. Modern detectors, including most of the YOLO family, ship with one of these variants as the default box loss.
The same geometry appears again at inference. Non-maximum suppression compares predicted boxes against each other and discards any that overlap a higher-scoring neighbor beyond a set threshold, which is how a model with hundreds of raw proposals returns one box per object.
Where IoU Falls Short
Treating the score as ground truth about quality causes real problems in production.
Scale sensitivity: an identical pixel offset barely dents the score on a large box and devastates it on a small one, so small object performance looks worse than it functionally is.
Shape blindness: the metric counts area and ignores what the region contains, which limits its usefulness for fine structures in medical or defect imaging.
Crowded scenes: heavily overlapping instances confuse both matching and suppression, producing counts that drift from reality.
Threshold cliffs: a prediction at 0.499 and one at 0.501 are graded as opposites despite being indistinguishable in practice.
Confidence blindness: IoU says nothing about how certain the model was, so a lucky low-confidence hit outranks a well-calibrated near miss.
Annotation dependence: the metric measures agreement with a human-drawn box, so inconsistent labels cap achievable scores regardless of model quality.
The metric also drives tracking. Frame-to-frame association in most multi-object trackers works by computing overlap between newly predicted boxes and each track’s last known position.
IoU in Agentic Computer Vision
Detection is increasingly one layer in a larger system rather than the finished product. In agentic computer vision, a detector supplies structured observations that a reasoning layer, often a vision language model, then interprets, questions, and acts on. That shift changes what overlap is for.
Offline, IoU remains a benchmark. Online, it becomes a continuous consistency signal. An agent can compare successive frames and ask whether a track’s boxes remain geometrically coherent, whether a newly proposed region agrees with what object tracking already established, and whether two model heads agree about the same object. Sustained low overlap between what a system expects and what it observes is an early warning of drift, occlusion, or a miscalibrated camera, all of it detectable without a single new label.
In an agentic pipeline, IoU stops being a report card and starts being a sensor. It is one of the few perception signals available at runtime that requires no ground truth at all.
This is where deployment infrastructure matters. Running these checks continuously across many cameras means computing overlap at the edge, in real time, at scale. Viso Suite provides the platform layer for that work, handling annotation, training, edge deployment, and monitoring so teams can track localization quality in production rather than only at benchmark time.
Non-maximum suppression uses the same geometry in reverse. Instead of scoring accuracy, it removes duplicate detections that overlap by more than a threshold, so each object in a scene yields exactly one box.
FAQs
Above 0.5 is conventionally acceptable and above 0.7 is considered good, but the right target depends entirely on the task. Counting people tolerates loose boxes, while robotic grasping does not.
Yes. IoU is the Jaccard index applied to bounding boxes or segmentation masks. The names are interchangeable, though computer vision literature almost always uses IoU.
Find the intersection rectangle by taking the maximum of the left and top edges and the minimum of the right and bottom edges, compute its area, then divide by the combined area of both boxes minus that intersection.
Start at 0.5 for comparability with published results, then tune based on the relative cost of a missed object versus a false alarm in your application.
GIoU is an extension that penalizes the empty space in the smallest box enclosing both rectangles. It provides a useful training gradient even when the predicted and ground truth boxes do not overlap.
Yes. For image segmentation, the same ratio is computed over pixel masks rather than rectangles, and results are usually reported as the mean IoU across classes.
A single IoU value describes one box pair. Average precision aggregates thousands of those decisions across confidence levels and classes into one score, which is why object detection tasks are almost always ranked on mAP rather than on overlap alone. Both metrics measure the same underlying computer vision capability at different resolutions.
On this page
On this page
Build real-world AI systems
Skip months of data, training, and coding. Build your AI vision system in an hour.
