A Decision Maker's Guide to Computer Vision Data Labeling: Bounding Boxes, Semantic Segmentation, and How to Choose

Ask three annotation vendors whether a project needs bounding boxes or segmentation masks, and expect three confident answers that track what each vendor happens to sell. The tooling company whose product manages masks explains why masks are superior. The services firm with a polygon team presents a technique menu. Nobody referees, because every author has a position to defend.

This guide takes the referee’s seat, because computer vision data labeling has a correct answer per project and it is not a matter of taste. Three moves follow. First, a task-first framework: what the model must do decides the annotation type, before cost enters the conversation. Second, honest economics, including a shift the ranking pages have not caught up with, where segmentation foundation models generate masks from box-level prompts and turn tracing work into correction work. Third, quality written as contract language, with thresholds and audit rules that belong in an agreement rather than in a vendor adjective.

Nothing here argues that one technique beats another. That claim is what the market already sells, and it is what leaves teams paying mask prices for a detection problem or shipping boxes into a model that needed exact boundaries. The stakes run past a single line item. Eight in ten companies¹ name data limitations as the roadblock to scaling AI systems, and fewer than 10 percent have moved past pilots into work that returns measurable value. The technique map settles the question first, because a cheap dataset that cannot teach the model its task is the most expensive dataset available.

What Is Computer Vision Data Labeling?

Computer vision data labeling is the process of adding structured annotations to images and video so supervised models can learn to recognize what those annotations describe. The annotations take specific forms: class tags, bounding boxes, polygons, pixel masks, keypoints, and cuboids, each encoding a different kind of information about what appears in the frame and where.

Labeling sits in the middle of a three-part pipeline, and its quality depends on the two parts around it. Collection produces the raw imagery, and the conditions under which that data was gathered, including consent, licensing, and documented provenance, follow the dataset for its entire life. Teams scoping the collection stage will find the sourcing, specification, and provenance questions treated separately in the AI data collection practice; this guide assumes raw data already in hand and picks up where labeling begins.

Between collection and labeling sits the label schema, which is where most annotation projects are quietly decided. The schema defines the classes, writes the definition of each one in language an annotator can apply at 2 a.m. without asking a question, and rules on the edge cases before they arrive: partially visible objects, reflections, objects behind glass, the boundary between two adjacent classes, and the minimum size worth annotating at all. Ambiguous class definitions are annotation’s version of an underspecified data request. They do not announce themselves as errors. They surface as inconsistent labels, then as a model that behaves unpredictably on exactly the cases the schema never resolved.

Against a settled schema, the technique menu is short. Boxes locate. Polygons trace irregular outlines cheaply. Pixel masks describe exact shape and area. Instance masks separate individual objects. Panoptic labeling covers the whole scene. Keypoints capture pose. Cuboids place objects in three dimensions. Choosing among them is the subject of the next section, and it is a modeling decision rather than a budgeting preference.

Minimum-size rules close the list, because objects below a threshold contribute noise rather than signal.

The Technique Map: Matching Annotation to Model Task

The framework that governs this entire guide fits in one sentence: the model’s task chooses the annotation technique, and budget then negotiates volume and assistance, never the technique itself. Applied honestly, it resolves most annotation arguments before they start.

Two consequences follow from reading the table honestly. The first is that no technique is superior to another; each is correct for the task that requires it, which is precisely the statement a vendor with a technique to sell cannot make. The second is that most production projects are mixed. A retail shelf-monitoring system uses boxes for the majority of product classes and reserves masks for the few where shape or occupancy drives the decision. An autonomous driving stack segments drivable surface while boxing vehicles and pedestrians. Mixed-technique projects are the professional norm rather than a compromise, and a vendor who quotes a single technique across a heterogeneous class list has not read the schema closely enough.

The rest of this guide takes the two techniques buyers argue about most and gives each a full working treatment before putting them side by side.

Bounding Box Annotation

Bounding box annotation marks an object with a rectangle defined by two corner coordinates and a class label. It is the cheapest useful annotation in computer vision and the input format behind the detection model families most teams deploy, including the YOLO and R-CNN lineages, which learn to predict a box and a class for every object they find.

Four practices separate competent box work from the kind that quietly degrades a model.

• Tightness discipline comes first: every box hugs the object, because slack rectangles teach the model that background pixels belong to the class. A dataset of loose boxes produces a detector that fires on shadows and adjacent clutter, and the failure appears at inference rather than in review.
• Occlusion and truncation rules come second, decided in the schema rather than by each annotator: whether a partially hidden object gets a box around its visible extent or its inferred full extent, and whether an object cut by the frame edge is annotated at all.
• Crowding protocols come third, defining the point at which a dense group becomes a region label instead of 200 individual boxes.
• Minimum-size rules close the list, because objects below a threshold contribute noise rather than signal.

The known weakness deserves stating plainly. A rectangle always includes background for irregular, rotated, or elongated objects. A diagonal power line, a bent pipe, or a person mid-stride fills a small fraction of the box drawn around them, and everything else inside that rectangle is labeled as the class. When the ratio of object to enclosing rectangle drops far enough that the surrounding pixels start teaching the wrong lesson, polygon annotation is the next step up, and it costs considerably less than full pixel masks.

Box quality is measured with Intersection over Union, which compares an annotator’s rectangle against a reference box and reports the overlap as a ratio between zero and one. Auditing runs on three tracks: IoU against a golden set of expert-drawn boxes, tightness sampling to catch systematic slack before it spreads across a batch, and class-confusion review on the pairs a schema makes genuinely hard to tell apart. Boxes also serve a second purpose worth flagging now, because they are the input tier that model-assisted mask generation runs on.

Semantic Segmentation and Its Siblings

Semantic segmentation labels every pixel in an image with a class, producing masks that capture exact object boundaries rather than enclosing rectangles. Unlike bounding boxes, it represents true shape and area; unlike instance segmentation, it does not distinguish between individual objects of the same class, so five adjacent cars become one region labeled “car.”

The technique earns its cost where boundaries carry meaning. Medical imaging needs the contour of a lesion, not a rectangle containing it, because the margin is the clinical finding. Autonomous systems need drivable surface described exactly, since the edge of the road is the decision. Manufacturing defect detection needs the shape and extent of a flaw to grade severity. Agricultural analysis needs canopy area, which is an occupancy measurement a box cannot produce. In each case the model’s output is a measurement, and measurements require pixels.

Two siblings complete the picture the comparison keyword hides.

• Instance segmentation produces a separate mask per object, which adds the counting capability semantic segmentation structurally lacks. If the requirement is how many, semantic segmentation cannot answer it at any budget, and the escape hatch is instance segmentation rather than either side of the usual debate.
• Panoptic segmentation unifies both views, assigning every pixel to either a countable thing, such as a vehicle or a person, or an uncountable stuff region, such as sky, road, or vegetation. The things-versus-stuff distinction is the practitioner’s signal for which of the three a project actually needs.

Cost is the honest caveat, and pixel-level work sits at the top of the annotation price list. Researchers annotating high-resolution steel microstructure imagery measured 170 hours² of manual labeling to complete a single segmentation dataset, on images large enough that every pixel decision compounds. That figure is the fully-manual baseline, and the section on model-assisted labeling explains why a quote built on it deserves a second look.

Mask quality is governed by mean Intersection over Union, which averages the per-class IoU across every class in the dataset, paired with boundary-focused measures that check whether edges land correctly rather than whether bulk area roughly matches. Class-consistency sampling completes the audit, since the most common mask defect is a correct shape assigned to a neighboring class.

Bounding Boxes vs. Semantic Segmentation: The Decision

The bounding boxes vs. semantic segmentation question resolves in a fixed order, and reversing that order is what produces expensive mistakes. Task requirements come first, economics second, project modifiers third.

Start with what the model must output. A model that answers “where is it” needs localization, and boxes carry localization completely. A model that answers “what is its exact shape, area, or extent” needs boundary precision, and only masks carry that. A model that answers “how many are there” needs neither, because counting separate objects of one class requires instance segmentation, and a project that discovers this after buying semantic masks has bought the wrong dataset at the highest available price.

The economics behind that last column are real. A box is two corner points and a class, while a mask is a per-pixel decision across an entire object boundary, and the industrial figures cited earlier put a single manual segmentation dataset at 170 hours of annotator time. Pixel-level labeling has been among the largest line items in applied deep learning for a decade, which is why the assumption that segmentation is unaffordable became conventional wisdom.

Four modifiers move the decision once the task allows either technique.

• 1. Image complexity matters, since a cluttered scene with heavy occlusion raises box ambiguity and narrows the gap.
• 2. Class count matters, because per-class mask consistency degrades faster than per-class box consistency as the taxonomy grows.
• 3. The downstream model’s accuracy tolerance matters, and a system with a wide margin for localization error rarely justifies masks.
• 4. Regulatory and safety stakes matter most, since a boundary error in surgical planning or obstacle avoidance carries consequences a detection error does not.

Applying this in a scoping conversation takes about an hour and saves considerably more. Write the class list, and against each class record what the model must output for it: a location, a boundary, or a count. Classes cluster quickly, and the clusters are the specification. A defect-inspection program might land on boxes for eight component classes, masks for the three defect types where severity grading depends on extent, and instance masks for the one class an operator has to count per unit. That specification is what goes to vendors, and it changes the conversation from a technique pitch into a priced response to a fixed requirement.

The verdict this section refuses to give is a winner. The verdict it gives instead is a match, project by project and often class by class within one project. One column of the table above, though, has changed since most segmentation budgets were written.

Model-Assisted Labeling: The Economics Shifted

Segmentation foundation models changed the cost structure of mask annotation, and the change is documented in the research literature rather than in vendor marketing. Models in the Segment Anything family accept prompts as light as a bounding box, a point, or a short noun phrase and return a mask, which converts annotation from tracing work into correction work.

The current generation reports the effect directly. The SAM 3 data engine³, published in late 2025 and revised in March 2026, pairs the model with AI annotators and verifiers working alongside human reviewers, and its authors report roughly double the annotation throughput of the human-only pipeline it replaced, with fine-grained domains running about 36 percent faster on positive prompts and around five times faster on negative ones. The same engine produced 4 million unique concept labels across images and video.

Work outside the model labs points the same direction. The industrial segmentation study cited earlier cut annotation for its dataset from 170 hours to 37 hours using unsupervised pre-annotation, a reduction of roughly 78 percent, on exactly the high-resolution imagery that makes manual masking expensive.

The honest boundaries matter as much as the headline. Assisted labels inherit the model’s biases, and performance degrades on domains far from the pretraining distribution, on fine structures such as wires and hair, and on the boundary-critical applications where masks were commissioned in the first place. Medical imaging, novel industrial inspection, and safety-relevant perception still demand human-heavy review. Validation sampling is not optional, and assisted labels pass through the same acceptance gates as manual ones, at the same thresholds. What has genuinely changed is the starting assumption. A segmentation budget quoted on fully-Qualmanual figures should be re-quoted, and “masks are out of reach” is now a claim worth re-checking rather than a constraint to plan around. Bring that question, and the decision table above, to the next annotation quote, including any quote from an annotation services partner.

QA and Acceptance Criteria

Quality in annotation is a contractable property, and the teams who get it write the criteria into the agreement before the first image is labeled. Vendor adjectives cannot be enforced. Thresholds can. Skipping this step shows up later as abandoned work. Gartner⁴ expects organizations to abandon 60 percent of AI projects through 2026 for want of AI-ready data, a finding drawn from a survey in which 63 percent of data management leaders said they either lacked the right practices or could not tell.

Metrics attach to techniques rather than to projects. Boxes are governed by IoU thresholds, with tightness matched to the downstream model’s tolerance, since a detector feeding a robotic gripper needs a stricter threshold than one counting vehicles for traffic analytics. Masks are governed by mIoU together with a boundary-quality measure, because bulk overlap can look acceptable while edges are systematically wrong. Instance work adds per-instance IoU and counting accuracy, which catch the merge-and-split errors that dataset-level averages conceal.

Process gates carry the rest of the weight. Golden-set audits run at a stated frequency against expert-annotated reference images the vendor does not see in advance. Inter-annotator agreement is measured on a sampled overlap, where multiple annotators label the same images and the disagreement rate exposes schema ambiguity rather than individual carelessness. Adjudication rules decide who resolves disagreements and on what basis. Per-class acceptance gates are the clause most worth insisting on: a dataset averaging 95 percent can hide a rare class failing at 40 percent, and rare classes are frequently the ones that matter, since they tend to be the anomalies, defects, and hazards the model exists to catch.

The agreement terms complete the device. Specify rejection and replacement so failed batches are re-annotated rather than credited, re-review rights on any batch that misses threshold, and schema-change handling that states who absorbs the cost of relabeling when a class definition shifts mid-project. Name the sampling rate as a number rather than a practice, since “regular audits” and “5 percent of every batch, stratified by class” describe different commitments. Set the golden-set refresh interval too, because a reference set that never changes eventually gets learned rather than passed, and its scores drift upward while real quality holds flat. Annotation provenance closes it: a record of who labeled what, with which assistance, under which schema version, and reviewed by whom. That lineage is what makes a dataset auditable years later, and it aligns annotation with the governance expectations set out in Damco’s Trustworthy AI framework. Provenance is inherited from the collection stage and extended through labeling, not invented at the end.

Where Damco Fits

Damco runs annotation as an engineering discipline rather than a staffing arrangement. Engagements start with a schema workshop, because class definitions and edge-case rules determine more about final dataset quality than any downstream review step. Technique selection follows the task-first framework in this guide, which regularly means a mixed specification: boxes across most classes, masks where boundary precision drives the model’s decision, instance masks wherever counting is the requirement.

The published capability covers the techniques this guide treats, including bounding boxes, semantic segmentation, and polygon annotation across image, video, and text data annotation services. Model-assisted pipelines are applied where they genuinely pay and reviewed at the same acceptance thresholds as manual work, with the validation burden treated as part of the method rather than as an optional extra. Human judgment stays in the loop at annotation review, which is where schema ambiguity and edge cases actually surface.

Labeled datasets are a means rather than an end, so the annotation practice is paired with the data engineering pipelines that move and version the data and with the AI development work the datasets ultimately feed. Teams scoping the stage before this one will find the collection-side treatment in the linked data collection practice, and the operational view of annotation workflows in Damco’s companion image annotation guide.

The acceptance criteria described in Section 8 are not a checklist offered to buyers and exempted for the vendor. They are the criteria Damco expects to be held to, written into the agreement, measured on golden sets, and reported per class. Talk to Damco about scoping annotation with task-first technique selection, current model-assisted economics, and contracted QA.

What to Take Into the Next Annotation Quote

Three questions settle most computer vision data labeling decisions, and they run in a fixed order. What must the model output for each class: a location, a boundary, or a count? That answer picks the technique, and it picks it before anyone quotes a price. What does the work cost now, given that assisted pipelines moved the mask baseline within the last year? That answer belongs in a fresh quote rather than in an assumption carried over from an older project. What quality gets measured, at which threshold, on which classes, and audited how often? That answer belongs in the contract rather than in a capability deck.

Teams that run those three questions in order tend to buy the dataset their model needed. Teams that start from price tend to buy the cheapest dataset that cannot teach the task, then pay again to relabel it. The techniques will keep improving, and the assisted-labeling figures quoted here will look conservative within a year. The discipline that turns them into a working dataset will not.

References:

• 1. McKinsey
• 2. Arxiv
• 3. Arxiv
• 4. Gartner