Annotation quality needs a definition of disagreement

An annotated dataset can be technically valid while containing conflicting judgements about the same driving scene. Different reviewers may classify an object differently, infer an uncertain intention or draw boundaries according to different conventions. Those differences become part of the data used to train or evaluate a system.

Quality management should therefore address interpretation as well as format. The objective is a reproducible annotation policy that fits the intended engineering question. Perfect agreement is not a credible assumption when observations are incomplete or genuinely ambiguous.

Write rules for difficult cases

Define the objects, actions and relationships the task requires. Include guidance for occlusion, partial visibility and other ambiguities relevant to the dataset. Ordinary examples establish a starting point; difficult cases reveal whether the policy actually supports consistent work.

ASAM OpenLABEL defines an annotation format and labelling methods for objects and scenarios. A shared format helps exchange data, but it cannot resolve a team's judgement about an ambiguous observation. The annotation instructions still need to explain how uncertainty should be represented.

Distinguish error from uncertainty

A reviewer disagreement may reveal a mistake, a vague policy or evidence that supports more than one interpretation. Give the review process a route for distinguishing those possibilities. Forcing every uncertain observation into a confident label can hide information the model or evaluation process needs.

Record the reason for adjudication where it affects the policy. If a decision changes the meaning of a class, update the instructions and identify existing annotations that may require review. A one-off correction should not create a second convention that only experienced reviewers know.

Give reviewers a way to escalate a new kind of ambiguity before delivering the batch. That route prevents an improvised convention from spreading through a dataset and becoming expensive to unwind.

Sample according to engineering risk

Review should pay attention to the cases that matter to the intended function, including difficult or infrequent conditions. A random sample can provide useful information, but it may overlook categories whose errors are consequential. Explain why the review selection is appropriate to the dataset's planned use.

Also examine consistency across batches, providers and time. A policy can drift as reviewers change or instructions evolve. Preserve versions and compare the interpretation used in successive deliveries. Otherwise an apparent change in model performance may reflect a changed annotation convention rather than a changed capability.

Keep history attached to the dataset

Retain the annotation policy, tool context and significant revision history alongside the relevant dataset version. Evaluation needs to know which labels were used and whether later corrections alter the conclusion. A dataset name without its interpretation history is incomplete engineering evidence.

Use disagreements to improve both instructions and upstream collection. Some uncertainty may be reduced by better sensor context; some should remain explicitly represented. The useful quality question is whether the labels are appropriate, interpretable and consistently produced for their purpose. File validation is a necessary operational check, but the engineering value comes from understanding the judgements encoded inside the files.