Annotation Metrics
Understand annotation progress and agreement rates across your datasets.
Annotation metrics provide insights into how your annotation projects are progressing and how consistent annotators are with each other.
Dataset Level Metrics
Dataset level metrics provide a high-level overview of annotation progress and quality across the entire dataset.
Progress
The dataset progress metric shows the percentage of datapoints that have received annotations from at least the minimum required number of annotators (as configured in the annotation project).
For example, if your project requires at least 2 annotators per datapoint, the progress shows what percentage of datapoints have been annotated by 2 or more annotators.
Overall Agreement Rate
This metric measures the average concordance between annotators who have annotated the same datapoints across the dataset. It represents the percentage of matching annotations when comparing each pair of annotators who worked on the same datapoints.
A high agreement rate (above 80%) typically indicates clear annotation guidelines and consistent understanding among annotators.
Subset Level Metrics
When a dataset is divided into subsets for annotation, you can view metrics for each individual subset.
Progress
The subset progress shows the percentage of datapoints within the selected subset that have received the minimum required number of annotations.
Overall Agreement Rate
For the selected subset, this shows the average percentage of matching annotations when comparing pairs of annotators who worked on the same datapoints within that subset.
Comparing agreement rates across different subsets can help identify parts of your dataset that may have ambiguous examples or unclear annotation guidelines.
Annotator Level Metrics
Annotator level metrics help you evaluate individual annotator performance and workload.
Progress
The annotator progress shows the percentage of assigned datapoints the selected annotator has completed across all their dataset assignments.
This metric focuses only on the datapoints assigned to the specific annotator, not the entire dataset.
Overall Agreement Rate
The annotator agreement rate measures the average percentage of annotations from the selected annotator that match with other annotators on the same datapoints.
An annotator with a significantly lower agreement rate than others may need additional training or clarification on the annotation guidelines.
