Don't want to read? We made you a video:
This article covers the technical best practices for reviewing span annotations in LightTag. If you haven't read the intro to review, you really should. If you're all set, here we'll cover
-
Choosing the example to review
-
Accepting/Rejecting/Adding an annotation
-
Filtering and Automation of Review
The Review Screen
The review screen is split into three columns. On the left you have the list of examples that need to be reviewed, on the right you have different filtering options for annotations, and in the middle you have the annotated text.
Choosing Examples To Review
The list on the left will show you a list of all remaining annotations.
Annotation Source
By default, it will restrict itself to examples that were annotated by humans, but you can show annotations made by models or both by adjusting the source.
Show only annotations made by a particular person or model
If you want to hone in an a particular annotator or model, you can click the filters button to restrict the list to examples they had seen.
Sorting
The list is sorted by the number of unreviewed annotations. Examples with no annotations needing review have a green text.
Accepting/Rejecting/Adding an annotation
LightTag will display all annotations with any filters (see below) you've chosen applied.
-
An annotation that has been marked as correct will be colored.
-
A rejected annotation will not be shown.
-
An annotation that has not been reviewed will have a border and no fill.
-
When you hover over an annotation you'll be able to accept or reject it as shown below
Changing the tag of an annotation
For both accepted and un-accepted annotations, you can hover over the annotation and click on the tag name to change the annotations tag.
Adding a new annotation
If an annotation is missing, you can add it by annotating as usual and changing the tag as needed.
Resolving Conflicts
A conflict is when two or more annotators annotated an overlapping text differently. Conflicting annotations are displayed with red crossed out text
To resolve the conflict, click on the annotation to open the conflict box then accept the correct annotation
Filtering and Majorities
LightTag helps you automate the review process by letting you filter annotations down to those you trust.
You can filter annotations to show only those made by 2 or more people, and filter them by a particular person. This means that you can accept any annotation your trust, whether because it had a majority vote or made by a particularly qualified person.
Accepting A Particular Annotators Annotations
The recording below shows accepting all annotations made by Renee Carter, which implicitly rejects any annotation that conflicts with her
Accepting By Number of Annotators
The recording below shows how we filter our annotations to those made by two or more annotators, and then automatically accept them. This implicitly rejects annotations that conflict.
