Don't want to read? We made you a video:

This article covers the technical best practices for reviewing span annotations in LightTag. If you haven't read the intro to review, you really should. If you're all set, here we'll cover

  1. Choosing the example to review
  2. Accepting/Rejecting/Adding an annotation
  3. Filtering and Automation of Review

The Review Screen

The review screen is split into three columns. On the left you have the list of examples that need to be reviewed, on the right you have different filtering options for annotations, and in the middle you have the annotated text.

Choosing Examples To Review

The list on the left will show you a list of all remaining annotations.

Annotation Source

By default, it will restrict itself to examples that were annotated by humans, but you can show annotations made by models or both by adjusting the source.

Show only annotations made by a particular person or model

If you want to hone in an a particular annotator or model, you can click the filters button to restrict the list to examples they had seen.

Sorting

The list is sorted by the number of unreviewed annotations. Examples with no annotations needing review have a green text.

Accepting/Rejecting/Adding an annotation

LightTag will display all annotations with any filters (see below) you've chosen applied.

  • An annotation that has been marked as correct will be colored.
  • A rejected annotation will not be shown.
  • An annotation that has not been reviewed will have a border and no fill.
  • When you hover over an annotation you'll be able to accept or reject it as shown below

Changing the tag of an annotation

For both accepted and un-accepted annotations, you can hover over the annotation and click on the tag name to change the annotations tag.

Adding a new annotation

If an annotation is missing, you can add it by annotating as usual and changing the tag as needed.

Resolving Conflicts

A conflict is when two or more annotators annotated an overlapping text differently. Conflicting annotations are displayed with red crossed out text

To resolve the conflict, click on the annotation to open the conflict box then accept the correct annotation

Filtering and Majorities

LightTag helps you automate the review process by letting you filter annotations down to those you trust.

You can filter annotations to show only those made by 2 or more people, and filter them by a particular person. This means that you can accept any annotation your trust, whether because it had a majority vote or made by a particularly qualified person.

Accepting A Particular Annotators Annotations

The recording below shows accepting all annotations made by Renee Carter, which implicitly rejects any annotation that conflicts with her

Accepting By Number of Annotators

The recording below shows how we filter our annotations to those made by two or more annotators, and then automatically accept them. This implicitly rejects annotations that conflict.

Did this answer your question?