During annotation, LightTag manages a queue of items to be annotated. During review, LightTag offers a second queue of items to be reviewed. Every annotation Job can have a Review Job associated to it and you can configure how many examples should be reviewed, who the reviewers are and a minimal number of annotators that annotated an example before it gets placed in the review queue.

Like with regular annotation Jobs, the Review queue is prioritized, and you can control which review job is currently being reviewed.

Adding A Review Job

To create a review job go to Review, select the Review Jobs section on the left and then press the plus icon on the Review Jobs table.

A form will appear asking you to define the Review Job.

Configuration

Job To Review: Review Jobs review regular Jobs. To set up a review job first select a job to be reviewed.

Reviewing Team: You can choose which team will be able to review the job. You can leave this field blank and the job will be assigned to Everyone. You can change this later.

Participant Threshold: This setting defines when a particular example can be added to the review queue. It asks how many annotators have already annotated that example, and it only adds examples to the queue that meet or exceed the threshold. This will be set by default to the underlying jobs Annotators Per Example and you can make it less.

Sample % and Sample Count: Control how many examples from the job will be reviewed. This will default to the number of examples, but you can change this

Working on Review Tasks

One a review job has been defined reviewers will be assigned work accordingly. To begin working on a Review Job (and receive a review task) reviewers should go to Review and select My Review Task

This will lead them to the review screen.

Reviewers can accept or reject an annotation by either hovering other it or CTRL+click to accept or ALT+click to reject.

The Accept Unanimous Spans on the right will automatically accept any annotation that was agreed on by all annotators who annotated the example.

Annotators must accept or reject all annotations before being able to submit and get the next task. The submit button will be grayed out until they do so

No Tasks Assigned

When a reviewer goes to the "My Review Tasks" page, they will be issued the highest priority review task that they can work on. It can happen that no such tasks are availble, for example if you set a threshold of 3 annotators per example before review but at most 2 have annotated any given example.

In such cases, continue annotating and revisit the review task later, or modify a review job so have a lower threshold.