LightTag

Build your own labeling tool?

Do you want to spend
A lot
$0.00
and wait
Almost Forever
0 working days
to start labeling?
Well, that's what it takes to build a good labeling tool for teams. Don't take our word for it. Here's a partial spec and estimates. You can change the estimates, and the hourly rate of an engineer and see what you come up with.

Assuming an engineer's cost per hour is:    

$

Setup a web serviceNon Functionals
Provision server(s), Set up DNS and networking , Provision Database, Configure security, Deployment Process, Set up monitoring, Configure Logging

Days

User AuthenticationNon Functionals
Flows for Signup, Login, Password Reset, Email integration to send notifications,

Days

Span Annotation InterfaceCore Experience
Support for span annotations, Support RTL and CJK languages, Annotations of phrases, Subword Annotations, Change the tag of an existing annotation, Annotate with double click, Annotators commit to review when no annotations were found

Days

Classification InterfaceCore Experience
Support document classification, Support multiple classes per document

Days

Relationship InterfaceCore Experience
Support binary relationships, Support phrase structure relations, Support dependency structure relations, Drag and drop interface for relationships, Highlight tokens that are part of a relationship, Highlight all items related to a token, Recursive components to display arbitrary trees, Recursive components to display arbitrary trees

Days

Data BackendCore Experience
Associate annotations with a task as well as annotator, Server side validation of span annotations (out of bands etc), Server side validation of classification (out of bands etc), Server side validation of relationship structures (no loops), Reject duplicate and malformed data, Group annotations by dataset, schema, annotator or team

Days

Workflow ManagementProject Management
Dynamically assign tasks to users, Optimize workflow for quantity/inter annotator agreement, User defined combinations of dataset and schemas, Maintain seperation of annotations made in seperate tasks, Serve users the next task when they log in, Support arbitrary concurrent users, Change priority of tasks, Prioritize tasks for seperate teams

Days

TeamsProject Management
Set up access control and seperation via teams, Assign multiple users to multiple teams, Default Everyone team, Measure team performance and compare labeler to cohort

Days

Keyboard ShortcutsCore Experience
Keyboard shortcuts to switch between tags, Keyboard shortcuts to navigate text, Apply annotations with keyboard only

Days

Provide context to labelersCore Experience
Show related annotations together and in order, UX to quickly navigate between related examples, Store and display metadata related to an example

Days

Annotator AnalyticsAnalytics
Provide analytics on annotator throughput, Provide analytics for annotator quality, Measure annotators against peers and normalize for group baseline, Annotator Analytics UI

Days

Task Level AnalyticsAnalytics
Inter Annotator Agreement metrics for your task, Confusion matrices to understand annotator disagreement, Error Analysis to understand types of disagreement (span, span overlap, tag), Annotator Analytics UI

Days

Immutable Audit LoggingNon Functional
Who viewed which data, Who made or changed an annotation, Maintain all versions of data (no delete), Log all requests, failed/successful authentications etc

Days

User Provided SuggestionsAI
Mechanism to upload user defined annotaiton suggestions, Track annotator accept/reject of suggestion, Error Analysis of models / disagreement with annotators, Protect from ambiguous suggestions (overlap) , Enforce non overlapping suggestions

Days

AI Powered SuggestionsAI
Implement algorithm that learns quickly from small set of annotations, Implement online learning, Support any language and tokenization scheme

Days

Docs and User GuideCore Experience
Document how your system works for internal staff, Document how to use your system for managers and annotators

Days

REST APIData Consumption
Token Provisioning and authentication, Dynamic paths, Upload data via API, Define Schemas via API, Define Tasks via API, Download Results, Retrieve Analytics, Append Data, Annotate via API

Days

Data UploadData Consumption
UI To upload and manage datasets, Client side validation of data, Decide on data format, Tools to get data into correct format

Days

Data DownloadData Consumption
UI to download data, Aggregate annotations from multiple labelers, Indicate whether label was provided by a suggestion, Include classifications alongside span annotations, Include all metadata about each example, Aggregate annotations by majority vote

Days

AutosavingCore Experience
Save annotations client side to prevent lost work, Synchronize tasks when user switches workstations

Days

Review modeProject Management
View what your annotators labeled and correct/confirm, Search for particular tags/errors

Days