Build your own labeling tool ?

Do you want to spend
A lot
$0
and wait
Almost Forever
0 working days
to start labeling ? Well that's what it takes to build a good labeling tool for teams. Don't take our word for it. Here's a partial spec and estimates. You can change the estimates, and the hourly rate of an engineer and see what you come up with.
Assuming an engineer costs

$

an hour
Setup a web serviceNon Functionals
Provision server(s)Set up DNS and networking Provision DatabaseConfigure securityDeployment ProcessSet up monitoringConfigure Logging

days

User AuthenticationNon Functionals
Flows for Signup, Login, Password ResetEmail integration to send notifications

Days

Span Annotation InterfaceCore Experience
Support for span annotationsSupport RTL and CJK languagesAnnotations of phrasesSubword AnnotationsChange the tag of an existing annotationAnnotate with double clickAnnotators commit to review when no annotations were found

days

Classification InterfaceCore Experience
Support document classificationSupport multiple classes per document

days

Relationship InterfaceCore Experience
Support binary relationshipsSupport phrase structure relationsSupport dependency structure relationsDrag and drop interface for relationshipsHighlight tokens that are part of a relationshipHighlight all items related to a tokenRecursive components to display arbitrary treesRecursive components to display arbitrary trees

days

Data BackendCore Experience
Associate annotations with a task as well as annotatorServer side validation of span annotations (out of bands etc)Server side validation of classification (out of bands etc)Server side validation of relationship structures (no loops)Reject duplicate and malformed dataGroup annotations by dataset, schema, annotator or team

days

Workflow ManagementProject Management
Dynamically assign tasks to usersOptimize workflow for quantity/inter annotator agreementUser defined combinations of dataset and schemasMaintain seperation of annotations made in seperate tasksServe users the next task when they log inSupport arbitrary concurrent usersChange priority of tasksPrioritize tasks for seperate teams

days

TeamsProject Management
Set up access control and seperation via teamsAssign multiple users to multiple teamsDefault Everyone teamMeasure team performance and compare labeler to cohort

days

Keyboard ShortcutsCore Experience
Keyboard shortcuts to switch between tagsKeyboard shortcuts to navigate textApply annotations with keyboard only

days

Provide context to labelersCore Experience
Show related annotations together and in orderUX to quickly navigate between related examplesStore and display metadata related to an example

days

Annotator AnalyticsAnalytics
Provide analytics on annotator throughputProvide analytics for annotator qualityMeasure annotators against peers and normalize for group baselineAnnotator Analytics UI

days

Task Level AnalyticsAnalytics
Inter Annotator Agreement metrics for your taskConfusion matrices to understand annotator disagreementError Analysis to understand types of disagreement (span, span overlap, tag)Annotator Analytics UI

days

Immutable Audit LoggingNon Functional
Who viewed which dataWho made or changed an annotationMaintain all versions of data (no delete)Log all requests, failed/successful authentications etc

days

User Provided SuggestionsAI
Mechanism to upload user defined annotaiton suggestionsTrack annotator accept/reject of suggestionError Analysis of models / disagreement with annotatorsProtect from ambiguous suggestions (overlap) Enforce non overlapping suggestions

days

AI Powered SuggestionsAI
Implement algorithm that learns quickly from small set of annotationsImplement online learningSupport any language and tokenization scheme

days

Docs and User GuideCore Experience
Document how your system works for internal staffDocument how to use your system for managers and annotators

days

REST APIData Consumption
Token Provisioning and authenticationDynamic pathsUpload data via APIDefine Schemas via APIDefine Tasks via APIDownload ResultsRetrieve AnalyticsAppend DataAnnotate via API

days

Data UploadData Consumption
UI To upload and manage datasetsClient side validation of dataDecide on data formatTools to get data into correct format

days

Data DownloadData Consumption
UI to download dataAggregate annotations from multiple labelersIndicate whether label was provided by a suggestionInclude classifications alongside span annotationsInclude all metadata about each exampleAggregate annotations by majority vote

days

AutosavingCore Experience
Save annotations client side to prevent lost workSynchronize tasks when user switches workstations

days

Review modeProject Management
View what your annotators labeled and correct/confirmSearch for particular tags/errors

days