Tutorials for popular tasks
The tutorials will show you how to post tasks in Toloka. Choose the tutorial that is most similar to your task.
Before you start
Make sure you're registered as a requester. We recommend that you go through tutorials in the sandbox.
List of examples
-
Images
-
Image classification — A multiple choice task. Examples are moderating content or grouping images by category.
-
Side-by-side image comparison — The Toloker compares pairs of images, illustrations, or interfaces.
-
Object recognition and area selection — The Toloker labels the areas on an image that contain the specified object. These tasks are easier to run in the web version.
-
Object recognition and area selection (example with decomposition) — An example of dividing a task for selecting objects into three projects: filtering out inappropriate images, selecting objects, and reviewing assignments.
-
-
Video
- Video classification — A multiple choice task. For example, moderation of content, evaluation of video for noise and defects, or video classification by specified categories.
-
Audio
-
Audio transcription — The Toloker receives an audio recording and types the text they hear. Checking such tasks can be assigned to other Tolokers as a classification task.
-
Audio classification — A multiple choice task. For example, evaluation of audio recordings for noise and defects, or audio classification by specified categories.
-
Voice recording — The Toloker receives a phrase or text fragment to record and has to attach the audio file as the response. These tasks are easier to run in the mobile app for Android and iOS.
-
-
Texts
- Sentiment analysis and content moderation — The Toloker checks the text for compliance with the rules.
-
Data enrichment
-
Searching for product characteristics — A task in which Tolokers search for data about businesses online.
-
Product photo search (example with decomposition) — Type of task for searching and processing relevant data. For example, find similar shoes in an online store or determine which of the found objects is more similar to the original one.
-
-
Surveys
-
Survey with Toloka template — Create a form with questions and run a survey in Toloka.
-
Survey on third-party platform — Use Toloka to run a survey created on any third-party platform.
-
Survey based on Qualtrics — Use Toloka to conduct your academic Qualtrics-based survey.
-
-
Field tasks
Field tasks are completed in the Toloka mobile app for Android and iOS.
-
Collect offline data — The Toloker goes to a specified point on the map, checks something, and takes photos.
-
Monitor prices — The Toloker goes to the location and takes photos of the store's front, a particular product on the shelf, and the price tag.
-
Monitor products — The Toloker goes to the store, takes a photo of the front, finds the relevant section inside, and takes a photo of the rack with a certain brand of products.
-
Collect data on businesses — The Toloker goes to the point, takes a picture of the outside of the organization, and photographs a particular object inside from different angles.
-
Monitoring field objects — The Toloker goes to the point, finds an object on the street, photographs it from different angles, and takes a picture of the address sign.
-
-
Level 1. Tutorials using a single project are suitable for beginners. Go through at least one tutorial that is most similar to your task.
-
Image classification — A multiple choice task. Examples are moderating content or grouping images by category.
-
Side-by-side image comparison — The Toloker compares pairs of images, illustrations, or interfaces.
-
Object recognition and area selection — The Toloker labels the areas on an image that contain the specified object. These tasks are easier to run in the web version.
-
Audio transcription — The Toloker receives an audio recording and types the text they hear. Checking such tasks can be assigned to other Tolokers as a classification task.
-
Audio classification — A multiple choice task. For example, evaluation of audio recordings for noise and defects, or audio classification by specified categories.
-
Voice recording — The Toloker receives a phrase or text fragment to record and has to attach the audio file as the response. These tasks are easier to run in the mobile app for Android and iOS.
-
Video classification — A multiple choice task. For example, moderation of content, evaluation of video for noise and defects, or video classification by specified categories.
-
Sentiment analysis and content moderation — The Toloker checks the text for compliance with the rules.
-
Survey with Toloka template — Set requirements for respondents and take a poll.
-
Survey on third-party platform — Use Toloka to run a survey created on any third-party platform.
-
Survey based on Qualtrics — Use Toloka to conduct your academic Qualtrics-based survey.
-
Searching for product characteristics — A task in which Tolokers search for data about businesses online.
-
-
Level 2. Field tutorials — If you need to post field tasks, go through a tutorial at this level. If not, you can skip it.
Field tasks are completed in the Toloka mobile app for Android and iOS.
-
Collect offline data — The Toloker goes to a specified point on the map, checks something, and takes photos.
-
Monitor prices — The Toloker goes to the location and takes photos of the store's front, a particular product on the shelf, and the price tag.
-
Monitor products — The Toloker goes to the store, takes a photo of the front, finds the relevant section inside, and takes a photo of the rack with a certain brand of products.
-
Collect data on businesses — The Toloker goes to the point, takes a picture of the outside of the organization, and photographs a particular object inside from different angles.
-
Monitoring field objects — The Toloker goes to the point, finds an object on the street, photographs it from different angles, and takes a picture of the address sign.
-
-
Level 3. Tutorials using several projects — Learn how to decompose a task.
-
Object recognition and area selection (example with decomposition) — An example of dividing a task for selecting objects into three projects: filtering out inappropriate images, selecting objects, and reviewing assignments.
-
Product photo search (example with decomposition) — Type of task for searching and processing relevant data. For example, find similar shoes in an online store or determine which of the found objects is more similar to the original one.
-
If you didn't find a suitable example, use general instructions for launching and setting up projects.
You can contact us or our partners if you need help with your project overall or with individual steps like editing tasks.
Learn more about how to set up pricing.
Troubleshooting
You can create such an assignment based on the classification template. Show a product image and ask the question: “Does the product match the query?” Add two radio buttons for responses: “Yes” and “No”.
Generate the screenshots of pages and manually mark up areas using the “Object selection in an image” template.
To create a shortcut, add the following action to the "onKey" method:
onKey: function(key) {
var el = this.getDOMElement().querySelector(".image-annotation-editor__shape-polygon");
if (key === 'D') {
el.click();
el.classList.add('image-annotation-editor__shape_active')
}
Use the C shortcut for closing areas.
You can also use the library to customize the keyboard shortcuts for your tasks.
We recommend that you break down your object selection project into three projects in Toloka:
-
Sorting images containing an object.
-
Create a task using the “Image classification” template.
-
Sort the images containing the object you are looking for.
-
Show the image to the Toloker and ask if the object is in the image. Response options: Yes/No.
-
-
Selecting objects in images.
-
Select the object in the images you obtained after the previous project. You already have such a project. Run the task with manual review.
-
Use the quality control rules: fast responses, manual review, and post-review re-assessment. Description of rules with examples.
-
-
Reviewing object selection assignments.
-
Create a task using the object selection template.
-
Hide the editor and ask whether the object is selected correctly. Response options: Yes/No.
-
In the input data, pass the images and coordinates of the labeled objects from the previous task.
Now you can run the resulting pool with an overlap of 3–5 or with dynamic overlap. After that, you can aggregate the results and then upload the data for review to Project 2.
To prevent the Tolokers who worked on the second project from doing the review, assign a skill to them. Use this skill as a filter in the pools of the third project.
-
Main options:
-
Select an arbitrary area in the image (for example, put a square in the upper-right corner). In this case, the project instructions for reviewers should also reflect this.
-
Ask the Toloker to skip the task and report it in a personal message. Messages are reviewed by the requester. If the object is truly missing, the task is deleted from the pool by resetting the overlap.
-
Add an additional “No object” checkbox to the interface. Make sure that your JS checks that either the object is selected or the checkbox is enabled. In this case, add information about the checkbox value in the review task interface.
In the case of crowdsourcing, it's better to break down this task. The simpler the task, the cheaper it is and the better the quality of the final result. The cost of labeling a single class of objects in photos might be about $0.01.
Base your task on the “Object selection in an image” template. See the step-by-step guide for creating this type of project on this page.
The editor used in the template lets you add a drop-down list for labeling the selected object. See how to do this in the editor description (Dropdown list tab).
You can create a selection + drop-down list with category selection. See how to do this in the editor description (Dropdown list tab).
The coordinates are relative to the image.
In the standard template with an area selection editor, you can't use the control tasks, because in order for the assignment to be accepted by the system as correct, the object selected by the Toloker must exactly match the control object. This is almost impossible. Therefore, you can leave the GOLDEN field empty in the task file or simply delete all the columns except INPUT.
In the standard template with an area selection editor, you can't use the control tasks, because in order for the assignment to be accepted by the system as correct, the object selected by the Toloker must exactly match the control object. This is almost impossible. Therefore, you can leave the GOLDEN field empty in the task file or simply delete all the columns except INPUT.
You can't use training and the main pool with the Training type in an area selection project because for the response to be correct, the object selected by the Toloker must exactly match the control object. This is almost impossible.
Such tasks are usually run with manual review: the Toloker submits an assignment, and then the assignment is rejected or accepted after the review.
For pre-selection of users, you can use “examination tasks”. Review the assignments and assign skills based on the percentage of accepted assignments. For this purpose, add the “Results of assignment review” rule to the pool. To make sure that only the good Tolokers are admitted to the main pool, put a skill-based filter to the pool.
-
Use the Image comparison (Side-by-side) template.
-
In the file with tasks, specify the links to the compared images.
-
In the file, create the tasks where all the images will be compared in pairs:
-
Image 1 and Image 2
-
Image 1 and Image 3
-
Image 2 and Image 3
-
-
Process the results.
You can also edit the project so that the Toloker sees 3 images at once and selects one of them.
You can structure your text classification task using the source text and radio buttons. It can also be text and checkboxes if you use multiple subjects.
We recommend that you base it on the “Text classification” template.
You can also create your own interface. All the available tools are described in the documentation.
You can see how it's implemented in the “Transcript of audio recordings” template where a string-type output field is used. In the “Text recognition from an image” template, you can view how to describe text input fields. Please note that if you later want to use the Dawid-Skene aggregation method, you must specify allowed values.
No, but you can add links to them.
To the component that inserts the image, add the parameters: real-size=true
and screenshot=true
.
-
Create a project from an empty template.
-
Write your question in the HTML block.
-
If you need an extensive response, add the required number of text entry components.
-
If you need a short response, add the required number of string entry components.
-
Come up with a name for each of the components and create a matching number of string-type output fields with the same names. They will be used to save responses.
-
Make all the output fields mandatory.
-
Make the input field auxiliary. It will only be used to create a TSV file with tasks.
-
Come up with a name for the input field and set the string type for it (see the step-by-step guide on creating a survey).
The survey includes an auxiliary input field. You can use it to pass any information, and it won't be visible to Tolokers.
Upload one task and use the overlap parameter to enter the number of people you want to survey.
If your task contains many objects of different types, you should break it down. For example, you can ask users to select numbers in the first project, doorways and windows in the second project, walls in the third project, and plumbing in the fourth project.
The simpler the task, the cheaper it is and the better the quality of the final result. Set the cost of labeling a single class of objects in photos at about $0.01.
Use the Object selection in an image template. You can open this template in the editor and add a drop-down list for labeling the selected object. See how to do this in the editor description (Dropdown list tab).
You can use JavaScript to add assignment validation depending on a checkbox. An example is provided in the “Search for information online” template.
To open the camera instead of the gallery when the Toloker taps the image upload button, in the Image upload button component specify camera=true
.
In the mobile apps, Tolokers can add photos from the default gallery (iOS) or Google photo (Android). To limit the capacity of adding online images, specify in the component requiredCoordinates=true
. In this case, the system won't let the Toloker add images without geotags.