If everything is OK, close the preview tab. If not, check the data you inserted in the code blocks.
Save the changes.
Write instructions for Tolokers:
Instructions:
Look at the image and answer the question: **Are all traffic signs outlined correctly?**
If there are, click **Yes**.
If there isn't, click **No**.
For example, road signs are outlined correctly, so the correct answer is **Yes**.
Tip
If you want to add a task examples in the instruction, complete it yourself in the preview mode. Take screenshots, upload them to photo hosting or cloud storage and insert image links in the instructions by clicking the button on the toolbar.
Click Create project.
Create a pool
Open the project page titled Are the traffic signs outlined correctly?.
Click Add a pool on the project page.
Set the Pool name (visible only to you) field. Only you will see this pool name on the project page.
Specify the pool description which will be displayed instead of the project description in the task list for Tolokers. By default, Tolokers see the description from the project settings. To use a different description, uncheck the Use project description box and set Public description. If necessary, click + Private comment to add a private project description that only you will see.
Click Create.
At the Select the audience for your task step, set up filters to select Tolokers for your pool.
Use the Languages filter to select Tolokers with knowledge of the language.
Clear My tasks may contain shocking or pornographic content if your project has none of those.
Tasks in pools will be automatically available in the web version of Toloka and the mobile app. If you want to change the default settings and limit the visibility of the task for any of the versions, add the Client filter and select the desired value: Toloka web version or Toloka for mobile.
Add another filter. Click Add filter.
Find the Skills block in the list and select the My skills skill.
In the Skills field, select Area selection.
In the ? field specify =.
Leave the Missing field blank.
(optional) In the Speed/quality balance section, specify the desired quality level. Improving quality may reduce the speed of task completion because the pool will be available to fewer Tolokers. Learn more in the Speed/quality balance section.
At the Set the task price and overlap step, set up how much a single task will cost for you.
In Price per task suite, $, set the amount of money to pay per task suite done by one Toloker. For example, 0.01.
Enter 3 in the Overlap field.
At the Setup quality control step, set quality control rules for more accurate results:
Click Add a quality control rule.
Find the Rules block in the list and choose Majority vote.
In the Accept as majority field, soecify 2.
Set the rule: if number of responses≥ 10 and correct responses (%)< 50, then ban the Toloker on project for 10 days. Specify the reason Doesn't match the majority.
Note
The rule takes effect when the number of responses for the task is equal to the overlap.
You can prepare the file in Notepad. To transfer data to Microsoft Excel, use Text Import Wizard and disable the Text qualifier option.
Upload the file to the Are the traffic signs outlined correctly? pool:
Open the Are the traffic signs outlined correctly? pool.
Click Upload.
Click Drop file here or select, and upload the file you’ve just made.
Click Continue.
Choose Set manually.
In the Tasks per suite field, specify 10.
Click Combine tasks into suites.
On the pool page, click Preview. Make sure that the task displays images with the selected objects.
Tip
If there are no outlined objects, make sure that quotes are properly used in the INPUT:selection column in the file. Make sure that the interface parameters are set correctly.
Since the manual review option is enabled in the pool settings for project 2, you should check the Tolokers' responses within the time limit set in the Review period field.
You can check the results in two ways:
In the results file.
In the pool interface.
Review assignments in the results file
Review assignments in the pool interface
In the text or spreadsheet editor, open the file you received after aggregating the results.
Prepare the file:
Add a column named ACCEPT:verdict with the review results.
Add a column named ACCEPT:comment with comments for Tolokers if responses were rejected. For example, comment on which part of the instructions wasn't followed.
Rename the INPUT:assignment_id column to ASSIGNMENT:assignment_id.
Fill in the ACCEPT:verdict: and ACCEPT:comment: columns:
If the aggregate result of the assignment is OK, put + to accept it.
If the aggregate result of the assignment is incorrect or it doesn't open, put - to reject it. Enter the reason for rejecting the task in the ACCEPT:comment: field, for example, The object isn't selected or is selected incorrectly.
Tip
You can use the awk commands to outline images on Linux and MacOS devices:
awk 'BEGIN {FS=OFS="\t";} NR>1 {if($4~"OK"){ print $1, "+", ""; }else{ print $1, "-", "The object isn't selected or is selected incorrectly.";}}' <post_accept_res>.tsv > <review_res>.tsv
In the window that opens, choose the results file you want to upload and click Open.
In the window that opens, compare the number of tasks in the Processed successfully and Total submitted fields on the pool page.
Click Add.
In the window that opens, click Close.
When setting up a pool in the second project you turned on the Recompletion of the rejected tasks option.
In this case, the pool automatically reopens and the assignments are reassigned to other Tolokers. When they're completed, send the results for review. Then download the results, check them, and upload the reviewed results. You can reject assignments as many times as you want to get more accurate results.
Hover over the line of the assignment you want to check.
In the Status column, you will see the buttons for accepting () or rejecting () the completed assignment. If you reject the assignment, enter a comment in the window that opens and click Done.