Public datasets

This dataset was collected for the KDD'2020 paper «Prediction of Hourly Earnings and Completion Time on a Crowdsourcing Platform». This dataset contains over 18 million tasks performed by 161,377 users in Yandex.Toloka. Data was collected over a 3-month period (September-November, 2018) to include all sessions on the main page of the web platform where a user chose a task, and all tasks that were completed by users. The session data includes the timestamp, user ID, ID of the project chosen by the user, ID of the task assigned to the user, and additional information. The task data includes start and completion timestamps, the project ID, the ID of the assigned user, the reward for completing the task, the number of micro tasks involved in the task, the amount of input data and output data, and additional information. Some static characteristics of users and projects are also provided, such as user registration dates and the length of project instructions. The data has been anonymized: no real internal ids, personal user data, proprietary requester data (including task results), or textual info (including project names and descriptions) have been used.

For this dataset, commissioned by the Yandex Business Directory, we prepared 10,000 photos of organizations’ information signs and a text file with INN (Taxpayer ID) and OGRN (Main State Registration Number) codes shown on these signs. The computer vision model can learn from this data to recognize these number sequences in images. First we launched the task in the Yandex.Toloka mobile application, where we asked performers to go to the address marked on the map, find the specified organization and take a photo of its information sign. Field tasks like this help us to keep the Yandex Business Directory information updated. Then the quality of completed tasks was checked by other performers. The photos containing INN and OGRN codes were sent for reсognition. Yandex.Toloka performers typed the numbers from the photos, and then we processed the results and formed a dataset. Download Sample 

This dataset of 10,000 dialogues will help dialogue systems researchers to develop approaches for training chat bots. Created in collaboration with iPavlov, MIPT’s Neural Networks and Deep Learning Lab’s project for conversational AI research (which also develops DeepPavlov, an open-source library for chatbot technology), this dataset contains profiles of imaginary personalities with descriptions and dialogues between participants who are given a random profile and instructed to mimic a described personality. A chat bot that is trained on the dataset will be able to communicate on behalf of a certain persona and get to know people by chatting with them on general topics.

This dataset contains about 60,000 crowdsourced labels for 1,000 tasks and ground truth labels for almost all the tasks. The task was to classify websites into five categories based on the presence of adult content. Additionally, each task has 52 real-valued features that can be used to predict the category. The key quality metric is accuracy of aggregated labels, which is estimated as the percentage of aggregated labels that match the golden labels for the golden set.

Researchers can use this dataset to explore different methods of quality control in crowdsourcing. The dataset contains around 0.5 million anonymized crowdsourced labels that were collected in the Relevance 2 Gradations project in 2016. It includes the labels from individual performers and golden labels that help to measure the quality of their answers. The dataset contains anonymized information about how performers evaluated a particular document, and in some cases, whether their evaluation was correct. By studying this dataset, you can find out how the opinion of individual performers affects the quality of the final assessment, what aggregation model is most effective, and how many opinions you need in order to get an accurate evaluation.

The key quality metric is accuracy of aggregated labels, which is estimated as the percentage of the aggregated labels that match the golden labels for the golden set.

This dataset is similar to the previous one, but rather than a binary choice for rating label relevance, it uses a five-point scale in the Relevance 5 Gradations project. The task was to assess the relevance of a document for a query on a 5-point scale. Some tasks in this dataset have more than one golden label. In these cases, all the golden labels are considered equally correct.

The key quality metric is accuracy of aggregated labels, which is estimated as the percentage of the aggregated labels that match one of the golden labels for a given task from the golden set. In addition to the crowdsourced labels, there is also information about performers who were banned for a certain reason. For each banned performer, the reason for banning is provided as one out of four ban types (details about each ban type are not given). The dataset contains more than 1 million labels.

This dataset consists of 9515 tweets describing health issues. Each tweet is labeled for whether it contains information about an adverse side effect that occurred when taking a drug. Twitter's privacy policy does not allow storing and distributing tweets in the public domain, so the files only contain the tweet ID, class number, and a script for collecting the source text. The dataset was created as part of The Social Media Mining for Health Applications (#SMM4H) Shared Tasks in a competition for automatically extracting information about the side effects of drugs from tweets. The dataset was a joint effort with the UPenn HLP Center and the Chemoinformatics and Molecular Modeling Research Laboratory at Kazan Federal University.

This dataset, assembled by Dmitry Ustalov in 2017, contains the opinions of Russian native speakers about the relationship between a generic term (hypernym) and a specific instance of this term (hyponym). A set of 300 most frequent nouns was extracted from the Russian National Corpus. Then each method or resource (including RuThes and RuWordNet) produced at most five hypernyms, if possible. This resulted in 10,600 unique non-empty subsumption pairs, which were annotated by seven different performers whose mother tongue was Russian and who were at least 20 years old as of February 1, 2017. As a result, 4,576 out of 10,600 pairs were annotated as positive, while the remaining 6,024 were annotated as negative. Interestingly, the performers were more confident in the negative answers than in the positive ones.

This dataset, assembled by Dmitry Ustalov in 2017, contains human-annotated sense identifiers for 2,562 contexts of 20 words used in the RUSSE’2018 shared task on Word Sense Induction and Disambiguation for the Russian language. 2,562 contexts were annotated by humans trained on 80 pre-annotated contexts so that each context was annotated by nine different annotators. After the annotation, every context was additionally inspected (curated) by the organizers of the shared task.