Open datasets

This dataset of 10,000 dialogues will help dialogue systems researchers to develop approaches for training chat bots. Created in collaboration with iPavlov, MIPT’s Neural Networks and Deep Learning Lab’s project for conversational AI research (which also develops DeepPavlov, an open-source library for chatbot technology), this dataset contains profiles of imaginary personalities with descriptions and dialogues between participants who are given a random profile and instructed to mimic a described personality. A chat bot that is trained on the dataset will be able to communicate on behalf of a certain persona and get to know people by chatting with them on general topics.

This dataset, assembled by Dmitry Ustalov in 2017, contains human-annotated sense identifiers for 2,562 contexts of 20 words used in the RUSSE’2018 shared task on Word Sense Induction and Disambiguation for the Russian language. 2,562 contexts were annotated by humans trained on 80 pre-annotated contexts so that each context was annotated by nine different annotators. After the annotation, every context was additionally inspected (curated) by the organizers of the shared task.

Researchers can use this dataset to explore different methods of quality control in crowdsourcing. The dataset contains around 0.5 million anonymized crowdsourced labels that were collected in the Relevance 2 Gradations project in 2016. It includes the labels from individual performers and golden labels that help to measure the quality of their answers. The dataset contains anonymized information about how performers evaluated a particular document, and in some cases, whether their evaluation was correct. By studying this dataset, you can find out how the opinion of individual performers affects the quality of the final assessment, what aggregation model is most effective, and how many opinions you need in order to get an accurate evaluation.

The key quality metric is accuracy of aggregated labels, which is estimated as the percentage of the aggregated labels that match the golden labels for the golden set.

This dataset is similar to the previous one, but rather than a binary choice for rating label relevance, it uses a five-point scale in the Relevance 5 Gradations project. The task was to assess the relevance of a document for a query on a 5-point scale. Some tasks in this dataset have more than one golden label. In these cases, all the golden labels are considered equally correct.

The key quality metric is accuracy of aggregated labels, which is estimated as the percentage of the aggregated labels that match one of the golden labels for a given task from the golden set. In addition to the crowdsourced labels, there is also information about performers who were banned for a certain reason. For each banned performer, the reason for banning is provided as one out of four ban types (details about each ban type are not given). The dataset contains more than 1 million labels.

This dataset, assembled by Dmitry Ustalov in 2017, contains the opinions of Russian native speakers about the relationship between a generic term (hypernym) and a specific instance of this term (hyponym). A set of 300 most frequent nouns was extracted from the Russian National Corpus. Then each method or resource (including RuThes and RuWordNet) produced at most five hypernyms, if possible. This resulted in 10,600 unique non-empty subsumption pairs, which were annotated by seven different performers whose mother tongue was Russian and who were at least 20 years old as of February 1, 2017. As a result, 4,576 out of 10,600 pairs were annotated as positive, while the remaining 6,024 were annotated as negative. Interestingly, the performers were more confident in the negative answers than in the positive ones.

For this dataset, commissioned by the Yandex Business Directory, we prepared 10,000 photos of organizations’ information signs and a text file with INN (Taxpayer ID) and OGRN (Main State Registration Number) codes shown on these signs. The computer vision model can learn from this data to recognize these number sequences in images. First we launched the task in the Yandex.Toloka mobile application, where we asked performers to go to the address marked on the map, find the specified organization and take a photo of its information sign. Field tasks like this help us to keep the Yandex Business Directory information updated. Then the quality of completed tasks was checked by other performers. The photos containing INN and OGRN codes were sent for reсognition. Yandex.Toloka performers typed the numbers from the photos, and then we processed the results and formed a dataset. Download Sample 

This dataset contains about 60,000 crowdsourced labels for 1,000 tasks and ground truth labels for almost all the tasks. The task was to classify websites into five categories based on the presence of adult content. Additionally, each task has 52 real-valued features that can be used to predict the category. The key quality metric is accuracy of aggregated labels, which is estimated as the percentage of aggregated labels that match the golden labels for the golden set.