Instructor toolkit
Use our learning materials to hold a crowdsourcing workshop at your institution! It can be a one-time event or you can include it in your course on Machine Learning, Crowdsourcing, Research Methods, or any other.
Introduction to Crowdsourcing
Here’s a suggested plan for an introductory workshop about crowdsourcing and its basic principles. It consists of a theoretical part, which covers the main aspects of the crowdsourcing approach, and a practical task where students follow a step-by-step guide and launch a real crowdsourcing project.
Goals, outcomes and assessment
Here’s a suggested plan for an introductory workshop about crowdsourcing and its basic principles. It consists of a theoretical part, which covers the main aspects of the crowdsourcing approach, and a practical task where students follow a step-by-step guide and launch a real crowdsourcing project.
The goal
Give students a basic understanding of the principles and applications of crowdsourcing.
Expected outcomes
Students are able to:
- Explain what crowdsourcing is.
- Explain what role crowdsourcing plays in ML and related fields.
- Tell what parts a typical crowdsourcing project consists of.
- Run a simple crowdsourcing project.
To assess whether these outcomes have been achieved, we recommend:
- Holding a quiz that covers the theoretical part of the workshop.
- Checking the quality of labels collected by students by comparing them to benchmark labels.
Teaching scenarios
| Lecture | Data for ML | 10 mins |
| Introduction to the crowdsourcing approach | 10 mins | |
| Instructions & interfaces | 15 mins | |
| Practice | Setting up a project in Toloka – step 1 of the guide | 30 mins |
| Lecture | Quality control | 10 mins |
| Practice | Creating a task pool on Toloka – steps 2-5 of the guide | 30 mins |
| Lecture | Result aggregation | 10 mins |
| Case study | 10 mins | |
| Practice | Getting results step 6 of the guide | 10 mins |
| Reflection | Discussing the perspectives and applications of the crowdsourcing approach | 10 mins |
| Assessment | Quiz | |
| Assessment | Checking results (contact us for details) |
Reach out to request learning materials, set up quality assessment and get extra support