Fractal

Instructor toolkit

Use our learning materials to hold a crowdsourcing workshop at your institution! It can be a one-time event or you can include it in your course on Machine Learning, Crowdsourcing, Research Methods, or any other.

Introduction to Crowdsourcing

Here’s a suggested plan for an introductory workshop about crowdsourcing and its basic principles. It consists of a theoretical part, which covers the main aspects of the crowdsourcing approach, and a practical task where students follow a step-by-step guide and launch a real crowdsourcing project.

Goals, outcomes and assessment

Here’s a suggested plan for an introductory workshop about crowdsourcing and its basic principles. It consists of a theoretical part, which covers the main aspects of the crowdsourcing approach, and a practical task where students follow a step-by-step guide and launch a real crowdsourcing project.

The goal

Give students a basic understanding of the principles and applications of crowdsourcing.

Expected outcomes

Students are able to:

  • Explain what crowdsourcing is.

  • Explain what role crowdsourcing plays in ML and related fields.

  • Tell what parts a typical crowdsourcing project consists of.

  • Run a simple crowdsourcing project.

To assess whether these outcomes have been achieved, we recommend:

  • Holding a quiz that covers the theoretical part of the workshop.
  • Checking the quality of labels collected by students by comparing them to benchmark labels.

Teaching scenarios

LectureData for ML10 mins
Introduction to the crowdsourcing approach10 mins
Instructions & interfaces15 mins
PracticeSetting up a project in Toloka – step 1 of the guide30 mins
LectureQuality control10 mins
PracticeCreating a task pool on Toloka – steps 2-5 of the guide30 mins
LectureResult aggregation10 mins
Case study10 mins
PracticeGetting results step 6 of the guide10 mins
ReflectionDiscussing the perspectives and applications of the crowdsourcing approach10 mins
AssessmentQuiz
AssessmentChecking results (contact us for details)

Tutorials

Launch a crowdsourcing project by following a step-by-step guide: classifying images, text, and audio

Reach out to request learning materials, set up quality assessment and get extra support

Contact us
Fractal