In typical applications of machine learning (ML), humans typically enter the process at an early stage, in determining an initial representation of the problem and in preparing the data, and at a late stage, in interpreting and making decisions based on the results. Consequently, the bulk of the ML literature deals with such situations. Much less research has been devoted to ML involving “humans-in-the-loop,” where humans play a more intrinsic role in the process, interacting with the ML system to iterate towards a solution to which both humans and machines have contributed. In these situations, the goal is to optimize some quantity that can be obtained only by evaluating human responses and judgments. Examples of this hybrid, “human-in-the-loop” ML approach include:
- ML-based education, where a scheduling system acquires information about learners with the goal of selecting and recommending optimal lessons;
- Adaptive testing in psychological surveys, educational assessments, and recommender systems, where the system acquires testees’ responses and selects the next item in an adaptive and automated manner;
- Interactive topic modeling, where human interpretations of the topics are used to iteratively refine an estimated model;
- Image classification, where human judgments can be leveraged to improve the quality and information content of image features or classifiers.
In this workshop in December 2014, we focused on the emerging new theories, algorithms, and applications of human-in-the-loop ML algorithms.