doi: 10.17706/jsw.18.3.117-129
The Effect of Performance-Based Compensation on Crowdsourced Human-Robot Interaction Experiments
Abstract—Social scientists have long been interested in the relationship between financial incentives and performance. This subject has gained new relevance with the advent of web-based “crowd-sourcing” models of production. In recent decades, recruiting participants from crowd-sourcing platforms has gained considerable popularity among human-robot trust researchers. A large number of outliers due to lack of enough attention and focus of the experiment participants or due to participants' boredom and low engagement, especially in crowd-sourcing experiments, has always been a concern for researchers in the field of human-robot interaction. To overcome this problem, financial incentives can be a solution. In this study, we examine the effects of performance-related compensation on the experiment data quality, participants' performance, and accuracy in performing the assigned task, and experiment results in the context of a human-robot trust experiment. We designed an online human-robot collaborative search task and recruited 120 participants for this experiment from Amazon's Mechanical Turk (AMT). We tested participants' attention, performance, and trust in the robotic teammate under two conditions: constant payment and performance-based payment conditions. We found that using financial incentives can increase the data quality and help prevent the random and aimless behavior of the participants. We also found that financial incentives can improve the participants' performance significantly and causes participants to put more effort into performing the assigned task. However, it does not affect the experiment results unless any measures are directly associated with the compensation value.
Index Terms—Amazon mechanical turk, crowdsourcing, financial incentive, human-robot interaction, human-robot trust, MTurk
Cite: Zahra Rezaei Khavas, Monish Reddy Kotturu, Russell Purkins, S. Reza Ahmadzadeh, Paul Robinette, "The Effect of Performance-Based Compensation on Crowdsourced Human-Robot Interaction Experiments," Journal of Software vol. 18, no. 3, pp. 117-129, 2023.
Copyright © 2023 by the authors. This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited (CC BY 4.0)
General Information
-
Apr 26, 2021 News!
Vol 14, No 4- Vol 14, No 12 has been indexed by IET-(Inspec) [Click]
-
Nov 18, 2021 News!
Papers published in JSW Vol 16, No 1- Vol 16, No 6 have been indexed by DBLP [Click]
-
Dec 24, 2021 News!
Vol 15, No 1- Vol 15, No 6 has been indexed by IET-(Inspec) [Click]
-
Nov 18, 2021 News!
[CFP] 2022 the annual meeting of JSW Editorial Board, ICCSM 2022, will be held in Rome, Italy, July 21-23, 2022 [Click]
-
Aug 01, 2023 News!