Design and evaluation of crowdsourcing platforms based on users' confidence judgments.
Crowdsourcing
Group decision making
Majority voting
Meta-cognition
Journal
Scientific reports
ISSN: 2045-2322
Titre abrégé: Sci Rep
Pays: England
ID NLM: 101563288
Informations de publication
Date de publication:
08 Aug 2024
08 Aug 2024
Historique:
received:
10
02
2023
accepted:
25
06
2024
medline:
8
8
2024
pubmed:
8
8
2024
entrez:
7
8
2024
Statut:
epublish
Résumé
Crowdsourcing deals with solving problems by assigning them to a large number of non-experts called crowd using their spare time. In these systems, the final answer to the question is determined by summing up the votes obtained from the community. The popularity of these systems has increased by facilitating access for community members through mobile phones and the Internet. One of the issues raised in crowdsourcing is how to choose people and how to collect answers. Usually, users are separated based on their performance in a pre-test. Designing the pre-test for performance calculation is challenging; The pre-test questions should be selected to assess characteristics in individuals that are relevant to the main questions. One of the ways to increase the accuracy of crowdsourcing systems is by considering individuals' cognitive characteristics and decision-making models to form a crowd and improve the estimation of their answer accuracy to questions. People can estimate the correctness of their responses while making a decision. The accuracy of this estimate is determined by a quantity called metacognition ability. Metacoginition is referred to the case where the confidence level is considered along with the answer to increase the accuracy of the solution. In this paper, by both mathematical and experimental analysis, we would answer the following question: Is it possible to improve the performance of a crowdsourcing system by understanding individuals' metacognition and recording and utilizing users' confidence in their answers?
Identifiants
pubmed: 39112555
doi: 10.1038/s41598-024-65892-7
pii: 10.1038/s41598-024-65892-7
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
18379Informations de copyright
© 2024. The Author(s).
Références
Maniscalco, B. & Lau, H. Signal detection theory analysis of type 1 and type 2 data: meta-d, response-specific meta-d, and the unequal variance sdt model. In The cognitive neuroscience of metacognition (eds Fleming, S. M. & Frith, C. D.) 25–66 (Springer, 2014).
doi: 10.1007/978-3-642-45190-4_3
Howe, J. The rise of crowdsourcing. Wired magazine 14(6), 1–4 (2006).
Li, H., Bin, Yu. & Zhou, D. In ICML Workshop: Error rate analysis of labeling by crowdsourcing (Machine Learning Meets Crowdsourcing. Atalanta, Georgia, USA. Citeseer), (2013).
Sautter, G. & Böhm, K. High-throughput crowdsourcing mechanisms for complex tasks. Soc. Netw. Anal. Min. 3, 873–888 (2013).
doi: 10.1007/s13278-013-0114-z
Fleming, S. M. Hmeta-d: hierarchical bayesian estimation of metacognitive efficiency from confidence ratings. Neurosci. Consciousness 1, nix007 (2017).
Veenman, M. V. J., Elshout, J. J. & Meijer, J. The generality vs domain-specificity of metacognitive skills in novice learning across domains. Learn. Instr. 7(2), 187–209 (1997).
doi: 10.1016/S0959-4752(96)00025-4
Mazancieux, A., Fleming, S. M., Souchay, C. & Moulin, C. Retrospective confidence judgments across tasks: domain-general processes underlying metacognitive accuracy, (2018).
Sheng, X., Tang, J., Xiao, X. & Xue, G. Sensing as a service: Challenges, solutions and future directions. IEEE Sens. J. 13(10), 3733–3741 (2013).
doi: 10.1109/JSEN.2013.2262677
Faggiani, A., Gregori, E., Lenzini, L., Luconi, V. & Vecchio, A. Smartphone-based crowdsourcing for network monitoring: opportunities, challenges, and a case study. IEEE Commun. Mag. 52(1), 106–113 (2014).
doi: 10.1109/MCOM.2014.6710071
Yefeng, L., Lehdonvirta, V., Mieke, K., Todorka, A., Hiroaki, K. & Nakajima, T. A crowdsourcing based mobile image translation and knowledge sharing service. In 9th International Conference on Mobile and Ubiquitous Multimedia, Limassol, Cyprus, (2010).
Ntalianis, K., Tsapatsoulis, N., Doulamis, A. & Matsatsinis, N. Automatic annotation of image databases based on implicit crowdsourcing, visual concept modeling and evolution. Multimed. Tools Appl. 69(2), 397–421 (2014).
doi: 10.1007/s11042-012-0995-2
Von Ahn, L. & Dabbish, L. Labeling images with a computer game. In Proceedings of the SIGCHI conference on Human factors in computing systems, Gaithersburg, Maryland, USA, (2004).
Ribeiro, F., Florencio, D. & Nascimento, V. Crowdsourcing subjective image quality evaluation. In 2011 18th IEEE International Conference on Image Processing. Brussels, Belgium, (2011).
Ghadiyaram, D. & Bovik, A. C. Massive online crowdsourced study of subjective and objective picture quality. IEEE Trans. Image Process. 25(1), 372–387 (2015).
doi: 10.1109/TIP.2015.2500021
pubmed: 26571530
Shin, H., Chon, Y., Kim, Y. & Cha, H. A participatory service platform for indoor location-based services. IEEE Pervasive Comput. 14(1), 62–69 (2015).
doi: 10.1109/MPRV.2015.1
Chen, S., Muyuan, L., Kui, R. & Chunming, Q. Crowd map: Accurate reconstruction of indoor floor plans from crowdsourced sensor-rich videos. In 2015 IEEE 35th International conference on distributed computing systems. Columbus, OH, USA, (2015).
McDuff, D., El Kaliouby, R. & Picard, R. W. Crowdsourcing facial responses to online videos. IEEE Trans. Affective Comp. 3(4), 456 (2012).
doi: 10.1109/T-AFFC.2012.19
Gatautis, R. & Vitkauskaite, E. Crowdsourcing application in marketing activities. Procedia Soc. Behav. Sci. 110, 1243–1250 (2014).
doi: 10.1016/j.sbspro.2013.12.971
Zhang, X. et al. Free market of crowdsourcing: Incentive mechanism design for mobile sensing. IEEE Trans. Parallel Distrib. Syst. 25(12), 3190–3200 (2014).
doi: 10.1109/TPDS.2013.2297112
Liu, W., Yang, Y., Wang, E., Han, Z. & Wang, X. Prediction based user selection in time-sensitive mobile crowdsensing. In 14th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON), San Diego, USA, (2017).
Pu, L., Chen, X., Xu, J. & Fu, X. Crowdlet: Optimal worker recruitment for self-organized mobile crowdsourcing. In IEEE INFOCOM 2016-The 35th Annual IEEE International Conference on Computer Communications. San Francisco, USA, (2016).
Liu, X. et al. Cdas: a crowdsourcing data analytics system. Proc. VLDB Endowment 5(10), 1040–1051 (2012).
doi: 10.14778/2336664.2336676
Nguyen, A. T., Lease, M. & Wallace, B. C. Explainable modeling of annotations in crowdsourcing. In IUI, (2019).
Kuldeep, Y., Ponnurangam, K., Atul, G., Ashish, G. & Vinayak, N. Smsassassin: crowdsourcing driven mobile-based system for sms spam filtering. In Proceedings of the 12th Workshop on Mobile Computing Systems and Applications. ACM, (2011).
Kamar, E., Hacker, S. & Horvitz, E. Combining human and machine intelligence in large-scale crowdsourcing. In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems, Valencia, Spain, 1, (2012).
Yang, J., Drake, T., Damianou, A. & Maarek, Y. Leveraging crowdsourcing data for deep active learning an application: Learning intents in alexa. In World Wide Web Conference, Lyon, France, 23–32, (2018).
Abernethy, J. D. & Frongillo, R. M. A collaborative mechanism for crowdsourcing prediction problems. Adv. Neural Inf. Process. Syst. (2011).
Gadiraju, U., Fetahu, B., Kawase, R., Siehndel, P. & Dietze, S. Using worker self-assessments for competence-based pre-selection in crowdsourcing microtasks. ACM Trans. Comp.-Human Int. (TOCHI) 24(4), 30 (2017).
Zhao, D., Li, X.-Y. & Ma, H. How to crowdsource tasks truthfully without sacrificing utility: Online incentive mechanisms with budget constraint. In IEEE INFOCOM 2014-IEEE Conference on Computer Communications. Toronto, Canada, (2014).
Ran, B. Zheng, X. & Guozhen, T. Optimal assignment for deadline aware tasks in the crowdsourcing. In IEEE International Conferences on Big Data and Cloud Computing (BDCloud), Social Computing and Networking (SocialCom), Sustainable Computing and Communications (SustainCom)(BDCloud-SocialCom-SustainCom), (2016).
Mazor, M., Charles, L., Mor, R. Maimon & Fleming, S. Paradoxical evidence weighting in confidence judgments for detection and discrimination, (2022).
Livingston, J. A. Metacognition: An overview. (2003).
Fleming, S. M. & Daw, N. D. Self-evaluation of decision-making: A general bayesian framework for metacognitive computation. Psychol. Rev. 124(1), 91 (2017).
doi: 10.1037/rev0000045
pubmed: 28004960
pmcid: 5178868
Maniscalco, B. & Lau, H. A signal detection theoretic approach for estimating metacognitive sensitivity from confidence ratings. Conscious. Cogn. 21(1), 422–430 (2012).
doi: 10.1016/j.concog.2011.09.021
pubmed: 22071269
Fleming, S. M. & Lau, H. C. How to measure metacognition. Front. Hum. Neurosci. 8, 443 (2014).
doi: 10.3389/fnhum.2014.00443
pubmed: 25076880
pmcid: 4097944
Baird, B., Smallwood, J., Gorgolewski, K. J. & Margulies, D. S. Medial and lateral networks in anterior prefrontal cortex support metacognitive ability for memory and perception. J. Neurosci. 33(42), 16657–16665 (2013).
doi: 10.1523/JNEUROSCI.0786-13.2013
pubmed: 24133268
pmcid: 6618531
Sadeghi, S., Ekhtiari, H., Bahrami, B. & Ahmadabadi, M. N. Metacognitive deficiency in a perceptual but not a memory task in methadone maintenance patients. Sci. Rep. 7(1), 1–8 (2017).
doi: 10.1038/s41598-017-06707-w
Klaus, F., Rakefet, A. & Chiara, S. Metacognition: Monitoring and controlling one’s own knowledge, reasoning and decisions. The psychology of human thought: An introduction, 89–111, (2019).