Repository logo

Infoscience

  • English
  • French
Log In
Logo EPFL, École polytechnique fédérale de Lausanne

Infoscience

  • English
  • French
Log In
  1. Home
  2. Academic and Research Output
  3. Journal articles
  4. Computing Crowd Consensus with Partial Agreement
 
Loading...
Thumbnail Image
research article

Computing Crowd Consensus with Partial Agreement

Nguyen, Quoc Viet Hung  
•
Huynh, Huu Viet
•
Nguyen, Thanh Tam  
Show more
2018
IEEE Transactions on Knowledge and Data Engineering

Crowdsourcing has been widely established as a means to enable human computation at large-scale, in particular for tasks that require manual labelling of large sets of data items. Answers obtained from heterogeneous crowd workers are aggregated to obtain a robust result. However, existing methods for answer aggregation are designed for \emph{discrete} tasks, where answers are given as a single label per item. In this paper, we consider \emph{partial-agreement} tasks that are common in many applications such as image tagging and document annotation, where items are assigned sets of labels. Common approaches for the aggregation of partial-agreement answers either (i) reduce the problem to several instances of an aggregation problem for discrete tasks or (ii) consider each label independently. Going beyond the state-of-the-art, we propose a novel Bayesian nonparametric model to aggregate the partial-agreement answers in a generic way. This model enables us to compute the consensus of partially-sound and partially-complete worker answers, while taking into account mutual relationships in labels and different answer sets. We also show how this model is instantiated for incremental learning, incorporating new answers from crowd workers as they arrive. An evaluation of our method using real-world datasets reveals that it consistently outperforms the state-of-the-art in terms of precision, recall, and robustness against faulty workers and data sparsity.

  • Files
  • Details
  • Metrics
Type
research article
DOI
10.1109/Tkde.2017.2750683
Web of Science ID

WOS:000418067900001

Author(s)
Nguyen, Quoc Viet Hung  
•
Huynh, Huu Viet
•
Nguyen, Thanh Tam  
•
Matthias, Weidlich
•
Hongzhi, Yin
•
Xiaofang, Zhou
Date Issued

2018

Published in
IEEE Transactions on Knowledge and Data Engineering
Volume

30

Issue

1

Start page

1

End page

14

Subjects

Crowdsourcing

•

Nonparametric Models

•

Bayesian Models

•

Answer Aggregation

Peer reviewed

REVIEWED

Written at

OTHER

EPFL units
LSIR  
Available on Infoscience
September 7, 2017
Use this identifier to reference this record
https://infoscience.epfl.ch/handle/20.500.14299/140554
Logo EPFL, École polytechnique fédérale de Lausanne
  • Contact
  • infoscience@epfl.ch

  • Follow us on Facebook
  • Follow us on Instagram
  • Follow us on LinkedIn
  • Follow us on X
  • Follow us on Youtube
AccessibilityLegal noticePrivacy policyCookie settingsEnd User AgreementGet helpFeedback

Infoscience is a service managed and provided by the Library and IT Services of EPFL. © EPFL, tous droits réservés