Search results

Over the past decade, many research works in various disciplines have benefited from the endless ocean of people and their potentials (in the form of crowdsourcing) as an effective problem-solving strategy and computational model. But nothing interesting is ever completely one-sided. Therefore, when it comes to leveraging people's power, as the dark side of crowdsourcing, there are some possible threats that have not been considered as should be, such as recruiting black hat crowdworkers for organizing targeted adversarial intentions. The purpose of this paper is to draw more attention to this critical issue through investigation of its different aspects.

To delve into details of such malicious intentions, the related literature and previous researches have been studied. Then, four major typologies for adversarial crowdsourced attacks as well as some real-world scenarios are discussed and delineated. Finally, possible future threats are introduced.

Despite many works on adversarial crowdsourcing, there are only a few specific research studies devoted to considering the issue in the context of cyber security. In this regard, the proposed typologies (and addressed scenarios) for such human-mediated attacks can shed light on the way of identifying and confronting such threats.

To the best of the authors' knowledge, this the first work in which the titular topic is investigated in detail. Due to popularity and efficiency of leveraging crowds' intelligence and efforts in a wide range of application domains, it is most likely that adversarial human-driven intentions gain more attention. In this regard, it is anticipated that the present research study can serve as a roadmap for proposing defensive mechanisms to cope with such diverse threats.

As a relatively new computing paradigm, crowdsourcing has gained enormous attention in the recent decade. Its compliance with the Web 2.0 principles, also, puts forward unprecedented opportunities to empower the related services and mechanisms by leveraging humans’ intelligence and problem solving abilities. With respect to the pivotal role of search engines in the Web and information community, this paper aims to investigate the advantages and challenges of incorporating people – as intelligent agents – into search engines’ workflow.

To emphasize the role of the human in computational processes, some specific and related areas are studied. Then, through studying the current trends in the field of crowd-powered search engines and analyzing the actual needs and requirements, the perspectives and challenges are discussed.

As the research on this topic is still in its infancy, it is believed that this study can be considered as a roadmap for future works in the field. In this regard, current status and development trends are delineated through providing a general overview of the literature. Moreover, several recommendations for extending the applicability and efficiency of next generation of crowd-powered search engines are presented. In fact, becoming aware of different aspects and challenges of constructing search engines of this kind can shed light on the way of developing working systems with respect to essential considerations.

The present study was aimed to portrait the big picture of crowd-powered search engines and possible challenges and issues. As one of the early works that provided a comprehensive report on different aspects of the topic, it can be regarded as a reference point.

Allocation of complex crowdsourcing tasks, which typically include heterogeneous attributes such as value, difficulty, skill required, effort required and deadline, is still a challenging open problem. In recent years, agent-based crowdsourcing approaches focusing on recommendations or incentives have emerged to dynamically match workers with diverse characteristics to tasks to achieve high collective productivity. However, existing approaches are mostly designed based on expert knowledge grounded in well-established theoretical frameworks. They often fail to leverage on user-generated data to capture the complex interaction of crowdsourcing participants’ behaviours. This paper aims to address this challenge.

The paper proposes a policy network plus reputation network (PNRN) approach which combines supervised learning and reinforcement learning to imitate human task allocation strategies which beat artificial intelligence strategies in this large-scale empirical study. The proposed approach incorporates a policy network for the selection of task allocation strategies and a reputation network for calculating the trends of worker reputation fluctuations. Then, by iteratively applying the policy network and reputation network, a multi-round allocation strategy is proposed.

PNRN has been trained and evaluated using a large-scale real human task allocation strategy data set derived from the Agile Manager game with close to 500,000 decision records from 1,144 players in over 9,000 game sessions. Extensive experiments demonstrate the validity and efficiency of computational complex crowdsourcing task allocation strategy learned from human participants.

The paper can give a better task allocation strategy in the crowdsourcing systems.

Human or machine, which one is more intelligent and powerful for performing computing and processing tasks? Over the years, researchers and scientists have spent significant amounts of money and effort to answer this question. Nonetheless, despite some outstanding achievements, replacing humans in the intellectual tasks is not yet a reality. Instead, to compensate for the weakness of machines in some (mostly cognitive) tasks, the idea of putting human in the loop has been introduced and widely accepted. In this paper, the notion of collective hybrid intelligence as a new computing framework and comprehensive.

According to the extensive acceptance and efficiency of crowdsourcing, hybrid intelligence and distributed computing concepts, the authors have come up with the (complementary) idea of collective hybrid intelligence. In this regard, besides providing a brief review of the efforts made in the related contexts, conceptual foundations and building blocks of the proposed framework are delineated. Moreover, some discussion on architectural and realization issues are presented.

The paper describes the conceptual architecture, workflow and schematic representation of a new hybrid computing concept. Moreover, by introducing three sample scenarios, its benefits, requirements, practical roadmap and architectural notes are explained.

The major contribution of this work is introducing the conceptual foundations to combine and integrate collective intelligence of humans and machines to achieve higher efficiency and (computing) performance. To the best of the authors’ knowledge, this the first study in which such a blessing integration is considered. Therefore, it is believed that the proposed computing concept could inspire researchers toward realizing such unprecedented possibilities in practical and theoretical contexts.

Underlying much recent development in data science and artificial intelligence (AI) is a dependence on the labour of precarious crowdworkers via platforms such as Amazon Mechanical Turk. These platforms have been widely critiqued for their exploitative labour relations, and over recent years, there have been various efforts by academic researchers to develop interventions aimed at improving labour conditions. The aim of this paper is to explore US-based crowdworkers’ views on two proposed interventions: a browser plugin that detects automated quality control “Gold Question” (GQ) checks and a proposal for a crowdworker co-operative.

The authors interviewed 20 US-based crowdworkers and undertook a thematic analysis of collected data.

The findings indicate that US-based crowdworkers tend to have negative and mixed feelings about the GQ detector, but were more enthusiastic about the crowdworker co-operative.

Drawing on theories of precarious labour, this study suggests an explanation for the findings based on US-based workers’ objective and subjective experiences of precarity. The authors argue that for US-based crowdworkers “constructive” interventions such as a crowdworker co-operative have more potential to improve labour conditions.

The purpose of this paper is to propose a novel collaborative trend prediction method to estimate the status of trending topics by crowdsourcing the wisdom in web search engines. Government officials and decision makers can take advantage of the proposed method to effectively analyze various trending topics and make appropriate decisions in response to fast-changing national and international situations or popular opinions.

In this study, a crowdsourced-wisdom-based feature selection method was designed to select representative indicators showing trending topics and concerns of the general public. The authors also designed a novel prediction method to estimate the trending topic statuses by crowdsourcing public opinion in web search engines.

The authors’ proposed method achieved better results than traditional trend prediction methods and successfully predict trending topic statuses by using the crowdsourced wisdom of web search engines.

This paper proposes a novel collaborative trend prediction method and applied it to various trending topics. The experimental results show that the authors’ method can successfully estimate the trending topic statuses and outperform other baseline methods. To the best of the authors’ knowledge, this is the first such attempt to predict trending topic statuses by using the crowdsourced wisdom of web search engines.

Academic scholars have leveraged crowd work platforms such as Amazon Mechanical Turk for human subjects research for almost two decades. However, few scholars have reflected or questioned this mode of academic research. This paper aims to examine three fundamental problems of crowd work and elaborates on their lasting effects on impacting the validity and quality of human subjects research on crowd work.

A critical analysis is conducted on the characteristics of crowd work, and three fundamental problems of crowd work since its origin were identified, namely, the position of “Human-as-a-service,” the confusion of terminology and crowd work platforms’ abdication of responsibilities.

This paper explains that the three identified fundamental problems of crowd work render at least two lasting problems in crowd work-based research: first, the negligence of the teleological difference between crowd work and academic research; second, the ontological schism between scholars and institutional review boards (IRBs) in their ethical concerns and practices.

This paper critiques the foundation of crowd work-based research that has become growingly popular, extolled and taken for granted. Such a critique is deficient in literature and may seem a bit peculiar. However, we hold that it is time to take research ethics seriously in crowd work because we need to introspect and question ourselves as scholars: What is our motive or ethical stance in using crowd work for human subjects research? Is it for advancing scientific knowledge, promoting crowd workers’ welfare, or predominantly for benefiting ourselves from the fast, cheap and “good” data via crowd work?

Image annotation plays an important role in image retrieval process, especially when it comes to content-based image retrieval. In order to compensate the intrinsic weakness of machines in performing cognitive task of (human-like) image annotation, leveraging humans’ knowledge and abilities in the form of crowdsourcing-based annotation have gained momentum. Among various approaches for this purpose, an innovative one is integrating the annotation process into the CAPTCHA workflow. In this paper, the current state of the research works in the field and experimental efficiency analysis of this approach are investigated.

At first, and with the aim of presenting a current state report of research studies in the field, a comprehensive literature review is provided. Then, several experiments and statistical analyses are conducted to investigate how CAPTCHA-based image annotation is reliable, accurate and efficient.

In addition to study of current trends and best practices for CAPTCHA-based image annotation, the experimental results demonstrated that despite some intrinsic limitations on leveraging the CAPTCHA as a crowdsourcing platform, when the challenge, i.e. annotation task, is selected and designed appropriately, the efficiency of CAPTCHA-based image annotation can outperform traditional approaches. Nonetheless, there are several design considerations that should be taken into account when the CAPTCHA is used as an image annotation platform.

To the best of the authors’ knowledge, this is the first study to analyze different aspects of the titular topic through exploration of the literature and experimental investigation. Therefore, it is anticipated that the outcomes of this study can draw a roadmap for not only CAPTCHA-based image annotation but also CAPTCHA-mediated crowdsourcing and even image annotation.

The majority of existing studies about named entity recognition (NER) concentrate on the prediction enhancement of deep neural network (DNN)-based models themselves, but the issues about the scarcity of training corpus and the difficulty of annotation quality control are not fully solved, especially for Chinese ancient corpora. Therefore, designing a new integrated solution for Chinese historical NER, including automatic entity extraction and man-machine cooperative annotation, is quite valuable for improving the effectiveness of Chinese historical NER and fostering the development of low-resource information extraction.

The research provides a systematic approach for Chinese historical NER with a three-stage framework. In addition to the stage of basic preprocessing, the authors create, retrain and yield a high-performance NER model only using limited labeled resources during the stage of augmented deep active learning (ADAL), which entails three steps—DNN-based NER modeling, hybrid pool-based sampling (HPS) based on the active learning (AL), and NER-oriented data augmentation (DA). ADAL is thought to have the capacity to maintain the performance of DNN as high as possible under the few-shot constraint. Then, to realize machine-aided quality control in crowdsourcing settings, the authors design a stage of globally-optimized automatic label consolidation (GALC). The core of GALC is a newly-designed label consolidation model called simulated annealing-based automatic label aggregation (“SA-ALC”), which incorporates the factors of worker reliability and global label estimation. The model can assure the annotation quality of those data from a crowdsourcing annotation system.

Extensive experiments on two types of Chinese classical historical datasets show that the authors’ solution can effectively reduce the corpus dependency of a DNN-based NER model and alleviate the problem of label quality. Moreover, the results also show the superior performance of the authors’ pipeline approaches (i.e. HPS + DA and SA-ALC) compared to equivalent baselines in each stage.

The study sheds new light on the automatic extraction of Chinese historical entities in an all-technological-process integration. The solution is helpful to effectively reducing the annotation cost and controlling the labeling quality for the NER task. It can be further applied to similar tasks of information extraction and other low-resource fields in theoretical and practical ways.

Along with the various beneficial uses of artificial intelligence (AI), there are various unsavory concomitants including the inscrutability of AI tools (and the opaqueness of their mechanisms), the fragility of AI models under adversarial settings, the vulnerability of AI models to bias throughout their pipeline, the high planetary cost of running large AI models and the emergence of exploitative surveillance capitalism-based economic logic built on AI technology. This study aims to document these harms of AI technology and study how these technologies and their developers and users can be made more accountable.

Due to the nature of the problem, a holistic, multi-pronged approach is required to understand and counter these potential harms. This paper identifies the rationale for urgently focusing on human-centered AI and provide an outlook of promising directions including technical proposals.

AI has the potential to benefit the entire society, but there remains an increased risk for vulnerable segments of society. This paper provides a general survey of the various approaches proposed in the literature to make AI technology more accountable. This paper reports that the development of ethical accountable AI design requires the confluence and collaboration of many fields (ethical, philosophical, legal, political and technical) and that lack of diversity is a problem plaguing the state of the art in AI.

This paper provides a timely synthesis of the various technosocial proposals in the literature spanning technical areas such as interpretable and explainable AI; algorithmic auditability; as well as policy-making challenges and efforts that can operationalize ethical AI and help in making AI accountable. This paper also identifies and shares promising future directions of research.