A recent study utilized crowdsourcing to measure the risks that city buildings would face in the event of a natural disaster. The project, “Building Detective for Disaster Preparedness”, leveraged crowdsourcing capabilities by allowing the public to identify architectural features of buildings such as roofs, windows, and chimneys; these labels would be used to train future models. With over a thousand volunteers and 20,000 images annotated, the team hopes their work can be used for large-scale natural disaster risk management. Crowdsourcing sources information at a larger scale, allowing projects to leverage human capital from all over the world. However, this broad scope means that there are certain key compromises in data accuracy, time, and cost.


A study presented at the Open Data Science Conference found that the managed workforce had higher accuracy in three identical tasks: basic transcription, sentiment assessment, and data tagging. When transcribing numerical data, the crowdsourced workers were 2,300 percent more likely to make errors. In the sentiment assessment task, both teams had to predict the stars rating for a written review; the crowdsourced workers team had only 20 percent accuracy. In the data tagging task, the managed workforce outperformed the crowdsourced workforce with 25 percent higher tagging accuracy.


These findings are not to suggest that crowdsourcing is without purpose. We employ crowdsourcing services daily. From food delivery services such as Postmates to technology and design services such as Fivver, crowdsourcing delivers unique services while promoting independent workforces. But when it comes to servicing AI projects through data labeling, research proves that a managed workforce nearly always outperforms a crowdsourced workforce.


So why was the disaster preparedness project successful? The reason lies in the power of AI. The project combined crowdsourcing manpower with its AI tool called BRAILS: Building Recognition using AI at Large-Scale. The BRAILS framework uses computer vision and machine learning to extract building information, and accounts for human error in the image annotation process. At the company and individual level, crowdsourcing fails to maintain similar levels of efficiency and accuracy because it requires AI systems to accommodate the uncertainty of an untrained workforce. Compared to a managed workforce which already integrates a human-in-the-loop approach to maximize high quality trained data.


Crowdsourcing means an untrained workforce that makes more mistakes on average with a high likelihood of redoing the work. Research concludes that a managed team labels data with 25 percent higher quality than crowdsourcing. When accuracy and quality are priorities for your project, trust in a managed workforce that strategically deploys people and technology.