freelance-work-ftr

Inside Amazon’s clickworker platform: How half a million people are being paid pennies to train AI

Reading Time: 17 minutes

Internet platforms like Amazon Mechanical Turk let companies break jobs into smaller tasks and offer them to people across the globe. But, do they democratize work or exploit the disempowered?

Each morning when she wakes up, Kristy Milland powers up her home computer in Toronto, logs into Amazon Mechanical Turk, and waits for her computer to ding.

Amazon Mechanical Turk (AMT), which has been around for over a decade, is an online platform where people can perform small tasks for pay.

Milland is looking for job postings, or “HITs”—and the alerts tell her when a listing matches her criteria. “The alerts go off once a minute,” Milland said. “I break from what I’m doing to see if it’s a good HIT before I accept the job.”

Sometimes, a group of HITs is posted. “If a batch comes up and it’s lunchtime, or I have a doctor’s appointment, or my dog needs to go out,” said Milland, “I drop everything and do it. I’m literally chained to my computer. If this is how you feed your children, you don’t leave.”

She has been doing this for 11 years.

Milland is one of more than 500,000 “Turkers”—contract workers who perform small tasks on Amazon’s digital platform, which they refer to as “mTurk.” The number of active workers, who live across the globe, is estimated to run between 15,000 and 20,000 per month, according to Panos Ipeirotis, a computer scientist and professor at New York University’s business school. Turkers work anywhere from a few minutes to 24 hours a day.

Who are Turkers? According to Ipeirotis, in October 2016, American Turkers are mostly women. In India, they’re mostly men. Globally, they’re most likely to have been born between 1980-1990. About 75% are Americans, roughly 15-20% are from India, and the remaining 10% are from other countries.

“Requesters”—the people, businesses, and organizations that outsource the work—set prices for each task, and the tasks vary widely. They include, but are not limited to:

  • data categorization
  • metadata tagging
  • character recognition
  • data entry
  • email harvesting
  • sentiment analysis
  • ad placement on videos

For instance, a recent task for Milland was to transcribe the contents of a receipt. According to Milland, the company that asks for that work will then sell the information to marketing and research departments at companies like Johnson & Johnson, P&G, and others. (The pay for that specific task was three cents.)

The early days of AMT

Milland calls herself a digital native. “I hit puberty, [and] I was on the internet,” she said. And Milland said she’s always “hustled online,” using platforms like eBay for extra income. So when she came across an article about the opportunity to do click work when Amazon Mechanical Turk launched in 2005, it seemed like a perfect fit.

In those early days, Milland saw it as “more of an experiment” than real work, she said. But during the 2008-2009 recession, that changed. Milland, who had been running a daycare center, had to move—and lost her income. At the same time, her husband lost his job. She began working on AMT full-time. For Milland, that meant 17 hours a day, seven days a week.

“We started viewing it as work,” she said. “And we really started questioning it as work.”

Rochelle LaPlante, based in Los Angeles, has been working on AMT full-time since 2012. Echoing Milland, LaPlante agreed that the work is unpredictable. “You never know when work will be posted,” she said. “It could be at 3 am. And there’s absolutely nothing to do at 9 am.”

“I’m not as hardcore as some people,” LaPlante said, “because I do value my sleep.” Others, she said, set alerts. “If a requester posts at 3 am, their computer will ding, their phone will ding, and they’ll get out of bed to do that work. It completely controls their day.”

Neither Milland or LaPlante experience a “typical” day—primarily because they’re usually setting a goal for how much money they need to make. During a normal day, LaPlante may work eight hours. “But it’s 10 minutes here, 20 minutes there—it all runs together,” she said.

So what do Turkers make, on average? It’s hard to say. But Adrien Jabbour, in India, said “it’s an achievement to make $700 in 2 months of work, working 4-5 hours every day.” Milland reported that she recently made $25 for 8 hours of work, and called that “a good day.” Just over half of Turkers earn below the US federal minimum wage of $7.25 per hour, according to a Pew Research Center study.

LaPlante talked about the difficult choices she needs to make, juggling work and life. “I have to decide: Do I take that job, or do I go to my family dinner?”

“For people living paycheck-to-paycheck on this kind of thing, on the edge of being evicted,” she said, “those decisions are difficult.”

Master Turkers

For those working on AMT, there’s a frustrating reality: Not all Turkers are created equal.

Amazon’s system designates certain workers “Master’s Level.” When a new requester posts a HIT, it’s automatically defaulted to find Turkers at this level—which costs more for the requester, and pays more for the worker.

If you don’t have that designation, you are eligible for far fewer jobs.

One weekday in March, Milland said, there were 4911 available tasks on Mechanical Turk. She was eligible for 393 of them—just 8%.

So how does one attain a “Master’s Level” designation? No one knows.

Milland has seen unqualified people—those with a low number of completed tasks, low approval ratings, false accounts, or suspensions—all earn a Master’s Level.

“There does not seem to be any rhyme or reason,” she said.

Amazon won’t reveal their criteria to attain this level. (TechRepublic reached out to Amazon for comment, but after initially agreeing, the company later declined to be interviewed for this story.)

There are various theories floating around on Turker forums about how to get to Master’s Level. Sometimes, a batch of HITs will be posted, and high-performers on that batch break into the Master’s Level. “It’s a matter of being in the right place at the right time,” said Milland.

Beyond the Master’s designation, where you live also disqualifies you for certain jobs. Being outside of the US is an obstacle, for instance, since many requesters restrict their tasks to US-only.

Getting paid

“No two Turkers are alike,” said LaPlante. “Some come for essential income, and some people just use it for play money.”

William Little is a moderator for TurkerNation, an online community for Turkers from Ontario, Canada. He uses AMT for extra cash. Little aims to make $15 a day for three hours of work. “Most of the time, I can achieve that,” he said, “which is better than someone starting out.”

Still, the payment process is a major issue for many Turkers.

Right now, only Turkers in the US and India are paid in cash. All others, including Milland and Little in Canada, are paid via Amazon gift cards.

Little will drive 45 minutes to a US border store, where he can receive free shipping from Amazon, to pick up his packages. There are also workarounds for those who actually need the cash—although they mostly involve taking a loss on the earnings. Different websites, like purse.io, can convert the Amazon gift cards into bitcoins, for instance.

“You put your ‘wish list’ up on purse.io. I see that list and say ‘I’ll buy that for Hope.’ I purchase that product and ship it to you,” said Little. “The bitcoins are held in escrow. When you receive the product, I receive the bitcoins.”

Then Little could sell the bitcoins, receive cash by PayPal, and transfer it to his bank. “I’m taking a loss on the transaction twice,” he said. “It’s not really worth it.”

Another problem? Unpaid labor. A job might be rejected with no explanation. And beyond that, Turkers often spend time assessing whether a job is good. Searching, looking up the requestor. Loading scripts, adding tools, checking statistics.

Download this article as a PDF (free registration required).

Slaves to the machine

Miland and LaPlante are part of an invisible, online workforce—one that is increasingly in demand for their vital role in helping train intelligent machines.

Smart systems are gradually coming into everyday use, as artificial intelligence (AI) begins to be put to use across society. Today’s narrow version of AI powers everything from voice-controlled virtual assistants, such as Amazon’s Alexa and Microsoft’s Cortana, to the computer vision systems that underpin the Autopilot in Tesla automobiles.

These systems are being taught to carry out tasks that historically would have been too complex for a computer, tasks that can range from understanding spoken commands to spotting a person crossing the road.

A common technique for teaching AI systems to perform these tricky tasks is by training them using a very large number of labeled examples. These machine learning systems are fed huge amounts of data, which has been annotated to highlight the features of interest. These examples might be photos labeled to indicate whether they contain a dog or written sentences that have footnotes to indicate whether the word “bass” relates to music or a fish.

This process of teaching a machine by example is called supervised learning and the role of labeling these examples is commonly carried out by Turkers and other online workers.

Training these systems typically requires vast amounts of data, with some systems needing to scour millions of examples to learn how to carry out a task effectively. Training datasets are huge and growing in size—Google’s recently announced Open Images Dataset has about nine million images, while its labeled video repository YouTube-8M links to eight million labeled videos. ImageNet, one of the early databases of this kind, has more than 14 million categorized images. Compiled over two years, it was put together by nearly 50,000 people—most of whom were recruited through Amazon Mechanical Turk—who checked, sorted and labeled almost one billion candidate pictures.

Because of the scale of these datasets, even when the labeling is spread across many workers, each individual can be repeating essentially the same simple action hundreds of times. It’s menial and often mentally wearing work.

Beyond labeling, Turkers and other online workers also clean up the often messy datasets ready for use in training machine learning systems—deduplicating, filling in gaps, and other tasks needed to sanitize the data.

As AI becomes ubiquitous, every big name firm in the tech industry is engaging people in this sort of microwork to support their machine learning efforts. Amazon, Apple, Facebook, Google, IBM and Microsoft—all of the major tech companies—either has their own internal crowdworking platform or contracts tasks to external alternatives, such as the two biggest, Amazon Mechanical Turk and CrowdFlower.

These internal microwork platforms, such as Microsoft’s Universal Human Relevance System (UHRS) or Google’s EWOK, are used for a considerable amount of work. Around the time UHRS was launched, some five years ago, the platform was listed as being used within Bing and across various product teams in Microsoft and as orchestrating 7.5 million tasks per month.

According to Mary Gray, senior researcher at Microsoft, the firm’s UHRS is “very similar” to Amazon Mechanical Turk. Gray said the firm uses UHRS to source labor in regions where “Amazon Mechanical Turk doesn’t have the best reach” or where the work is sensitive and needs to be carried out in secret.

“Every company that has an interest in automating a service has access to or uses some sort of platform like Amazon Mechanical Turk. Indeed, many of them use Amazon Mechanical Turk,” she said.

Chris Bishop, laboratory director for Microsoft Research Cambridge, said that UHRS gives Microsoft “a little bit more flexibility” over third-party platforms such as AMT, saying the firm is using AI to automatically identify strengths and weakness in crowdworkers, such as relative levels of expertise, which in turn helps Microsoft decide whether to attach more or less importance to different workers’ results.

Beyond helping train AI, platforms like AMT are used by household names, everyone from eBay to Autodesk, to offload an assortment of repetitive, small-scale grunt work, which has made up the bulk of microtasks on AMT for many years.

This low-skilled, monotonous works spans many tasks: the occasionally traumatic process of screening user-generated images and other content, completing marketing and academic surveys, deduplicating entries and checking product descriptions and images for online retailers—Amazon created Mechanical Turk to help with its inventory management, categorizing images and products, writing website descriptions, extracting names from emails, translating text, transcribing text from speech or images, correcting spellings, verifying geolocations, giving feedback on web design, leaving reviews for products, choosing thumbnails for videos, or letting companies track which part of an ad you view.

How did we get here?

The idea of humans working to help machines carry out tasks they would otherwise find impossible is nothing new.

While the recent AI explosion has magnified demand for data labeling and curation, these sorts of microtasks date back more than two decades, said Gray, when work revolved around trying to improve spelling and grammar checking in word processors like Microsoft Word.

The wider history of click working and microtasks goes back to the rise of online retailers during the dotcom bubble of the late 1990s and early 2000s.

In 2001, Amazon, looking for new ways to more efficiently organize products on its rapidly growing store and solve difficult inventory problems that lay beyond the ability of computers, patented a hybrid machine/human system.

Four years later, Amazon realized its goal of building a digital platform to provide on-demand access to the huge pool of labor available online, with the launch of Amazon Mechanical Turk.

Being able to tap into Amazon’s pool of “artificial artificial intelligence”—Amazon’s description of Mechanical Turk’s USP—appealed to a broad range of companies, everyone from online retailers to porn sites looking for affordable ways to sort their products, particularly at the low price for which Turkers would carry out microtasks.

In 2015, an average of 1,278 people or organizations were posting jobs to Amazon Mechanical Turk each day. While the amount of work being carried out by crowd laborers is increasing, particularly via sites like CrowdFlower, the exact amount remains unclear, since much work goes unrecorded or is contracted out multiple times.

And while more than 500,000 people may be signed up to work for Amazon Mechanical Turk, according to Amazon’s website, these numbers don’t reveal how people use crowd-working platforms—whether it’s a full-time gig or something people do to earn cash on the side.

The Oxford Internet Institute estimated that the two large microtask platforms—Amazon Mechanical Turk and CrowdFlower—had a combined annual gross revenue of about $120 million in 2013. Professor Vili Lehdonvirta, associate professor and senior research fellow at the institute, estimates this is about 5% to 10% of the overall online crowd labor market, but again stresses the difficulty of sourcing accurate figures that account for employment via non-English language platforms globally.

The other cost of click work

Beyond the dull nature of click work for the people doing it, there can be more costly consequences. It can take a severe toll on the physical and mental health of some workers.

“I would wake up, ignore everything else,” said Milland. “My family would prepare food and leave it here for me so I could eat while I worked. I would eat at the computer and I wouldn’t see my family. If my daughter needed homework help she’d have to go to her dad. It got so bad that I developed a ganglion cyst in my wrist. I’ve got a repetitive strain injury in my arm, but that’s what you do.”

“I was lucky that I was doing it at the peak when my husband was home, because he was unemployed,” said Milland. “If anybody hears the ‘ding’ that indicates high-paying work, they say, ‘Go, go, go!'”

A Turker from southern India, Manish Bhatia, has been a volunteer moderator for MTurk Forum for almost two years, and currently moderates two forums.

The strangest thing he’d been asked to do? Film himself lying in a bathtub with rose petals. “That was really weird,” he said. In terms of the graphic content, Bhatia also reports seeing disturbing images. “You don’t get to know beforehand,” he said. “You can opt out afterwards.” But, then you don’t get paid for a job you don’t complete and it wastes time.

Milland reported similar experiences. “People say to me ‘Oh my god, you work at home? You’re so lucky,'” said Milland. “You can’t tell them ‘I was tagging images today­­—it was all ISIS screen grabs. There was a basket full of heads.’ That’s what I saw just a few months ago. The guy on fire, I had to tag that video. It was like 10 cents a photo.”

Milland isn’t the only one tasked with tagging graphic or grotesque images.

“In the YouTube batch yesterday,” said LaPlante in March, “there were a lot of beheadings. There’s a check-box at the bottom that says ‘Inappropriate Content,’ and you push ‘Submit,” she said.

This kind of work can be important, since it has the potential to prevent objectionable material from appearing online. Still, it can be pretty traumatic to the people doing it, and the pay doesn’t necessarily match the value it’s providing to YouTube or its users.

Little said he would often have to tag photos or videos for pornographic content. “The only time I would take any exception to that is if there was child pornography,” said Little. “Then I would report that to the requester and Amazon.”

But in terms of gore or mutilation, for instance, it’s “par for the course to see stuff like that,” said Little.

Once a task is completed, it’s impossible to know what happens with the result. “I wonder, is somebody going to review this? Hopefully, this is going to be reported or removed,” LaPlante said. “Someone came across some child pornography, and they checked the box, but is that going to be ever checked into or looked into? You just don’t know.”

Since requesters use pseudonyms, no one knows who is asking for this work to be done. LaPlante calls it “the wild, wild west.” And while requesters rate Turkers, there is no way for Turkers to rate or review requesters.

“You could be tagging faces in a crowd, but maybe something is being built for a malicious purpose or something,” she said. “You don’t know what you’re doing, exactly, because there’s no information.”

“It’s called vicarious traumatization,” said John Suler, a psychology professor at Ryder University who specializes in behavior in cyberspace. “The same thing happens with first responders, and this is another example. When people see horrible images online, they become traumatized.”

But we are not always aware of the psychological toll, he said. “Our conscious mind goes numb,” said Suler. “But our unconscious mind doesn’t—it picks up on things. We’re underestimating how all these things we see online impact us at a subconscious level.”

Workers have found online community forums to connect with each other and share stories, commiserate, and support each other. “There are so many questions around things like pay [and] content moderation,” said Milland.

“It’s a place to find social support,” she said.

Each of the community platforms has a slightly different vibe. MTurk Forum has a “water cooler feel.” On the other hand, Mturkgrind “seems to be more focused on production and efficiency and work,” Milland said. At TurkerNation, she said, “the focus seems to be on answering questions and helping new users navigate and understand the system. They’re a little more production-minded.”

There’s also a closed Facebook group called Mturk Members, with 4,436members. The group uses the page to ask questions, post earnings, and cheer each other on.

LaPlante and three other women created MTurk Crowd, a worker forum for Turkers that helps them locate resources to enable them to do the best work they can on the platform. And there are many other forums, subReddits, and organizing platforms online, as well.

There’s also an organizing site for workers: WeAreDynamo.org. It’s where the “Dear Jeff Bezos” campaign was initiated. That campaign was an attempt to humanize Turkers, giving a voice to people actively engaged in the platform, where they stated their experiences and voiced concerns about the nature of their work.

Unfortunately, the campaign had very little impact: Although Indian workers were able to receive bank transfers after the campaign, neither Amazon nor Jeff Bezos ever directly addressed the initiative.

Communicating with Amazon is, for all practical purposes, virtually impossible. “The lack of support we have is disturbing,” said Bhatia. “There’s no live chat, no phone number.” The only way a Turker can make contact is through email, which prompts a boilerplate response.

“I am utterly baffled by the choices they are making,” said Little, “and the number one choice they are making a mistake in is a lack of communication. Why do they want to be so hands off? It can’t because of the risk of a lawsuit, because their terms of service clearly state ‘No class action lawsuits.'”

Little talked to lawyers, but “none of them would ever take on a single worker against Amazon,” he said. And Amazon still refuses to talk. “Not even about rejections, not about improvements, not about how we think they could make more money,” he said. “Nothing.”

Lilly Irani, who teaches at UC San Diego, explores the “cultural politics of high-tech work practices.”. Irani co-authored a 2013 study looking at forums for Turkers. The work aimed to understand how collective actions could work, looking at things like Dynamo, the collective platform for Turkers, and Turkopticon, which allows Turkers to review and rate available jobs. In a paper called “Turkopticon: Interrupting Worker Invisibility in Amazon Mechanical Turk,” the authors stated: “We argued that AMT is predicated on infrastructuring and hiding human labor, rendering it a reliable computational resource for technologists.”

Despite the poor working conditions, people like Milland rely on the income AMT provides. And she has a disability that makes her doubt her chances of being hired for a traditional job. “I applied at McDonald’s, and they won’t hire me,” she said.

Download this article as a PDF (free registration required).

Humans working alongside AI

Helping fuel demand for this piecemeal, on-demand employment, predicts Microsoft’s Gray, will be the growth of human/AI systems that promote a symbiotic relationship between people and machines.

She cites the emergence of virtual assistants such as Facebook M or customer service chatbots like IPsoft’s Amelia, where humans either handle queries with the aid of AI or an AI handles queries and the human takes over when there is an issue the machine can’t handle. Over time these smart systems can also learn from human responses, and gradually increase the breadth of queries they can tackle.

Services that use narrow AI to handle easier tasks and humans to handle the more complicated demands are on the rise. One of the major hubs for crowdsourced labor, CrowdFlower, recently launched a machine learning platform to automate certain tasks that previously would have been carried out by manual workers, leaving human workers to “focus on the harder cases and help the [machine learning] models learn”. This approach results in significant amounts of manual work being automated, but the more optimistic forecasts predict that, while on a job-by-job level the human’s share of the work is reduced, overall employment opportunities won’t fall, due to increased demand for these joint AI-human services.

How long will machines need people?

But, for how long will humans have a role to play in training the smart systems of tomorrow?

As intelligent systems gain the ability to perform tasks that once had to be carried out by people, the nature of the work offloaded to humans on platforms like AMT changes.

In 2006, one year after AMT launched, Amazon CEO Jeff Bezos said a human was needed to spot whether a person was in a photo, a task that can now be carried out by deep learning, neural networks run by the likes of Baidu, Facebook, Google and Microsoft. So does this mean that the microtasks that offer employment today will gradually be taken over by machines?

The Oxford Internet Institute’s Lehdonvirta sees little prospect of demand for AI-focused microtasks being sated. He forecasts that as machine learning is applied to more and more tasks, there will be an ever-increasing amount of data needing to be labeled.

“It’s a moving target. There are so many applications that I don’t think we’ll be running out of that work anytime soon,” he said.

Microsoft’s Bishop said that in the near-future, AI systems will likely be trained using a mix of human-led, supervised learning and unsupervised learning. Microsoft’s Gray believes there will be a long-lasting need for humans in the loop: “If anything, we would predict they’re going to go up because the amount of things we’re trying to automate is going up,” she said.”If we take those early cases of natural language processing and image recognition as a bellwether or as a benchmark, we see a pretty steady amount of work in the system.”

The University of Southampton’s Ramchurn uses the example of image recognition to illustrate just how much data will still need labeling.

“We are nowhere close to hitting a limit. Image labeling for example still relies on human labeling for every type of context in which pictures are taken,” he said.

Such is the range of different settings in which images can be captured—in light, in shadow, obscured, unobscured—that “even after classifying 50 million pictures, only very few items in pictures will be accurately classified in all possible contexts,” he said.

Expand that need for data to be labeled in a multitude of contexts to speech, natural language understanding, emotion recognition, and all of the many areas that machine learning is being applied, and there is no danger of the work drying up, he said. Especially as society finds new uses for machine learning.

“Demand is likely to keep growing and we will see more systems combining human and machine intelligence in novel ways to address real-world problems.”

Jobs as a service

Whether or not people are needed to help train AI in the long run, the rise of platforms like Amazon Mechanical Turk reflects a wider, ongoing shift in working practices.

Just as the advent of faster global telecommunications links in the late 1990s made it possible to outsource and offshore a far greater range of business roles, so online crowd labor platforms and an abundance of people with access to broadband and a computer at home will again reshape the world of work, said Microsoft’s Gray.

“We are able to, for better and for worse, break up jobs that use to be full-time occupations and turn them into work that can be done 24×7 by a range of people in different time zones, in different locations,” she said.

“We’ve not so much dismantled or deskilled the work that we do, so much as we’ve created modules of it that different groups of people can pick up.”

In the long run, Gray sees individuals chopping and changing between microtasks as being a much more common way of working. The practice of software managing the task of splitting jobs into chunks and then farming out the resulting microtasks to individuals via online platforms, as and when the need arises, is a natural progression from the outsourcing practiced by firms today, she said.

“Technologically, we are there and we have been there for the last decade in some areas of customer service,” she said, citing the shifting of customer relations from call centers to live web chats and referencing a similar proportion of software-managed manual roles in retail, marketing and events industries.

As these online platforms become better at rapidly connecting employers with the skills they need for specific tasks, a key appeal of the practice for firms today, so the use of microwork will grow, she said.

“We see this industry of work that’s sourced, scheduled, managed, paid for, and shipped through an API [Application Programming Interface],” said Gray. “It’s exploding underneath our noses.”

The Oxford Internet Institute’s Lehdonvirta, shares Gray’s perspective that it will become increasingly common for computer systems to orchestrate labor.

“Some of these same practices and ways of organizing the work, the computer mediation—the use of platforms to mediate the working relationship—those kind of things seem to be on the rise,” said Lehdonvirta.

As online connectivity and crowd labor platforms continue to grow, enabling full-time jobs to be broken into smaller packets of contract work, it’s time for governments to start paying attention to the human impact of this shift in the way we work, said Gray.

“We have yet to grapple in any substantive way with how they completely reorient the vast majority of people on this planet to how they work,” she said.

“It’s been going on for the last 30 years,” Gray said. “We weren’t paying attention because, frankly, it didn’t hit the kinds of jobs that people in power have and their children have.”

Credit for image at top: Sam Santos/George Pimentel Photography

http://www.techrepublic.com/article/inside-amazons-clickworker-platform-how-half-a-million-people-are-training-ai-for-pennies-per-task/

Leave a Reply

Your email address will not be published. Required fields are marked *