A new beginning in familiar territory

Earlier this year I made the decision to end my twelve year tenure at Amazon to pursue something new. It was incredibly hard to leave, not least because it meant leaving my role leading business development for the Amazon Mechanical Turk and SageMaker Ground Truth teams where I had been for the last three years. One of the things I loved most about that role was that I had an opportunity to help customers find unique ways to use crowd workers to solve a host of really fascinating problems. That was only a small part of my job and I always wished I could spend more time helping customers design and build the pipelines and tasks that would allow them to utilize MTurk or Ground Truth to solve their problems. I loved the multidisciplinary aspects of working on crowd tasks because it exercised many of the skills I’ve developed over the years in data science, software, product, business, and process development.

I had so much fun working with Amazon customers on these types of projects that I’ve decided to make it my next business. I am excited to say that I’m returning to my roots in consulting and will be working with clients interested in using crowds to gather the data and insights they need to grow their business. It has been exciting getting started with this work and being able to focus 100% on helping clients be successful. It will be a challenging adventure and I will be sharing progress and insights on this blog as I go.

What is the crowd?

When I talk about the crowd I’m generally talking broadly of any group of distributed workers who are going to contribute to completing a task. The MTurk crowd is a Public crowd made up of workers that work on your task to earn money. There are also Private crowds made up of volunteers, members of your team, or others that wish to contribute to your project. The real power of crowds comes in the distributed nature of these workforces. MTurk, Ground Truth, and other services all start by breaking a large set of data down into atomic tasks so that they can be distributed amongst workers. In doing so, projects that would have taken weeks or more to complete can be efficiently distributed to hundreds of workers and completed in a matter of hours.

The difficulty has always been that using crowd services such as MTurk requires a diverse set of skills most folks simply don’t have. To be successful at any task requires both technical skills to develop the task interface and data pipeline for the task, as well as an understanding of interface design and general product management skills to build tasks that can be well understood by workers. In addition, using the crowd requires understanding the audience of workers who are going to be completing your tasks. A generic interface geared toward the MTurk Public crowd is likely going to be inefficient for a Private crowd comprised of members of your team.

Note that this type of work is generally called crowdwork or microwork and is a subset of crowdsourcing. Crowdsourcing is a broad category of activities that allow crowds to contribute in a variety of ways. Common examples include Wikipedia, Google Maps, hackathons, and a variety of others.

Can’t I just use AI and machine learning?

While it’s true that AI and machine learning are supplanting some uses of crowds, I’ve generally found that crowds are actually an important and growing complement to ML. To begin with, any project to train a new model has to start with ground truth training data that provides the algorithm with input data to train with. To train a model to detect cats and dogs in images, you need to start by providing your algorithm with thousands of images with labeled cats and dogs. Crowds are ideally suited to gathering the labels and annotations that can be used for training.

Once you have a model, the crowd can provide a useful backstop to help you ensure you are getting the accuracy you need. For example, if your model is only 64% confident that an image contains a dog, the crowd is an easy solution for adding human review to ensure accuracy. The validation provided by workers can then be used to re-train the model to better handle edge cases.

Of course there are also a range of scenarios where building an ML model isn’t possible or efficient and it’s just easiest to use crowds to complete the tasks manually. This is often the case where accuracy is crucial and there is a lot of variability in how to interpret inputs. For example, if you needed to find the websites for two thousand businesses, crowds are ideal because they take advantage of the human capability to sift through search results and find the right answer.

How are you going to serve clients?

I’ve found that most crowd projects involve the following four stages and I am able to assist clients in all or just a few of these phases depending on their needs. Each project will be unique and I’ll work with my clients to determine how best to allocate my time.

The first step is to clearly define the goals of the project and design a task flow to get data into the format you want. In some cases a project is relatively straightforward and can be accomplished with a single crowd task, but many task benefit from breaking them into multiple stages. A simple example would be to break an image annotation task into two parts. In the first stage you might have workers simply draw boxes around all of the animals in an image, and in the second stage you would have workers label them as cats and dogs. Breaking tasks down in this way can be a powerful way to improve the quality of results and can actually lower costs by reducing the complexity of the task for Workers. Other scenarios might involve pairing Private and Public crowds by having your staff annotate out sensitive data in the first stage, and then have Public workers complete the task in the second stage. Finally, you can insert existing ML models into your workflow to reduce the amount of time workers need to spend on simple tasks. Thinking through how best to structure and design a process will ensure you’re using resources as efficiently possible and get the results you want.

In the Prototype and Test phase you’re building out your data pipeline and defining the tasks that you want workers to complete. This is easiest if you can use one of the off-the-shelf templates that MTurk and SageMaker Ground Truth provide. If not, and your task can’t be built with simple HTML, you’ll need to build a custom interface in HTML and JavaScript. I’ve built dozens of unique interfaces and am happy to help you build a task interface that meets your needs. Once you have an interface and your data prepared, you can send test data to workers to get initial feedback and check the accuracy of the results. This is often an iterative process that requires tweaking your interface and making adjustments to ensure workers can provide the results you’re looking for.

Once we’ve completed our task, the final step is to move it into production. For most ML projects this simply means running it with the full set of data you wish to have annotated. For other projects, this might include plumbing it into existing processes, adding monitoring, and other activities. In either case, it is important to plan for ongoing monitoring of the results you’re getting from workers. The quality of your results can drift over time for a variety of reasons so it’s important to review your results periodically.

I’m excited to begin this new adventure and look forward to sharing my story. If you have a project that would be benefit from my assistance, please don’t hesitate to reach out on LinkedIn.