Amazon Go Stores Could Watch, Listen, and Remember Your Every Move

December 6, 2016

THE VERGE - There’s a lot of excitement about Amazon Go, the company’s new kind of grocery store that lets customers take what they need and leave without ever waiting in line, but there weren’t a whole lot of specifics on how this system works. An Amazon patent filed in 2014 gives us a glimpse of how this could all operate, and it involves — you betcha — hella cameras.

The patent describes a system where cameras could capture you as you walk into the store, then identify who you are based on an ID card that’s associated with your Amazon account (in Amazon Go’s case, an app that you scan when you enter). Facial recognition may also be used.

Once you’re in the store, cameras could follow you around as you navigate the aisles to see where you’re browsing and what items you pick up. “The user's location within the materials handling facility and identification of the inventory location within the user's location may also be determined,” the patent details. “Microphones may record sounds made by the user and the computing resource(s) may process those sounds to determine a location of the user.” In addition to microphones, Amazon also listed cellular triangulation as a potential method for locating a customer’s location.



As you’re shopping, sensors in the inventory shelves monitor when which items are being picked up. The sensors watch for whether the item gets put back down on the shelf or not, and may even check if the weight of the item changes from its original state. This is likely designed so people are correctly charged if they pick up a food or drink and begin consuming them while they’re still in the store.

Static cameras watching the shelf also determine whether multiples of the same item are taken, such as several sticks of gum or a few bags of chips. If you pick up several bags of chips just to get the one on the inside, the camera is supposed to know that you ended up putting the other bags of chips back as well. If Amazon is not confident about how many items you took or what exactly you took (i.e., a Colgate versus Crest toothbrush) it may end up confirming with you to charge the right amount.

The cameras can even go so far as to determine your skin tone color. The patent says this is used to identify the shopper’s hand to see whether they actually pick up anything off a shelf, but combine that with the fact that Amazon knows what you’re buying and who you are down to your skin color and this is pretty next-level market research data. (Not to mention... kind of creepy?)


The patent provides a good look at how Amazon Go could work, but there are still some unanswered questions: what if you go shopping with a friend and they pick up something and hand it to you? Whose account gets charged for that item? How does Amazon determine if two people who enter the store together are family members in the same household account? What happens if they separate in the store and leave at different times? We’ve reached out to Amazon for clarification, but the company is not currently offering press interviews.

It’s important to note that these are all just ways Amazon could implement the technology — this is not the final version of what is going to be done in-store, which has only opened in beta to Amazon employees. Still, the patent does describe basically everything the company teased with its announcement yesterday, so this seems rather close to how Amazon Go will work in practice. If you’re still curious, feel free to read through the full patent here.