Seattle Seahawks fans win no matter who their team is facing on the field. Their secret weapon? Artificial intelligence (AI) from Amazon.
Last year, the Seahawks’ Lumen Field became the first NFL stadium to introduce Just Walk Out technology by Amazon. It allows fans to enter a concession stand, grab whatever they want, and quickly get back to their seats, without having to wait in a checkout line or stop at a cashier.
The feat is a combination of computer vision, object recognition, advanced sensors, deep machine learning models, and generative AI—a type of artificial intelligence that has recently captured the public’s imagination.
Amazon researchers found a way to make these technologies work together seamlessly, in the background, in order to figure out almost instantly which customer took a tuna sandwich and who got one with chicken salad, charge them correctly, and allow them to simply walk away with their items.
“Without knowing the technology, it feels like magic,” says Gérard Medioni, vice president and distinguished scientist at Amazon. “But creating that magic—determining who took what—is harder than you think.”
Medioni led the team of research scientists who designed the technology, an effort that began eight years ago. Today, Just Walk Out technology is available at more than 70 Amazon-owned stores, and at more than 85 third-party locations across the U.S., UK, and Australia, including sports stadiums, airports, grocery stores, convenience stores, and college campuses.
“People don’t like to wait in line when they have a plane to catch or a game to watch,” Medioni says. “We built Just Walk Out technology for shoppers who want to get in, find the items they need, and get out, without waiting for the person in front of them to count their change.”
At Lumen Field, Just Walk Out technology from Amazon has scored a resounding victory. After installing Just Walk Out technology in September 2022, District Market—a convenience store that offers beer, soda, and gameday snacks and cuisine from the stadium's community neighbors—more than doubled sales, compared to the previous store in the same location, and fan satisfaction survey scores hit an all-time high. Lumen Field has since added Just Walk Out technology to eight additional stadium concession areas, the most of any sports venue the world.
“We saw a dramatic increase in throughput in one of our tightest concourses, with less congestion and steadier flow at full capacity,” says Zach Hensley, vice president of operations for the Seattle Seahawks and general manager of Lumen Field. “Just Walk Out is a seamless technology that lets fans get their food and drinks during a timeout as quickly as possible, so they can get back to their seats.”
Here's a closer look at the various technologies that make Just Walk Out technology tick:
Computer vision and machine learning enable Just Walk Out technology and preserve customer privacy
You enter a store like Lumen Field’s District Market with your credit card at the entry gate, or by scanning your palm over an Amazon One device. Amazon One is Amazon’s palm-based identity and payment system that uses AI and machine learning to link your credit card to your palm.
Once inside the store, you will find several specialized cameras mounted to the ceiling with a view of the entire store area. Jon Jenkins, vice president of Just Walk Out technology, says a combination of computer vision and machine learning allows the system to know “who takes what and charge them correctly when they walk out.”
When you pick up a can of soda, the Just Walk Out technology adds it to your virtual cart. If you decide to return it to the shelf, the technology will automatically remove it.
“Our tech is able to distinguish shoppers from one another, without collecting or using any of their biometric information,” Jenkins adds. “Just Walk Out technology detects when a shopper’s hand interacts with a product on the shelf. When that happens, machine learning algorithms make sure the correct item is added to the virtual cart—all without any specific knowledge about the person.”
Just Walk Out technology operates independently from Amazon One, and doesn’t use or collect any biometric information of shoppers. Customer trust and privacy are paramount to the experience. Just Walk Out technology simply links a customer with their payment instrument. When shoppers enter the store, the technology assigns them a temporary numeric code, which serves as the shopper’s unique digital signature for that shopping trip. The system preserves the code throughout the shopper’s time in the store. When they exit, the code disappears, and if they come back, they get a new code.
“In every image, we have a number of pixels that belong to a person,” Medioni says. “If a shopper takes off their jacket, for example, their signature updates as they move around the store. Just Walk Out tech doesn’t collect any biometrics. All we need to know is where that person is on the floor, and where their hands are in relation to the store’s merchandise.”
The technology is so accurate that it can also track groups of shoppers. More than one person can enter a store with a single credit card. The system tracks each individual, but associates the group with the same payment instrument, which can be useful for family shopping trips. When the group walks out, the system knows which shoppers were using the same card and generates one receipt for the collective transaction.
“We had a tour bus that came in one day, and they had 90 people all paying with a single credit card,” Medioni adds. “Even if people leave the store separately and we can still keep track of the group’s purchases.”
Shelf sensors and sensor “fusion” for accurate receipts
Just Walk Out technology uses object recognition to accurately identify sandwiches, bags of potato chips, and other merchandise. Medioni says the system has been trained to know what it is looking at—“When you return an item to the wrong place by mistake, the system adjusts accordingly,” he says. The technology also alerts a store attendant to put the misplaced item back in the right place.
For smaller, harder-to-see products, like chewing gum or lipstick, weight sensors on the shelf detect when customers pick something up.
“Very small items are an interesting challenge for a computer vision system, because the camera—the primary sensors—can only see so much,” Jenkins says. “Having additional sensors enables us to do some really cool things in terms of solving problems that would be otherwise very difficult to solve with cameras only.”
A fusion of input from the weight sensor, known as a “load-cell sensor,” on the shelves and the cameras helps the system figure out what item was picked up by the shopper.
“The thing that’s important about Just Walk Out technology is that we always look at each store uniquely, to ensure we’re deploying the right set of technologies,” Jenkins says.
Leveraging generative AI and synthetic data to build a robust system
Before generative AI became a buzzword, Amazon research teams used it to help develop Just Walk Out technology by creating photorealistic sets of synthetic data—information about situations that can be hard to find in real life and difficult to teach a computer.
“This is very similar to the self-driving car, where the basic case is easy, but there is a long tail of complex cases,” Medioni says. “If the system has never seen an event like this, it may not do well. We wanted to have many examples, so we generated video clips of synthetic shoppers performing similar activities, then trained the system to make the right decisions.”
Medioni and his team used a type of generative AI called generative adversarial network (GAN) to create synthetic data for training Amazon’s Just Walk Out technology.
“With GANs, you can control absolutely every movement that the virtual shopper is performing,” he says.
The team used datasets from millions of AI-generated synthetic images and video clips mimicking realistic, and sometimes rare, shopping scenarios—including variations in store format, lighting conditions, and even crowds of shoppers. This training allowed them to scale the original idea for the Just Walk Out technology to a system that can recognize millions of actions without making mistakes.
“When the customer exits, having an accurate account of their purchases is critical,” Medioni says.
Just Walk Out technology based on computer vision is already available around the world, with new locations launching every month. But Jenkins says that Amazon has “barely scratched the surface” in terms of where Just Walk Out technology can work.
“We've started with some of the most obvious use cases, where people are in a hurry, but we believe it can apply almost anywhere in the long run, and that there are cases beyond retail,” he says. “The fundamental technology is inherently transferable.”
Learn more about Just Walk Out technology and Just Walk Out technology powered stores near you.
