Today, I spoke to the audience at the Amazon re:MARS 2022 event about how my team is using computer vision and machine learning to deliver easier and faster in-store shopping experiences for customers. I’m excited about the ways that we use artificial intelligence (AI) in our technology services, including Just Walk Out technology, Amazon One, and Amazon Dash Cart, and I want to share the top four takeaways from my presentation.
1.We continue to evolve, improve, and build new machine learning algorithms
Since we use computer vision algorithms across all our physical retail technologies, pushing the state of the art forward in this space has always been important to us. In the case of Just Walk Out technology, which allows shoppers to save time by skipping the checkout line in many Amazon stores, select Whole Foods Market stores, and several third-party retailer stores, we’ve made ongoing inventions at all layers, including sensors, optics, and machine vision algorithms. As a result, we’ve reduced the number of cameras required in Just Walk Out technology-enabled stores to make them more cost-effective, smaller, and capable of running deep networks locally. Our sensors and algorithms have evolved to detect a broad range of products and differences in shopping behavior in full-sized grocery stores, all while ensuring an effortless customer experience. We also increased the diversity of environments our algorithms have to account for as we deploy Just Walk Out technology to third-party retailers.
Similarly, when our team built the Amazon Dash Cart, a smart shopping cart that helps customers skip the checkout line in many of our U.S. Amazon Fresh stores, they developed a robust set of computer vision and sensor fusion algorithms to detect items while in motion, including accurately capturing weight and quantity. The machine vision algorithms also have strict latency budgets, as we keep track of a customer’s receipt in real time.
At Amazon Style, our physical apparel store that offers a personalized, convenient shopping experience for customers to find styles they love at great prices, we built new technologies and algorithms to make this experience effortless for customers. Our algorithms use the information a customer provides—whether that’s a detail they entered into our Style Survey or the items they scanned while shopping on the floor of the store—to build a diverse set of recommended items, balancing similarity to their current choices with a diverse set of options. The system also generates complementary selections, such as a shirt to match a pair of jeans to create a complete outfit. We went to great lengths to keep the fun in shopping while elevating the experience through machine learning algorithms.
2.We use synthetic data to bolster our algorithms
When my team set out to reimagine the in-store shopping experience for customers, one challenge we faced was getting diverse training data for our AI models to ensure high accuracy. To address this challenge, our research teams built millions of sets of synthetic data—machine-generated photorealistic data—to help build and perfect our algorithms and provide a seamless customer experience. Combining real-world and synthetic data allowed us to build truly robust and smart algorithms that were ready to take on all types of scenarios in stores.
In the case of Just Walk Out technology, we built synthetic datasets to mimic many variations of realistic shopping scenarios. For example, we used synthetic data to create variations in lighting conditions to account for sunlight differences in stores, and we used it to create a crowd of synthetic characters to ensure that we could handle many customers in the store at one time. We also used synthetic data to generate virtual models of larger grocery stores for testing before we scaled to that size, so we knew when we launched Just Walk Out technology to the public, we were delivering a highly accurate experience for shoppers.
For Amazon One—a fast, convenient, contactless way for people to use their palm to make daily activities like paying at a store or entering a location more effortless—we needed data to train and test our AI algorithms across demographics, age groups, temperatures, and even variations like calluses and wrinkles that are unique to one’s palm, to enable the service to correctly determine whose palm was hovering over the device. As we started to build Amazon One, we soon realized the limited availability of public datasets consisting of palm and vein images to help train our algorithms. So, our research scientists further advanced already state of the art technologies to build huge volumes of diverse, realistic synthetic palm and vein images to train our AI models and prepare the service for a wide variety of users.The Innovations Powering Amazon One | Amazon News
3.We design all our technologies with security top of mind
All our technology services are designed from the ground up with security top of mind and with a secure backend infrastructure, so all data follows Amazon’s stringent security approaches. Our services also have multi-layered security controls built into the hardware, software, and cloud infrastructure for end-to-end security.
For example, Just Walk Out technology cameras are custom-built devices that incorporate hardware-backed security capabilities and end-to-end encryption of data both locally and while being sent between our services. Amazon Dash Cart’s image data captured during item scanning is obfuscated and then encrypted to protect confidentiality. Meanwhile, for Amazon One, we built tamper and intrusion detection capabilities into the device hardware, allowing us to render the device unusable if we detect any attempts to compromise it. The Amazon One device processes all biometric data in hardware secure zones, and it’s built so it only connects to and sends data to our cloud services, which only accepts data that originates on our devices.
4.We’re continuing to innovate as we scale
Since launching Just Walk Out technology in 2018, we’ve scaled rapidly and introduced new services that redefine in-store shopping experiences for customers. Today, Just Walk Out technology is available in more than 30 Amazon Fresh stores in the U.S. and UK, over 25 Amazon Go stores in the U.S., and two Whole Foods Market store locations. The Amazon Dash Cart is available in many Amazon Fresh stores, and Amazon One is available in select Amazon Go, Amazon Fresh, and Whole Foods Market stores across the U.S.
We’ve equipped more than a dozen U.S. and UK third-party locations with Just Walk Out technology, Amazon One, or a combination of both. Sports venues, like Climate Pledge Arena and TD Garden, and travel retailers, like Hudson and WHSmith, have enabled their stores with our technologies to make the shopping experience more enjoyable and efficient for their shoppers. In addition to our own Amazon Fresh stores in the UK with Just Walk Out technology, UK grocer Sainsbury’s has opened SmartShop Pick & Go, a Just Walk Out technology-enabled convenience store in London.
This kind of scaling comes with its own challenges. For example, over the past few years, we’ve increased the complexity of use cases where Just Walk Out technology is deployed—from 1,800 square foot Amazon Go stores to 40,000 square foot, full-size grocery stores—and had to account for differences in shopping behavior. For example, at an Amazon Fresh store, people tend to rummage for items like produce versus a grab-and-go salad or sandwich from Amazon Go. Similarly, expanding Just Walk Out technology to third-party retailers such as sports venues and travel retailers meant solving for unique challenges, such as being open 24/7, or hosting games and events where shoppers rush in at once at certain times. All these scenarios increase the complexity for our algorithms, and my team continues to innovate to meet the requirements of our customers and retailers.
Delivering on our vision
When we took on the challenge of reimagining in-store shopping experiences for customers several years ago, our vision was to introduce new levels of ease and convenience to a human experience that has remained largely unchanged over time. As my team continues to build and scale new technology services, the hardest part is ensuring that the technology recedes into the background, so when customers come in to shop, it’s a seamless and magical experience. It’s still so exciting for me to see shoppers experience Just Walk Out Shopping, Amazon One, and Amazon Dash Carts for the first time.
As I look back on the progress my team has made, I’m reminded of an Amazon saying that “it’s always Day 1,” and it certainly is still Day 1 for us in physical retail and technology. It feels like we’re just getting started in tackling some of the complex challenges in the physical retail world, and I’m excited to see what the team does next to push the boundaries of AI forward.
If you’re interested in learning more, tune in to watch my full re:MARS keynote.