Will Advanced AI Step Up Amazon’s Just Walk Out Technology?

Amazon’s Just Walk Out technology received an upgrade with advanced artificial intelligence technology promising to significantly improve accuracy, helping minimize receipt delays and increasing the ease of deployment of the technology for retailers.

In a blog entry, Jon Jenkins, VP of Just Walk Out technology in AWS Applications, said Just Walk Out tech launched in 2018 with the latest AI and machine learning available at the time to figure out: “Who took what?”

The system analyzed shopper behavior “sequentially” — their movement, their location, what they picked up, and the quantity of each item — with each action “processed one after another.”

Jenkins added, “However, in unusual or novel shopping scenarios (such as if a camera view was obscured due to bad lighting or a nearby shopper), the sequential approach could take time to determine purchases with confidence, and sometimes required manual retraining of the model.”

The new advanced AI-driven system analyzes all sensor data “simultaneously, rather than sequentially,” said Jenkins. Multiple cameras, weight sensors, and other data help the system prioritize “what’s most important to accurately determine the variety and quantity of items selected.”

He adds, “For example, a shopper might pick up and put down multiple varieties of yogurt, in different combinations, and as they are doing so, another customer might reach for the same item, or the freezer door could fog up, obscuring the cameras’ view. In complex situations like these, the new model can quickly and accurately determine the actual items taken by each shopper.”

The new multi-modal foundation model further increases accuracy by using the same transformer-based machine learning models underlying many GenAI applications online and applying them to physical stores. Transformers are a type of neural network architecture that processes sensor data into outputs, such as receipts for checkout-free shopping.

News of the upgrade reaffirmed Amazon’s commitment to scaling Just Walk Out after an early April report from The Information that the technology was being removed from Amazon Fresh stores led to questions about its viability.

Jenkins said Just Walk Out is currently available in over 170 third-party locations, including airports, stadiums, universities, and hospitals, in the U.S., UK, Australia, and Canada. Plans call for more than doubling the number of third-party stores using the technology this year. Some Fresh stores in the UK and Amazon Go convenience stores also use the tech. Jenkins said, “As we scale, the system will continue to learn from everyday shopping scenarios and raise the bar for accuracy and convenience, delivering the benefits of AI to retailers and customers around the world.”

The Information report found that despite its initial portrayal as a fully automated solution, it heavily relied on human reviewers, often leading to delayed receipts for customers. Critics of Just Walk Out technology in the past have questioned the costs and structural challenges related to installing and maintaining video cameras and other hardware in store ceilings.

In a follow-up interview at the time with Sucharita Kodali, VP and principal analyst at Forrester, Jenkins said the shift at Fresh was largely due to strong shopper appeal for its Dash Cart smart carts. He said The Information’s finding that 1,000 people in India were watching and labeling videos to manually label transactions had “no factual basis” but noted that the technology is working better at small-format stores, especially where planograms were largely identical and little to no human intervention is required.

Kodali wrote, “This recent news confirms the concern people have had all along with making the technology cost-effective in large, low-margin environments like grocery stores. It seems to have better success in high-margin, small formats.”