Machine learning solution for checkout-free shopping

Client

A Milan-based startup that creates AI-powered cashierless checkout technology for physical stores

Industry

Retail

Services

Product Development, UX/UI, AI/ML, QA

Tech

TypeScript, ReactJS, Node.js, MongoDB

Challenge

Artificial intelligence is reshaping the shopping experience in brick-and-mortar stores, enabling customers to pick up the items they want and walk out the door without having to scan codes or waiting in line to pay. More and more retailers are embracing autonomous shopping technology as AI-powered systems have made it easier to adopt. Our team helped the client harness machine learning for developing an AI solution that allows making stores checkout free without the need to redesign them. The solution tracks shoppers’ movements and detects the items they take from the shelves using ceiling-mounted cameras.

We took on the following challenges:

●

Apply our skills and expertise to contribute to a rather complicated and innovative project already at a late implementation phase

●

Deliver effective visualization for data used by the client’s annotation software for training its machine learning model to recognize shoppers’ actions and track their movement

●

Facilitate the management of videos and sensor-provided raw data

●

Conduct manual testing on three target browsers

Solution

●

A web app front end built in fewer than four months that serves as an efficient annotation tool facilitating training ML models to monitor customers’ movements in a multi-camera store setting, and can run on browsers such as Google Chrome, Mozilla Firefox, and Safari.

●

A back-end solution created using Node.js that fetches data from the client’s server

●

A video and text file storage system relying on MongoDB

●

A 3D reconstruction and skeleton visualization capability for action recognition

●

A keypoint creation/edit feature that is vital to delivering error-free pose estimation, tracking and action recognition

●

A multi-view function providing better occlusion handling and thereby boosting the ML model’s ability to correctly track customers’ movement

●

A simple yet functional UI built with ReactJS that speeds up annotation and improves its efficiency

●

A rigorous UI/UX testing process resulting in a few enhancements that improved the app’s performance

Impact

●

Our solutions helped the client take its machine learning model training to the next level and make big strides in the fast-growing autonomous shopping technology market.

●

The client’s technologies attracted investor interest, resulting in its acquisition by a San Francisco-based autonomous retail company.