Human-in-the-loop design

Improved data annotation workflow

Redesigned the behind-the-counter item flow in Grabango's internal computer-vision annotation tool

Company: Grabango (grocery-tech startup)
Role: Product & UX Design Intern
Timeframe: Jun – Nov 2023
Tools: Figma

Improved data annotation workflow hero image

Overview

Visitbase is Grabango's internal tool for training computer vision models on checkout-free grocery footage: human annotators watch shopper video and label actions like item pickups, so the model learns to recognize them automatically. I worked closely with the Senior Director of Product to improve the annotation flow for behind-the-counter (BTC) items — cigarettes, nicotine products, and other items that require ID verification and a clerk hand-off rather than a simple shelf pickup.

The problem

The legacy BTC flow was error-prone. Annotators had to manually create a time window around the clerk hand-off and mark it using a special pick up action label called "clerk pickup". The system then guessed which scans belong to which pick up action and how many items belong to that pickup. Data showed that scanner hardware captures too many false positives and that annotators sometimes missed "clerk pick up" action due to lack of knowledge of existing scans and missed scan windows.

"How can we automate manual parts of this task and make the parts that still need human review faster and less error-prone?"

Goals

Automate

Remove manual time-windowing completely. Use a data-driven rules engine to capture all scans that are potentially relevant to the visit and rank results based on timestamp.

Unify

Replace the special-cased 'Clerk Pick Up' action with the generic 'Pick up' action used everywhere else, for consistency.

Enable

Create a tab that flips from product search to available product scans to empower an annotator to manually complete product ID tasks for BTC items. The quantity is determined by the annotator based on the number of scans per item and the accompanying video.

Walkthrough

A shopper hands an item to a clerk at the counter. The annotator's job is to correctly log the pickup and identify the item.

Empty annotation state with welcome message for first-time annotators — 01 First-time annotators land on a short welcome explainer before the visit loads.

Video feed with logged pickup and put-down actions — 02 The annotator scrubs the visit video and logs the clerk hand-off as a generic pick up.

Product Search tab for identifying the item — 03 Product Search is the default tab for item pick up.

New Clerk Scans tab showing scanner data ranked by relevance — 04 For BTC items, Clerk Scans tab lists scanner reads — timestamp, scanner ID, price.

Clerk Scans tab with a scan selected and quantity set to 3 — 05 The annotator selects the matching scan and sets quantity directly from scan data.

Completed action panel with the correct item, price, and quantity attached — 06 The action panel updates with the real product, price, and UPC.

Process

Grabango is a small team, and I was handed the problem from the Product team and worked independently to scope the opportunity, evaluate tradeoffs, and design the solution, checking in regularly to discuss direction before moving forward.

Outcome

I handed off finalized mockups and specs to engineering before my contract ended in November 2023.