Modern e-commerce offers instant inventory lookup, personalised recommendations, and dynamic media. Physical retail, by contrast, is hard-bounded by floor space, local inventory, and fitting room friction.
To solve this gap without introducing heavy, high-maintenance hardware to physical stores, I engineered Magic Mirror. A zero-install, dual-screen system that lets shoppers virtually try on full online catalogues in real time using a kiosk screen and their own smartphone.
Here is an architectural breakdown of how I built the platform, managed low-latency state synchronisation across devices, and integrated real-time generative video models into a production retail stack.
System Architecture & State Synchronisation
The core challenge was creating a seamless, low-latency link between two distinct hardware targets: an unattended, public-facing kiosk display (the "Mirror") and an unauthenticated, personal mobile device (the "Controller").
Architectural Data Flow
- Customer Mobile Interface: The shopper scans a dynamic QR code on the kiosk screen, launching a lightweight Next.js web application. This app queries the Shopify Storefront API directly via GraphQL to pull real-time inventory and product variants.
- Orchestration & State Engine: A Node.js backend manages bi-directional WebSocket channels, pairing the specific mobile session to the kiosk screen. When a user selects a garment, event payloads dispatch instantly to trigger stream generation.
- Display & AI Inference: The kiosk runs a browser-based frontend that streams real-time webcam feeds through Decart's generative video model. Generated frame outputs render back onto the screen via WebRTC streams with sub-second latency.
Key Engineering Decisions
- Zero-Install Client Architecture: To eliminate onboarding friction, the phone controller runs entirely as a web application. Scanning the dynamic session QR code opens a web session immediately, bypassing app stores, account creation, or local bluetooth pairing.
- Bi-Directional Session Routing: I implemented a lightweight WebSocket orchestration layer to link the Mirror display and the user's phone. When a user taps a garment on their mobile interface, state mutations dispatch instantly to trigger stream generation on the mirror.
- Shopify Storefront API Integration: Rather than maintaining an isolated product database, the mobile controller queries the merchant's Shopify GraphQL API directly. Live pricing, inventory variants, and asset URLs remain completely synchronized with the brand's online store.
AI Pipeline & Real-Time Video Processing
Rendering virtual garments onto a live camera feed requires high-throughput, sub-second inference pipelines.
- Generative Model Pipeline: Integrated Decart’s real-time generative video architecture (Lucy) to process inbound webcam frames and apply target product overlays dynamically.
- Frame Rate & Latency Optimization: Leveraged WebRTC stream negotiation to minimize end-to-end transport latency, preserving high framerates while maintaining a continuous "mirror" feel for moving shoppers.
- Stateless Kiosk Design: The mirror runs inside a standard, full-screen browser context. No customer biometric data or video feeds are stored locally on the kiosk device.
Security, Lifecycle & Edge Edge-Case Management
Operating software in public spaces requires strict session boundary management:
- Ephemeral Session Lifecycle: When a session starts, a unique, short-lived session token is generated. Tap-to-render actions are bound exclusively to that token.
- Autonomous Teardown & Reset: To ensure privacy and seamless handoff between customers, sessions automatically expire after 60 seconds of inactivity or immediately upon an explicit "Done" action, resetting the mirror to a default idle loop.
- Network Isolation: Communication between the controller and mirror is strictly mediated through backend session channels, keeping the display client safe from untrusted inputs.
Key Technical Takeaways
Building Magic Mirror highlighted several core principles about deploying interactive systems in physical environments:
- Decouple Compute from Hardware: Moving the interactive logic to the user’s personal phone eliminated the need for expensive, high-maintenance shop-floor touchscreens.
- Real-Time WebSockets Over Polling: Sub-second visual feedback is critical for natural mirror interactions; event-driven state syncing was essential to prevent input lag.
- Hardware-Agnostic Web Standards: Building the display layer around standard web technologies (Next.js, WebRTC, WebSockets) allows the system to run on virtually any smart TV, commercial monitor, or tablet setup.
Technical Summary
- Project Name: Magic Mirror
- Core Stack: TypeScript, Next.js, Node.js, WebSockets, WebRTC
- External APIs & Models: Shopify Storefront API (GraphQL), Decart Lucy AI (Real-Time Video Generation)
System Capabilities & Execution Flow
- Session Initialization: Kiosk generates a dynamic session token → User scans QR code to establish a WebSocket channel.
- Catalog Synchronization: Mobile Web App fetches live Shopify GraphQL schema → Displays available product variants in real time.
- Inference Pipeline: User selects product → Node.js orchestrator signals Decart AI pipeline → Augmented WebRTC video stream renders on the Mirror display.
- Session Cleanup: Teardown timer triggers after 60 seconds of inactivity → Flush session token and reset client state.
If you want to see more you can visit: https://www.vslabs.co.uk/blogs/r-d/magic-mirror-poc