Can AR plus WebRTC let a remote expert point at things in your space — live, across phone, and web?
We built a prototype to show how AR and WebRTC — across mobile and web — can solve problems over distance. One person needs help; the other provides it. The assistant immediately sees the user's environment through their camera and can point at specific elements, draw, and place 3D markers in real time.
Remote Assist App
2021
Phone and chat support force the customer to describe a malfunction verbally — slow, error-prone, and frustrating. Wanted to test if the AR and real-time-video layers could be made to work together reliably across platforms.
A cross-platform prototype that lets expert sees the user's space and can annotate it: draw in their own color and place 3D objects on detected surfaces. Pairing is anonymous — no login, email, or phone number; users just share a code.
The problem space
Frame the frontier that motivated the R&D, grounded in real data.
The cost of "I can't see what you're seeing" is enormous — longer support calls, unresolved issues, unnecessary truck rolls, and field engineers travelling hundreds of miles to point at a part. AR remote assistance collapses that distance: the expert sees the real environment and annotates it directly. The frontier question for 2021 was feasibility — could consumer-grade AR and browser-grade real-time video be combined into something stable and cross-platform enough to be useful?
smartphone users worldwide (2021)
of those devices were AR-capable
AR-capable iOS vs. Android devices — a huge, ready installed base for spatial assistance
Technology choices
What we evaluated, what we chose, and why.
We deliberately learned WebRTC deeply rather than leaning on a ready-made service, so we could configure codecs and behaviour to our exact requirements for AR-grade video.
Flutter gave us one mobile codebase — but no existing Flutter module met our AR-quality bar, so we wrote our own native modules to handle the connection between users and AR drawing.
Native AR frameworks underneath, for surface detection and placing 3D objects in the real environment.
Ideal for building an efficient consultant-side web app quickly, so experts can assist from a browser without installing anything.
Handles packet routing/signalling between users.
Rejected. Off-the-shelf solutions didn't give us the control we needed over the real-time pipeline; we resigned from them to meet AR-specific requirements.
Rejected. No available Flutter AR module met our quality requirements — which is why we built native modules instead.
The POC in action
The working thing — capabilities, not a scope list.
The consultant sees the user's back-camera view and can draw in their assigned color and place 3D objects on detected surfaces — pointing at the exact part in question.
No login, email, or phone number. The user generates a code, shares it with a consultant, and the session connects — friction-free and privacy-respecting.
The person needing help is on a phone; the expert can assist from a browser, widening who can act as a consultant.
After the call, the user can rate the service and give feedback — useful for a support-quality loop.
Results & takeaways
Honest feasibility findings.
A remote expert can reliably see and annotate a user's environment — the core experience works cross-platform.
Going deep on raw WebRTC (instead of Twilio) is what let us hit compatibility and stability across many codecs and three platforms — and configure image transfer for smooth, AR-grade frame-to-frame quality.
No off-the-shelf Flutter AR module met our quality bar, so we built custom native modules. Connecting AR to WebRTC for stable, smooth image transfer was the central engineering challenge — not a plug-and-play integration.
Field-service machinery repair, car service, call-centre/tech support, and remote training are where see-what-I-see assistance pays for itself fastest — the natural path from prototype to product.
