Mobile
Web
AR

Remote Assist App

mobile phone viewing an object in augmented reality

Overview

We’ve built a prototype of a mobile app to show how technologies such as AR and WebRTC for mobile and web can be used to solve problems over distances.

main technologies
ARCore
ARKit
WebRTC
Flutter
Metal
Ruby on Rails
React
Year
2021
Service
Mobile app development
Design
Augmented reality
Technologies
ARCore
ARKit
WebRTC
Flutter
Metal
Ruby on Rails
React

Augmented Reality Devices for Remote Assistance

The global number of smartphone users in 2021 was 3.8 billion. Out of that number, there’s an estimated 2 billion devices with AR capability (1.25 billion running iOS and 851 million with Android), which makes smartphones an excellent and highly affordable vehicle for augmented reality. 

With so many AR-enabled smartphones, access to mobile remote assist becomes significantly greater, giving everyday users a convenient tool that lets them easily connect with specialists (or their friends) to get advice. 

Remote Assist App brings together two people: one in need of help, the other willing to provide it. When a person in need of help gets connected with an assistant, the assistant immediately sees the user’s environment and can point at specific elements in that environment.

Help centers could use Remote Assist App to instruct customers how to troubleshoot problems with a router or other appliance. This could potentially decrease the time to solve the problem and eliminate many friction points that occur when the assistant can’t see the device in question.

Use Remote Assist App

Try on web as a consultant
or download it from
App store app buttonGoogle Play app button
people in front of a glass wall with notes

Building Remote Assist App

Building Remote Assist App was a highly demanding task and an inspiring challenge. During development, we encountered many issues:

  • Overcoming WebRTC's entry barrier. We had to invest some time to understand what’s going on under the hood. We wanted to acquire an in-depth knowledge of WebRTC to be able to use it according to our requirements. That's why we resigned from ready-made solutions such as Twilio.
  • Ensuring compatibility and stability between iOS, Android, and web. WebRTC is a flexible technology with many codecs — we learned how to configure WebRTC to make the system efficient and compatible across platforms.
  • Connecting AR with WebRTC. An optimal image transfer for a smooth frame to frame transition and stability of the app.
  • Making the app more UX friendly. We learned how to design the conversation screen and take advantage of the whole user screen.

Detailed application breakdown

We built the mobile apps for different platforms in Flutter. We wanted to rely on Flutter entirely but none of the existing modules met our expectations with AR quality. This led us to writing our own native modules responsible for the connection between users and AR drawing.

The web client is written in Next.js — it’s perfect for building efficient prototypes quickly.

All that is supported by our backend. The backend was written in Ruby on Rails, and it’s responsible for packet handling between users.

Use Remote Assist App

Try on web as a consultant
or download it from
App store app buttonGoogle Play app button

Remote Assist App use cases

1

Remote training

Remote Assist App helps teachers provide better learning opportunities for students by holding lessons everywhere, without the need for physical contact. By using the app, teachers simply walk the students through the subject in question, pinpointing elements that need more explanation.

2

Machinery repairs

Remote Assist App fits into contexts where the help of skilled engineers is needed on-site, but they are hundreds of miles away. With the assist app, engingeers can instruct on-site workers and guide them through the fixing process of highly specialized machinery.

3

Car service

Cars increasingly rely on electronics and computer-operated maintenance. Car shops may often lack the specialized workforce to perform repairs according to the manufacturer's requirements. Remote Assist App is an easy way through which mechanics can concult with engineers and service a car.

4

Call centers and technical support

Call center operators have a carefully designed guidelines on how to troubleshoot customer problems. However, they lack a visual on the problem that could let them pinpoint the exact problem faster and improve issue resolution times. Remote Assist App lets customers show the problematic hardware and apply the consultants advice faster.

5

6

What Is Remote Assist App?

The app is simple and intuitive — it doesn’t require login, email confirmation, or phone number. Everything is anonymous: user pairing happens when the user shares a code with a consultant.

After launching the app, the user can choose the mode:

  • “I need a consultant’s help” — this mode generates a code that the user has to share with the consultant. The user also shares his back camera view with the consultant.
  • Consultant mode — the consultant assists and helps the user solve a problem. The consultant enters the code the user shared with him and shares the front camera.

After giving the code, the user starts the connection by clicking “start a call” and waits for the consultant.

After the connection is established, the consultant and the user can hear each other. Both can see the user’s back camera view. They can draw in their assigned color and place 3D objects on detected surfaces.

After the call, the user can rate the service and give feedback on the app.




View prototype in Figma
a user flow of remote assist app

Use Remote Assist App

Try on web as a consultant
or download the app to use both modes
App store app buttonGoogle Play app button

Validate your tech idea in our boutique R&D department

Contact us
next project

PONS XR Interpreter

Mobile
AR

PONS XR Interpreter is an augmented reality proof of concept for communication and real-time translation. Built for Magic Leap, XR Interpreter lets professionals work together across distances and languages.

View project
Remote Assist app view

Validate your tech idea in our boutique R&D department

Contact us
See the product in your space
View in AR
Want to view the product to see how AR works?
Show product in AR
Visualize it better in AR
See in your space
On a laptop? Scan the QR code below to see how it works.