How mixed reality could spice up bricks-and-mortar shopping

Imagine walking into a store, picking up a canned drink and instantly hearing what it sounds like when poured into a glass. If that wouldn’t whet your thirst, what if there were also videos, reviews and information related to the drink appearing right before your eyes. How would it be possible? With a technology known as mixed reality (MR).

MR is the blending of physical and digital worlds through the use of devices such as head-mounted displays (HMDs) and smartphones. The idea is that text, sound or video from the online world could be fused with and made to respond to objects in your immediate surroundings.

Digital meets physical

“I believe mixed reality is the future of computing,” says Dr. Kelvin Cheng, a research scientist in the Rakuten Institute of Technology’s Computational Interaction Group. “We had desktops, and then laptops and now we have mobile phones. So what comes next? I think MR is the next thing in terms of interacting with computers.”

According to Cheng, MR encompasses both augmented reality (AR), which adds some data or graphics to real-world views, and virtual reality (VR), which is a fully immersive, complete virtual representation of a world.

The AR and VR paradigms have been around for years. What makes MR different is that it allows for responses to what is going on in the physical world. So for example it can be made to recognize and respond to objects and their orientation. Cheng has taken advantage of this feature, known as context awareness, to enhance the shopping experience.

MR-Shoppingu, as the system is known, draws upon Rakuten’s vast experience with e-commerce. Cheng gave Rakuten.Today a demo using a Microsoft HoloLens and some cans of Yona Yona Ale. After putting on the viewer, Rakuten.Today picked up a can and noticed a glowing purple circle appear around it in 3D. As we rotated the can, its different faces triggered different kinds of displays, all of which appeared magically alongside the product in front of us. Pricing information, customer reviews and even a video of the beer being poured into a glass could be seen through the headset – making for a far more compelling pitch than, say, a poster showing a glass of frothy beer beaded with condensation.

Using natural gestures

While HMDs such as HoloLens are currently rather bulky, they’re expected to become as lightweight and unobtrusive as eyeglasses in the future, replacing smartphones as consumers’ main mobile devices.

“These glasses will be able to detect who you’re with and what you’re interacting with in the sense of physical activities like going to the supermarket and picking up products,” says Cheng. “In addition, one of the elements within context awareness is the fact that the system can detect natural actions and behaviors from users. These actions might trigger online content such as a video or product reviews. This means users don’t need to learn any new interaction or gestures.”

Just like with Rakuten Ichiba today, the system would learn user preferences over time, making recommendations upon entering a bricks-and-mortar shop, for instance. It could also direct users to items they’re searching for in large stores, such as supermarkets or pharmacies. Merchants could use it to instantly see product information such as sales, returns and inventory. Additionally, it could recognize clothing or other goods spotted on the street and call up the brand, model, price and availability.

Building an interactive world

With a background in human-computer interaction research at the University of Sydney and Australia’s Commonwealth Scientific and Industrial Research Organisation, and experience lecturing on Interaction Design at the University of Singapore, Cheng is profoundly interested in how computing devices will change in the future. He has also worked on interaction methods for wall-sized tiled displays, tabletop systems, and collaborative digital walls using personal tablet devices with multiple users.

“Ever since the beginning of my graduate studies, I’ve been fascinated with the question of what if all the walls, ceilings, floors or tables in the room were interactive rather than just projection surfaces,” he says. “How would we interact with them naturally without a mouse and keyboard? MR headsets are the perfect tool that enables us to not only interact with the room around us, but also the world around us.”

MR-Shoppingu was introduced at the IFIP International Conference on Entertainment Computing (ICEC 2017) last month, and received the Best Poster Award. Details of this work can be found here: K. Cheng, M. Nakazawa, S. Masuko, MR-Shoppingu: Physical Interaction with Augmented Retail Products using Continuous Context Awareness