RIT connects the real and the virtual at CEATEC

Fancy being able to flick through books at bookstores with the ease of a Google search? Or mix and match clothing suggestions from online stores before you buy? Or virtually clink glasses with fellow patrons at a bar?

Well, you’re in luck! The Rakuten Institute of Technology’s booth at this year’s edition of one of Japan’s largest tech conferences, CEATEC (Combined Exhibition of Advanced Technologies), exhibited all three of these technologies, and thrilled many of the event’s estimated 130,000 visitors in the process.

RIT’s Takako Yoshida, who was attending the booth, explained that the book-scanning application, known as zapzap, was particularly popular with visitors.

“We received a lot of attention for zapzap because of the way it combines different types of technologies and data,” she said.

Users are invited to select a real-life book and place it on a touchscreen. The book cover is then captured by an overhead camera and the book is identified using the Rakuten Kobo database of eBooks. Keywords from the book’s content are then displayed around the physical book on the touchscreen and, when they are clicked, relevant quotations from within the book’s text are displayed.

“The idea is that it improves on the old practice of flicking through a book in a bookstore,” Yoshida explained. “Now you can immediately identify and display the parts of the physical book most likely to interest you.”

zapzap caught the eye of a group of journalists from the United States who had been invited to CEATEC to judge the hundreds of corporate displays and award prizes. zapzap was chosen as a finalist for both the “Home Entertainment” and the “Software Computer and Networking” categories.

The judges seem to have been impressed with the way the program combines image recognition, big data and content-generation. As RIT’s Masahiro Sanjo explained, “We’ve achieved a cycle from physical product, to data and then to digital content – in the form of the quotations.”

Another RIT display that was popular with visitors was the KiTeMiROOM. The program displays composite images of clothes, shoes, hats and accessories that it recommends based on the user’s gender and also the color of clothes they are wearing at the time.

“Visitors are most impressed by the fact that the program uses real images taken from the Rakuten Ichiba marketplace,” Yoshida said. “The products can really be bought.”

While the version of the program on display uses images from a single merchant, the same approach could possibly be applied to pull images from the vast database of clothing images available on Rakuten Ichiba. The program automatically removes the backgrounds and resizes the resulting product images so they can be assembled into a composite as though they are being worn by a mannequin.

Another point that impressed visitors was the quality of the recommendations. “People were impressed with how well the suggested accessories went with the suggested clothes, and so on,” Yoshida said.

It turns out that the secret is AI. “The program incorporates AI and we have actually trained it to make appropriate recommendations based on the same kinds of color theories that human fashion coordinators apply,” she explained.

And, last but not least, the virtual glass-clinking technology, PhySig, attracted its fair share of attention, too. The idea is that customers in a bar, for example, would be able to use their smartphones to pick up and hold virtual drinks displayed on a shared public screen, and then clink glasses of other patrons.

“The key technology is one that RIT has been applying in a number of other projects – whereby multiple users are able to use their own smartphones to control a cursor on a screen in a public place,” Yoshida explained.

CEATEC 2016, which finished today, lived up to its billing as one of Japan’s leading tech showcases. The event was covered widely in the international media, with robots attracting the majority of attention.

One of the most popular new products on display was Toyota’s tiny companion robot called Kirobo Mini. At just 10 centimeters in height, it’s small enough to fit on your hand or your bag, but also smart enough to carry out conversations and read facial expressions. Importantly, it can remember places it has visited and also recall previous conversations–abilities that Toyota hopes will encourage users to develop affection for it. The robot mimics the intelligence and speaking habits of a 5-year-old child.