VR surgery is becoming a reality, thanks to Holoeyes

In Japan’s hospitals, virtual reality (VR) is becoming… reality.

“My company provides VR and MR (mixed reality) applications for doctors, using 3D images from patients’ CT scan data,” says Naoji Taniguchi, whose company Holoeyes won the “Technology Award” at last October’s Rakuten Technology Conference 2018 in Tokyo. Taniguchi sat down with Rakuten Today to discuss his company’s role in the future of medical technology.

Innovation in the operating theater

Hospitals have traditionally used 2D images – CT scans, or CAT scans – to guide surgeons before, during and after surgical procedures. Holoeyes was born when Taniguchi and his co-founder sensed that two dimensions might not be enough for such delicate work.

Holoeyes’ technology is in line with modern medical trends, as hospitals strive to improve the quality and safety of the care they offer patients by embracing new techniques and technologies. “Today, surgery tends to be of the minimally invasive variety, like laparoscopic surgery, for example. But for surgeons, the field of vision is very narrow.”

Holoeyes uses layered CT scans to construct 3D figures of patients’ organs which surgeons are able to examine in detail. Taniguchi emphasizes the need to have precise 3D data for each individual patient.

“Every person has a different face, right? Our body is the same,” he explains. “We all have different vein structures, our bones are shaped differently and so are our organs.”

Using these figures, surgeons can then visualize the 3D organs using VR and MR headsets.

“Mixed reality works well during surgery, because doctors can still see the operative field with the headset on,” Taniguchi says. By overlaying 3D images on top of the surgeon’s field of vision inside the operating theater, the patient’s organ data can be consulted as they work.

“VR is good for before and after surgeries. Our application can record the movements of the headset and controller made during surgeries, and it allows doctors to draw lines in the 3D space,” Taniguchi explains. “It’s a good way to record surgical procedures, or any ideas the surgeon might have.”

Outsider innovation

Taniguchi himself hails from outside the medical world. “My background is in computer graphics research and development, and as a video game developer and designer,” he shares. It wasn’t until he met his co-founder Dr. Maki Sugimoto that the idea for Holoeyes came about.

Taniguchi was building a database for a Japanese publishing company looking to publish a new medical dictionary. “I was surfing the web when I found an article from Dr. Sugimoto,” he recalls. “In the article, he talked about how using super high-resolution video or motion sensors (like a Kinect) would become important for the future of medical technology.”

It was technology that Taniguchi was already familiar with, so he dug deeper. “I searched his name and found his Twitter account. I mentioned him in a Tweet and we met up.”

The path to mainstream acceptance

Holoeyes is wasting no time in the development stage – its software is already in use at several Japanese hospitals for procedures like orthopedic and liver surgeries. “We also want to use it for brain surgeries and urology, or kidney surgeries,” Taniguchi adds.

Getting approval for use of the technology in the operating theater was no easy feat. “This is a totally new technology for medicine. In Japan, there is no technology like it,” Taniguchi explains, comparing the process to the medical world’s gradual adoption of deep learning for medical imaging.

While use of the software is currently restricted to Japan, the company has plans to expand to Singapore and Germany in the near future.

The next step

Taniguchi isn’t restricting his technology to the operating theater. He has a grand vision for putting the Holoeyes to even more uses – something that is already becoming a reality in the classroom.

For Taniguchi, Holoeyes is more than just VR. “As a next step, we want to provide a library of 3D data to share with other companies and hospitals,” he says. “My concept is basically GitHub for doctors. We want our company to be a platform for exchanging skills and ideas between doctors and students.”

For more information about the other speakers and events at Rakuten Technology Conference 2018, visit here.