How Virtual Reality is Changing Surgery in Japan – Holoeyes

Links from the Founder

• Everything you ever wanted to know about Holoeyes
• Follow Naoji on Medium
• Follow him on twitter @tani_yang
• Friend Naoji on Facebook
• See Holoeyes in action
  • https://www.youtube.com/watch?v=nrYlsSldXSM
  • https://www.youtube.com/watch?v=Fu9RU03PPho
  • https://www.youtube.com/watch?v=ANN64JeUjog&t=2s

Transcript from Japan

Disrupting Japan, episode 82.

Welcome to Disrupting Japan, straight talk from Japan’s most successful entrepreneurs. I’m Tim Romero, and thanks for joining me.

The medical industry is one of the world’s most highly regulated and hard to disrupt. And for the most part, that’s a good thing. But there are a number of innovative start-ups that have ways of improving things. Not disruptive change, mind you, but simple, more cautious, incremental change that will make life better for everyone. Holoeyes is one of those questions. And today we sit down with Naoji Taniguchi and we talk about how their VR solution is winning over doctors all over Japan, and changing the way surgery is done.

Holoeyes builds up a virtual reality model of organs from CT scans, and let’s doctors analyze and discuss these matters much more directly and efficiently than they could before. We’ll get into the details during the interview. But one of the things that impressed me the most about Holoeyes, is that is is already in use today. So much VR tech and so many VR companies have an amazing wow factor, but only the promise of future applications.

But you know, Naoji tells that story much better than I can. So let’s hear from our sponsor and get right to the interview.

Tim: So I’m sitting here with Naoji Taniguchi of Holoeyes.

Naoji: Yeah.

Tim: This is an application that uses AR and VR for medical training, and thanks for sitting down with me.

Naoji: Okay.

Tim: Can you tell me a bit more about the application and how it’s used?

Naoji: Holoeyes make customized model for each patient. For VR, our mixed reality, our product helps communication between doctor, surgery team members, or training senior doctors and new doctors.

Tim: So let’s just walk through from start to finish how it’s used. So how do you build up this VR model?

Naoji: Partially use Diacom Viewer. Diacom Viewer is viewer of CT scan image. Now we are trying to use deep learning to automate, create, make part of a model from CT scan image.

Tim: Okay, so it’s laterally taking a CT scan and building up the VR image kind of slice by slice?

Naoji: Yes, yes.

Tim: Alright, that makes sense. And then the doctors can use this to communicate and to show the model.

Naoji: Yes.

Tim: Now the demo I saw was basically viewing the model and using AR to zoom in and to rotate. What is the application for that? Would that be for example, teaching—

Naoji: What they call 3-D movement. I think 3-D movement is very important for surgery. Before our product, doctors have to write in tickets some procedure of surgery. So it is very complex. So using our product, doctors do surgical procedure with game controller and head-mount display this way. They record the whole procedure of surgery.

Tim: So for example, the lead surgeon could run the entire operation with the surgical team before the surgery, and say, ‘This is what we’re going to do.’ And then everyone will be able to see it. And then when it comes time for the surgery, everyone understands?

Naoji: Yeah, so it’s like private mode and car racing game.

Tim: Okay.

Naoji: Do you know?

Tim: Yeah, yeah, yeah.

Naoji: It’s like that. And in using VR we can see the movement at any angle.

Tim: Okay, so it’s also used after the surgery to evaluate how it went—

Naoji: Yeah, yeah, yeah.

Tim: —and if there were any problems? Okay.

Naoji: We archive VR model on our crowd server. So we will be GitHub of surgery.

Tim: Okay.

Naoji: So GitHub has open public repository. Also GitHub has private repository to read. So if they shall want their data to be public—

Tim: Well, that’s what I wasn’t asking. But there are very strong laws all over the world about sharing patient data?

Naoji: Yeah. So our patients, they’re objective is cure so data is not important for them. But for doctors, it’s a treasure.

Tim: Sure, yeah. It’s extremely important. So far have you found patient’s being very willing to share their data.

Naoji: Nowadays the data is thought about very personal, but we use polygon. Polygon data don’t have personal information. Name, or age, or the patient’s living address.

Tim: Okay, so if you reduce the model to all polygons, and take off all personal information, then it’s fine? Oh, okay. So what platform are you using for this? Both the hardware and the software side.

Naoji: Well, hardware we are using HTC Vive using Windows PC.

Tim: Okay.

Naoji: And Microsoft Hololens using smart phone. Both device has good point and bad point. For HTC Vive, it’s a little bit expensive, but smartphone is very cheap and everyone have. But HTC Vive can display very detailed, how polygon model, and good personal tracking.

Tim: Right, right.

Naoji: But smartphone is very cheap. We use crowd server to store our data. The device download that data. Why is a high polygon called HTC Vive, and design a polygon model for a smart phone?

Tim: Medical imaging is one of the most demanding and challenging areas for image processing.

Naoji: Yeah.

Tim: The resolution has to be much, much higher than almost any other type of application. There’s very little tolerance for the artifacts of glossy compression. So is the resolution of your VR models, is it high enough for medical use?

Naoji: Some part is enough. For the liver we provide like a map. It’s not a photo. So we will use map to go 3-D.

Tim: Okay, so it’s really more of a reference, and it’s not a diagnostic tool so it doesn’t have to be that accurate.

Naoji: Yeah, yeah. So we will show thick veins. So if we provide all veins in liver, doctors will confuse.

Tim: Alright. Okay, okay. That makes sense. A little later I want to dive deeper into the application, but for now let me ask you a bit about you and your co-founder. So your background is in physics and aeronautical engineering.

Naoji: Yeah, yeah, yeah, yeah:

Tim: And your co-founder, Sugimoto-san is a surgeon?

Naoji: Yes.

Tim: How did you two come together on this project?

Naoji: I have a friend. He’s working as an editor. The publisher has 36 book on medical for families. The old data is digital. So he ordered me, can you create some digital service using this data. So  I thought the data in the book is like databased. During the project, I researched a lot of medical information. And I found article of Sugimoto-san. He saying modern medical needs 8K video image, so at same time I was doing a lot of interactive project. So I thought, ‘I’m interested.’ And I thought we would get on well. I found his Twitter account.

Tim: Okay.

Naoji: And I messaged him. And after that, we talked together at Tokyo.

Tim: Oh, alright. Does he act more as an advisor, or is Sugimoto-san involved in the day-to-day operation of the company?

Naoji: Yeah, day-to-day he does operation. So he use our product at hospital and looking for doctors who will use our product.

Tim: So right now you’re still in the development phase. You haven’t started charging for—

Naoji: Yeah.

Tim: —the use of Holoeyes yet? Right? Can you tell me a bit about your users and your partners? Who’s using Holoeyes today?

Naoji: Our first customer is Doctor Sugimoto and second is Bokuto Hospital in Tokyo. Some will be Sano Hospital. He’s a dentist. It is very useful for implants to see the bones.

Tim: To have like a false tooth implanted.

Naoji: Yeah, yeah, yeah, yeah.

Tim: Right. So is the main application now the planning of surgeries?

Naoji: Yeah. First is planning surgery, and next is training tool or education tool for young doctor.

Tim: So you were also apart of the Recruit Holding’s accelerator.

Naoji: Yes.

Tim: Was that valuable for you?

Naoji: Yes. For medical they don’t do nothing, because Recruit is not medical company.

Tim: Right.

Naoji: But our business is for medical, but our business model is for internet technology company. We are using crowd server, and we will stocl our patients’ data, we make value from the data. So it’s like—

Tim: So it helps you kind of understand how to make a business around it?

Naoji: Yeah, yeah, yeah, yeah, yeah. And they introduce me some with media.

Tim: So it helped you with getting the word out? Promotions and stuff. Okay, getting back to Holoeyes, there’s so many really cool VR, AR start-ups in Japan now. But what I want to ask you is—I can understand why Holoeyes is interesting—

Naoji: Thank you.

Tim: —and kind of fun to use, but what is the big problem that it solves?

Naoji: Our body’s 3-D but the doctors using 2-D monitor. So it’s a difference of dimension.

Tim: How would you measure if it’s successful? So for example, would it make surgeries go faster? Would it result in fewer surgical errors? What’s the real world benefit?

Naoji: We don’t have a quantity system made. Doctors saying, ‘Very useful.’ And maybe it’s training course. And maybe it’s time to communicate. Surgery is teamwork.

Tim: Right.

Naoji: So seeing a doctor and a younger doctor—

Tim: Okay, so the big problem it solves is it makes the planning for surgery faster, more accurate, and therefore cheaper?

Naoji: Yeah, yeah.

Tim: That makes sense.

Naoji: Our goal is making archival patient data. So it’s make big value for our archive. So it helps Asian country into grow up young doctors.

Tim: Right. So doctors in poor countries that don’t have access to cadavers can use this and train more quickly?

Naoji: Yeah, yeah, yeah.

Tim: That makes sense. So is your goal with the modeling—Are you trying to develop a complete anatomical map of the human body? Or are you also trying to model very specific—this particular tumor—

Naoji: Yes, yes, yes.

Tim: This particular kidney tumor.

Naoji: Yeah, yeah, yeah, yeah. Yeah, yeah, yeah, yeah.

Tim: So both?

Naoji: First, we focus on the particular organs: liver and kidney, or jaws. There’s a lot of patients.

Tim: Well, sure. Every start-up knows you need to start with a niche.

Naoji: Yes.

Tim: So you start with the liver, you start with the jaw. The jaw makes sense for dentistry, but why a liver and kidney?

Naoji: Liver and kidney is static organ.

Tim: Static?

Naoji: Yeah, yeah. For example, heart is always moving.

Tim: Oh, okay. So it’s easy to get a good scan?

Naoji: Yeah, yeah, yeah.

Tim: Okay.

Naoji: And a scan. And now a CT scan can cat scan heart, but it’s just snapshot.

Tim: So liver and kidney was a technological reason?

Naoji: Yeah, yeah, yes. It’s a very static organ.

Tim: Right now one of the main limitations you mentioned was processing power for the computers and for the smart phones. And every year that’s getting better and better.

Naoji: Yes.

Tim: And cheaper and cheaper. So hopeful time and intel will solve that problem for you.

Naoji: Yes, I think so.

Tim: But on the input side with moving organs like the heart and the lungs. Is that something you think technology will be solving for you in the next few years?

Naoji: Yes, I think so. So opponents of CT scan market is getting better. And technology for taking scan is getting better because CT scan makers’ goal and our goal is same. The goal is more clear image or they want to find tumors by imaging.

Tim: Today’s CT scans are very detailed.

Naoji: Yes.

Tim: So you can see very tiny veins throughout the liver for example. So just out of curiosity, how much memory is required to store your 3-D model of, say, the liver?

Naoji: Not so many.

Tim: Really?

Naoji: Yeah, yeah, yeah, yeah. Today’s video data on YouTube is more bigger.

Tim: Okay.

Naoji: And CT scan images is just a greyscale image. Greyscale is 0.2 millimeter a piece. It’s limited. It’s small as some movie.

Tim: Okay. Alright, so definitely in the range of what smart phones can handle.

Naoji: Yeah, yeah, yeah.

Tim: Now before we were talking about kind of the next step application, which is kind of debriefing after surgery. And taking 3-D movies of surgeries in progress. Last year in April 2016, an English doctor I believe is the first person who broadcast a surgery recorded in virtual reality.

Naoji: Oh, so it’s a surgery 360 movie? It’s not 3-D? Usually 3-D is we will see depths.

Tim: So the surgery last year simply used a 360-degree virtual reality camera. But what you’re talking about is actually making 3-D models of what happens during the surgery itself?

Naoji: Taking 360 video is really easy today. There are a lot of solutions on camera.

Tim: But before you were mentioning that your technology could also be used after surgeries too.

Naoji: And before.

Tim: And before. So how would it be used after? Would you just build another model from a new CT scan?

Naoji: So doctors have to write report of surgery. Usually two doctors—

Tim: So they could use your tool in the same way as before the surgery—

Naoji: Yeah, yeah, yeah, yeah.

Tim: —they use the tool to explain what’s going to happen. They can use the same tool to explain what they did and any surprises they had. Okay. So does your tool have an ability for annotations or—

Naoji: Yes, yes.

Tim: So it can connect directly into the tools that doctors already use?

Naoji: Yeah, yeah, yeah, yeah.

Tim: Oh, okay.

Naoji: Doctors can draw alright on 3-D space or doctors can put voice notation to 3-D space.

Tim: Okay, so in that way, someone reviewing the surgery would actually be able to see the virtual reality representation with the doctor’s comments, and notes, and annotations as they go through?

Naoji: Yes, yes.

Tim: I can see that being a huge benefit. Do you think that Holoeyes would also be used in actual surgical training?

Naoji: Yes, I think so.

Tim: So I noticed for example, Stanford has a project which is virtual reality training and actually have haptic feedback for the fingers.

Naoji: Yeah, yeah, yeah, yeah.

Tim: Is that a direction you’re thinking of moving into.

Naoji: Yes, it’s one direction. Nowadays doctor doing a lot of lathroscopic—Do you know?

Tim: Lathroscopic.

Naoji: Yeah, yeah, yeah, yeah.

Tim: Be the little, tiny cameras.

Naoji: Yes, yes, yes. Or robotic surgery so they can’t touch directly. Only use instrument. So now we are focusing on that because nowadays lathroscopic is very important because it’s less harm to patient.

Tim: Right. You don’t have to cut them open.

Naoji: Yes, yes.

Tim: So of all the potential. So far what have the doctors found most appealing? What are your users most excited about?

Naoji: Precision 3-D surgical image, and reducing time, and sharing the successful surgery to other doctors or patients.

Tim: Okay, so it is that planning and review functionality?

Naoji: Yes, yes.

Tim: Medical devices and hospital equipment is some of the most strictly regulated in the world. And that’s a good thing.

 

Naoji: Yes.

Tim: So have you had to get special licenses and approvals?

Naoji: Not because our tool is not diagnosis tool.

Tim: Well, that’s true. It’s not a diagnostic, it’s not treatment, but it’s being used as a medical record.

Naoji: Yes, it’s like Excel on PC.

Tim: Oh, okay. Well, I never thought of it that way. Okay. So it doesn’t need any special—

Naoji: Yes.

Tim: Alright.

Naoji: And the other reason we not get a license is for speed for the development we are using gaming technologies Unity. It’s a game engine.

Tim: Right.

Naoji: And it’s originally for video game. Gaming technology is –evolution of gaming technology is very high.

Tim: Well yeah. Gaming technology has been pushing the edge of PC performance since the first PC, right?

Naoji: Yeah, yeah, yeah, yeah, yeah. First game on IPhone 3G very poor, but nowadays high quality Android and iPhone 7 game is very beautiful.

Tim: That’s true. Let’s talk a bit about Japan in general.

Naoji: Okay.

Tim: Even if you don’t need any specific license or certifications to use your technology for medical records, the medical profession in Japan is very conservative.

Naoji: Yes, that’s so.

Tim: So do you find doctors and hospitals willing to work with the new technology or is there a lot of resistance?

Naoji: Sugimoto-san’s a doctor.

Tim: Right.

Naoji: So he loves technologies, and he’s always searching new technology outside medical.

Tim: Having Sugimoto inside pushing new technology helps I’m sure, but he’s one doctor.

Naoji: Yeah, yeah, yeah. So I think every industry’s same. Some game developer is interested in medical. Some doctor interested in interactive technology. Some scientists interested in this technology.

Tim: I don’t know. I understand what you’re saying but I think there’s a big gap between being interested and paying money and making it a part of the workflow. So what advice do you have to other start-ups who want to sell to hospitals and doctors.

Naoji: Yes, the important thing is my co-found is besides doctor. Connection to doctor is very close. They here a lot of information on the doctors.

Tim: So the advice is have a doctor as an advisor.

Naoji: Yes, yes.

Tim: Someone who’s very excited and an advocate about your technology.

Naoji: Yeah, yeah, yeah, yeah. So other start-ups in Japan in medical member co-founder is doctor. For example, there’s a start-up namely is MediWeb.

Tim: Oh, I know MediWeb well. That’s right. Their founder’s a doctor.

Naoji: Yes, so they have own clinic.

Tim: Okay. My impression is that it’s very unusual for doctors to be working with start-ups in Japan, but is it happening more and more now? Is there a lot of doctors doing this?

Naoji: Yeah, I think so. Because life of a doctor is very static.

Tim: That’s true.

Naoji: Some doctors have questions. Some young doctor has question. One intern, he is a young doctor. He hear about Sugimoto-san. He contact me. I told how to use Unity and how to build application from Microsoft HoloLens ].

Tim: So is he working with you guys now?

Naoji: Yes.

Tim: Excellent. That is excellent. That’s really encouraging. When do you plan to start charging for Holoeyes and do you know about what the pricing’s going to be?

Naoji: Beginning of next year. This year is marketing and development. So research with hospital and the hospital present our paper at a conference with us.

Tim: Excellent.

Naoji: So it is way to keep us in medical.

Tim: Have you decided the pricing for the Holoeyes system yet?

Naoji: No, I’m considering.

Tim: Okay. It sounds like you’ll have an opportunity to use a very SaaS-like approach where it’s a low-cost subscription. Like charge per seat per user.

Naoji: Yeah, I think so.

Tim: Well listen, before we wrap up, I want to ask you what I call my “magic wand” question. So and that is if I gave you a magic wand and I said you could change one thing about Japan. Anything at all. So the way people think about risks, the education system, the legal system, the medical system—anything at all to make it better for start-ups in Japan. What would you change?

Naoji: I want to change businessman to think more positive.

Tim: To have businessmen think more positively?

Naoji: Yeah, yeah, yeah, yeah. So when I show a demo for middle aged business man, they always tell about risk or luck. I always know. I want to hear about good solution from him.

Tim: That’s a really good point, yeah. I agree. I’ve been in so many meetings where everyone would basically go around the room and say, ‘Here’s why I think it won’t work.’ And, ‘Here’s why I think it won’t work.’ Almost no one comes up with one idea of, ‘Oh, this might improve it.’ Right?

Naoji: Yeah, yeah, yeah.

Tim: That’s a great answer. I believe that would be a huge benefit.

Naoji: Yeah, yeah, yeah, yeah. I already know about negative things. Right?

Tim: So instead of giving just negative ideas, also give positive ideas.

Naoji: Yeah, yeah, yeah, yeah.

Tim: That’s a good one. Okay, excellent. Well listen, thank you so much for sitting down and talking with me today.

Naoji: Okay. You’re welcome.

And we’re back.

I was pleasantly surprised at how interested in trying new start-up technology Japanese hospitals and doctors are. Medicine is a conservative industry everywhere in the world, but in Japan it’s particularly so. Of course, the fact that the other half of the Holoeyes founding team, Sugimoto-san, is a doctor, makes that acceptance a lot easier. Both by giving Naoji and the rest of the developers a resident domain expert, but also somewhat validating the product in the eyes of the medical establishment.

Oh, and Naoji also wanted me to point out that the importance of being able to run this system on smart phones was not that their business models revolved around smart phone use, but because designing for a relatively low spec device enables a greatest future flexibility and opportunities for them.

But really, I think Holoeyes has two things that most VR start-ups lack. First, they have an existing user base that is using their system today to do the job that it was designed to do. And second, they’re solving a very specific problem in a way that their users agree is a huge step forward. Five years from now, it’s almost impossible to imagine surgeons not using a system like Holoeyes to more accurately plan and document their surgeries.

If you have a great idea about how VR should be used in medicine, Naoji and I would love to hear from you. So come by DisruptingJapan.com/show082 and tell us about it. And when you come by the site, you’ll see all the links and notes that Naoji and I talked about and much, much more in the resources section of the post.

And before we go, I want to tell you a bit about our sponsors. No, no, don’t worry. We’re done with the ads for today. But over the past few months, I’ve had a number of people comment that I seem a too effusive in my praise of our sponsors. Now of course their sponsorship is what allows me to introduce you to all these amazing founders. But more than that, all of our sponsors really do contribute to the community here. And yes, I’ve turned down a couple of companies that I didn’t feel were a good fit. So hey, check them out, drop by their sites, and let them know that you hear about them here.

But most of all, thanks for listening. And thank you for letting people interested in Japanese start-ups know about the show. I’m Tim Romero, and thanks for listening to Disrupting Japan.