Surgery in virtual reality: How VR could give trainee doctors the feel of real patients
A virtual operating theatre is helping train up surgeons on new procedures.
Virtual reality is often touted as a way of creating fantasy universes, but it could also turn out to be an effective way of teaching skills that are hard to practice in the real world.
Take training up the doctors of tomorrow, for example. US university Case Western has already announced it plans to do away with its anatomy labs, and the cadavers that go with them, and teach medical students with Microsoft's HoloLens 'mixed reality' system instead. Aspiring doctors will be able to wear HoloLens headsets, and view the different layers of a body -- skin, muscle, blood vessels, and so on -- in 3D.
But going one step further, one UK company is trying to recreate the hands-on aspects of surgery in a VR setting, allowing students to get a sense of how the human body feels in using haptic feedback.
Fundamental VR, based in London and Guildford, has added a haptics element to virtual reality to allow medics to train without having to test out their nascent skills on an actual patient.
The system combines the HoloLens headset and the company's software with a stylus connected to a standard-issue mechanical arm.
The stylus appears as a syringe in the VR world the wearer sees, with one button to empty the syringe, and another to refill it.
Moving the stylus in the real world moves the syringe in the simulation, and when the virtual needle meets the virtual skin, flesh, or bone, the varying resistance of the material is transmitted through the stylus to the user, giving them a powerful facsimile of a real-live body.
The idea is that encountering different elements of the body -- like fat or bone -- should feel very different.
The first system, set up to resemble a total knee arthroscopy, was custom-built for the drug company Pacira to teach clinicians how to do a procedure using one of its products, an anaesthetic called Exparel.
Unlike traditional most anaesthetics, where a larger dose is injected in one go and spreads out widely from the injection site, Exparel is injected in several doses and stays largely where it's put. For some surgeons, the change in procedure was difficult to grasp, and so the VR teaching tool was born.
The imagery for the system was created by taking a series of photos of a knee to build up its 3D counterpart. Off-the-shelf haptic hardware is used to stand in for the syringe in the Pacira system, but could equally act as any medical tool that's needed.
Surgeons weigh in
Building a system that could faithfully recreate the experience of surgery required a mixture of human and technological smarts. In order to build the VR setup for knee replacement surgery, the company canvassed the opinions of orthopaedic surgeons on the steps that make up each procedure.
"Surgery is about science, but also about art, and where there's art, there's opinion. Getting to a common standard where people agree what's the right way to do that and on best practice, that took some time. Once we had got that, we were ready to start embracing some of the challenges of texture and tissue types and how those change throughout the procedure," Richard Vincent, cofounder of Fundamental VR, told ZDNet.
Next the surgeons were enlisted to help convert the real-life experience of surgery into a virtual version, with the company's haptics development engine the bridging the real and VR world.
"We built a calibration tool: it's the core of our software that allows us to quickly translate what are quite difficult things to communicate into numbers... we started with people saying 'it's like sticking a needle into an orange' or 'it's like chicken', then you can basically adjust it in real time until they agree it's how they feel and average that out," Vincent said.
Dr Stan Dysart, an orthopaedic surgeon that specialises in joint replacements at Georgia-based Pinnacle Orthopaedics, was among the surgeons that contributed their first-person perspective of knee operations to help the system recreate the authentic feel of surgery, assigning each element of the human body a number that corresponds to a certain texture.
"The haptic device has a scoring system, and I helped them decide what a needle feels like in capsule, what it feels like in muscle, in fat, in periosteum, and what it feels like on bone," he said. Fat, for example, is extremely forgiving, while the capsule has a fibrous, plastic-like texture.
"The capsule has a certain resistance, and when you go through the capsule, resistance releases, so you can score that -- you can score that [level of haptic] feedback, and score a different feedback for every part of the knee. You give it a number, and computers understand numbers -- the higher the number, the greater the resistance the surgeon will feel," Dysart said.
Once the surgeons have agreed on the haptic-feedback rating for each layer of the body, from muscle to bone, the haptics system can translate that back into the level of feedback the VR wearer will feel when they apply the virtual syringe -- a matter of balancing the amount of processing that the scenario needs with the abilities of the GPU underpinning the system to produce a smooth experience for the user.
The total knee arthroscopy-related system is already being used by surgeons in centres across the US, and it's helping surgeons refine their techniques, according to Dysart.
"Surgeons love it. They enjoy the experience, they enjoy practicing without potentially damaging a live patient. That's where it's important. Everything we do in live surgery has a consequence -- how deep do you cut? where do you cut? where do you inject? -- because there are nerves and arteries all about the knee.
"In virtual reality, if you plunge the needle too deeply, nothing is injured. You realise you've done it incorrectly, and you can do it over and over until you have the right technique. That's the beauty of this," he said.
Alongside the total knee arthroscopy, Fundamental VR has several more custom setups in the pipeline that it expects will go live at some point this summer.
Teaching tool
Fundamental VR is already talking to educational institutions about how haptics-based systems could be used to teach students to improve their skills or help established doctors learn new procedures before they try them out on the wards. For now, Fundamental VR is concentrating on the US market, though it has had conversations with teaching facilities both in London and abroad.
As well as building more specific one-off systems for clients in future, the company also expects to create a library of common procedures that can be accessed on a subscription basis. Long with 'standard' anatomy, the company could potentially create variants to introduce students to some of the rarer anatomical variations or conditions.
"[Removal of] the appendix is still the most performed operation, so having a better way of teaching that would be useful for lots of people, but on the flip side, there's a lot of opportunity [for doctors] to be around that and observe that," Fundamental VR's Vincent said.
"But if you go into neurology, there may be something you only see three times a year, but it's a life-and-death situation. The number of people that need that training is much less and it might be harder to make the business case, but the human case in much stronger," he continued.
While haptics system might not go over well with all surgeons -- some more senior clinicians found the simulations a bit too close to computer gaming -- the company prefers to liken it to the way pilots use flight simulators.
"We go to lots of conferences where we talk to lots of surgeons about how, say, when you face this bleed at this moment, and you've got five minutes to deal with it, it's never going to make that a less traumatic moment when it happens, but if you go through a simulator that gets you close to it a few times, that has got to be good thing," Vincent explained.
Practicing on VR, not patients
Once mixed and virtual reality become cheaper and more common, haptics and VR could be used to create models of individual patients before they undergo surgery.
"If you could create from those scans something where we could share the kidney, move it around, agree how to get in there, what's the plan, how do we make the surgery the quickest and most effective, that would be good for patient safety," Chris Scattergood, Fundamental VR's co-founder, said.
The future of medical VR, then, will be a mix of teaching students and professionals how to do high-volume, routine operations of the kind that are done hour after hour in hospitals across the world, as well as to understand niche procedures that clinicians at the highest level may only see once or twice in their lives.
Either way, while doctors are practising the skills they need to perform the procedures, they'll be learning virtually, making their mistakes away on a computer system and perfecting their techniques long before they get to use them on their patients.
Read more about medical technology
- Google's DeepMind and the NHS: A glimpse of what AI means for the future of healthcare
- #WeAreNotWaiting: Diabetics are hacking their health, because traditional systems have failed them
- Playing House: How IBM's Watson is helping doctors diagnose the most rare and elusive illnesses
- Bioprinting bones and muscles: The inkjet cell printers shaping the future of transplants
- More by Jo Best