Augmented Reality in Healthcare: Applications, COVID-Related Examples, & Deployment Tips

What is augmented reality, and why is it getting traction in healthcare?

Augmented reality (AR) is a technology that blends elements from the real world with the virtual ones by overlaying computer-generated content on top of a live user environment.

Speaking of the medical sector, AR aggregates health-related information, such as patients’ medical records or 3D models of internal organs composed from patients’ medical imaging data, and projects all this as a virtual component for the doctor to view while examining a patient in real time. AR is just one type of existing immersive technologies. Extended reality (XR) is the umbrella term that covers all immersive technologies: AR, VR, and MR. Virtual reality (VR) removes users from their surroundings and fully immerses them in an artificial digital environment with the help of VR headset. Mixed reality (MR), or hybrid reality, allows virtual and real-world objects to coexist and interact together in real time. During the surge of COVID-19, augmented reality is more relevant than ever.

With the lockdown and social distancing measures in place, it is difficult to arrange the presence of external consultants during surgical procedures and to conduct group trainings for new healthcare practitioners. Moreover, hospitals are overloaded, and doctors have less time to spend on each appointment. Augmented reality with its use cases in healthcare caters to the needs of the medical sector during the pandemic, as you will learn from the following section.

5 medical applications of augmented reality

AR helps conduct complex surgeries
AR improves mental health therapy
AR facilitates diagnostic procedures
AR takes medical training and education to another level
AR contributes to pain management

1. AR helps conduct complex surgeries

It has been a challenge to bring AR to surgery, as superimposing computer-generated images (CGI) on top of a camera or smart glasses’ view leads to performance and perception issues. If you want to put virtual objects into focus, then real objects will become blurry and appear at different distances, which is a recipe for disaster during medical procedures. Another option for augmenting surgeries involves displaying virtual information on a separate screen, which would cause surgeons to constantly take their eyes off the patient. Incorporating augmented reality into healthcare allows projecting virtual objects within the doctor’s line of sight and it doesn’t interfere with how real-life objects appear. The first-ever augmented reality-assisted surgery took place in February 2020 at Policlinico Universitario S. Orsola of Bologna. The lead surgeon wore VOSTARS, which is a visor integrated with AR. The device enabled him to view virtual elements, such as the patient’s heart rate, body temperature, breathing rate, and some CT and MRI scans taken prior to the surgery. Dr Philip Pratt, a Research Fellow in the Department of Surgery and Cancer at Imperial College London, speaks about the benefit of augmented reality tools for surgery:

“This technology allows us to experience the data that we have collected from patients before their operation in the most realistic and natural way. You look at the leg and essentially see the inside of it.”

Augmented reality in spine surgery One FDA-approved augmented reality tool for spine surgery is Xvision Spine System developed by Chicago-based Augmedics. This is a navigation system that guides surgical tools’ movements. Xvision determines the position of surgical tools in real time and builds a virtual trajectory imposing it on the patient’s CT data. The whole model is then projected on the surgeon’s retina through the headset. That way, the surgeon can see both the patient and the navigation data at once. Dr. Frank Phillips, Director of the Division of Spine Surgery at Rush University Medical Center, was the first to use Xvision to perform a lumbar fusion with spinal implants. The tool enabled Phillips to see the patient’s spine anatomy in 3D through the skin. Here’s how Dr. Phillips described his experience:

“Traditional surgical navigation platforms have been shown to improve accuracy of implant placement, however using augmented reality allows for the advantages of traditional (non-3D) navigation plus the ability to visualize the patients spinal anatomy in 3D through the skin.”

Augmented reality in brain surgery An example of AR-assisted brain surgery comes from LA-based Surgical Theater. The healthcare startup has partnered with Medtronic, a medical device company, to enhance its augmented reality technology with Medtronic’s StealthStation S8 surgical navigation system. This new technology will enable neurosurgeons to visualize brain structures and test different surgical equipment virtually before entering the operating room. Augmented reality in heart surgery EchoPixel, a medical visualization software provider, introduced an intraoperative program that delivers a 3D anatomic visualization of the human heart. This model is composed of different types of a patient’s medical images and displays blood vessels and other heart structures. This invention allows the surgical team to interact with the 3D model as if it was a real-world object. They can see patient-specific anatomy, capture accurate measures and distances.

Plastic surgery with augmented reality With the help of augmented reality in healthcare, patients can see how their appearance changes after a reconstructive surgery even before entering the operating room. One example comes from dentistry. New Jersey-based SmarTek developed an augmented reality medical app, which uses a smartphone’s camera to apply virtual depictions of an improved set of teeth to the current patient’s photo. That way, the patient knows what to expect, and together with the dentist, they can adjust the features based on the patient’s preferences.

Augmented reality surgery in the COVID-19 era The pandemic presents a unique challenge for the healthcare software development companies and medical industry, as the novel virus is highly contagious and the risk of infecting the surgical team is rather high. Also, due to limitations imposed on travel, senior surgeons are not always able to attend and advise their junior colleagues. Augmented reality-assisted telesurgery looks like a viable option in these circumstances. The British Journal of Surgery describes the case of a 59-years old patient who was admitted to a hospital with a persistent cough and medical history of diabetes and other disorders. His COVID-19 test turned positive, and the patient needed an immediate surgical intervention to alleviate COVID-related complications. Despite the difficulty of this case, a junior attending surgeon performed the operation. He used an AR-enabled telesurgery platform, which let him consult two other surgeons who could not be physically present.

2. AR improves mental health therapy

Augmented reality is diverse in its medical applications as it spans over the realm of mental well-being. For example, augmented reality can treat phobias. In safe settings, doctors can help patients immerse into a virtual reality that would trigger negative responses, such as fear and anxiety. Under a professional’s supervision, patients learn to control their feelings, thereby overcoming the mental health condition. Additionally, AR can be used for relaxation and meditation. Augmented reality treats mental illnesses during the pandemic The coronavirus infection and the subsequent lockdowns created extreme circumstances and took a toll on people’s psychological behavior. During the pandemic, four out of ten adults reported symptoms of depression or/and anxiety, compared to only one in ten experiencing similar symptoms during the period of January-June 2019.

The Journal of Medical Internet Research published a paper describing the positive impact of playing augmented reality games during the pandemic. Within this study, three out of four participants reported that playing AR games was beneficial for their mental health. The participants also mentioned that AR games encourage them to exercise more often. Just recently, the University of Alberta developed YOU-AR-OK, an augmented reality healthcare app that prompts users to reflect on their feelings. This program features an avatar in the form of a dog. As soon as users create an account, this virtual dog is transported into their physical space. The animal asks users questions such as “how are you feeling?”, “did you get enough sleep?”, etc. With each question, people will open up without fear of judgment and recognize their psychological patterns.

3. AR facilitates diagnostic procedures

Clearly communicating the symptoms plays a vital role in accurate diagnosis. However, many patients struggle to describe their symptoms realistically, without downplaying or amplifying them. Augmented reality in healthcare can help both patients and doctors understand medical conditions better. Moreover, AR can help doctors handle other diagnosis-related tasks, such as taking medical notes. Making veins easier to spot Every person’s anatomy is different, and with some patients, it is rather hard to spot their veins. According to statistics, 40% of intravenous injections miss the vein on the first attempt. AccuVein, a global producer of medical imaging solutions, has solved this problem. The company developed a device that uses AR to illuminate veins, enabling medical professionals to view the vein map clearly on patients’ skin while administering intravenous injections. Bringing important data into view While diagnosing patients, physicians must constantly look up relevant information on their screen, scattering their attention between their PC and the patient. Atheer, a California-based startup, developed augmented interactive reality (AIR) glasses that are connected to an Android-powered computer. These glasses can display relevant patient data within physicians’ field of view during checkups. Additionally, doctors can interact with the presented information via voice commands and hand gestures. Enabling self-diagnosing Augmented reality-based healthcare apps permit users to perform self-checks, monitor their conditions, and contact a professional when needed. SkinVision uses machine learning, a subset of AI, and augmented reality to assess any suspicious spots or moles on the skin. All you need to do is to take a photo of the affected area using the app, which will analyze the spot and classify it as high/medium/low risk. You can set a reminder to repeat the procedure to continuously monitor the progress. All these images are stored in the SkinVision database.

Augmented diagnosis in the coronavirus era Taking notes during patient examination is a time-consuming task that prolongs the time allocated per patient. During the pandemic, hospitals are trying to reduce the duration of each office visit to be able to serve more patients in one day. Augmedix, a startup headquartered in San Francisco, provides tools for remote medical documentation. This product is designed specifically for the busy emergency department environment during the pandemic. All the virtual scribes are properly trained and use Augmedix note builder technology to produce consistent documentation. They capture key information from a patient's speech and suggest any educational material that a patient might find beneficial. To let physicians benefit from augmented reality in healthcare, the company integrates their remote scribing tool with Google Glass to display relevant patient information to the doctor in real time.

4. AR takes medical training and education to another level

This is one of the most relevant augmented reality examples in healthcare, given the lockdown and ubiquitous shift to eLearning. AR for students and residents training Augmented reality in healthcare can create 3D models of human organs. Medical students and trainees can interact with these projections, try to examine, cut, and treat them. Some advanced tools even allow physicians to perform complex surgeries virtually for the purpose of training, without the need to interact with the real human tissue. For example, TouchSurgery, a health tech company headquartered in London, developed a groundbreaking augmented reality surgery training platform that offers step-by-step guidance for over 300 surgical procedures. It is used globally in 160 surgical residency programs.

Patient education with AR AR-based solutions for patient education help individuals better understand their anatomy and the impact a disease or a treatment can have on their body. Medical Augmented Intelligence, a digital health company headquartered in Taiwan, developed a tool called DigiTwin that enables patients to learn from their digital twins. The solution converts a patient’s medical images, such as CT and MRI scans, into a 3D model in a matter of 30 seconds. Patients engage with their digital twins under doctors’ supervision. They experiment and decide on treatments together. Augmented reality-supported medical education in the COVID-19 era Since the pandemic hit, employers are looking for novel ways to conduct classes without gathering people in one location. With augmented reality-based medical training, every physician can learn individually from their office or home. Trainees can virtually practice medical procedures as described above. And they can also learn how to use medical equipment remotely by means of digital twins that mirror the functionality of real-life complex medical devices.

5. AR contributes to pain management

Augmented reality in healthcare is emerging as a strong drug-free pain management tool. This technology allows patients who experience persistent pain to immerse themselves in a safe, relaxing environment with audible and visual stimulations. Such virtual surroundings alter people’s experience, providing a distraction from pain. In addition to visualization, deep breathing is another powerful remedy against pain. Neon, an immersive technology provider, developed an application that enables people with chronic pain to use deep breathing as a pain relief technique. When patients put the Neon headset on, they are asked to inhale deeply. Upon exhaling, they see the surrounding virtual leaves move in response to their breath. This tool offers both visual and breath-based pain distractions. Pain management with AR during the pandemic COVID-19 and the consequent lockdown add stress and anxiety, making chronic pain sufferers even more vulnerable. Several medical clinics are opting for virtual and augmented reality in healthcare to support pain management. For example, in response to the pandemic, Stanford Children Health Pediatric Rehabilitation Program supplies its young patients with VR and AR equipment to use during their telehealth sessions with doctors, as well as independently.

medical applications of augmented reality

Tips to get started with augmented reality in healthcare

If your medical facility is looking to adopt augmented reality to deliver superior patient care, provide accessible training, and improve internal workflows, we have some tips that will help you get started.

Start small and simple: many augmented reality tools and vendors are available on the market, making it easy for AR newbies to lose focus. The only way to avoid costly mistakes is to start small and concentrate your attention on one tool or solution with a well-defined business case. Purchase/develop this tool, test it with your clinicians to see if it brings about the expected benefits (e.g., shortens patient examination time, supports decision making during surgery, etc.). Investigate whether your training program was successful and if your medical staff are willing to use the tool. Start with a simple tracking approach that has proven itself, like marker-based or marker-less AR driven by SLAM algorithms, which deliver perfect tracking quality in most cases.
Think about legacy system integration: when introducing a new augmented reality solution in healthcare, check if it can be seamlessly integrated with your existing systems. If your hospital uses legacy software, choose an AR solution that can merge with the old system without disrupting other internal operations.
Upgrade your technology: augmented reality in healthcare is becoming increasingly sophisticated. So, medical facilities need to make sure their infrastructure (software, hardware, cloud services, etc.) can keep up.
Select the right software development kit (SDK) vendor: a vendor’s SDK will determine the set of features you can expect from your augmented reality application. If your AR tool requires a complex feature, such as recognizing a dynamic target, you need to discuss this in advance to ensure your selected vendor offers a kit that can support this requirement.
Start with a Proof of Concept (PoC): this phase allows you to verify if the chosen technology/SDK works as expected before you commit time and resources to a full-blown project.
Determine your content strategy: decide where to obtain content for your AR tool and how often it will be updated (if relevant). Research the prices of content creation. Sometimes, content costs are significantly higher than AR development costs.
Work on safety and security: this aspect is particularly relevant if your AR data will be stored in the cloud. You need to secure data transmission between the cloud and the AR device. Think about how much of a problem Internet interruption can cause and how likely it is to happen. Can your solution work in offline mode (e.g., with the hospital’s restricted network)? When choosing a cloud vendor, opt for someone who is familiar with the healthcare sector and its demands including compliance and data sensitivity

What to expect for AR in the medical field?

Even though augmented reality in healthcare is still in its early stages, it evolves fast and will soon make its way to operating rooms, doctor offices, and medical universities. According to Richard Chua, an MD from Northwest NeuroSpecialists, Arizona, As with many of our other enabling technologies, AR will find a place and penetrance. It may initially be best for training surgeons, but as high tech gets more and more integrated with artificial intelligence, machine learning and data analytics, AR may become the new image guidance or robotic-assisted technology to gain popularity.” While thinking about improving your clinic and building a business case, remember that acquiring AR-based medical software solutions is not only a matter of development costs. You also need to consider deployment and merging with the existing systems, staff training, and content creation.

Wondering if augmented reality can help you streamline clinical processes and benefit your patients? Drop us a line! Our AR and healthcare experts will develop a solution that corresponds to your unique needs.

How augmented reality can transform healthcare in the COVID-19 era

What is augmented reality, and why is it getting traction in healthcare?