This report surveys the state of the art in applications of virtual environments and related technologies for health care. Applications of these technologies are being developed for health care in the following areas: surgical procedures (remote surgery or telepresence, augmented or enhanced surgery, and planning and simulation of procedures before surgery); medical therapy; preventive medicine and patient education; medical education and training; visualization of massive medical databases; skill enhancement and rehabilitation; and architectural design for health-care facilities. To date, such applications have improved the quality of health care, and in the future they will result in substantial cost savings. Tools that respond to the needs of present virtual environment systems are being refined or developed. However, additional large-scale research is necessary in the following areas: user studies, use of robots for telepresence procedures, enhanced system reality, and improved system functionality.
This paper highlights recent and ongoing research related to applications of virtual environments and related technologies in the health-care arena. The purpose is to provide an overview for those reviewing research proposals in this field. It also provides a general introduction to virtual reality especially as it relates to health care.
Virtual environments and related technologies are allowing medical practitioners to help their patients in a number of innovative ways. The following are some examples:
Virtual environments (VEs) present a unified workspace allowing more or less complete functionality without requiring that all the functions be located in the same physical space. "Virtual environments [can be defined] as interactive, virtual image displays enhanced by special processing and by nonvisual display modalities, such as auditory and haptic, to convince users that they are immersed in a synthetic space" (Ellis, 1994, p. 17). Less technically, "a virtual world is an application that lets users navigate and interact with a three-dimensional, computer-generated (and computer-maintained) environment in real time. This type of system has three major elements: interaction, 3-D graphics, and immersion" (Pratt et al., 1995, p. 17).
Satava (1993, pp. 203-205 and 1995, p. 337) has identified five elements that affect the realism of a virtual environment for medical applications:
However, realism of the virtual objects is not enough. In addition, the human-computer interaction must provide a realistic environment with which the user can interact (Barfield and Hendrix, 1995, p. 27).
Virtual reality (VR) is an emerging technology that alters the way individuals interact with computers. "Virtual reality is a fully three-dimensional computer-generated 'world' in which a person can move about and interact as if he actually were in an imaginary place. This is accomplished by totally immersing the person's senses...using a head-mounted display (HMD)" or some other immersive display device, and an interaction device such as a DataGloveTM or a joystick (Satava, 1993, pp. 203-05). "User immersion in a synthetic environment distinctively characterizes virtual reality (VR) as different from interactive computer graphics or multimedia. In fact, the sense of presence in a virtual world elicited by immersive VR technology indicates that VR applications may differ fundamentally from those commonly associated with graphics and multimedia systems" (Hodges et al, 1995, p. 27).
The head-mounted display (HMD) is a helmet fitted with paired wide-angle television screens placed in front of the eyes and stereophonic speakers placed over the ears, so that a person wearing it can see and hear only what is generated by the computer. For some medical applications, the isolating helmet has been supplanted by 3-D glasses or helmets that allow the wearer to peer at real objects by looking down rather than only at the screens in the helmet. "Several alternatives to HMDs are in the wings. The most prominent is the Cave Automatic Virtual Environment developed at the Electronic Visualization Laboratory of the University of Illinois, Chicago. The CAVE system uses stereoscopic video projectors to display images on three surrounding walls and on the floor, and the participants wear glasses with LCD shutters to view the 3D images" (Bowman, 1995, p. 58).
The DataGloveTM is an input device (a glove that works like a joystick) that appears as a hand in the virtual world. The human user moves through the world by pointing in a direction for travel, or the user can pick up and manipulate objects by making grasping motion with the glove. A world can be anything from an abdomen to an operating room.
Telepresence surgery is performed by manipulating equipment at remote sites; image-directed surgery, on the other hand, maintains a direct connection with the real world, rather than total immersion in an artificial data space. Both involve manipulating an environment in real time, but image-directed surgery is by no means remote. Computer-assisted image-directed systems could be considered rudimentary artificial reality applications.
Partial immersion is a hybrid of digital and real environment spaces. These types of environments have been described as composite reality or augmented reality (Doyle, 1995, p. 96). Augmented reality returns simulated cues to the operator; it is also known as enhanced reality.
Telemedicine is the real-time or near-real-time two-way transfer of medical information between places of greater and lesser medical capability and expertise. It uses technologies such as telecommunications, high-resolution graphics, imaging, and video to allow medical personnel to diagnose and treat patients without seeing them in person. Telemedicine does not use virtual environments. Telepresence systems which use a full VE for the user interface and telemedicine are often confused; the Appendix contains a brief survey of telemedicine applications.
1.4 Organization of this Paper
This paper is organized as follows:
Section 1: Introduction -- presents the purpose of the paper, background on the uses of virtual environments and related technologies, definitions of key terms, and the organization of the paper.
Section 2: Health-Care Applications of Virtual Environments and Related Technologies -- describes applications of virtual environments and related technologies in seven health-care related areas. The areas discussed are surgical procedures, medical therapy, preventive medicine and patient education, medical education and training, medical database visualization, skill enhancement and rehabilitation, and design of health-care facilities.
Section 3: Discussion of Current Applications -- addresses the value added of current applications of virtual environments and related technologies, summarizes some problems and limitations, and describes tools now under development.
Section 4: Areas for Further Research -- highlights areas were additional research is needed.
Appendix: Medical Diagnosis and Monitoring (Telemedicine) -- gives some specific applications of telemedicine being explored in current projects.
Bibliography -- provides a comprehensive listing of sources on virtual environments and related technologies as applied to health care.
2. Health-Care Applications of Virtual Environments and Related Technologies
The terminology and categories used in this section are an amplification of Satava's schema for health-care applications of virtual environments and related technologies (Satava, 1995, pp. 335-37). The following categories of applications are discussed:
2.1 Surgical Procedures
Remote Surgery or Telepresence
Telepresence applications link research in robotics and virtual environments. Telepresence systems are used in medicine to manipulate equipment at remote sites. The surgeon or other medical practitioner has the sense of actually being at the site performing the procedure (Bowman, 1995, p. 65).
SRI International's Green Telepresence Surgery System was designed to allow surgeons to participate in battlefield operations from sites removed from the front line. The system consists of the "remote operative site and a surgical workstation...[that includes] 3-D vision, dexterous precision surgical instrument manipulation, and input of force feedback sensory information." The surgeon operates on a virtual image, and a robot on the battlefield reproduces the surgeon's movements. This one-to-one coupling between the surgeon and the machine has been demonstrated to work from 150 yards away with a fiber-optic connection. The next goal for the military is to replace the fiber-optic connection with a wireless signal (Satava, 1995, pp. 335-36). As promising as this system is, it will take about 3 years for it to be approved for investigational trials on humans (Satava, personal communication, August 1995).
Recently, supervisory staff at medical schools such as Johns Hopkins have been using telepresence. The supervising staff member remains in his/her office and holds a laparoscope remotely for the surgeon. The remote interaction is facilitated by two monitors and a hard-wired connection (Satava, personal communication, August 1995).
The use of virtual environments and remote surgery opens up new possibilities. Besides the military and the academic examples discussed above, other civilian uses are possible. A specialist could assist a local surgeon by remote connection, or surgery could be performed, via a medical center, in a rural setting, a ship at sea, an airplane in flight, or even a space station. Besides solving the distance problem, telepresence offers other benefits, such as minimizing the exposure of surgeons to diseases and reducing potential costs as a result of reduced trauma (Satava, 1992, pp. 376-77).
The military surgery system and academic application described above fit the narrow definition of telepresence or remote surgery as the manipulation of equipment at a remote site. However, many of the procedures described in the following subsections will lend themselves to robotic telepresence in the future.
Augmented -- or Enhanced -- Reality Surgery
Augmented -- or enhanced -- reality surgery is being used for minimally invasive surgery. Augmented surgery includes fusing computer images with real time images or superimposing them on the body using video technology or using robots directed by surgeons physically present in the operating room. Table 1. Applications for Augmented Reality Surgery
|Endoscopic surgery||Cholecystectomies (gall bladder) removal|
|Robots used locally||Orthopedics, joint replacement|
|Fusing scanned images and real time video image||Brain surgery: MRI overlaid on a video image of the patient's head|
Traditionally, surgery is performed by making incisions and directly interacting with the organs and tissues. Recent innovations in video technology allow direct viewing of internal body cavities through natural orifices or small incisions. As with remote surgery, the surgeon operates on a virtual image. The manipulation of instruments by the surgeon or assistants can be direct or via virtual environments. In the latter case, a robot reproduces the movements of humans using virtual instruments. The precision of the operation may be augmented by data/images superimposed on the virtual patient. In this manner the surgeon's abilities are enhanced.
Surgical practice, particularly in orthopaedics, presents excellent opportunities for robotic and computer-based technologies to improve clinical techniques. Procedures such as total joint replacements are performed in large volumes and at significant cost. The clinical success of these procedures is very dependent on the proper placement and fit of the implants within bony structures. Important contributions to surgical planning and execution can be made by surgical robots and pre-operative planners that utilize computer simulations. Use of robotic assistants significantly augments the skill of the surgeon (DiGioia III et al., 1995, p. 88)
In 1989, video technology was employed by a team of surgeons to perform a laparoscopic cholecystectomy (removal of a gallbladder). The use of this minimally invasive therapeutic technique has become standard: "about 90% of all cholecystectomies performed in the last two years have been done using laparoscopic techniques" (Blumenfeld, 1995, p. 45). Use of endoscopes (instruments that use video techniques for visualizing) are now commonly used for surgery.
In 1993 surgeons at Brigham Women's Hospital in Boston, "with help from engineers from General Electric Co.'s imaging and visualization laboratory in Schenectady, N.Y.,...began modeling work to assist operations in real time. As of early November , 17 operations had been performed" with the surgeons using a monitor. In one example, a magnetic resonance image (MRI) taken earlier was overlaid on a real time video image of the patient's head. (Adams, 1994, pp. 70-71). This technique can provide an X-ray view of a tumor that might otherwise not be visible if it is deeply embedded in the brain tissue (Satava, 1995, p. 336). Similarly, "UNC, Chapel Hill, is overlaying ultrasound images on live video that is then viewed in an HMD [head-mounted display]. [Again], this system essentially gives the viewer the feeling of having X-ray vision." Aligning images in real time is an area for major research (Bowman, 1995, p. 64).
A low-cost interactive image-directed neurosurgical system is being used in epilepsy surgery. The system has been successfully used in 20 epilepsy surgery cases. The epilepsy surgery is directed by the physiological and functional data obtained during the preoperative evaluation. Computer-assisted image-directed techniques are particularly well-suited for this surgery. "An image-directed system melds the preoperative data to the surgical anatomy by defining all the relevant data topographically in 3D-space, and coregistering this data to the anatomical image and the patient space" (Doyle, 1995, p. 96).
"Dr. Henry Fuchs of the University of North Carolina has created a [virtual reality] model which allows the physician to visualize the tumor (from reconstructed 3-D CT scans) inside the individual patient and plan various radiation trajectories to allow lethal doses of radiation to the tumor while avoiding damage to normal organs" (Satava, 1995, p. 336).
Additionally, magnetic resonance "allows the visualization of temperature changes for 'thermal surgery' procedures, a capability never previously available non-invasively. For the first time, 'real-time' 3D images of temperature changes can be obtained. As well, [magnetic resonance] images show a 'phase change' upon heating, and it is postulated that this occurs when the tissue protein is denatured (i.e., when coagulation takes place). These two capabilities provide MR with the ability to monitor and control thermal therapies. Tissue ablation can be monitored during the deposition of energy via interstitial laser or focused ultrasound therapy, RF ablation or cryosurgery" (Blumenfeld, 1995, p. 46).
Planning and Simulation of Procedures Before Surgery
Virtual environment technology can also be used to improve the way surgeons plan procedures before surgery. When used to simulate a procedure on a particular patient, the technology can be used to integrate the information provided by diagnostic sensors into a realistic model of the actual environment in which the surgeon will perform the proposed procedure (Satava, 1992, p. 376-77). The realistic model or surgical simulator "must have accurate detail and must be highly interactive. Specifically, the image must be anatomically precise and the organs must have natural properties such as the ability to change shape with pressure and to behave appropriately in gravity. All the body parts represented must be able to be manipulated by grasping, clamping, or cutting, and they must be able to bleed or leak fluids" (Satava, 1993, p. 204).
Virtual environments can make a critical contribution to the planning of a surgical procedure (see Table 2). For example, "Netra has been used for various precision, computer-assisted surgical procedures. Neurosurgeons use Netra to plan precision biopsies, laser-guided tumor resections, surgery for Parkinson's Disease and other motor disorders, and surgical implantation of electrode arrays for epilepsy" (Goble et al., 1995, p. 20). What is unique with this system is its user interface. Users manipulate objects, such as a doll's head, to cause movement of the virtual image on the monitor.
|Virtual reality face model with deformable skin||Plastic surgical procedure and demonstration of final outcome||Dr. Joseph Rosen, Dartmouth University Medical Center|
|Virtual leg model||Tendon transplant surgery and walking the repaired leg to predict consequences||Dr. Scott Delp|
|Creating 3-D images from a CT scan||Cranio-facial dysostosis repair||Dr. Altobelli, Brigham Women's Hospital|
|Netra system, used for various precision computer-assisted surgical procedures||Biopsies, laser-guided tumor resections, surgery for Parkinson's disease and other motor disorders, surgical implants of electrode arrays for epilepsy||University of Virginia, Department of Neurosurgery|
Simulators are being developed for all types of surgery. Many of them are used for planning particular procedures. Dr. Altobelli's (Brigham Women's Hospital) system "creates 3-D images from the CT scan of a child with bony deformities of the face (cranio-facial dysostosis); using the model created from the CT scan, the bones can be correctly rearranged to symmetrically match the normal side of the face." The procedure can be practiced repeatedly (Satava, 1995, p. 336).
Some of the simulations designed for planning a procedure also include a predictive component. Data that could be used to correct a condition is introduced into the model and the results of the proposed actions are calculated and visualized. For example, Dr. Joseph Rosen's (Dartmouth University Medical Center) "[virtual reality] model of a face with deformable skin...allows the practicing of a plastic surgical procedure and demonstrates the final outcome." Another example is Dr. Scott Delp's "virtual model of a lower leg upon which he can practice a tendon transplant operation and then 'walk' the leg to predict the short and long term consequences of the surgery" (Satava, 1995, p. 336).
An application allowing evaluation of the probable results of a procedure is being developed by the Fraunhofer-Institute for Industrial Engineering and the Orthopaedic University Clinic Heidelberg. They have "developed an application for the planning of osteotomy operations. The surgeon, with the aid of tracked shutter-glasses and a 6D input device, is able to view and rotate the femur and the hip joint." Using a simulated X-ray of the bones, the medical practitioner can determine the angle, allow the computer to perform the surgery, and view the simulated results on the screen (Bauer et al., 1995, p. 29). Planning for surgery is always important. With computer-assisted surgical planning, an exact picture can be drafted "of the relative positions of the cartilages, the abrasion of the cartilage and the position of the femur in the hip joint, [to help] determine the alteration parameters. The planning of an osteotomy can be optimized by using a 3D image because the operator is more able to evaluate the result. An increase in the rate of successful operations is conceivable" (Bauer et al., 1995, p. 30). In the future, the application should serve in the planning of patient-specific osteotomies requiring tomographic data from the relevant bones and cartilage (Bauer et al., 1995, pp. 29-35).
Simulation techniques for improved preoperative planning are of value for orthopedic surgery with or without a robot. It is
...useful for a surgeon to realistically simulate surgery beforehand to help determine what implant size to use, and what would be its optimal position. Existing planners permit the surgeon to template the implant based upon 3D geometry, but give no indication of the consequences of the proposed surgery on the initial stability of the system, the presence of implant-bone interface gaps, and the changes in the mechanical environment that are induced in the bone. The inclusion of biomechanical simulations would permit the surgeon to make appropriate changes in the initial surgical a plan, changing such parameters as the implant placement, specifics of the bone preparation, and type and size of implant. Without a robotic tool, however, there is no method for a surgeon to accurately implement a preoperative plan. For example, the simulation may help indicate an 'optimal' bone cavity shape and implant location, but the surgeon will be unable to accurately perform this plan without a robotic device. In this manner, surgical robots actually improve the clinical usefulness of realistic surgical simulations (DiGioia et al., 1995, pp. 88-89).
The predictive element is important not only for the medical team, but also for the patient's family. It can serve as a model on which to base informed consent of the patient and his or her family (Kobayashi et al., 1995, p. 175). Further, these simulation applications could be used for medical education and training.
2.2 Medical Therapy
Virtual environments are being used for a variety of purposes loosely categorized as medical therapy (see Table 3). These include diverting patients' attention during medical intervention sessions, treating phobias and other psychological conditions, providing an appropriate environment for physical exercise and/or relaxation, and providing a means for patients to describe their experiences from within altered states.
A new trend in virtual environments is their use in medical edutainment. Dentists are using virtual environments to divert patients' attention while in the chair. Using 3-D glasses with headphones hooked up to a tuner and a VCR, patients can watch 2-D and 3-D videos or educational programs, listen to music, and play video games (Buckert-Donelson, 1995, p. 6). Another example is its use in a medical support program for cancer patients. Cancer patients may experience insomnia and unrest, especially when undergoing chemotherapy. The purpose of the Psycho-oncological VR Therapy (POVRT) application is to diminish these psycho-oncological problems through virtual environments. In the virtual space, the patients can feel as if they are outside the hospital. Further, "some patients memorize the experience of having had nausea and vomiting when they were first treated with chemotherapy, and they complain in the second chemotherapy [session] even before the treatment" starts. Such anticipatory emesis may be reduced with virtual environment applications (Oyama et al., 1995, pp. 433-37).
|VR/3-D eyeglasses and headphones for movies or educational programs||To distract dental patients|
|Controlled virtual environment -- sand play||To treat autistic children||Japan|
|Controlled virtual environments||To treat acrophobia||Georgia Institute of Technology, Graphics, Visualization, and Usability Center; Ralph Lamson, Kaiser-Permanente, San Rafael, CA|
|Controlled virtual environments||To provide competition while exercising||Japan|
|Controlled virtual environment as part of a relax/refresh system||To relax users while monitoring their pulse rate||Matsushita, Japan|
|Building a virtual reality experience from within an altered state||To provide understanding and analysis of hemianopia, proprioceptive disorientation, or muscular damage||Rita Addison, New Media Arts, Palo Alto, CA|
Another application is VR Bike, a stationary bike that allows the user to race against the computer or other people, changing gears, braking, and tilting the seat to steer. A variety of on-screen terrains offers different degrees of pedaling difficulty; going faster causes the feeling of simulated wind resistance (Black, 1994, p. 43). Another product is a "massage chair that uses [virtual reality] images and sound to help relax users; it monitors pulse rate and automatically controls images and sound to match the user's body condition and state of consciousness" (Kahaner, 1994, p. 75).
Still to be developed is the concept of using virtual reality as a medical and patient education and evaluation tool. The idea is to allow a virtual reality experience to be built from within altered states, including hemianopia, proprioceptive disorientation, or muscular damage (Addison, 1995, pp. 1-2).
2.3 Preventive Medicine and Patient Education
The areas of preventive medicine and patient education lend themselves to virtual environment applications. Video games are not necessarily virtual environments although some use VEs. VEs could become an important educational tool, as well as provide edutainment. For example, a video game series (Health HeroTM) is being developed for pediatric patient education. The video games are expected to enhance young people's self-esteem; self-efficacy; communication about health; health knowledge and skills; and motivation to learn about health, and how these factors can influence self-care behavior and health outcomes (Lieberman and Brown, 1995, p. 201).
This is an area that will benefit from work done in other areas of virtual environments, although to be effective, products must be tailored to the particular applications and users.
2.4 Medical Education and Training
Virtual reality allows information visualization (the display of massive volumes of information and databases). Through 3-D visualization, students can understand important physiological principles or basic anatomy (Satava, 1995, p. 336). Table 4 lists some applications of virtual environments and related technologies for medical education. In this paper, education is used for applications that do not allow the learner to practice a procedure while training is used where practice is the objective of the application.
One application for the study of anatomy allows the abdomen to "be explored by 'flying' around the organs (a bird's-eye view from the diaphragm) or behind them (watching the bile duct pass through the pancreas under the duodenum) or even inside them." For example, one might start inside the esophagus, go down into the stomach, etc. The extraordinary perspectives provided by such a learning tool "impart a deeper understanding and appreciation of the interrelationship of anatomical structure that cannot be achieved by any other means, including cadaveric dissection" (Satava, 1993, p. 205).
Virtual environments provide "both a didactic and experiential educational tool. A demonstration mode could give a 'tour' of the intended subject, and then an exploration mode would allow the student to actually experience the environment" (Satava, 1995, p. 336).
|Exploring anatomy by "flying" through the body||Providing an understanding of the organs by "flying" around them, behind them, or even inside them|
|"Touring" a topic of study and then exploring it||Seeing a visual representation (e.g., of shock and navigating through the arterial tree)|
|Adding the fourth dimension of time (archived information in multimedia format) to the three dimensions of a virtual world (3-D space); in essence, multimedia virtual reality||"Flying" into an organ and
grabbing something (e.g.,
going into the stomach,
seeing an ulcer and
"grabbing" it for biopsy)|
|Dr. Helene Hoffman, University of California, San Diego|
Dr. Helene Hoffman of the University of California, San Diego, is working to create 4-D, "the three dimensions of a virtual world (3-D space) and the fourth dimension of time (archived information in multimedia format), in essence, multimedia virtual reality (MMVR). For example in a MMVR simulator of the gastrointestinal tract, a student could 'fly' down into the stomach, see an ulcer and 'grab' it as if for a biopsy, this would bring up the histologic micrograph of an ulcer, or play a video tape of a Bilroth 2 operation for ulcer disease, or perhaps demonstrate (predict) the healing in response to medication. In this fashion the multiple layers of understanding could be rolled into one, and the change of the processes over time can be graphically represented and personally experienced" (Satava, 1995, p. 336).
The following are additional examples of systems being used in education:
The use of computers in medical schools is increasing. Table 5 lists some examples of applications of virtual environments and related technologies for medical training.
In medical research, there is a shortage of cadavers; for students, being able to summon a detailed, lifelike computer image of a human body is a workable replacement. To improve training, companies are developing 3-D computer simulations of medical and surgical procedures (Edelson, 1995, p. 90). According to Jon Merrill, "these 3-D simulations allow rather more realistic training than do cadavers... . If a trainee pinches a nerve or slices through a blood vessel in a cadaver, nothing happens. Dead tissue is usually harder, color is changed, and arteries no longer pulsate. Once something is cut in a cadaver, it cannot be reattached. But in digital format, procedures can be repeated ad infinitum and stored for later group analysis. Some organs can also be made transparent, 'crisis' drills can be created, and various types of organs can be modeled" (Adams, 1994, pp. 71-72).
A system built at Talisman Dynamics, Inc. simulates the rather rare open cholecystectomy procedure. Because the laparoscopic procedure is being addressed by a variety of applications for medical training, Talisman felt that a simulation of the open procedure would lend itself to an overview of the abdominal anatomy in 3-D context that could be useful in anatomical education. Simulating the open procedure allowed the developers "to simulate a large number of different tools and interactions, and to provide a comprehensive spatial context" (Lasko-Harvill et al., 1995, p. 183).
High Techsplantations' initial work involved the development of a "virtual abdomen." The virtual abdomen or "uro man" could be rotated using a three-dimensional mouse. After a year of research and development efforts, a laparoscopic lymph node dissection simulation was completed. In parallel, High Transplantations "simulated other procedures, including an angioplasty procedure (sponsored by Marion Merrell Dow). This simulation allows the user to use a simulated balloon catheter to practice angioplasty. The software allows for various complications including transection of the coronary vessels, rupture of the balloon, as well as resistive feedback to the end-user. A specially designed catheter, equipped with position sensors and feedback devices was constructed to afford a high-fidelity tactile simulation" (Merril et al., 1995, 251).
|Virtual reality laparoscopic surgery simulator, consisting of a torso into which the handles of laparoscopic instruments are mounted and providing force feedback||The virtual abdomen (liver and gall bladder) are graphically displayed upon the video monitor, and the apprentice surgeon practices specific laparoscopic procedures.||Woods and D. Hon|
|Allows the trainee to guide a balloon catheter through a hollow guiding wire to the obstruction, inflate the balloon to expand the artery, and restore regular blood flow.||High Techsplantations Inc., Rockville, MD|
|Virtual abdomen||Created for the immersive, traditional helmet-mounted display (HMD) and DatagloveTM.||Richard M. Satava, Advanced Research Project Agency, Arlington, VA|
|Limb trauma simulator||Will lead to a virtual reality environment.||MusculoGraphics, Inc., Evanston, IL|
|Simulating surgery complete with feedback on the force being exerted||Simulates surgery on the human eye.||Georgia Institute of Technology, Atlanta|
"Woods and Hon have developed virtual reality laparoscopic surgery simulators. These consist of a simple plastic torso into which the handles of laparoscopic instruments are mounted (to provide force feedback); the virtual abdomen (liver and gall bladder) are graphically demonstrated upon the video monitor, and the apprentice surgeon can practice the specific laparoscopic procedure" (Satava, 1995, pp. 336-337). Satava took a different approach. A virtual abdomen has been created for the immersive, traditional head-mounted display (HMD) and DatagloveTM. Using virtual scalpel and clamps, the abdominal organs can be operated upon. This same abdomen can be 'explored' by a student in the manner described above (Satava, 1995, p. 337).
The following are examples of additional issues being addressed by simulators recently or currently under development:
In addition to building simulators for specific procedures, generic problems are being addressed, such as the introduction of olfactory stimuli in virtual environments. Surgical simulations need to "provide the proper olfactory stimuli at the appropriate moments during the procedure. Similarly, the training of emergency medical personnel operating in the field should bring them into contact with the odors that would make the simulated environment seem more real and which might provide diagnostic information about the injuries that simulated casualty is supposed to have incurred" (Krueger, 1995, p. 180).
Another generic issue is the development of low-cost systems for mass distribution. "[Virtual reality] surgical simulators will someday be valuable in medical education, both to reduce training costs and to provide enhanced physician training." However, "many surgical simulators rely on real-time volume rendering techniques, and run on hardware costing several hundred thousand dollars." The Departments of Surgery and Computer Science at Stanford University "have created a low cost, interactive [virtual reality] application for the IBM PC that provides a much clearer perception of surgical anatomy than a surgical atlas while remaining comparable in price. This program could be used to teach hundreds of other surgical procedures" (Daane et al., 1995, p. 79).
2.5 Visualization of Massive Medical Databases
"Virtual reality is a way to visualize, manipulate, and interact with computers and extremely complex data. Interactive image-directed neurosurgery may be considered a primitive form of virtual reality since it essentially does this in the context of performing surgery. With interactive computer image-directed techniques, the neurosurgeon navigates through anatomic, functional, and physiologic 3D space defined by digital images obtained from and representing the patient. The image space is a virtual reality data set representing functional as well as anatomic digital data" (Doyle, 1995, p. 91). This group of applications represents data as visual objects in a virtual environment. Table 6 lists some applications of virtual environments and related technologies for visualization of massive databases.
Under funding from the Department of Defense, the Ohio State University Hospital's Department of Anesthesiology is "creating a system for teaching a specific method of regional anesthesia, the epidural technique... . The system will enable the resident to investigate various three-dimensional reconstructed data sets in a non-threatening environment. The system can be cued through voice activation to provide additional information in text, audio, or graphical form. Furthermore, the system incorporates the necessary components to allow the resident to 'feel' the technique as performed by the expert" (McDonald et al., 1995, p. 238).
|Epidural technique||Allowing residents to investigate various 3-D datasets||Ohio State University Hospital, Department of Anesthesiology|
|War injury statistics||Providing a cyberspace representation of war injuries from the Viet Nam Database; allowing complex combinations of war wounds, organ systems injured, mortality, etc. to be visualized as clusters of data points||J. Henderson|
"Henderson has created a cyberspace representation of the war injuries from the Viet Nam Database. Using a 3-D cube to plot 3 axes of information, complex combinations of war wounds, organ systems injured, mortality, etc. can be visualized as clusters of data points. These clusters can illustrate and reveal important relationships which otherwise cannot be discovered. Navigating in three dimensions allows different perspectives of the data, permitting different interpretations. This application for visualization has not been exploited sufficiently and holds promise for the field of medical informatics" (Satava, 1995, p. 337).
2.6 Skill Enhancement and Rehabilitation
Applications of virtual environments and related technologies for skill enhancement and rehabilitation include those that provide training in the use of equipment, those that allow the exploration of virtual space, those that augment physical abilities, and those that teach skills. Table 7 lists some examples.
|Training in use of equipment||Training disabled children to control motorized wheelchairs||Dean Imman, University of Oregon|
|Exploration of "physical space"||Providing virtual environments for exploration in a wheelchair||Greenleaf Medical Systems, Palo Alto, CA|
|Empowerment of the disabled using an eyetracker device||Providing a quadriplegic child the opportunity to develop interactions with the outside world before her disability causes her to become too introverted to communicate||David Warner, Human Performance Institute of Loma Linda University Medical Center, CA|
|Use of virtual reality to enhance vision of the visually impaired||Providing a virtual computer
monitor that moves the user's
line of sight across an
enlarged virtual monitor|
Providing vision enhancement
-----------------------------------Using glasses that display a television image to help Parkinson's disease patients overcome their halting, hesitant gait
-----------------------------------University Applied Physics Laboratory at The John Hopkins University, Baltimore, MD
Suzanne Weghorst, University of Washington
|Train to Travel project||Substituting virtual reality bus rides for the real thing to train individuals to use the public transportation system independently||University of Dayton Research Institute, OH, and Miami Valley Regional Transit Authority, FL|
"Dean Imman at the University of Oregon is using a [virtual environment] to train disabled children to control motorized wheelchairs. For a child who is learning to use a motorized wheelchair but who has limited control over his body, for instance, a [virtual environment] overcomes several potential safety problems. This [virtual environment] training might also be motivational, because children accustomed to constant care by others can feel threatened by a wheelchair's mobility" (Bowman, 1995, p. 64).
The use of virtual environments for rehabilitation in medicine has grown dramatically, resulting in an annual conference devoted to the subject. "Greenleaf Medical Systems has created virtual environments for exploration in a wheelchair. For example, an individual with cerebral palsy who is confined to a wheelchair can operate a telephone switchboard, play handball, and dance within a virtual environment" (Greenleaf, 1995, p. 419). "Warner has utilized an eyetracker device from BioControl, Inc. in a quadriplegic child in an effort to provide her the opportunity to develop interactions with the outside world before her disability causes her to become too introverted to communicate. In these circumstances, [virtual reality] is being used to empower those individuals with disabilities" (Satava, 1995, p. 337).
The visually impaired have difficulty using conventional computer display products. Current products for the visually impaired limit the amount of enlarged imagery to as little as 1 percent of a document page. Further, it is difficult to determine the cursor position on the page. "A prototype system was designed using a virtual computer monitor (VCM) that moves the user's line of sight across an enlarged virtual document, instead of vice-versa, using a virtual reality head-mounted display (HMD). The wearer's head position is sensed using a head-tracking device, and translated into mouse output in software. The simulated mouse data is used to scan the enlarged computer output across the HMD field-of-view in the opposite direction from the wearer's physical head movement, causing the impression of moving one's view about a fixed, enlarged document" (Zwern and Clark, 1995, p. 406).
The University Applied Physics Laboratory at The John Hopkins University, Baltimore, Maryland, is working on an Interdivisional Sensory Engineering Program in cooperation with the Schools of Medicine, Engineering, and Arts and Sciences for the use of virtual reality to enhance vision of the visually impaired (Buckert-Donelson, 1995, 11).
"A primary symptom of Parkinson's Disease...progressively restricts the ability to walk... . Virtual reality technology provides a way...to enable walking, by presenting virtual objects overlaid on the natural world. Normal stride length, and even running, can be enabled, provided the virtual clues are spaced properly and simulate the appearance of staying stable on the ground as the person moves" (Riess and Weghorst, 1995, p. 298). Glasses displaying a television image "help Parkinson's disease patients overcome the halting, hesitant gait that characterizes the disease... . The...glasses project a track of objects, currently yellow blocks, at stride-spaced intervals. Wearers step across the track, facilitating nearly normal walking patterns" (Dutton, 1994, p. 34).
"The Train to Travel project, sponsored by the University of Dayton (UD) Research Institute in cooperation with the Miami Valley Regional Transit Authority (RTA), substitutes [virtual reality] bus rides for the real thing. Students learn independently in the classroom, eliminating the need for a teacher to accompany or follow them on real trips" (Buckert-Donelson, 1995, 4). "In the project, students use interactive multimedia to recognize landmarks and learn what to do in case of an emergency. When they master basic skills, they progress to the [virtual reality environment, where they use a system with head tracking to look around a computer-generated landscape" (Buckert-Donelson, 1995, p. 5).
2.7 Architectural Design for Health-Care Facilities
Virtual environment testing of architectural designs for health-care facilities could save both time and money. However, considerable research on interaction methods and software system design will be needed before those savings can be realized (Bowman, 1995, p. 64). To illustrate what is being done, "Dr. Kenneth Kaplan of the Harvard Graduate School of Design is beginning to apply [virtual reality] to architectural design in a project for the operating room (OR) of the future" (Satava, 1995, pp. 337-38).
In one example of using a virtual environment for design purposes, "a person in a wheelchair puts on a Virtual Research helmet equipped with electromagnetic motion trackers. Objects are manipulated via a Dataglove... . The architect can point to the representation of a window sill, for example, and make it higher or lower, or widen a door if it's too narrow" (Krumenaker, 1994, p. 48).
Companies are also using virtual reality to sell their health-care products. For example, one "has adopted a relatively low-cost, custom-built system to present interactive layouts of its healthcare equipment to potential clients in realtime" (Buckert-Donelson, 1995, 4).
Table 8 lists examples of applications of virtual environments and related technologies for architectural design for health-care facilities.
|Architectural design testing||Applying virtual reality to
architectural design of the
operating room of the future|
Allowing designers to perform work within a virtual environment
|Dr. Kenneth Kaplan, Harvard
Graduate School of Design|
|Sale of health-care products||Presenting interactive layouts of health-care equipment to potential clients in real time|
3. Discussion of Current Applications
This section presents a discussion of the current applications of virtual environments and related technologies described in Section 2. It summarizes the value added of these applications for health care, notes some problems and limitations, and describes some tools currently under development.
3.1. Value Added
Virtual environments and related technologies add value to health care in the areas of cost savings, improved services, and savings in material resources. Table 9 summarizes examples of value added in these three areas.
|Cost savings||Trauma units in emergency rooms could improve operating efficiency and reduce costs by using telepresence. Doing so would conserve resources by limiting the need for part-time specialists to be physically present in trauma units.|
|Improved services||Simulations allow surgeons to develop new techniques, to
practice unfamiliar techniques, and to predict results of
particular surgical procedures.|
The success of joint replacement depends on the proper placement and fit of implants within bony structures. Surgical robots and preoperative planners using computer simulations can improve surgical techniques and accuracy.
Advantages offered by telepresence systems include enhancing task performance in remote manipulation; allowing controlled application of extremely large or small forces; improving operator perception of the task; and facilitating manipulation in hazardous environments.
|Savings in material resources||The use of simulators saves precious resources such as cadavers and animals.|
Trauma units in emergency rooms could improve operating efficiency and reduce costs by using specialists who are not physically present. With telepresence, experts could be linked to remote patients. This is effective whether distances are great or not. Because the need for specialists in emergency situations cannot be predicted, making use of remote specialists in time of need limits the staffing needs of the trauma units without limiting their effectiveness. Doing this would conserve resources by limiting the need for part-time specialists to be physically present in trauma units.
Virtual reality is impacting and improving surgical results. Examples include the use of laparoscopic simulators for training, the development of applications that simulate human response to medication (for example, simulator systems helping to train anesthesiologists), and the development of imaging tools that guide surgical tools through brain tissue to the site of a tumor (Chinnock, 1995, p. 26).
Advantages offered by telepresence systems include enhancing task performance in remote manipulation through increased positioning resolution or range; allowing controlled application of extremely large or small forces; improving operator perception of the task; and facilitating manipulation in hazardous environments by isolating the environment from the operator, or manipulation in clean environments by isolating the operator from the environment. Common to all telepresence systems is a human operator in a manual or supervisory control loop overseeing task performance. Application areas include operations in radioactive, underwater, space, surgical, rehabilitation and 'clean-room' environments, as well as the manufacturing and construction industries.
Non-Renewable Resource Savings
Finally, the use of simulators saves precious resources such as cadavers and animals. By allowing medical personnel to train using simulators, the demand for non-renewable resources can be drastically reduced. The trainee can practice over and over using a realistic, virtual environment without reducing the supply of non-renewable resources.
3.2 Problems and Limitations of Current Applications
Many of the current virtual environment applications in health care have problems that limit their effectiveness. Some of these limitations are due to the state of the art of the supporting technologies.
As noted earlier, the sense of smell in virtual environment systems has been largely ignored. Both Krueger and Keller are developing odor-sensing systems. (Keller et al., 1995, p. 168 and Krueger, 1995, p. 180). Smells are extremely important. Not only do they help distinguish specific substances, but also they give a sense of reality to a situation. The absence of odor is a serious limitation of current telepresence and training systems.
Another major research problem relates to overlaying ultrasound images on live video that is then viewed in a head-mounted device application. The research issue to be addressed is the alignment of images in real time (Bowman, 1995, p. 64).
"The senses of vision and touch are the two main feedback mechanisms used by surgeons when performing a surgical procedure. Improved cameras, HDTV, head mounted displays, and stereoscopes have advanced the sensing and displaying of vision; however, there have been few developments in the area of tactile feedback. The ability to feel tissue is a valuable tool. Procedures that require palpitation, such as artery localization and tumor detection, are extremely difficult when the only form of haptic exploration is in the form of forces transmitted through long, clumsy instruments. The ability to remotely sense small scale shape information and feel forces that mesh with natural hand motions would greatly improve the performance of minimally invasive surgery and bring a greater sense of realism to virtual trainers" (Peine et al., 1995, p. 283).
"According to experiments conducted by the British Defence Research Agency virtual reality can make you sick... . [This confirms] what some users of [virtual reality] headsets have long suspected. Researchers found that 89 of 146 otherwise healthy adults suffered temporary nausea, dizziness, or impaired vision after using a [virtual reality] helmet-mounted display (HMD) for just 20 minutes. Eight became so nauseated they could not finish the 20-minute period... . Virtual reality researchers attribute the apparent effects to a couple possible causes. First, [virtual reality] systems usually have a lag time between when the user moves and when the display is updated; users, accustomed to this lag, may become confused when they take the helmet off. Moreover, [virtual reality] creates an illusion of three dimensions, even though the screen actually remains at a fixed distance from a wearer's eye. This may disrupt depth perception later" (Langreth, 1994, p. 49).
Today simulations trade off less realism for more real-time interactivity because of limited computing power, but the future holds promise of a virtual cadaver nearly indistinguishable from a real person. "Initial research over the past 5-10 years in video technology, graphics, computer-aided design (CAD) and virtual reality has given us an insight into some of the requirements for a virtual-reality surgical simulator. If graphic images are used, it is estimated a rate of 5,000,000 polygons/s would be required for a realistic reconstruction of the abdomen; current high-level computer-graphics workstations generate 60,000-100,000 polygons/s. Using CT or MRI scan images would require substantially more computer power. The algorithms for object deformation and gravity are available and continue to evolve. For motion, at least 30 frames/s are necessary to eliminate flicker and response delays; this level of interactivity is available on standard [virtual reality] systems. The computational power for sensory input and for object reactivity has not been determined" (Satava, 1993, p. 204).
Fujita Research has created a situation in which a construction robot in America can be controlled from Japan. If this can be done with a construction robot, it will not be long before this is applied to telepresence surgery. "The biggest problem now is the time delay involved in long distance communications. Currently, there is a delay of a second or more from the time the command is given until the robot actually moves. The same is true when sending a camera image from the robot. Fujita hopes to cope with the delay problem by employing computer simulations of the robot movements and displaying predicted motions on the operator screen" (Kahaner, 1994, p. 76).
Virtual reality systems used for interfacing to "training equipment require the user to be isolated from the real world and surrounded by the virtual world generated by the computer. This is normally achieved using a head mounted display...enclosed in a rugged casing" (Rowley, 1991, p. 12/1). One of the challenges for virtual environments is to allow the user to move freely between the virtual and the real worlds.
3.3 Tools Under Development
There are many types of tools under development that will facilitate the use of virtual environments in health care. These tools can be categorized as providing computer input, computer output, or both. Examples of these are listed in Table 10.
|Computer input: facial||Using eye movements and
Using image processing to recognize gestures and facial expressions as input
|David Warner, Human
Performance Institute of Loma
Linda University Medical
Systems Research Laboratory, Advanced Telecommunications Research Institute, Kyoto
|Computer input: props||User manipulating props such as a doll's head to cause the screen image to move||University of Virginia, Department of Neurosurgery|
|Computer input: virtual perambulator--bodily movement coordinated with visual images||Coordinating bodily movement with visual images, creating a lifelike model||Hiroo Iwata, Institute of Engineering Mechanics, University of Tsukuba|
|Computer input: visualization of data||VRASP: allowing surgeons
to interactively visualize 3-D
renderings of CT and MRI
Providing real time MRI and CT scans for the operating room
General Electric Co., Imaging and Visualization Laboratory, Schenectady, NY
|Computer output: ability to feel objects in virtual environments||Spidar (Space Interface
Device for Artificial Reality):
providing haptic display
(device for presenting tactile
and force sensation)|
PHANToMTM: actively exerting an external force on the user's finger(s) creating the illusion of interaction with solid virtual objects
Argonne Remote Manipulator: providing force feedback for molecular modeling applications
Reflecting force with a pen-shaped, six-degree-of-freedom device
Reflecting force with a four-degree-of-freedom manipulandum
Teletactile hand allowing doctors to reach out and touch patients electronically at a physical distance
|Makato Sato, Tokyo Institute
of Technology, Precision and
Intelligence Laboratory |
SensAble Devices, Inc., Cambridge, MA
University of North Carolina, Chapel Hill
Hiroo Iwata, Institute of Engineering Mechanics, University of Tsukuba
Michael Burrow, Georgia Institute of Technology, and colleagues at the Medical College of Georgia
|Computer output: ability to feel the visual image of column data on the screen||Volume haptization techniques representing higher- dimensional scientific data by force sensation||Hiroo Iwata, Institute of Engineering Mechanics, University of Tsukuba|
|Computer input and output: ability to smell objects in virtual reality||Allowing transmission of olfactory data||Paul E. Keller, Pacific Northwest Lab, Richland, WA; Myron W. Krueger, Artificial Reality Corp.|
"David Warner at the Human Performance Institute of Loma Linda University Medical Center, California, is employing [virtual environment] technologies to provide greater interactivity to people with severe disabilities. He has developed computer interfaces that use eye movements and muscle potentials. Severely disabled people have been able to navigate through [virtual environments] using biological signals from muscles over which they retain control" (Bowman, 1995, p. 64).
"Researchers at the Systems Research Laboratory of the Advanced Telecommunications Research (ATR) Institute in Kyoto are working on a system that uses image processing to recognize gestures and facial expressions as input. Such a system would provide a more natural interface without requiring the operator to wear any kind of special gear" (Kahaner, 1994, p. 75).
Hiroo Iwata of the Institute of Engineering Mechanics at the University of Tsukuba "has a virtual perambulator under development. In this project he wants to build an apparatus that will take the previously separated bodily sensations and handle them together. The perambulator uses a head-mounted display. The walker's upper body is fixed in place, and a goniometer detects the position of the head. Ultrasonic wave generators are attached to the walker's toes. The time required for the ultrasonic waves to reach receiver units determines the positions of the walker's feet. These motion data are sent to a computer that generates a virtual space, and the view through the HMD changes in real time in response to the walker's movements... . The perambulator is made so that the tension on wires attached to the feet produces the sense of reaction or resistance associated with climbing or descending stairs. When the walker ascends a step, the wire length is regulated so that the take-off foot feels the force of resistance. To represent the reaction force involved in opening a virtual door, the virtual perambulator includes a manipulator having six degrees of freedom. 'With this apparatus,' said Iwata, 'bodily movement is coordinated with visual images. A more lifelike model can be experienced' " (Kahaner, 1994, 77).
According to Kirby Vosburgh, manager of the General Electric's Imaging and Visualization Laboratory, "GE is working on real-time MRI and CT scans for the operating room. If that succeeds, surgeons could have high-quality views of the body's interior as they worked" (Adam, 1994, pp. 70-71).
The ability to feel objects in virtual environments can markedly enhance the effectiveness of many applications, particularly for training, scientific visualization, and telepresence. "Haptic displays, devices for presenting tactile and force sensations, are being developed in several laboratories, but are not yet widely used elsewhere. Most of the haptic displays being developed are electro-mechanical devices that deliver force feedback to the hand or arm within limited ranges of movement. For example, Makato Sato of the Tokyo Institute of Technology's Precision and Intelligence Laboratory has developed a force-reflecting system called Spidar (Space Interface Device for Artificial Reality). With this system, the user inserts his or her thumb tips and index finger into a pair of rings, each of which have four strings attached to rotary encoders. The encoders are located at the corners of a cube. String movements can be restricted with brakes, providing touch sensations. Spidar is being applied in a [virtual environment] for collaborative design" (Bowman, 1995, p. 60; Spidar is also discussed in Kahaner, 1994, p. 77).
The PHANToMTM actively exerts an external force on the user's finger(s) creating the illusion of interactions with solid virtual objects. Users can actually feel the tip of the stylus touch virtual surfaces. Surgeons can practice procedures on virtual patients. "The Phantom [sic] consists of a finger thimble mounted on the end of a small robot arm. When the thimble hits a position corresponding to the surface of a virtual object in the computer, three motors generate forces on the thimble that imitate the feel of the object... . The Phantom [sic] can duplicate all sorts of textures, including coarse, slippery, spongy, or even sticky surfaces. It also reproduces friction. And if two Phantoms [sic] are put together a user can 'grab' a virtual object with thumb and forefinger." Researchers are using the PHANToMTM for surgical training, prototyping, and drug design (Langreth, 1995, 45).
Other force feedback tools include a system under development at the University of North Carolina, Chapel Hill, which "uses an Argonne Remote Manipulator (ARM) to provide force feedback to the arm for molecular modeling applications. Hiroo Iwata of the University of Tsukuba has developed a pen-shaped, 6-degree-of-freedom, force-reflecting device. A 4-degree-of-freedom, force-reflecting manipulandum has been developed by Northwestern University" (Bowman, 1995, p. 60).
Recently, Hiroo Iwata of the Institute of Engineering Mechanics at the University of Tsukuba has developed "'volume haptization' techniques that represent higher dimensional scientific data by force sensation. The system maps voxel data to the three dimensions of force and three dimensions of torque in the force display. The operator thus feels the visual image of column data on the screen" (Kahaner, 1994, p. 77).
Addressing the issue of incorporating the sense of smell in virtual environments, Keller et al. have developed a "chemical vapor sensing system for the automated identification of chemical vapors (smells)... . An electronic nose will potentially be a key component in olfactory input to a telepresent virtual reality system. The identified odor would be electronically transmitted from the electronic nose at one site to an odor generation system at another site. This combination would function as a mechanism for transmitting olfactory information for telepresence. This would have direct applicability in the area of telemedicine since the sense of smell is an important sense to the physician and surgeon" (Keller et al., 1995, p. 168).
"[Engineer Michael] Burrow [of Georgia Institute of Technology] and his colleagues at the Medical College of Georgia are developing a 'teletactile hand' that will someday enable doctors to electronically reach out and touch someone in remote areas with few medical specialists" (Wickelgren, 1995, p. 21). A related project at Georgia Tech uses tactile feedback to help students practice surgery (Edelson, 1995, pp. 62-65).
Practicing Mayo Clinic surgeons are committed to assisting with the development, evaluation and deployment of the VRASP system. "VRASP will bring to the [operating room] all of the pre-surgical planning data and rehearsal information in synchrony with the actual patient and operation in order to optimize the effectiveness of the procedure, minimize patient morbidity, and reduce health-care costs" (Robb and Cameron, 1995, p. 309).
4. Areas for Further Research
Virtual environment applications for health care are currently available and are being used to achieve the benefits discussed in Section 3 of this paper. The demands of surgery, training for surgery, and telepresence surgery require a high degree of realism as concerns transmitted images. Additional large-scale research is necessary in the following areas: user studies, use of robots for telepresence surgical procedures, enhanced system reality, improved system architecture, and improved functionality. Table 11 summarizes research needs in each of these areas.
There is need to do research among physicians to find out what their problems are and in what ways telepresence and other forms of virtual environments can help them perform their jobs better (Wagner, 1995, pp. 1-4).
"In terms of future research, it is necessary to upgrade the system by improving not only [virtual reality] related parts such as developing a high resolution [head-mounted display] or the enrichment of the [virtual reality] experience but also other parts such as a vibration pattern or massage contents. Also, it is necessary to evaluate physiological effects of [virtual reality] itself or to investigate more about the physiological meaning of [virtual reality] stimulations to the human" (Nakajima, Nomura, Fukushima et al., 1994, 150).
|User studies||Research among physicians to
find out what their problems are
and how virtual environments
can help them perform their
Evaluation of the physiological effects of virtual reality itself, or investigation of the physiological meaning of virtual reality simulations to humans
|Telepresence surgical procedures||Development of a telepresence
microsurgical robot for eye
Development of a simulated beating heart that would allow surgery on a beating heart
|Ian Hunter, MIT |
-----------------------------------Ian Hunter, MIT
|Enhanced system reality||Development of olfactory
The addition of spoken language for virtual environment commands
|Paul E. Keller, Pacific
Northwest Lab, Richland,
WA; Myron W. Krueger,
Artificial Reality Corp.|
SRI International, Menlo Park, CA
|Improved system architecture||Use of fast parallel hardware to solve the CT and MR image time bottleneck and the segmentation and classification problems|
|Improved functionality|| Hardware that runs faster|
A display that is brighter, sharper, and higher-resolution
System software that supports the faster hardware
Networking that supports real time multiuser environments
An interaction device that provides force feedback and supports 3-D navigation, object interaction, and flawless speech generation and recognition
Telepresence Surgical Procedures
"At MIT, Ian Hunter is developing VE technologies for teleoperated surgical procedures. The initial application uses a teleoperated microsurgical robot for eye surgery. A virtual model of the eye has been developed to practice teleoperated surgery under simulated conditions" (Bowman, 1995, p. 65). Hunter is also developing a simulated beating heart that would allow surgery on a beating heart (Richard M. Satava, personal communication, August 18, 1995).
Enhanced System Reality
While inclusion of the visual, aural, and tactile senses into virtual reality systems is widespread, the sense of smell has been largely ignored. A mechanism for transmitting olfactory information for telepresence is needed. "This would have direct applicability in the area of telemedicine since the sense of smell is an important sense to the physician and surgeon" (Keller et al., 1995, p. 168).
"The Virtual Perception Program at SRI International has been exploring the use of spoken language for [virtual environment commands with SRI's Decipher, a speaker-independent, continuous-speech recognition system. Spoken language proved very useful for discrete commands, such as selecting virtual objects and changing their attributes, but poor for continuous commands such as navigating through a [virtual environment]. Spoken commands for movement to specific locations, such as a chair, also were useful" (Bowman, 1995, p. 62).
Demand for enhanced-reality "services is being helped by minimally invasive surgery, such as endoscopy, where the doctor guides an optical-fiber light probe and tiny camera through the patient while looking at a video screen. Instruments are inserted through puncture holes and manipulated from outside the body. A 3-D environment to utilize human depth perception might aid this work, as might tactile feedback since often cancerous tissue is detected by touch" (Adams, 1995, p. 71).
Improved System Architecture
"There have been attempts of 3D visualization in medicine, most of them follow the vector based approaches of the VR world. The main sources of medical 3D data are CT and MR images" (Roux and Coatrieux, 1994, p. 397). Fast parallel hardware can and will solve the time bottleneck, and within the next few years, virtual environments in medicine will be realized if the segmentation and classification problems can be overcome (Roux and Coatrieux, 1994, p. 398).
To achieve improved functionality in virtual environments for health care, the hardware should run faster. The display should be brighter, sharper, and higher-resolution; when coupled with a head-mounted display, it should also be more lightweight and contain faster and more accurate head tracking. The system software should support the faster hardware. Networking should support real-time multiuser environments, eventually allowing an infinite number of users to network with real-time results (including speech) over a telephone line, for example. The interaction device should provide force feedback and support 3-D navigation, object interaction, flawless speech generation and recognition (Alexander et al., 1995, p. 17).
Appendix: Medical Diagnosis and Monitoring (Telemedicine)
There is a growing trend toward the development of telemedicine applications: "In 1994, 37 telemedicine projects were launched, joining some 20-odd telemedicine pilots already in progress" (Deborah Dakins, Telemedicine Newsletter, Miller Freeman Inc. Publisher, San Francisco, as reported by Laplant, 1995, p. 48). The potential for involvement ranges from what has become traditional telemedicine functions (diagnosis and monitoring) to remote surgery. Table 12 list some specific applications of virtual environments and related technologies for telemedicine being explored in current projects. Clearly the potential for expanded and enhanced telemedicine is great.
||Providing diagnoses and
treatment for Parkinson's
disease patients in rural areas
||University of Kansas, Kansas
City||Mobile units consisting of
communications and laptop
||Allowing nurses to receive
doctors' orders, chart
patients' progress, review
medical reference books, and
show educational videos to
patients at home
||Carnegie Mellon University,
Pittsburgh ||Videoconferencing pilot
||Enabling rural nurses and
doctors to treat 95% of
patients who would have
been referred to a specialist
hundreds of miles away|
Deconess Medical Center,
Billings||Home computer and modem
||Allowing patients to test
themselves daily, record the
results, and transmit them
weekly to a relational
||Stanley M. Finkelstein,
University of Minnesota||Interactive video and medical
diagnostic equipment in the
hospital patients suffering
from chronic conditions to
||Medical College of Georgia,
equipment provided to
||Applying telemedicine to
private obstetrics, monitoring
baby's heart rate and
mother's contractions for
babies with cord
||Jason Collins, Pregnancy
Institute, Slidell, Louisiana||Communications network|
|Linking three main medical
Linking the largest prison in North Carolina and two rural hospitals to diagnose and prescribe medications, and have access to a digital stethoscope, a graphics camera, and a miniature, handheld dermatology camera
Linking a prison project to a hospital
Linking Watauga Medical Center in Boone, NC, with Carolina Medical Center for a monthly "tumor board" meeting
Linking an emergency room/ trauma center to a major hospital
Allowing limited transmission of the reports and slides for which the Health Care Financing Administration will reimburse
Providing communications and telemedicine to "frontier areas"
East Carolina University School of Medicine, Greenville, NC
University of Texas, Medical Branch at Galveston
The Carolina Medical Center, Charlotte, NC
Letson, Cleveland Memorial Hospital in Shelby, NC
Oklahoma Telemedicine Network, Oklahoma University Health Sciences Center, Stillwater
High Plains Rural Health Network, Colorado
One key area of telemedicine applications is for the delivery of trauma care in the field. "For example, the health-care system in a battlefield situation demands timely and efficient evaluation, development and implementation of a treatment plan for severely traumatized patients." A delivery system should increase the efficiency of centrally located experienced trauma specialists by assisting medics at the remote site (O'Toole et al., 1995, p. 271).
Also challenging is the delivery of expert medical care in remote areas of the United States, as well as in developing countries. The publisher of the Global Telemedicine Report (AJ Publishing, Germantown, Maryland) recently published a map entitled "1995 Telemedicine Across the U.S." It shows telemedicine hubs in all but two states. Other reports indicate that the missing states have at least remote referral sites.
The use of digital compression to send high-quality video images from rural areas to big-city hospitals for diagnosis has been instrumental in bringing specialized care to rural America. "Packets of information, such as patient records, X-rays, or pathology slides, can be sent over the telecommunications link to be stored digitally for review by a specialist. These data can be simple text, the primary care physician's notes on an initial examination, for example, or contain high-resolution images from CT or MRI scans." In addition to interactive videoconferencing, which allows patient and doctor to see and hear each other, new devices let physicians listen to a patient's heart, take blood pressure readings, and otherwise thoroughly examine a patient without actual physical contact (Laplant, 1995, p. 48). Once the capability exists to display registered images at the remote site, an expert can guide a remote medic by manipulating the graphics that appear on the screen (O'Toole et al., 1995, p. 272).
As for developing countries, their needs are not being ignored. For example, the Cleveland Clinic Foundation is exploring opportunities for providing expertise abroad. Clearly telemedicine is a developed technology. However, telemedicine applications vary from limited electronic exchange of records for review to sophisticated interactive systems with extensive diagnostic equipment.
A "cost-cutting trend is to treat more patients at home using telemedicine technologies rather than in the hospital. A project by Carnegie Mellon University (CMU), Pittsburgh, has combined wireless data communication and laptop computers to help nurses provide care at patients' homes..,[the system allows] nurses to receive doctors' orders, chart patients' progress, review medical reference books, and even show educational videos to patients" (Adam, 1994, p. 70). Such systems allow trained medical personnel to provide home monitoring and also allow self-monitoring. A recent study by Stanley M. Finkelstein of the University of Minnesota illustrates the potential of such a system. In his experimental setup, "lung transplant patients, who benefit from monthly checkups after surgery, test themselves, record the results, and transmit the results reliably from their homes. Not only may costs be cut, but daily tests (of blood pressure and 'best blows' into a spirometer) could help detect trends before symptoms indicate a problem. All the measurements were performed daily, stored automatically, and transmitted weekly to a relational database" (Adams, 1994, p. 70).
The Medical College of Georgia in Augusta, Ga., is putting interactive video and medical diagnostic equipment in the homes of "revolving-door" hospital patients suffering from chronic conditions. "Dr. Jay Sanders, director of the telemedicine center, who in 1991 founded a telemedicine network that has grown to include 59 Georgia hospitals and clinics claims the network has allowed 86% of the patients who would have been transferred out of a rural hospital for specialized care to stay put" (Laplant, 1995 , p. 48). Another example of home care began with Collins' search for a way to better monitor babies with cord entanglement problems. "He performed a clinical study...in which the patients were given telemetry/fetal monitoring equipment so that they could perform the monitoring themselves, without a nurse." The doctor linked into the woman's computer and studied the baby's heart rate and the mother's contractions (Collins, 1995, p. 2).
Other specialized telemedicine units are also being developed. For example, "a 'virtual clinic' at the University of Kansas in Kansas City provides diagnoses and treatment for Parkinson's disease patients in rural areas, many of whom might otherwise go untreated" (Laplant, 1995, p. 48). Another example is the monthly "tumor board" meeting of the Carolina Medical Center in Charlotte, North Carolina, and the Watauga Medical Center in Boone, North Carolina. The team of expert oncologists, pathologists, and radiologists reviews the particularly complex cases at Watauga's cancer center from its specially equipped telemedicine room at the Carolina Medical Center. The facility has a video camera, communications capability for real-time video transmission, and a 60-inch color monitor. The experts are thus able to interact with the audience of 30 surgeons, chemotherapy specialists, and health-support workers in Boone (Laplant, 1995, p. 48).
Another innovative use of telemedicine is in prisons. "East Carolina University [ECU] is performing telemedicine consultations to the largest prison in N.C... . Originally established to provide only emergency consultations for trauma cases, this network's usage has expanded to include 31 School of Medicine physicians from 15 medical disciplines. Physicians see and talk to the patients via the telemedicine link and then diagnose and prescribe medications, when necessary. Practitioners have access to a digital stethoscope, a graphics camera and a miniature, handheld dermatology camera to aid in patient examinations" (Balch, 1995, p. 15). The network is being expanded to six rural hospitals and a large naval hospital. A similar telemedicine prison project at the University of Texas, Medical Branch at Galveston, handles 45-65 telemedicine cases per week (Brecht, 1995, p. 8).
There are a number of pilot projects involving regional telemedicine. The following are some examples:
Acknowledgements: This research project was supported by the ATP information infrastructure for health care focused program. I would like to thank the following individuals for their help in the development of this paper: Richard Satava, M.D., Steven C. Otto, Sanford Ressler, and Neva Carlson.
|Go to BIBLIOGRAPHY|