It was the capstone project of my undergraduate program. I cooperated with Yuan Feng, designed and developed the whole game system. I was mainly responsible for the research process, physical product design and the initial idea of the game planning, while Yuan was responsible for coding and development. After my graduation, we cooperated with two fellow students to write essay of this project. And the essay was accepted for HCII2018 Conference.
2017. 03 - 2017. 06
User Interview, Competitive Analysis, Game Design, Prototyping, 3D Printing
PS, AI, Arduino, Unity, Rhino, Keyshot
This project studies the combination of virtual reality (VR) technology and conventional stroke rehabilitation physiotherapy. Specifically, we propose a novel therapeutic device coupled with an immersive VR software environment to foster hand rehabilitation.
We first study the current state of the art in VR technology use in medical rehabilitation. Next, we investigate the conventional stroke rehabilitation process to integrate accepted methods of physical therapy into mobile games. The game system’s input device (V-rehab) is an improvement on existing rehabilitation equipment, designed to maximize interaction between user and game. We feature a prototype game system based on sensor hardware and a custom environment running on the Unity3D software platform. Finally, we show results of system testing and discuss the application of VR in stroke rehabilitation.
Stroke is a medical condition in which poor blood flow to the brain results in cell death [1]. China has the highest incidence of stroke in the world. In recent years, the number of stroke survivors has increased with the development and progress of medical technology. However, about 75% of patients still suffer from varying degrees of motor dysfunction, especially hand motor dysfunction, with up to 10% of patients living with severe disability. The primary stroke hand dysfunction defects are buckling contracture, weakened grip, side clips, and other functional loss to palm and fingers[2]. Hemiplegic hand dyskinesia significantly reduces quality of life and increases the economic and psychological burden of patients as well as their families [3].
Prior research has demonstrated that exercise rehabilitation training can not only promote spontaneous nerve function recovery in patients with hemiplegia and help patients to restore central nervous system control and control of limb movement, but it can also prevent atrophy from muscle disuse and improve the recovery of motor function [4]. Physical therapy has become one of the most commonly used methods for rehabilitating motor function in clinical practice. At present, motor-function rehabilitation in patients with hemiplegia is still carried out under the guidance of a therapist, who assists patients with passive or auxiliary exercise training.
These methods have obvious shortcomings:
(1) Waste of manpower and material resources. Rehabilitation therapists cannot simultaneously guide more than a small number of hemiplegia patients in technique, and this costly and specialized field is impractical in a huge developing country like China;
(2) Rehabilitation training is mechanically tedious: hemiplegic patients rarely participate enthusiastically, which leads to lapses in treatment;
(3) It is very difficult to obtain feedback to gauge the intensity and effectiveness of rehabilitation training [5].
Existing research shows that the introduction of virtual reality technology and games into rehabilitation training is helpful for the rehabilitation of patients with motor dysfunction [6]. This method can provide a variety of feedback during training and reasonable rehabilitation recommendations according to the patient’s specific condition. A virtual environment can greatly increase patients’ initiative to participate in therapy, so as to effectuate rehabilitation [7]. With its three primary characteristics of interaction, imagination, and immersion, VR technology uses a synergistic merger of virtual and real environments to give patients strong sensory stimulation [8], which can greatly improve their enthusiasm for training.
Our research method was primarily based on a user interview.
Interviewees: Patients with a stroke experience who are recovering or who have recovered.
Interview form: One-to-one interview
Interview purpose: To understand the basic physical condition of patients during rehabilitation, rehabilitation methods, and psychological and physical demands during the rehabilitation process.
Main questions:
- Medical history (including age, stroke time, affected part, duration of rehabilitation)
- Hand rehabilitation training method and auxiliary equipment
- The family’s role in healing
- State of mind and emotional state since suffering strokes
- Lifestyle and exercise before strokes
- Views and attitudes towards the combination of virtual reality technology and stroke rehabilitation
The following is an overview of interview records of two stroke patients:
In communicating with stroke patients, we learned that patients must remain at home for an extended period during the rehabilitation process, and they usually cannot meet the standard training requirements without the supervision of a doctor. Furthermore, the lack of positive feedback in the process causes patients to suffer some emotional distress. Patients generally regard feedback and incentives as important during the rehabilitation process; our contacts reported that they would appreciate more reward forms in the game design, so that they can get a sense of accomplishment in the game process, as well as to ensure adequate training exercise. Finally, rehabilitation equipment should ensure safety and comfort, and it should be able to be handled by patients independently.
The hardware used by patients for rehabilitation also serves as the operating handset for the game system. In form, this is essentially an improved version of existing mechanical hand rehabilitation equipment. Among the devices that are already available on the market, this article compares their sensitivity to hand motion and aspects of their operation, operationalized as follows:
Operation: Hand rehabilitation training methods can be divided into active training and passive training. Active training is exercise through patients’ own muscle flexion and extension, while passive training is hand movement assisted by the equipment via motors or mechanical force devices.
Flexibility: The degrees of freedom of hand movement and the number of joints involved.
In order to ensure the rationality of the rehabilitation equipment, the choice of equipment refers to the existing hand rehabilitation equipment and is evaluated by the scientific program of hand rehabilitation. To offer the user a greater sense of interactivity and fully autonomous exercises during the rehabilitation process, the device should to have higher initiative and sensitivity and should be able to precisely record data for each finger press. Therefore, we chose a device with four pressure elements that users will need to press with the four fingers.
The selection of a virtual reality gaming platform was based on two considerations, one being the cost of purchasing head-mounted devices, and the second being the compatibility of the device for different phone size and system version. In order to allow use of these devices on a variety of mobile phones and to make this stroke rehabilitation system affordable, we chose the cheaper, more compatible Google cardboard and Google Daydream as a basic platform; users may choose a higher-cost platform according to economic situation, but we decided on this as a baseline.
To operate a VR game, the user opens the virtual reality game application in the smartphone and inserts the phone into a Google Cardboard or Google Daydream visor. The content displayed in front of the user will change via the phone camera and sensing by the phone or visor’s built-in gyroscope as the user moves his or her head.
This project used the Unity3D game software development platform with the “Google Cardboard SDK for Unity” plug-in to develop mobile virtual reality games. Our mobile game system structure is shown below
Specifically, the game system is divided into three parts: users, hand equipment, and mobile VR games. The user’s finger movement is captured and recorded by the pressure sensor in the handgear, and the data is read by the Bluno module and transmitted to the handset program via Bluetooth. After the mobile program receives the input data, the system responds and changes the corresponding scene in the screen and feedbacks to the user. These form the complete interactive process for our system.
Because pressure is the main action in stroke rehabilitation training, a pressure sensor was deemed as the best choice for input sensing. The pressure sensor we chose was the FSR402 Resistive Film Pressure Sensor, which converts the pressure applied on the FSR sensor film area into a change in resistance to obtain pressure information. The higher the pressure, the lower the resistance, allowing it to be used at 0g-10kg pressure to meet our design needs. It is small enough and light enough to be used in equipment with high experience requirements.
Based on the requirements of our project, we needed a device to communicate with the Arduino microcontroller and mobile phone. The Bluno Bluetooth module suited our requirements as the communication unit between our mobile gaming and hardware device.
We implemented a hardware device consisting of an Arduino microcontroller, a Bluno Bluetooth module, and pressure sensors. Four pressure sensors serve as the proximal interface for the user; they receive the user’s movement information directly. In the game, we defined the meaning of each of the four finger presses, each representing different actions in the user’s view. The function of the pressure sensor is to converts finger pressure into an electrical signal through the resistance effect described above, and to send the data to the microcontroller via pins connected to the Arduino.
Because the handset it is hand-held equipment, we needed to ensure that all electronic components such as hand-pressure devices, sensors, control panels, batteries, switches, and other devices will fit into the product, and that the product can be held by the average-sized hand and comfortably fit the shape of the hand.
Depending on the size of the hand and the features of the product, our design was most concerned with the fit of the thumb muscle to the handset, as this contributes most to the degree of comfort the when users hold the product.
Stage 1:
The initial model used an ellipsoid as the main form with the original four finger pressing parts of the same length. Our primary goal was to achieve a good fit between the thumb muscles with the handset.
Stage 2:
In the second model we made more adjustments in the curve of the arc which took into account the state of the full finger press given differing heights of each of the four fingers. Thus, the length of each button is adjusted according to different height. At the same time, the hand grip has also been changed to a greater curvature, making an overall shape resembling a bear’s paw.
Stage 3:
After confirming the design of the finger portion and the design of the bottom surface, we made the bottom line clearer on the original model. At the same time, the buttons were enlarged, resulting in a slightly mellower whole shape. Meanwhile, corresponding concavities and convexities were added to the hand-fitting area, so that the handset fits the natural contours of the palm.
Stage 4:
The previous model had a rather inelegant appearance, so we made a few further adjustments.
Final Display
The shape of the finger areas was improved to make it more abstract and give up the rounded shape; we also performed better blending on the palm-contact surfaces, making the shape more concise and clear. The final model is shown in following
Stroke rehabilitation patients’ hand movement ability is impaired, so our game design needed to take full account of the patient’s finger movement, in careful combination with the hand acupressure rehabilitation equipment. To do so, it is necessary to fully mobilize the motion of the four fingers during game operation. We devised the game settings in an attempt to achieve optimally effective finger movement according to the sequence and frequency of practical finger rehabilitation exercises. In the game design, we developed two styles of game scenes based on user characteristics of different genders and ages.
Scene 1: “Aircraft War” features explosions and realistic effects, incudes awards or penalty.
Game scene: Simulation of shooting down enemy planes in an actual first-person shooting confrontation. The player’s perspective is to drive a tank equipped with a machine gun. An enemy set in the game will send out a low-flying fighter to prevent the player from moving forward and will cast bombs at the right place to injure the player. Players must operate the tanks and use the machine guns to shoot as many enemy planes as possible to crash them, one point for each. The game scene is as shown.
Scene 2: “Harvest Forest” is a more leisurely game offering players a more relaxed and pleasant experience.
Game scene: Players wander in the forest according to the established route, along the way catching falling fruit from the trees with a basket. The basket must be placed at one of four possible places to catch the fruit, which is controlled by four fingers. Players need to catch as much fruit as possible, each catch earning a gold coin.
Aside from the game portion of the application, there are also device connection and data recording modules present in the mobile application. As for the connection module, the mobile application will automatically search for the user’s V-rehab equipment via Bluetooth. After matching the equipment, the user may select a game and then follow instructions to put the mobile into a Google Cardboard or Google Daydream headset. The user may now perform rehabilitation training in the panoramic game.
Finally, after developing the prototype of the game system, we invited several volunteers to conduct user tests. We divided the volunteers into two groups: those whose hands exercise functionally and half–hand-impaired people. The user tests aimed to judge whether the game system is suitable for hand-impaired people to use. The test was divided into two parts, testing the handset and testing the gaming experience. Combined with the user’s performance and attitudes, this paper proposes an improvement solution for both components, and this can be further studied in the future.
This project is an attempt to apply virtual reality to stroke rehabilitation training. The rehabilitation target is limited to the hand, specifically the restoration of the gripping and squeezing functions of affected fingers. Stroke rehabilitation requires training of many body functions, especially for the hand in restoring grip, wrist rotation, and other aspects. It can further be improved by exploring the game design for rehabilitation of other body functions of stroke patients. For this experiment, we only developed a simple action game scene with a simple plot. Developers can design more complex scenarios and scenes in order to understand the characteristics of patients with stroke and help them recover. In addition, this study did not establish a complete recovery evaluation mechanism due to time constraints. Thus, comprehensively assessing the recovery effects for patients playing virtual reality games wearing head-mounted equipment is beyond the scope of this paper. Future research can verify the effect of virtual reality technology for rehabilitation of stroke patients with fuller attention to recovery assessment methods. In conclusion, our design is a small attempt to solve the traditional problems with new technologies. With the development of medical technology and virtual reality technology, virtual reality will undoubtedly continue to provide options for medical treatment.
1. What Is a Stroke? National Heart, Lung, and Blood Institute. 2014-03-26 [2015-02-26].
2. Gabriele W, RENATE S. Work loss following stroke [J]. Disability Rehabilitation, 2009,31(18):1487-1493.
3. Tierney N.W, Crouch J, Garcia H, et al. Virtual reality in gait rehabilitation [J]. Modsim World, 2007.
4. Lin Riwu, Lin Maoen, Pan Shihai, et al. Mandatory exercise therapy in the treatment of stroke hemiplegia clinical application [J]. Cardiovascular and cerebrovascular disease prevention and control, 2007, 7 (2): 97-98.
5. Zhang Dongrui，Geng Yanjuan，Xu Lisheng，Zhang Xiufeng，Li Guanglin. Design and implementation of virtual reality hand rehabilitation training system [J]. Integrated Technology, 2013,2 (4): 33-40.
6. Saposnik G, Mamdani M, Bayley M, et al. Effectiveness of virtual reality exercises in stroke rehabilitation(EVREST): rationale, design, and protocol of a pilot randomized clinical trial assessing the Wii gaming system [J]. Int J Stroke,2010,5(1):47-51.
7. Levanon Y. The advantages and disadvantages of using high technology in hand rehabilitation [J]. J Hand Ther, 2013, 26(2):179-183.
8. Saposnik G, Levin M. Virtual reality in stroke rehabilitation: a meta-analysis and implications for clinicians [J]. Stroke, 2011, 42(5): 1380-1386.
