The Use of Multimodal User Interaction in Cognitive Rehabilitation

Letter to the Editor

As Editorial Board Member, I am writing this letter in order to present the progress and brief summary of my current research topics.

During my current position at Centre for Automation and Robotics CAR (UPM-CSIC), the main topics I focus my research on are related to Cognitive Rehabilitation. The main goal is to get advantages from the current technological state of the art and know-how in Engineering to provide with solutions which improve the quality of living and increase independency of those people who find almost impossible to live without the help from a relative or caregiver. Especially, those people who suffer from stroke need continuous assistance to carry out what are called Activities of Daily Living (ADL) such as preparing food, drinks or grooming [1].

A stroke is consequence of lack of oxygen or nutrients which blood supplies to a specific region. This produces in the patient a wide range of symptoms including cognitive deficits focused on difficulties when thinking, learning or paying attention.

One of the most common diseases after stroke are called Apraxia and Action Disorganisation Syndrome (AADS). Apraxia turns out to be a neurological disorder of those purposive movement skills previously learned [2] while Action Disorganization Syndrome introduces cognitive errors when carrying out multiple-step activities [3]. So, the cognitive ability of patients who suffer from AADS while performing ADL is quite minimum and this introduces the necessity of continuous care while facing everyday tasks or even more, multiple visits to the corresponding rehabilitation center or hospital [4].

Considering that, depending on the severity of the disease, some patients experiment a very long-term recovery, they need permanent and daily rehabilitation sessions which make them feel uncomfortable in some cases, losing their whole independence. This is the main motivation of my contribution to this field which is framed in an European project called Cog Watch [5,6]. The main goal to achieve with this project is to step forward and provide with a novel concept for cognitive rehabilitation by developing a personalized and long-term platform to support these patients while executing everyday tasks at home. This platform is composed of several instrumented tools and specific software to automatically recognize and detect what the patients are carrying out and if case of errors, inform about how to correct them by using interactive and vibro-tactile cues.

Different scenarios based on ADL are considered while developing the platform. As mentioned before in the document, ADL involve tasks related to basic self-care (preparing food and drinks in the kitchen, or grooming in the bathroom) and the correct performance of them requires to accomplish a sequence of more basic actions whose proper execution makes successful the whole task. So, continuous monitoring and feedback during these sub-actions is essential to ensure a good finalization of the final goal.

Currently, two prototypes are being developed, one focused on providing cognitive rehabilitation while preparing a drink (especially tea), and other while tooth brushing. Patients interacts with an attractive interface which lets them, for example, to select among different options for extra ingredients in tea preparation (sugar, milk or both). In any case, the main purpose of the interface is to act as a virtual therapist and to show the patients only in case of error, the necessary information to correct it by using different multimodal cues, i.e. text/audio messages, still images or even, simulations and real videos.

Regarding the devices used for the implementation of the platform, we classify them into two categories: monitoring and feedback devices. Monitoring devices are in charge of supervising the behavior and movements of the patient to pass this information to the specific algorithms to automatic recognize what the patient is executing. Meanwhile, feedback devices are focused on providing the corresponding cues to make the patient aware of the errors committed. It is important to highlight that the platform does not guide the patient during the performance of the task, just inform him/ her in case of error and show how to behave in consequence.

On the one hand, monitoring devices are mainly composed by: KinectTM, sensorized objects and Leap, although the functionality of some of them differs depending on the prototype considered. Considering the first prototype for rehabilitation in tea preparation, KinectTM is used to track the hands of the patient while manipulating the corresponding objects involved in the task (kettle, cup, milk jug, etc.) [7]. Continuous tracking is achieved by taking advantages from the open source image processing libraries which let us detect and recognize hands and arms over a table surface, of course, with a proper post processing of the data to remove noise and false detections. Meanwhile, the sensor zed objects correspond to the common objects used for the completion of the task which we strategically attach some specific sensors to. The sensors used are accelerometers and force sensitive sensors in order to obtain relevant information about the movement of the objects or if the patient grasps them or not.

Regarding the second prototype focused on tooth brushing, the information provided by the sensitized objects is the same, although the new objects now are different from the previous ones such as toothbrush, toothpaste dispenser, or glass of water. However, KinectTM now is used not to track hands but to track the patient's face, especially, the mouth points. This makes possible to build a 3D region or workspace around the mouth to obtain a better detection of the sub-action focused on approaching the toothbrush to the mouth to start tooth brushing action. But, how we implement tracking of the tool used, i.e. the toothbrush? Well, for this purpose a new motion sensor called Leap [8] is used to make in this case the detection and tracking of the hands during the manipulation of the objects over the table or sink and also to track the tool grasped, especially, the tip of the toothbrush, to provide monitoring during the approach to the mouth, as mentioned before.

On the other hand, feedback devices are composed by: smart watch from Met watch [9] and an All in One computer or tablet. The smart watch turns out to be a novel watch which can be paired with smart phones or any mobile devices and provides via Bluetooth vibration cues to the patient when committing errors. Meanwhile, the objective of the computer or tablet is two-fold: to act as the main processor to store all the data, algorithms and the corresponding software needed for the communication between all the devices, and to provide the interface the patient interacts with, already mentioned. Except Kinect™ and Leap, which are connected to the computer via USB, the rest of devices send the data via Bluetooth to the computer so, the platform turns out to be quite simple and easy to install.

Although not used during the rehabilitation session, we dispose of an additional non invasive monitoring device which is a module designed by RGB [10] to measure some vital signs of the patients such as heart rate and blood pressure to provide data the level of stress before and after the session. This data is shown via Bluetooth to the computer to store the measurements and the module is activated and deactivated remotely from the computer/tablet.

Finally, as the final goal to achieve is to provide rehabilitation while patients are executing their common daily activities at home, all the performance and data recorded is also supervised remotely by the corresponding medical expert who is in charge of the rehabilitation process of the patient. For that purpose, the clinician has an interface which is installed on his/her computer to obtain in real time information about the behavior of the patient, data recorded, errors committed, etc. All the data recorded is used later by the experts to analyze the progress of the corresponding patients during the rehabilitation period.

A complete and exhausted evaluation of the first prototype during tea preparation has already done and more information about the results obtained can be found in [11].

References

1. Smania N, Aglioti SM, Girardi F, Tinazzi M, Fiaschi A, Cosentino A, et al. Rehabilitation of limb apraxia improves daily life activities in patients with stroke. Neurology. 2006; 67: 2050-2052
2. Rothi L.J.G, Heilman K.M. "Apraxia: The Neuropsychology of Action". Psychology Press: East Sussex, UK, 1997.
3. Morady K, Humphreys GW. Comparing action disorganization syndrome and dual-task load on normal performance in everyday action tasks. Neurocase. 2008; 15: 1-12.
4. Sunderland A, Shinner C. Ideomotor apraxia and functional ability. Cortex. 2007; 43: 359-367.
5. CogWatch.
6. Cogollor Jose M, Pastorino, Matteo; Arroyo, Javier; Fioravanti, Alessio; Wing, Alan; Arredondo, Maria Teresa; Ferre, Manuel; Breñosa, Jose; Hermsdörfer, Joachim; Teresa, Javier; Walton, Clare; Worthington, Andrew; and Giachritsis, Christos. "An Innovative Solution Based on Human-Computer Interaction to Support Cognitive Rehabilitation". V International Congress on Design, Research Networks, and Technology for all. 23rd-25th September, 2013, Madrid, Spain.
7. Cogollor JM, Hughes C, Ferre M, Rojo J, Hermsdörfer J, Wing A, et al. Handmade task tracking applied to cognitive rehabilitation. Sensors (Basel). 2012; 12: 14214-14231.
8. LeapMotion, Inc. 2014.
9. MetaWatch, Ltd. 2014.
10. RGB Medical Devices.
11. Pastorino Matteo, Fioravanti Alessio, Arredondo Maria T, Cogollor José M, Rojo Javier, Ferre Manuel, et al. "Preliminary Evaluation of a Personal Healthcare System Prototype for Cognitive eRehabilitation in a Living Assistance Domain." Sensors 2014; 14: 10213-10233.