Daniel H. Goodman Department of Media Arts and Sciences MIT Media Lab, Changing Places Group Cambridge, MA, USA dhgood@mit.edu
Abstract The Changing Places group of the MIT Media Lab recently completed a prototype for a small-scale (200ft 2 ) transformable apartment to address some of the problems related to high-density urban living. The small size of the space paired with the density of technology creates an ideal environment for continuous sensing of the user. Of specific interest to the author is the recording of gait data to predict likelihood of fall as well as document precursors to and progression of joint diseases such as osteoarthritis. To allow for easy integration into the technology ecosystem of the CityHome, the author designed and constructed a 22x32 pressure mat with 1.25x1.25 resolution making use of off-the-shelf hobbyist components and the Arduino microcontroller system. Pairing the mat with the backend software of the CityHome, the system enables real time display of mat pressure readings as well as speculative simulated windows related to joint health information addressable by voice command. KeywordsGait analysis, smart home, MIT Media Lab, pressure sensor, falls, osteoarthritis I. INTRODUCTION The CityHome prototype constructed by the Changing Places Group will be used as a testing laboratory for hardware and software applications intended for deployment in a smart home. Though the possibilities for applications can fall under many different categories, two very important areas of research are related to rehabilitation and geriatrics. In 2013, the CDC estimated that one in three adults over the age of 65 falls [2]. Many families have first hand experience with grandparents or elderly parents falling and severely injuring themselves. Injuries range in severity from bruises to hip fracture and head trauma. The cost to treat all of these injuries exceeded $30 billion in 2010
[2]. Many of these people are ambulatory before the fall, but the injury and subsequent healing often lead to long term negative impacts on quality of life and even risk of death. For those who do recover, many lose confidence in their stability while walking, and as a result avoid ambulation. In addition to falls, joint disease, and in particular osteoarthritis, is becoming more prevalent in the obese and elderly population. Osteoarthritis affects about 27 million people in the US and causes pain and limited mobility in the affected joints [3, 5]. It develops gradually and typically progresses over many years. There is currently no cure; treatments include physical therapy, weight loss, or in extreme cases joint replacement surgery. For both falls and joint disease, several warning signs have been identified that are manifested as alterations in the normal gait. Factors such as reduced gait speed, increased stride time, and increased stride length have been linked to a higher likelihood of falling while lower walking speed, lower cadence, shorter step length, and an adjusted gait have been linked to gonarthritis and osteoarthritis [1, 4, 6, 7]. A pressure and location sensitive carpet, especially in the home continuously monitoring the occupant, could continuously and unobtrusively capture a rich set of gait data. The data can be analyzed in real time and warning markers could be identified. Continuous monitoring of walking could be a boon for doctors and family members, so that any early warning signs could be identified and treatments could begin earlier. The home is an optimal location for a gait-sensing device embedded in the floor/carpet, because recordings can be taken daily under similar conditions. The CityHomes computer system, for example, could run continuous analyses on the incoming data and alert appropriate parties should any warning signs appear. To collect spatial and force data from walking, a matrix pressure mat would need to be employed. There are several companies that design pressure sensing mats for orthopedic applications, and though the resolution is more than adequate for gait analysis, the systems are often closed- source, requiring proprietary hardware and software to use [13, 14]. Large pressure mats, on the order of a floor carpet, are also prohibitively expensive and preclude use outside of medical facilities and university labs. In an effort to create a practical and low cost sensing platform for the home, the author set out to develop a pressure mat that could easily integrate with the CityHomes operating system. Once the pressure mat was designed, built, and running through the CityHomes computer, the author created speculative interfaces, which served as a demonstration of the value that continuous gait monitoring could bring to the user. Questions such as what is my likelihood of falling? or how did I walk this week? could be posed to the system, and with proper analytics of the incoming data, an answer could be generated. II. METHODS A. The CityHome
Fig. 1. The CityHome Prototype
Note the central furniature module. This module contains the bed, desk, and computer system for the home. The RGB lighting and projection system can be seen above the living room. The CityHome is a scalable strategy for creating highly- efficient technology-enabled apartments that can help make living more affordable, productive, enjoyable and creative for urban dwellers [18, 19]. By combining transformable mechanical elements, a centralized operating system and compatable peripheral devices, the CityHome aims to optimize and enhance the experience of living in a small urban space. Transformable furniture allows the user to reconfigure the room on demand and to fit the specific requirements at that point. The bed and desk fit into a main module and are deployed when needed. The module itself can move about the apartment space, creating a larger bathroom, or a large living room. Using the devices and software of the home, developers can design hardware and software apps to enrich the customizibility and interaction for the user. A CityHome app store can serve as a depository of apps for the home, to be downloaded and used with minimal effort on the part of the user. Hardware add-ons to the native hardware of the home can pair with specific apps or serve as additional building blocks for the home device ecosystem. This is where the pressure sensor-enabled carpet would fit. B. The Pressure Mat The pressure mat operates under the same principles as a Force Sensitive Resistor or FSR. A FSR typically consists of two strips of a conductive material sandwiching or contacting a layer of material with a force-dependent resistance, see Fig. 2 [10, 12, 17]. In this case, the material was Velostat, which is manufactured by 3M [11]. Velostat or similar materials consist of a polymer impregnated with carbon. As pressure is applied to the FSR material, the carbon particles become more densely packed, resulting in lower resistance, which can be easily measured as a voltage change.
Fig. 2. Force Sensitive Resistor and its Construction
The Arduino has the ability to write and read analog and digital signals. For the case of reading an FSR, the Arduino applies 5V (digital write) to one terminal, and reads the voltage at the other terminal (analog read) [8, 9]. Applying a force to the pad will vary the resistance and therefore the recorded voltage. Generally, a pull down resistor is used to divide the output voltage and bring the readings into the measuring range of the Arduino before being collected.
The schematic for a typical FSR seen in Fig. 2 is shown below. Note the polymer is denoted as Velostat in the diagram.
Fig. 3. Schematic For a Single FSR Connected to an Arduino
5V comes from the Arduino digital pin and A0 is the analog pin. R represents the pull-down resistor, usually 10k! for a single pad.
In order to collect data regarding the location of the force, a different configuration is needed. There are still two conductors sandwiching the polymer, but this time there are rows and columns of conductive material on either side of the sheet. Each row is connected to a digital write pin on the Arduino, and each column is connected to an analog read pin. By writing one row high (5V) and reading each of the columns, a scanning read of the entire mat can be achieved. This method gives row*column unique values [17].
Tekscan, a well-known pressure sensor manufacturer, uses this collection method in their pressure sensors. A key difference is that they use an inverting amplifier before reading the signal [14].
Fig. 4. Schematic For Row and Column Configuration of the Pressure Mat
Fig. 5. Schematic from Tekscan For Reading a Pressure Mat
Tekscan utilizes an inverting amplifier before reading the output signal. For the prototype described, the author uses pull down resistors [14].
Two iterations of the pressure mat were constructed, one that was ~12x12 and another that was 20x32. An Arduino Uno controlled the smaller version, and an Arduino Mega controlled the larger version. For the smaller iteration, a 0.25 thick square of acrylic served as the base. Strips of copper foil tape were laid down so that there were five columns. Using a 1/16 square of acrylic as the stepping surface, the five copper rows were placed underneath. A square of polymer was placed in between the intersecting conductors, and the termini of the strips were connected to their respective pins on the Arduino. For this configuration, the data read by the Arduino consisted of a string of 25 values between 0 and 1023 (analog read range) separated by a space. The resolution was fairly poor, with one node every ~2.5.
Fig. 6. The First Pressure Mat Prototype
The black material is the Velostat. The rows and columns can be seen above and below the Velostat.
The second iteration was closer in area to a small rug and was sized so that at least two steps could be captured. Instead of acrylic, the two main surfaces were thick paper. One of the lessons from the earlier design was that due to the more rigid nature of acrylic, any applied force was distributed over a larger area, even if the force wasnt being applied at the surrounding points. Paper is more flexible, so the applied forces dont bleed into neighboring nodes as much. The new pad had fifteen columns by twenty-three rows. The number of analog read pins limited the number of columns. Since the Arduino Mega has forty-eight digital pins, there were more than enough pins available for the rows [8]. The string of data was now 345 values long. The resolution of the new mat was about 1.25x1.25.
Fig. 6. Two Images of the Second Mat Prototype
C. Visualizing the Data with Processing Processing is a visual programming language, with a very similar syntax to Arduino (in fact, the Arduino syntax was modeled off of Processing). Data can be read by the program from the Arduino and plotted, manipulated, and visualized etc. [20].
The scanning read method the Arduino uses to collect the mat data produces an array of values that is row*column long. Processing arranges the data from the array into a matrix representing the spatial position of the specific value. The matrix is then plotted where a square represents a node of column and row intersection. Each square is colored on a scale of black to white depending on the magnitude of the recorded force on that node.
Fig. 7. Raw Visualization of Large Pressure Mat, Unloaded and Loaded
The first image is the unloaded case. In the second image, the author places his fingers on the mat; each of the black squares indicates a finger contact point.
The above method works well for visualizing point forces, but a way to construct footsteps from the squares was needed. Using the OpenCV functions within the Processing environment, the output in Fig. 7 is analyzed, and clusters of high force (indicating a step or distributed force) are pulled from the surrounding data. Using another function, the centroid of that specific cluster can be plotted and its coordinates saved. The centroid of the cluster approximates the center of the footstep. By logging the coordinates of each centroid, the path of a user across the mat can be plotted. In addition to x and y coordinates of the step, the average forces present as well as the time between steps can also be recorded. These data can be used to analyze the gait characteristics of the user.
Fig. 8. OpenCV Isolated Clusters of Force and their Centroids
The centroids are plotted once the user reaches the end of the mat, creating a walking step pattern. In the second image, the author used his hand to apply force and create the step path. D. Interfacing with the Gait Data This section is currently speculative, as there is insufficient accuracy in the pressure mat as well as insufficient training data to generate prognoses related to gait. This section assumes an accurate, floor carpet-sized pressure mat recording and analyzing gait data in real time and for several months. Given these assumptions, the author generated simulated outputs corresponding to potential user queries. Ideally, if ones home is constantly collecting walking data, he/she would want to be able to access it, and be shown trends and warnings if necessary. Using the CityHome, the user is able to ask for specific results, which the house could then project against the wall for the user to see. In the CityHome prototype, the author linked these simulated result pages to specific user vocal commands.
Figure. 8. CityHome, how was my walking this week?
By recording the gait data for the week, the CityHome can generate plots showing the trends in various gait features. The user can query the system using natural speech to obtain an analysis.
Fig. 9. CityHome, how am I walking?
This query is particularly interesting because it is qualitative. An answer to this question requires a large dataset (in this simulated example, it was a year) and advanced analytics that can identify trends in gait and link them to clinically validated warning markers. In this example, the CityHome notices the user has some signs of emerging osteoarthritis. It suggests ways of preventing its progression, and reassures the user that the physician has been sent the data as well. [16]. The user now can take an active role in checking in on gait health and be notified of problems as well as ways to improve.
Fig. 10. CityHome, what is my likelihood of developing osteoarthritis?
This question is exciting to ponder because with sufficient data and analytics, your home could accurately predict your chances of developing joint disease based on your walking pattern. In essence, your gait becomes a trackable biomarker, just like weight, blood pressure and other physiological parameters. In this theoretical situation, the CityHome would present a screen showing a percent and a message. The message could give encouragement and reinforce good behavior. Eating well, staying active etc. can be promoted as ways of reducing the likelihood of joint disease.
Fig. 11. CityHome, what is my likelihood of falling?
Similar to the previous question and Fig. 10, an answer to this question would involve a likelihood paired with a message. One of the problems associated with falling is that often the person loses confidence in walking. Obtaining feedback and encouragement from this system could have a positive impact on the user continuing to ambulate. Though much work is needed in the development of living space-scale sensing, generating predictive algorithms, and analyzing the gait data for statistically significant trends in real-time, the idea of the home providing diagnostics and coaching is very exciting and worth developing further.
III. RESULTS AND DISCUSSION A. The Constructed Pressure Mat The 22x32 mat proved to be directionally successful in the trajectory to develop a low cost system. The overall cost was less than $100 compared to the several thousand dollar cost for a similarly sized professional mat. Refining the design can bring the price down further while improving the resolution and repeatability. For example, adding a multiplexer could yield 0.5x0.5 or 0.25x0.25 resolution with the same Arduino chip. The current design with the 1.25x1.25 resolution is able to recognize 345 unique sites of force. The collected data can be sent to any number of programs via serial for analysis and visualization. Keeping the interface protocol as standard serial will allow more people to interface with the data and develop useful apps. Unfortunately, while the mat was able to recognize point forces very well, steps were harder to consistently identify. To identify these steps using the OpenCV functions, a large amount of filtering was needed to deal with the noisy signal. This prevented statistically significant results to be obtained from from the current mat. Another challenge was that the mat had difficulty recording clusters when a user stood on the mat. The electronics and circuitry need more development to improve the ability to capture higher distributed forces. B. Implications of Continuous Gait or Biometric Sensing within a Smart Home Looking at the CityHome as a model for potential future technology-driven homes, the author believes there is a significant opportunity to collect and analyze user data in such a space. Not only can the data be used to optimize the users preferences and home environment, but it could also be utilized in a number of related applications. In the scope of healthcare, these data can be used for patient monitoring, diagnostics and providing reassurance. Having a floor-sized pressure sensing carpet could provide value to both patients and healthcare providers. With robust algorithms and sufficient resolution, gait data collected continuously could be used to identify likelihood of falling or likelihood of developing debilitating conditions such as joint disease. These innovations could help millions of people and save billions of dollars by reducing the number of preventable falls, and identifying diseases such as osteoarthritis early enough to treat and slow further progression. More people can remain mobile longer. In addition to preventing falls or joint disease, continuous gait monitoring could provide encouragement to those who have already fallen or are suffereing from musculoskeletal disease. The ability to track progress and receive encouragement from the home could have a positive psychological impact on the user. Returning confidence in ones walking, and providing suggestions for improving symptoms could encourage increasing mobility and taking a more active role in their health management. The continuous monitoring doesnt have to be limited to gait. Imagine how heart rate monitoring or non-invasive cholesterol measurement, or caloric intake tracking could be used. Any ailment that requires a large dataset to track its progression could be monitored using embedded sensors within the home. Doctors checkups could be transformed because more data could be collected around the house and analyzed faster than in any doctors office visit. The role of the primary care doctor may shift from collector of data to manager of data thanks to the CityHome system. In the not too distant future, it is not unreasonable to imagine a scenario where physiological sensors, algorithms and analytics could be used to predict disease better than a doctor. The author is excited to continue exploring continuous biometric monitoring in the home. With large datasets of biometric data, the diagnosis and treatment of many chronic illnesses could be enhanced tremendously. Diseases can be identified earlier, treatments could begin earlier, and quality of life for millions can be improved. C. Future Work Several improvements need to be made to the pressure mat before it can reliably collect gait data. The electronics need to be improved so that higher forces on the order of body weight can be collected, because the existing design had trouble identifying clusters of body-weight force. Additionally, the material choices in the pressure mat need further investigation. Velostat and similar materials have large hysteresis and relatively slow settling times. For more rapid and accurate force measurements, other materials may need to be used. The author wishes to eventually design modular pressure tiles that can be connected together to create larger mats. This way, an arbitrary-shaped floor can be covered in pressure sensing carpet. The Arduino can be reduced to a very small PCB and with the aid of multiplexors, cut the number of output pins needed to control a single mat.
IV. CONCLUSION In this paper, the author discusses a design for an open-source, low cost pressure mat carpet system and visions for continuous gait monitoring within a smart home environment. The constructed pressure mat was a fraction of the cost of similarly sized professional mats and had reasonable resolution. It was able to identify 345 unique pixels and the data was easily collected and visualized with Arduino and Processing. Unfortunately, the accuracy of the mat was not good enough to reliably and consistently identify steps. Further development of the system is needed before running feasibility and other user studies and data collection experiments. Though simulated, the outputs for the user questions regarding walking and joint disease predict a very exciting world where the home will be able to identify diseases and suggest ways to improve the situation. By continuously analyzing biometric data, a smart home environment such as the CityHome could predict joint disease, fall likelihood or any number of other conditions earlier on. Alerting the user, users family, or doctor could prevent disease progression and improve quality of life. ACKNOWLEDGMENT The author would like to thank Professor Hugh Herr for the suggestion to explore this area of monitoring within the home. The author would also like to thank Jean-Franois Duval and Carson Smuts for their assistance in debugging and testing the prototype pressure mat. REFERENCES
[1] Brach JS, berlin JE, et. al. (July 26, 2005). Too much or too little step width variability is associated with a fall history in older persons who walk at or near normal gait speed. The Journal of Neuroengineering and Rehabilitation. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/16042812. [2] Falls Among Older Adults: An Overview. Centers for Disease Control and Prevention. Retrieved from http://www.cdc.gov/homeandrecreationalsafety/falls/adultfalls.html [3] Osteoarthritis. Centers for Disease Control and Prevention. Retrieved from http://www.cdc.gov/arthritis/basics/osteoarthritis.htm. [4] Montero-Odasso M, Shapira M, et. al. (2004). Gait velocity in senior people. An easy test for detecting mobility impairment in community elderly. The Journal of Nutrition, Health & Aging. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/15359349. [5] Osteoarthritis. Arthritis Foundation. Retrieved from https://www.arthritis.org/conditions-treatments/disease- center/osteoarthritis/. [6] Bejek Zoltan, Robert Paroczai et. al. (2006). Gait Parameters of Patients with Osteoarthritis of the Knee Joint. Physical Education and Sport. Retrieved from http://facta.junis.ni.ac.rs/pe/pe2006/pe2006-02.pdf. [7] Sarmini, Mahmoud M.D. Understanding Normal & Pathological Gait. Retrieved from http://www.medschool.lsuhsc.edu/physical_medicine/PPT/Normal_Path ological_Gait.ppt. [8] Arduino Mega. Arduino. Retrieved from http://arduino.cc/en/Main/arduinoBoardMega. [9] analogRead(). Arduino. Retrieved from http://arduino.cc/en/Reference/analogRead. [10] Fraden, Jacob. Handbook of Modern Sensors. 3rd ed. N.p.: Springer, 2004. Print. [11] 3M Velostat Conductive Bags and Films. 3M United Kingdom. Retrieved from.http://www3.3m.com/catalog/uk/en002/electronics_mfg/static_con trol_packaging/node_JV20CN7XFBbe/root_K3BHNB8005gv/vroot_5K R9K2C2KZge/gvel_JQGPTT9723gl/theme_uk_staticcontrolpackaging_ 3_0/command_AbcPageHandler/output_html. [12] Force Sensitive Resistor (FSR). Adafruit. Retrieved from https://learn.adafruit.com/force-sensitive-resistor-fsr. [13] http://www.tactonic.com/ [14] http://www.tekscan.com/tekscan-technology [15] Abellan van Kan G, Rolland Y, et. al. (2009). Gait speed at usual pace as a predictor of adverse outcomes in community-dwelling older people an International Academy on Nutrition and Againg (IANA) Task Force. The Journal of Nutrition, Health & Againg. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/19924348. [16] What is Osteoarthritis? National Institute of Arthritis and Musculoskeletal and Skin Diseases. Retrieved from http://www.niams.nih.gov/Health_Info/Osteoarthritis/osteoarthritis_ff.as p. [17] Woven Pressure Sensor Matrix. How to Get What You Want. Retrieved from http://www.kobakant.at/DIY/?p=4296. [18] http://www.media.mit.edu/research/groups/changing-places [19] http://cp.media.mit.edu/research/67-cityhome [20] http://www.processing.org/reference/