An Auditory Display System for Aiding Interjoint Coordination

Claude Ghez Department of Neurology Columbia-Presbyterian Medical Center cpg1@columbia.edu

Thanassis Rikakis, R. Luke DuBois Computer Music Center Columbia University than/luke@music.columbia.edu

Perry R. Cook Computer Science Dept. Princeton University prc@cs.princeton.edu

ABSTRACT

Patients with lack of proprioception are unable to build and maintain ‘internal models’ of their limbs and monitor their limb movements because these patients do not receive the appropriate information from muscles and joints. This project was undertaken to determine if auditory signals can provide proprioceptive information normally obtained through muscle and joint receptors. Sonification of spatial location and sonification of joint motion, for monitoring arm/hand motions, was attempted in two pilot experiments with a patient. Sonification of joint motion though strong time/synchronization cues was the most successful approach. These results are encouraging and suggest that auditory feedback of joint motions may be substitute for proprioceptive input. However, additional data will have to be collected and control experiments will have to be done.

Keywords

Data sonification, limb movements, joint coordination, proprioception, interface design, software synthesis.

INTRODUCTION

The nervous system depends upon a continuous stream of sensory signals to move our hand through a intended trajectory [1]. In most tasks vision provides critical information about targets and spatial constraints (e.g. obstacles) and plans movement in a representation of Cartesian space. However, to actually displace the hand these spatial plans have to be transformed into control signals driving muscles and producing torques at joints [2] and to take account of the complex time-varying forces resulting from the dynamic mechanical properties of the limb. An especially complex problem is posed by inertial interactions, which vary with acceleration and deceleration at each joint [3]. Recent experiments in our laboratory indicate that spatial and joint-level errors are processed in parallel during learning [4]. To compute appropriate muscle commands the nervous system relies on an ‘internal model’ of the mechanical properties of the limb [5, 6], which it learns through practice using information from muscles and joints [6-8]. When this information, known as proprioception, is lacking, as occurs in a neurological condition known as large fiber sensory neuropathy, movements become highly inaccurate: unless the hand is visible trajectories intended to be straight become curved and there are prominent errors in extent and direction [5]. A characteristic anomaly is the inability to reverse the direction of hand movement sharply: the elbow and shoulder become desynchronized [9] because of uncontrolled effects of inertial interactions produced at the by the motions of the shoulder [10]. Thus, the patient with sensory neuropathy is unable to predict where their hand will go in response to their voluntary commands.

Vision can, however, substitute partially for proprioception but not completely. First, while accuracy improves when patients can see their limb, movements remain less accurate than normal. Second, the improvement produced by vision persists for a few minutes after the limb can no longer be seen. Thus, errors are not produced by a failure to correct trajectories but because the patients ‘forget’ how their arm works. Third, even with vision such patients are unable to learn new dynamic properties as would be required to aim a hammer or a tennis racket.

Physiological studies indicate that muscle and joint receptors provide static information about joint angles and dynamic information related to the velocity and acceleration of joint motions [11, 12]. Residual motor and learning deficits in neuropathy patients could derive from the fact dynamic information is not readily monitored visually. However, this information could, in principle, be transmitted more efficiently through auditory channels. Auditory information is processed more rapidly [13] as it does not encounter the long delays or the high degree of low pass filtering of visual channels [14]. Moreover, many tasks involving sound perception, including language comprehension, music and dance, are based on the analysis and prediction of complex time series data. This project was therefore undertaken to determine if auditory signals can provide proprioceptive information normally obtained through muscle and joint receptors.

APPROACHES

We considered two basic approaches to the problem of generating auditory feedback reflecting limb movements: encoding hand position (or speed) in Cartesian space and encoding joint rotation.

Sonification of Spatial Location

We first attempted to encode target locations and hand positions in sound space. We devised a “pitch mapping” exercise with the pitch mapped on the vertical plane (x dimension) and the amplitude mapped on the horizontal plane (y dimension). The subject used the mouse to control the pitch and amplitude of a tone and moved this tone in the above space attempting to match the pitch and amplitude of regularly appearing target tones. The image of the sound space appeared on the computer screen. Matching exercises became harder as the test progressed.

Pilot studies using this set-up where realized in the summer and fall of 1999. They showed that the kind of auditory feedback described above could provide subjects with usable information about hand position and hand path curvature, however, accurate matching of hand to target position was quite difficult. Accuracy comparable to what is achieved using visual information did not seem to be feasible. Target matching tasks in space through auditory feedback could be learned but target matching through vision was much more accurate and involved much less training. This conclusion was in accordance with extensive bibliography showing that human auditory perception does not create/utilize auditory spatial maps [15]. Humans use inter-aural cues for generic localization of a sound source but for specific localization required for target matching humans mostly use vision. On the other hand, sound has been shown to be a very strong timing/synchronization cue [16] and music has been used throughout the centuries to provide accurate timing information for movement. From folk songs using phrasings characteristic of the task they accompany (harvest songs, songs for rowing or collecting fishing nets) to military marches to metrically complex musical scores for dance (Rite of Spring and Les Noces by Stravinksy –interestingly both deriving their rhythms from folk dances-). Furthermore, auditory perception is know to be able to encode/decode in real time fairy complex frequency and time patterns and abstract complex organization structures from this information [17, 18].

Sonification of Joint motion

We therefore decided that it would be more practical if auditory feedback of limb movements was to provide specific timing and structure cues and generic spatial cues. We temporarily abandoned the approach of providing accurate localization cues for the hand and we concentrated on providing subjects with auditory signals that varied with the motions of joints and on developing tasks that might benefit from the use of such feedback in neuropathy patients.

The subject was a 54 year old patient with severe large-fiber sensory neuropathy of unknown etiology who has been studied by our laboratory in several previous experiments [10, 19]. This patient has normal strength in her face, arms and legs as well as normal sensations of pain and temperature. On the other hand, she is unable to detect when the joints of her arms, hands and legs are moved passively at any speed. Nor can she assess how far she may have moved her hand when attempting to reach for an object without visually observing it.

Three initial sets of experiments were planned of which two have been carried out. In experiment 1 we examined whether the patient could use auditory information to control the timing of a direction reversal at the elbow during a two joint movement. Pilot data obtained in the experimentors indicated that normal subjects can do so. Experiment 2 examined whether the patient can process auditory information about joint motion while adapting to a novel inertial configuration of their limb in a visuomotor task.

EXPERIMENTS

Experiment 1

In experiment 1 the patient was to make a rapid out-and back movement of the hand mimicking the gesture of slicing a loaf of bread in time with an ascending and descending melody and a metrical musical beat. The movement involved extending and then flexing the elbow while the shoulder initially flexes then extends. A sharp reversal of the hand movement requires reversing simultaneously the directions of elbow and shoulder rotations. This is particularly difficult for patients with sensory neuropathy: the elbow tends to flex prematurely propelled by inertial interactions arising from movement of the upper arm [7][9]. Thus, we first wanted to determine if the patient could reverse the direction of an elbow motion at a predetermined time relative to an external timing signal.

The motion was to be realized over a two beat measure with the motion starting on the down beat, the extension of the arm happening over the first beat, the reversal happening right on the upbeat and the retraction happening during the second beat. Then there was a measure of rest. The testing sequence could be extended to the desired length by adding couplets of one measure of motion and one measure of rest. Beats were sounded by a metronome click. The motion was accompanied by piano arpeggios or scales that followed the broad outlines of the motion (upward pitch motion during the time the arm was to extend and downward pitch motion during the time the arm was to move inwards). We used two sounds to provide auditory feedback for the timing of the reversal. If the reversal was almost in total synchrony with the up beat and the reversal of the pitch contour in the piano (within +/- 20ms of the up beat) then the reversal triggered a bright piati sound and a cello line which doubled (at the octave) the notes of the piano. If the reversal was almost in synchrony with the upbeat (within +/- 65ms of the up beat) then only the cello line was triggered. The synchrony of the piano/cello duet was indicative of the “correctness” of the time of reversal.

Since we did not know if/how the piano accompaniment would influence the shape of the movement we created a number of different melodies to see whether differences in meter (a 6/8 swing meter or a 2/4 directional meter), melodic contour, contour reversal point or pitch distance would influence the ability to synchronize the desired movement to sound. (See Figure 3 for melodies and corresponding numbers). Testing of the melodies by the experimenters found that melodies 1 and 5 facilitated the synchronization of this movement to sound in 2/4 and 6/8 meter respectively. However, the experimenters found that they could learn to synchronize with all the melodies after a few tries. Thus, extensive, controlled testing of the melodies was not attempted. We chose to use the melodies identified as the easiest to learn and to start working with the patient immediately since the applications of this sound structure in neuropathy patients was our main concern.

The patient was first taught to make simple elbow flexion jerks whose onset was synchronous with the reversal in the melody. She was then instructed to perform the out and back ‘slicing movement’ that demanded elbow and shoulder rotations and she was asked to synchronize the reversal with the upbeat. Ten couplets of one measure of sound and one measure of rest (10 complete motions with rest between motions) were included in each run. 20 runs were completed. 15 runs using melody 1 (at a speed of 750ms per beat) and 5 runs using melody 5 (at a speed of 468ms per beat).

Figures 1 and 2, below, give a schematic representation of how auditory information related to movement (represented in the figures by an elbow/shoulder flexion/extension diagram) in experiment 1.

Despite the absence of proprioceptive information the patient was able to learn both tasks in approximately the same time as intact subjects. This shows that even in the absence of proprioception it is possible to initiate a voluntary contraction at a time specified by an internal cue linked to a metric beat. It also suggests that discrete auditory cues, associated with movement reversals can be used to counter inertial interactions predictively.

However, not all auditory display elements were of the same importance in her learning. Time/synchronization cues, (metric beats providing a fixed time length for the completion of the movement, down beats providing a starting cue, upbeats providing a cue for the exact reversal time, reversal synchronization feedback (piati and cello)) seemed to be of most use in achieving a synchronized reversal. It was also clear that timing information was being used at least at three levels of organization: at the level of the subdivision (providing a regular driving force), at the level of the beat (down beat to start the motion, up beat to reverse, reversal feedback on the up beat to synchronize the reversal) and at the level of the measure (where a measure was a symmetrical time unit that contained a complete, symmetrical motion with the outward motion being completed over the first part of the measure and the backward motion over the second part).

The suggestion that the auditory information was being perceived as a unit with multiple levels of time organization is reinforced by the fact that when the subject started the motion correctly (right on the down beat) her percentage levels of achieving the reversal at the correct time were considerably higher. This successful use of hierarchical, multi-level organization of time related sound cues by the subject might be showing that our ’internal model’ of the motion also includes hierarchical levels of organization. If that is so, a musical feedback with matching hierarchical levels of organization is very appropriate.

Taking away the piano melody did not seem to influence the performance considerably. Furthermore, when the same motion was attempted in very fast tempos (fast motions are more desirable for this task since they produce the most exaggerated results for patients with loss of proprioception) single notes of the piano melody became indistinguishable. The piano line became more of an outline of a contour rather than a melodic line with distinct notes. Thus, the issues that had driven the creation of different melodies seemed to be of little consequence for this particular experiment. It was clear that for fast tempos the patient was listening only at higher levels of time organization, paying attention to the beats and measures and not to subdivisions.

Figure 1: Experiment 1, using Melody 1 Figure 2: Experiment 1, using Melody 5

Figure 3: Melodies for Experiment 1 Figure 4: Experiment 2

Experiment 2

The conclusions from experiment 1 showed that for this task it would be best to a)use detailed auditory display for timing/synchronization cues and b)use pitch/melody cues not as indicative accompaniment but as auditory feedback for contour information and only in places where such information could be of use.

In experiment 2 we wanted to examine whether the patient can make use of this kind of auditory display of elbow motions to control the spatial trajectory of out and back hand movements. We wished to determine whether this sensory input can be used to learn a new inertial configuration of the limb. A major deficit in sensory neuropathy is an inability to detect the beginning and end of motions at joints. This leads to drifts when they attempt to maintain their hand in a steady position and to a lack of smoothness in the initial and final phases of the trajectory. Since muscle receptors encode the velocity of muscle shortening and joint motion quasilinearly over only a narrow range, we decided to map elbow joint velocity on to pitch space only at the beginning and end of movement. The mapping of the elbow joint velocity was performed by the piano. Very fast subdivisions were used (64^th notes) so that individual notes could not be easily distinguished thus encouraging perception of the contour of the piano line. In order to provide information that might be analogous to the dynamic transients likely to be generated by muscle receptors during direction reversals we had the elbow reversal of the subject trigger a chord. If the reversal was within +/- 140ms of the beat marking the ‘correct’ time of reversal (the up beat) then the chord was performed on steel drums. If the reversal was within +/- 70ms of the up beat then the chord was performed by trumpets. (If the reversal was not within 140ms of the up beat no feedback for the reversal was provided). The out and back motion was to be completed over one measure followed by two measures of silence. A faster tempo (432ms per beat) than experiment 1 was used since higher movement speeds produce more pronounced problems for patients with loss of proprioception. The downbeat of each measure was performed by a bass drum. The upbeat of each measure (the reversal beat) was performed by the piati. A regular driving rhythmic subdivision (16^th notes) was provided by the tam-tam. For an schematic description of sound correlation to movement in experiment 2 see Figure 4.

The task given to the patient was to move her hand out and back to a set of visual targets while the center of mass of her forearm was shifted unexpectedly from medial (i.e. left) to lateral (i.e. right). Her hand, wrist and forward were fixed in rigid support and levitated above the work surface using an airjet system to prevent friction. Targets and hand position were displayed in a virtual workspace superimposed on her actual hand using a projector and mirror [20]. A 1kg mass was attached to the support system via an outrigger 8cm medial or lateral to the forearm to alter the center of mass. This perturbs the interaction torques generated at the elbow by movement at the shoulder joint. Displacing the mass from medial to lateral deviates forward movements clockwise and produces large counterclockwise curves at movement reversals. Intact subjects adapt rapidly to these variant inertial configurations: over 30-40 movements the hand paths become increasingly straight and reversals become sharp. When the mass is displaced medially, outward movements are now deviated counterclockwise and movement reversals show large clockwise curvatures. Again, adaptation occurs over 30 to 40 movements. In a previous study in this patient we found that adaptation to analogous changes in inertial configuration is severely impaired whether or not the limb was visible[21].

As previously, the patient was first trained to recognize the note pattern produced by her elbow movements during extension and flexion and to produce simple elbow movements that reversed in time with the up beat. Then she was trained to adapt to medial and lateral mass distributions while moving to 3 different targets located at 10:00 12:00 and 2:00 o’clock from a common starting position. Ten tests with sixty motions per test (twenty per target) were run. A striking finding was that adaptation occurred at what appeared to be the normal rate over a sequence of mass displacements from medial to lateral to medial and back to lateral. After the experiment the patient reported that although her attention was focused on reducing the errors in hand paths in performing the task, she noticed and made use of the sounds to improve her performance. She reported that the most useful cues for her were the down beat, the reversal time cue and reversal time feedback which was totally in agreement with what we had observed during the first experimental run. Although she did not mention anything specific about the melodic contour performed by the piano, she did say that as the experiment was progressing she was getting used to how a ‘correct’ run sounded and she was trying to reproduce that. So although she might not have been able to analytically report on some aspects of the sound (like the piano melodic contour) it is possible that those sounds could have been contributing to her construction of the ‘correct’ sound model for the motion. As discussed earlier, many complex sound structures are perceived as units with multiple, hierarchically related, levels of organization. Some of the higher levels of organization can be perceived and controlled analytically while other levels are handled unconsciously. The possibility that motions are perceived in similar, or at least parallel ways strengthens the role of auditory feedback for motion related rehabilitation especially when auditory display is used to enhance the building of internal representations of motions. The possible parallels between motion and sound processing should allow auditory display to facilitate the building of these representations.

While these results are encouraging and suggest that auditory feedback of joint motions may be substitute for proprioceptive input, additional data will have to be collected and control experiments will have to be done.

Our current plan for the next experiments is to use the same basic paradigm as experiment 2 but to include an auditory mapping of shoulder motions and reversal time. We will concentrate heavily on sound cues and auditory feedback that might assist in a synchronized change of direction for the elbow and shoulder. This is one of the key aspects in achieving controlled out and back hand movements. During these next experiments we will determine whether auditory information can substitute for visual information by comparing performance in normal and neurapathy patients. Our working hypothesis is that when provided with auditory feedback the patient should be able to perform more closely to normal and better than with vision alone. We will then reexamine the rates at which variant inertial configurations are learned with and without audio feedback.

TECHNOLOGY

The data acquisition for positional information from the patient's elbow was accomplished using a potentiomenter being sampled at 14-bit resolution through the A/D converter on a Parallax Basic Stamp II microcontroller. The "Stamp" then took the sampled pot values and sent them out to a host computer as MIDI continuous controller data. The microcontroller was programmed to send the pot position information without any post-converter smoothing or computation; this was done to minimize processing overhead on the chip for an increased sampling rate. Therefore, virtually all computation occurs on the host computer. The host computer, an Apple Powerbook G3, was running customized interface software authored in the Opcode Max environment, using the MSP signal processing extensions developed at Cycling'74 (21) (22). The host computer took the sampled positional data and used it to derive velocity and acceleration data, from which it was able to detect joint reversals. The computer played the musical sequences written for the experiment using a combination of a software sampler written in MSP and a software synthesizer (the HeadSpace Audio Engine). The computer also took reversal data and compared it with the timing in the sequence to determine how closely the patient was reversing to the optimal point in the sequence. The two software interfaces used in the pilot studies were designed so that the computer operator had control over various parameters of the experiment, such as the number of runs (sequences) in each trial, the amount of rest time between each run, the volume and panning of the various musical elements in the sequences, which melody to use in the piano in the first experiment, the overall tempo of the sequences, the pacing of the metronome, etc.

REFERENCES

1. Ghez, C., The control of movement, in Principles of Neural Science, E. Kandel, J. Schwartz, and T. Jessell, Editors. 1991, Elsevier. p. 533-547.

2. Ghez, C., et al., Spatial Representations and Internal Models of Limb Dynamics in Motor Learning, in The Cognitive Neurosciences (2nd Edition), M. Gazzaniga, Editor. 1999, The MIT Press: Cambridge, Massachusetts. p. 501-514.

3. Hollerbach, J.M. and T. Flash, Dynamic interactions between limb segments during planar arm movement. Biological Cybernetics, 1982. 44: p. 67-77.

4. Krakauer, J.W., M.F. Ghilardi, and C. Ghez, Independent learning of internal models for kinematic and dynamic control of reaching. Nature Neuroscience, 1999. 2(11): p. 1026-31.

5. Ghez, C., et al., Roles of proprioceptive input in the programming of arm trajectories. Cold Spring Harbor Symposia on Quantitative Biology, 1990. 55: p. 837-847.

6. Shadmehr, R. and F.A. Mussa-Ivaldi, Adaptive representation of dynamics during learning of a motor task. Journal of Neuroscience, 1994. 14: p. 3208-3224.

7. Sainburg, R.L., C. Ghez, and D. Kalakanis, Intersegmental Dynamics are Controlled by Sequential Anticipatory, Error Correction, and Postural Mechanisms. Journal of Neurophysiology, 1999. 81(3): p. 1045-1056.

8. Ghez, C., J. Gordon, and M.F. Ghilardi, Impairments of reaching movements in patients without proprioception II. Effects of visual information on accuracy. Journal of Neurophysiology, 1995. 73: p. 361-372.

9. Sainburg, R.L., H. Poizner, and C. Ghez, Loss of Proprioception Produces Deficits in Interjoint Coordination. Journal of Neurophysiology, 1993. 70: p. 2136-2147.

10. Sainburg, R.L., et al., The Control of Limb Dynamics in Normal Subjects and Patients without Proprioception. Journal of Neurophysiology, 1995. 73: p. 820-835.

11. Matthews, P.B.C., Muscle spindles: their messages and their fusimotor supply, in Handbook of Physiology: Sec. 1. The Nervous System: Vol. 2. Motor Control, Part 1, V.B. Brooks, Editor. 1981, American Physiological Society: Bethesda, MD. p. 189-228.

12. Houk, J.C. and W.Z. Rymer, Neural control of muscle length and tension, in Handbook of Physiology: Sec. 1. The Nervous System: Vol. 2. Motor Control, Part 1, V.B. Brooks, Editor. 1981, American Physiological Society: Bethesda, MD. p. 257-323.

13. Luschei, E., C. Saslow, and M. Glickstein, Mucle potentials in reaction time. Experimental Neurology, 1967. 18: p. 429-442.

14. Shapley, R. and P. Lennie, Spatial frequency analysis in the visual system. Annual Review of Neuroscience, 1985. 8: p. 547-83.

15. Handel, S., Listening, MIT Press, Cambridge MA, 461-544, 1989

16. Kramer, J., The time of music, Schirmer Books, New York, 1988

17. Lerdahl, F., A Generative Theory of Tonal Music, MIT Press, Cambridge MA, 1985.

18. Tillmann, Barbara; Bigand Emmanuel, Influence of global structure on musical target detection and recognition, International Journal of Pyschology, Vol 33(2), Apr. 1998, 107-122

19. Ghez, C., W. Hening, and M. Favilla, Parallel interacting channels in the initiation and specification of motor response features, in Attention and Performance XIII: Motor Representation and Control, M. Jeannerod, Editor. 1990, Lawrence Erlbaum: Hillsdale, NJ. p. 265-293.

20. Wolpert, D.M., Z. Ghahramani, and M. Jordan, Are arm trajectories planned in kinematic or dynamic coordinates? An adaptation study. Experimental Brain Research, 1995. 103: p. 460-470.

21. Virji-Babul, N., R. Sainburg, and C. Ghez, Roles of vision and proprioception in providing contextual and feedback information for the learning of limb dynamics. In preparation.

22. Puckette, M., Combining Event and Signal Processing in the Max Graphical Programming Environment. Computer Music Journal, 1991. 15(3): p. 41-49.

23. Zicarelli, D., Cycling74 Website. www.cycling74.com, 1997.