The Virtual Rockstar Project

The results of this work can be found here.

I've put together a system that reads in guitar tablature and uses the fingering information to move end-effectors attached to a constrained IK model of a hand. This gives fairly bad results and motivates the rest of this research.

Download AVI [2,352k]

Research is progressing along. We decided to concentrate on creating a muscle based model for the hand. This model should be anatomically based and should be applicable to general hand animation (i.e. not restricted to our guitar model). To that end I've compiled a useful map of the muscles affecting the motion of the human hand. It can be found here.

Well, classes are over and course projects are all but out of the way. So it's time to get back to research full time. We have a tentative research schedule.

End of June: Have a system up and going where we can animate finger joints based on musical input.

End of September: Have a fairly realistic heuristic model for fingers moving across a guitar fret board based on musical input.

End of Decemeber: Have a written up Master's thesis.

Long term and possible Ph.D. goals: Set up a video capturing system of real guitar players and gather a large data set of finger joint angles. This data can be acquired from video by having a proper setup and using some vision techniques. Then, use the data and machine learning techniques to train the system to play guitar. The final result should be very realistic looking motion.

We met with two extermely nice ladies today from the Anatomy department at the university, and they kindly took the time to explain how a real human hand works. Much to my dismay, we were greeted with rubber gloves and told to "follow me". We were assured that all the cadavres are donated specifically to the University of Toronto with the full consent of their now departed owners. That, I'm sure, was meant to assuage whatever moral objection we may have had. I ashamedly admit that moral objections were not, alone, the cause of my trepidation.

Nonetheless, we were presented with three upper-limbs that were previously attached to someone. Our kind instructors went on to explain how the human hand functions, switching back and forth between the limbs that most adequately demonstrated the layer of discussion.

I was surprised to learn that there is actually very little known about the movement of the hands other than what is affected by the muscles. For example, there are flat, ribbon-shaped tendons called aponeuroses that certainly affect the constraints of motion of the hand, but of which little is known (at least from the stand-point of motion-constraints.)

We learned much, of note though, is the fact that muscle bellies branch out to several tendons, and tendons each have exactly one root on a finger bone. This close connection between the finger bones to tendons sharing the same muscle bellies seems to be an important factor in the dependence of finger joint movements on one another. Also, there are flexor tendons and extensor tendons that attach to the hand. The flexor tendons flex the hand at the wrist, while the extensor tendons allow fingers to extend. This is a very rough outline, but serves as a starting point for further research. (Research that should no longer require hands-on demonstrations. (Please forgive pun)).

All in all, it is amazing how much one can absorb eventhough the vast majority of the effort was expended on trying to keep lunch from making an unwanted appearance. Were it not for the friendliness of our wonderful instructors, said task would have been more challenging.

Given a musical score and a 3D model of a musician, we would like to be able to automatically generate an animation of the musician playing the score. We would like to do it in such a way that the virtual musician is believable, with convincing hand movements.

• Kiosks at record stores that have virtual musicians playing music.
• Educational: teaching people to play instruments.
• Animation

The controller is the "brain". It knows how to play the instrument and the capabilities of the hand. The music is streamed into the controller and the controller sends the appropriate signals to the hand.

Issues

• a study into the anatomy of the human hand; the musculature, nervous system and structure of linked multi-appendage structures.
• a model that captures how movement of fingers affect movement of other fingers in particular (that can be generalized to arbitrary linked structures).
• reaching algorithms using this model and such structures. issues such as relative strength of fingers, using fingers in tandem; in sequence to minimize things like movement of the palm, fingers getting in the way of each other, speed of movement etc.
• collaboration between fingers, grasping.

Specifics

• a virtual guitar player. the ability to translate automatically tablature/ a treble clef score into kinematic motion.
• case study examples...
• slides, bends, double bends, hammer ons, pull offs, ideal candidates for the issues to be addressed above.
• chords: eg. you are likely to play a C differently if preceeded by an Am than if preceeded by a G.
• an auto guitar simulation other than being of animation value could be a good instruction tool from a musical standpoint, allowing the exploration of alternate tunings and chord configurations.
• should be interesting to apply the technique to other instrument genres like the piano or trumpet.

• Anatomy of the Hand
• Humanoid Hand
• Virtual Hand Laboratory
• Developing a Computation Model of Human Hand Kinetics...
• Physical Modeling of Plucked String Instruments