Automated Lip Synch

[Sarah Thompson]

I'm about to start an animation job for a multimedia project. It requires
about 10 minutes of lip synch animation. The previous CD ROM used cel
animation which was hand drawn (not by me), whereas I've been asked to do
the new version using CGI. The animation itself is no problem, nor is bar
charting for lip synch. However, 10 minutes of audio is a lot of work. There
are a few Lightwave plugins which purport to take a WAV/AIFF/whatever file
and automatically generate lip shape information. I'm not so keen on using
one because the cost involved would eat up most of the project's budget
(mostly because I'd have to upgrade to Lightwave 6.0 and spend time learning
its new interface). So, what I fancy doing, as much out of interest as
anything else, is putting together an automatic lip shape generation tool
from scratch that can generate an object replacement script for Lightwave.

The basic idea is that the audio would be sliced into 1/n th second
intervals, where 'n' is the frame rate (12 fps for this project, but it
would normally be faster for 'real' animation to video or film). Each of
those slices needs to be recognised to derive a lip shape. Many animators
work with six basic lip shapes: closed, slightly open, more open, fully
open, 'w' and 'o'. Between these, you can produce pretty convincing results
if you get the timing right. It is possible to go further and use specific
shapes for 'th' and so on, but this puts you into diminishing returns
territory. Some manual cleanup of the output is inevitable, so I'm not
worried about needing to do that.

I'm sure I can come up with an ad-hoc approach that will work reasonably
well, but it would be neat if there is a published algorithm, perhaps aimed
at speech recognition, that could make a reasonable stab at phoneme
extraction. Any ideas?

--
Sarah Thompson
CEO, TSI Ltd.


dupswapdrop -- the music-dsp mailing list and website: subscription info,
FAQ, source code archive, list archive, book reviews, dsp links
http://shoko.calarts.edu/musicdsp/