How does automatic lip-sync work without AI?

StoryToon moves each mouth from the voice itself, and every plan gets the same lip-sync. On every frame it picks the mouth opening from the loudness of the audio and the shape from the vowels in the text, choosing between 8 drawn mouth shapes.

Cartoon mouths do not need a shape for every sound. Animators group sounds into a few visemes, and software can pick them from signals it already has, without any machine learning.

Updated

Lip-sync on every plan

  1. Measure loudness

    The voice audio is measured every 10 milliseconds.

  2. Open the mouth

    Louder moments open the mouth wider; silence closes it.

  3. Shape it

    Vowels in the line choose round (O, U) or wide (A, E) shapes, and letters such as m, b and p briefly close the lips.

Why it needs no AI

Everything comes from two things StoryToon already has: the sound of the line and its text. The letters are spread over the length of the line with a simple timing rule, so no speech recognition and no machine-learning model is needed.

With your own recorded voice

Lip-sync also follows recordings made with your microphone, so a recorded line animates the mouth just like a computer voice. Because the mouth opening only needs the loudness of the sound, it works for any language and any accent.

The mouth shapes

On every frame lip-sync picks one of these drawings: quiet moments return to rest, letters such as m, b, p, f, v and th choose the matching shape, and the vowels in the line choose A/I, E, O or U.

Mouth shapeSounds it shows
RestSilence, pauses between words
A/IOpen vowels, as in "cat" and "bite"
EAs in "bed" and "see"
ORounded, as in "go"
USmall and rounded, as in "you"
M/B/PLips pressed together
F/VTop teeth on the lower lip
L/THTongue visible behind the teeth

Frequently asked questions

How many mouth shapes does StoryToon use?

8, including the closed rest shape.

Does lip-sync work in Urdu or Arabic?

Yes. Lip-sync follows the loudness of any audio, in any language, including your own recordings.

Does the rest of StoryToon need AI?

No. Everything works without AI. StoryToon has optional AI features, off by default, that can write or improve a script with your own key, but lip-sync never uses them.

← Answers