Spine and lipsyncing

klarabara

Hi,
I'm using Spine to animate my characters and I have created several mouth shapes and want to use the lipsyncing feature for talking animations.
I saw an old discussion about this. I think it was from 2014 so I am wondering if there are any new features that would make this possible and how I should go about it. Any input would be helpful!

Might be useful to know that I am currently developing on a mac.

klarabara

I was referring to this post from 2016 https://www.adventurecreator.org/forum/discussion/4753/custom-lipsync

ChrisIceBox

Are you using a custom integration script? There's a couple on the wiki.

With AC's built-in animation engines, you can incorporate a "Phoneme integer" parameter into your Animator, which will be updated to the character's current speaking phoneme. But if you're using a custom animation engine, you'll need to incorporate it into it, by reading the character script's GetLipSyncFrame function.

klarabara

Yes it's a custom integration that uses the Sprites Unity animation. Meaning I'm using the AC animation logic and then I have assigned a script to every state that points to a skeletonanimation:

    public class SpineAnimationScript : StateMachineBehaviour {
        public string animationName;
        public float speed;
        public bool flipX;

        public override void OnStateEnter(Animator animator, AnimatorStateInfo stateInfo, int layerIndex) {
            SkeletonAnimation anim = animator.GetComponent<SkeletonAnimation>();
            anim.Skeleton.ScaleX = flipX ? -1f : 1f;
            anim.state.SetAnimation(0, animationName, true).TimeScale = speed;


        }
    }

ChrisIceBox

So your character's "Animation engine" is Sprites Unity?

Lipsync is handled automatically by that engine. It works by playing the talk animation at zero speed, mapping the phoneme index to normalized time.

For example, if you have 5 phonemes, and your talk animation has 5 frames, then phoneme "2" will cause the animation's 2nd frame to be played, etc.

You can try it with the 2D Demo's Brain2D Player prefab for an example.

If it's not compatible with Spine, you can try the Phoneme parameter approach I mentioned above. Switching your character's Animation engine to Sprites Unity Complex would allow for that. Again, you can find an example with Brain2D_SpritesUnityComplex.

klarabara

Yes Sprites Unity.

Aha so maybe it could work setting them up as frames in a Spine animation too. Didn't know if Unity would read as frames in the exact same manner.

I was however able to use the phoneme parameter together with the GetSpriteDirection (We lipsync in 3 of 6 directions for now). Still facing some timing issues though since it's voiced, it looks a bit off together with the voice. Got any tips there?

I'll give the other one a go too and let you know how that works out.

ChrisIceBox

Got any tips there?

What method are you using to set phoneme frames, and what exactly is the timing issue? The processing speed can be adjusted in the Speech Manager.

klarabara

Right now I'm using only from Speech text and the result together with speed is alright. I'm setting a speed of .8 and I have 10 phonemes.
The timing issue is that sometimes the sound is slower/faster than the text which is understandable since I'm using only from speech text.

I was just wondering if you had any input about options to Pamela, Papagayo etc that might make it look super smooth and also works with a Mac. If not I'll guess I have to prep that on another computer or tweak what I have. Will more phonemes make any difference?

ChrisIceBox

Generating from speech text will always come off as an approximation, no matter how many phonemes you provide.

For smoother lipsyncing, my personal choice is to use LipSync Pro on the Asset Store, but Papagayo is a free OSX alternative that AC also supports.

klarabara

Thanks for the tips! With some more tweaking its looking rather good now but I'll probably dive into LipSync Pro or Pamela. Cheers!