Forum rules - please read before posting.

Papagayo lip sync bug and feature request

edited February 2022 in Engine development

I've found out that Papagayo in AC is giving incorrect results.

Char.cs, method ProcessLipSync, line 3624 the timeIndex property is being treated as endingTime while it's meant to be a startingTime. Moho frames indicates the beginning of a lip sync frame. This needs to be changed to the following:

if (lipSyncShapes.Count > 1)
if (Time.time >= lipSyncShapes[1].timeIndex)

Could you please elaborate on Dialog.cs, method GenerateLipSyncShapes, line 714, where are the Papagayo shapes being generated. The "rest" frames are being ignored. This is resulting into playing the previous frame longer, during the time "rest" frame should be played. Why is it there? Could it be removed?

if (!searchText.Contains ("rest"))

Later in the code when the Papagayo lip sync frame from the .dat file is not being found in the AC's phonemes, frame 0 is returned, but only if the frame is not named "etc". This is not causing any problems, but why is it there?

if (!found && !searchText.Contains ("etc"))

Feature request

An option to output the frameNormalisedTime to the animator controller would be amazing. There's currently the option to output the actual frame as an integer, this would be a separate output. I'm using only the normalisedTime in my animator controller. I have a single animation with the speed set to 0 and having the motion time set by a float parameter.



  • Please include your AC version when supplying code line numbers - the lines you're mentioning don't match up with the latest release.

    This needs to be changed to the following

    I would need details on the specific issue you're looking to solve - ideally with a practical example.

    Could you provide a sample package with a typical sound / lipsync file / character that exhibits issues? The code is as it is based on my own testing with things.

    An option to output the frameNormalisedTime to the animator controller would be amazing.

    Sorry - normalized to what? I may be missing the obvious here, but please elaborate on what you're looking to make use of here.

  • I'm sending you an example package to DM. Specific information describing the lip sync issues is inside an included readme.txt file inside the package.

    Sorry - normalized to what? I may be missing the obvious here, but please elaborate on what you're looking to make use of here.

    Sorry I was just trying to keep this post short. This is the AC method I took the naming from.

    If using the Unity Sprite Complex, the lip sync frame number is being outputed to the animator controller. What I mean is the animation motion time that is representing the section where the specific frame is being displayed.

    (1f / frameCount) * desiredFrame

    One option would be to have an animation for each frame and then transition between them based on the integer value, the other option would be having a single animation with speed set to 0 and manipulate the animation motion time. I prefer the second option, but it requires to convert the integer frame number to the float motion time inside an update method. Outputing the value the same way the integer number is if a controller parameter name would be set would make this easier.

  • Thank you, I will look through the package.

    Regarding the motion time: are you referring to lipysnc frames (i.e. the number of phonemes listed in the Speech Manager), or the number of frames in the lipsync file?

    The former is what GetLipSyncNormalised refers to - as this can then be used to have a single animation clip that contains all phonemes in order, and then play that animation in "normalized time" based on the current phonene. So if the current phoneme is 1, or 5 possible phonemes, the animation will be played 20% along its length.

    If you're referring to the latter, I'm not clear on how that would be useful. I don't doubt your intent, but a conversion through a custom script's Update function would likely be the best approach.

  • I'm referring to the phonemes listed in the speech manager. Exactly what you're describing. One animation have all these phoneme frames in order. I believe this "normalized time" would be useful in the Unity Sprite Complex animator controller, as it's more simple than controlling the lip sync with the integer value. It's not a problem to convert the value inside a custom script, I'm only suggesting it could come handy.

  • Understood, thanks for clarifying.

    I'll give it some thought - it's true, this is only available for Sprites Unity currently.

    If a custom script was used, however, it wouldn't be a case of converting any integer - you can apply GetLipSyncNormalised to a float property directly:

    public AC.Char character;
    public Animator _animator;
    void Update ()
        _animator.SetFloat ("MyParam", character.GetLipSyncNormalised ());
  • I've been through the package you've sent - thanks for the clear demo and explanation.

    You're correct about the changes necessary for your setup - though the exact change will need to account for other lipsyncing options. I'll make the necessary changes, however.

    Your alternative calculation of GetLipSyncNormalised is also correct - I was under the impression that a value of 1 would equate to Unity playing the last frame of an animation, but it seems that this instead causes it to loop back and instead play the first.

  • Glad the package was of help.
    Thanks Chris, appreciate it.

Sign In or Register to comment.

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Welcome to the official forum for Adventure Creator.