Forum rules - please read before posting.

Wishlist: Extended "Character: Move to point" and thoughts on LipSyncing

In the Hotspot inspector you have this fantastic feature "Walk To Marker" with the option "Face after moving?" on the interactions which is really awesome and I use this a lot. When I do such a thing in an ActionList I have to use two Actions to do the same: "Character: Move to point" and "Character: Face object" with the option "Use object's rotation", both with the same Marker. Would it be possible to integrate the option "Copy rotation" into the "Character: Walk to point"-Action, because that would make the following "Character: Face object"-Action unnecessary? This is not a biggy but would enhance the workflow a lot.

Then I would like to share some thoughts on lip syncing:
I bought Rogo LipSync long time ago because I thought it was a good way to get automatic lip syncing done in a short time for the thousands of speech line in our game but I stopped using it because of some reasons: First- and that's the main reason - it only works with english speech samples. I hoped that handling of other languages than english would be supported in the future (which even was advertised by the developer) but it never happened and in the meanwhile I'm quite sure it never will. Then there are some other small issues with LipSync that make it no fun to work with.
I like the lip syncing features that are implemented in AC a lot. The system is very easy to use, is rock steady and works well with different languages. The possibility to have tokens in the speech text is great and very useful. The only thing that it is missing is handling of timing or so to say syncing the text to the played speech sample. What I mean is words that are for example spoken slow and long or short.
What I did in my own (and deprecated because of AC;)) adventure engine was, that I had this little editor with the waveform of the sample in it and with the spoken text written above it and was able to add timestamps into the speech text and connect this to a point in the waveform. This way the engine calculated the time between two timestamps to alter the lipsyncing speed between the two. So basically it changed the speed of the lip syncing (or processing of the text) between two timestamps. If we would have such an editor in AC I think it would make all other lip syncing solutions unnecessary. It would be fully integrated and perfect the already great feature set. To integrate such a feature into the current AC speech system I believe the timestamps could be added into the speech text using tokens, maybe a token like "[stamp:3550]" to mark that this part in the text should end at 3550ms play time of the played sound sample, which is what I did in my engine. I tried to make a screen shot of ths feature from my deprecated engine but it seems that it is not working anymore. Maybe because of Windows 10, that I'm using meanwhile or something with the Ogre graphics engine I was using. So I made up this little fake editor screenshot in Paint.net:
image
That feature was both simple and effective. Lemme hear what you think about it.

Mac

Comments

  • Re: Action extension

    There's a lot of such cases where certain Action features would be useful put together, but this is a good reason why ActionList parameters are so useful.

    Make a new ActionList asset, with both Character: Move to point and Character: Face after moving Actions.  Then define a new GameObject parameter, and assign that in both Actions as the object to move/face to.

    Name this asset "Turn and face".  Then, use this asset instead of new instances of those Actions each time you want the player to move.  When you use the ActionList: Run Action to run the asset, you'll be able to set the Marker object as the GameObject parameter's value, and this will be used by both of the Actions within the asset.


    Re: Lipsyncing

    I appreciate the kind words about AC's own feature set, but I do think that something as complex as lipsyncing will always be most refined in a dedicated asset.  Most of AC's built-in options rely on having a separate text asset with phonemes and frame times already built in (using e.g. Papagayo, SAPI or Pamela).  The only instance where "scaling" the processing speed would have an effect would be with the From Speech Text option, which is designed more for animation variety in games without audio, rather than a viable option for "true" lipsyncing.

    While I see the benefit of adding markers to a waveform, I think adding tokens to the text itself isn't as practical for official inclusion.  However, if you've already made the feature in your own engine, I'd be interested in introducing "custom" lipsyncing methods so that you're free to use whatever works for you.

    If Custom Script was an option in the Lip syncing dropdown, it would probably involve the user defining a method that AC calls when a speech line is played, and expects a series of phonemes as a 2D array ([phoneme, time]).  AC already has a struct for this in the form of LipSyncShape:

    LipSyncShape[] GetCustomLipSync (int lineID, string speechText, int languageIndex, AudioClip audioClip)

    How does that sound?  Would you need different parameters passed to make use of it?
  • Re: Action extension

    You're absolutely right. It's easy to "over-parametrize" Actions and that is a good example of using ActionLists with a parameter. I don't know why it didn't came up to my mind.

    Re: Lipsyncing

    Adding a lot of tokens to the speech text if altered in such an editor I suggested would not make it more readable, that's a good point. I filtered it in my engine or to be precise you were able to blend the markers/tokens in and out. Since I don't like any of the mentioned tools (or maybe I'm just to dumb to use them properly...) maybe I wrap up a new tool and use one of the mentioned file formats to have it integrated with AC easily. But since I'm not that Unity guru guy I would do it in a different developement environment.
    But implementing a Custom Script option to the Lip syncing isn't a bad idea I think. It would enable users to implement their own way of using the system and maybe come up with a cool new Lip syncing solution that we all do not think of right now. If AC already has an underlying system for it, it maybe is not that big amount of work to implement it (don't hit me if I'm wrong ;)).
  • Well, it's not in there now - but I agree that it would be useful for people to have that choice.

    I can implement such a system, but I will need to know what your requirements are regarding the parameters you'd need an overriding function to receive.  To my mind, the line ID, speech text, current language, and associated audio clip would be enough - what do you think?
  • Ad hoc, I don't know what parameters would else be needed.
  • Update me when you know and I'll look into it.
Sign In or Register to comment.

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Welcome to the official forum for Adventure Creator.