Rework of the Mutator system

[regzo2]

Overview

This draft entails a rewrite of VRCFaceTracking's Mutator system, to better expose the Mutator to the end user and let users configure each individual Mutation. This also genericizes the Mutator system to allow more explicitly defined Mutations to be created without needing to touch the Mutator class itself. This potentially opens the doors for modular implementations to the Mutator system, such as a Tracking Module adding new Mutations, or even allowing native modifiers to tracking. Here is a overview of the additions/changes that come with the rework:

Note: This is not the finalized styling or feature set

Additions

• TrackingMutation
  • Main interface that implements a Mutator layer
  • Contains several features to facilitate correct usage of the Mutator
  • MutationProperty
    • Property that contains a key-value, and is exposed to the UI that allows users to control the parameter in a defined way.
• UI
  • Mutator tab
    • Contains all loaded Mutations, and allows users to toggle them on/off
    • All properties are shown here

Goals

Currently, the functionality of the new Mutator system and the implementation into the UI is mostly complete, there are several things that need to be done to finalize. Here are several things I want to achieve before finalizing this PR:

• Genericize the MutationProperty into something like VRCFTProperty to be useable by other portions of the app in the future (modules, settings, etc) to allow more direct user input to developer-implemented interfaces like modules.
• Vastly improve the default Calibration, Smoothing, and Corrector features (at this time of writing, I have not changed the functionality of these features but plan to)
• Finalize styling to better fit within the rest of the app.

Conclusion

If there are any suggestions or changes that need to be made feel free to comment about anything! This is still very WIP with the functionality currently working.

[hyblocker]

I think this makes sense to wait for sandboxing to be finalised first. It'll require a reimplementation of the core of this once that gets merged and the sandboxing core is nearing competition now.

[regzo2]

I think this makes sense to wait for sandboxing to be finalised first. It'll require a reimplementation of the core of this once that gets merged and the sandboxing core is nearing competition now.

Yeah that's what I was thinking, I want to avoid as many unnecessary conflicts as possible 😅. My goal for this was originally to rebuild the Mutator system and expose it better to the UI but I realized how much more useful it would be if parts of VRCFT Core could implement to the UI in a more robust way. Currently all of the features are just designed for the Mutator system but I could see it being genericized and being used for modules in the future for example.

[regzo2]

Progress

The styling is much more in-line with the rest of the app so far, I drastically improved the calibration and (dormant) smoothing and added a new default Mutation. The dynamic property system is still WIP but it is very usable. Eventually as well when the module system gets genericized this system will allow third-party developers to dynamically and easily implement Mutations that sit in-between the Tracking modules and exported tracking data. I am still planning out how exactly new Mutations that might offer features akin to the default ones be able to replace or swap them out (to keep the UI as uncluttered as possible for users).

There are several more Mutations I would like to implement but for now I will have these as defaults.

Here's a quick video that goes over the features!

2024-08-20.20-55-47.mp4

Overview

Component System

The UI component system still needs some TLC. Currently parameters updated by the mutator itself don't propogate back to the Property version in the UI. The component system is also tailored to the Mutator at the moment but I would like to genericize it so that it's usable by any class.

Calibrator

For the calibrator, it still does the janky messed up face (for some reason) but I should be able to weigh in the new Confidence stat that each parameter gets processed with now. I also weighed in the calibrator to use a data set to drastically improve the consistency of the parameters, and the new Delta stat should allow parameters some wiggle room to avoid including noisy data (and let users intentionally calibrate their faces better). The new calibrator is still very WIP and I am thinking of bringing in other stats such as a calculated half-life value which should decay unconfident values and push users to actively calibrate the expressions instead of having to do a time-lapsed face exercise every time. Currently the calibrator is hands-off enough but I hope the introduction of the half-life stat will allow the calibrator to become completely autonomous without relying on any time-out system. I also exposed some debug-specific adjustments but I will probably expose these under a proposed Advanced Mutator Options setting in the future.

Data Filter (replaces dormant Smoothing feature)

The new filter Mutation implements the One-Euro Filter, which has been proven to be an amazingly effective data filtering algorithm. I think it really serves well for face tracking, and currently the default values implemented into this mutator are intended to clean up data as opposed to overly smoothing anything. It makes a lot of the face tracking look much more refined than without it. The parameters are exposed for user adjustment, but I think for the purposes of cleaning up noisy or choppy data it works extremely effectively. The original Smoothing Mutation was kept mostly hidden or outright disabled in some versions of VRCFaceTracking because we wanted to avoid overly smoothed and muddied-up face tracking. I think this current data filtering system is much more in-line with what we want as part of the app.

Parameter Adjustments

The Parameter Adjuster Mutation does exactly what it sounds like. Every Unified Expression parameter's ranges can be immediately adjusted to the user's preferences using a bunch of range adjusters. Currently it might be a bit overwhelming for casual users so in the future it might be simplified to common expression groups (so all the Brow expressions will be unified under a Brow range adjuster for example). The algorithm is what used to be in the old Calibrator Mutation:

[regzo2]

Since I have implemented a bare-bones framework that loads components from Core directly into the UI and plan to genericize it anyways, I want to extend this to be genericized for dynamically-loaded implementations like third-party tracking modules and Mutations

[regzo2]

Reduced scope of PR, going to focus on making the Mutator system as good as possible. the UI stuff will be another PR when the module sandboxing and this PR complete.

[regzo2]

I think it's in a good enough state now for merging. Have been using it for the past month in it's current state with no issues.

[benaclejames]

Looks good, though I feel as if the UI should be in settings as a section as opposed to having a whole new tab? That being said, there's definitely space for more tabs on the left should we feel them necessary.

If it is the case that you're sure it's better in the sidebar, would it be possible to move it toward the bottom of the list? Ideally, we want the tabs to be in functional order from what's happening right this very moment at the top, to settings and customization at the bottom.

[regzo2]

Looks good, though I feel as if the UI should be in settings as a section as opposed to having a whole new tab? That being said, there's definitely space for more tabs on the left should we feel them necessary.

If it is the case that you're sure it's better in the sidebar, would it be possible to move it toward the bottom of the list? Ideally, we want the tabs to be in functional order from what's happening right this very moment at the top, to settings and customization at the bottom.

I felt that having it as it's own tab would make it more prominent as a feature (I did mess with having it in the settings card but it cluttered it up too much with the new stuff added to vs how it was originally). There is a lot of real-estate for the tabs, though I do worry that opening it up to expansion might make the user experience a lot more involved.

I think at the most it will be this and the UI changes that #214 brings, and I also want to eventually open it up to user-generated features if that's something we want to move towards. And yeah it can be moved to the bottom of the tabs stack since it's not a necessary feature to get VRCFT running.

[regzo2]

With that being said there could be sections in the navigation tabs that separates the necessary tabs vs customization tabs vs settings, since there's already a minor settings separation from the rest of the tabs in the current builds. Currently there isn't much need to tinker with the new settings frequently, it's mostly a done-and-forget addition so it's not that necessary in the tab. Pretty much every person that I've had try it didn't really complain about having another tab so I think we are good 😁. We will have to add localization though 😎.