LG-14189: A/B test for allowing F/T setup and authentication on desktop #11347
Conversation
jmdembe
force-pushed
the
jd-LG-14189-ft-setup-desktop
branch
from
cdba74a
to
6e5027e
Compare
jmdembe
force-pushed
the
jd-LG-14189-ft-setup-desktop
branch
from
6e5027e
to
8adbf86
Compare
|
can you rebase this branch to pick up changes to the reviewapp deploy process? |
…ercentage to sign up screen
jmdembe
changed the title
app/javascript/packages/webauthn/webauthn-input-element.spec.ts
Outdated
Show resolved
Hide resolved
app/javascript/packages/webauthn/webauthn-input-element.spec.ts
Outdated
Show resolved
Hide resolved
config/initializers/ab_tests.rb
Outdated
| should_log: [ | ||
| 'WebAuthn Setup Visited', | ||
| 'Multi-Factor Authentication Setup', | ||
| ].to_set, |
There was a problem hiding this comment.
The validation metrics for this test require us to measure how many people successfully set up WebAuthn. I think we'll need to revise this list of events (e.g. "submitted" event rather than "visited").
It'd be a good idea to mock up the CloudWatch queries we expect to run to perform the metrics validation.
| @@ -36,4 +41,12 @@ def initial_hidden_tag_options | |||
| { class: 'js' } | |||
| end | |||
| end | |||
|
|
|||
| def show_desktop_ft_unlock_option? | |||
| if desktop_ft_unlock_option? && I18n.locale == :en | |||
There was a problem hiding this comment.
I think we need to explicitly check whether the bucket is the :desktop_ft_unlock_option_shown bucket somewhere. Otherwise, I'd be concerned that the default bucket :default is "present", and we'll be showing the option for 100% of desktop users.
This might be easy to miss in development since we're configuring the A/B test at 100% anyways.
Co-authored-by: Andrew Duthie <1779930+aduth@users.noreply.github.com>
Co-authored-by: Andrew Duthie <1779930+aduth@users.noreply.github.com>
🎫 Ticket
Link to the relevant ticket:
LG-14189-Implement: A/B test for allowing F/T setup and authentication on desktop
🛠 Summary of changes
This PR implements an A/B test for setting up F/T unlock on desktops. This test will only run for users who are signing up for authentication methods in English.
📜 Testing Plan
Prerequisites:
In
application.yml, setdesktop_ft_unlock_setup_option_percent_testedconfig value to100in developmentOptional: set
show_unsupported_passkey_platform_authentication_setupto true. This will make it so that when testing by switching to another language, the option does not appear at all.Once on http://localhost:3000, register for a new account
Follow steps to create a new account
Once on the authentication methods setup page, if you set desktop_ft_unlock_setup_option_percent_tested to 100, you will see the F/T unlock setup option
Switch to all other languages, you should not be able to see the setup option
Notes:
desktop_ft_unlock_setup_option_percent_testedto 0 andshow_unsupported_passkey_platform_authentication_setupto false. When following the testing instructions, you will not see the option at all.show_unsupported_passkey_platform_authentication_setupis set totruewhile testing, you will see the F/T unlock option in Spanish, French and Chinese shaded in red. This is not a regression as that config value is set to false in production, so it will not appear as such.