Tricky for Testers Who Don't Read the Manual
October 12, 2019
Tricky for Testers Who Don't Read the Manual

Score 6 out of 10
Vetted Review
Verified User
Overall Satisfaction with Optimizely
We use Optimizely mainly within our development group, though the framework is available to all departments. Its main function is to enable us to show/hide features to subsets of users in the field, and make those changes via server-side controls. We're also starting to use it for cohort testing, though we're still in the early stages of trying this.
Pros
- Developers seem to be able to set up experiments fairly easily.
- The dashboard allows users to control settings for individual devices for testing purposes.
- Functional on both iOS and Android.
Cons
- At least in our implementation, there didn't seem to be a great way to scale the number of individual testers who are able to individually control experiment settings.
- Differences in the experiment stage weren't terribly intuitive to figure out.
- When we first started using it, there was no support for booleans to define the audience. If we wanted to enable an experiment for anyone on version N or higher, we couldn't just say "if version >= N", but had to add each subsequent version individually. I believe this has been addressed in the new version.
- The development has had additional flexibility because of the ability to dynamically enable or disable the feature on the fly.
- We've been able to change our risk calculations, and release features more aggressively than before because of the ability to ramp up exposure to ensure no problems or revert if a problem is discovered after release.
I wasn't involved in evaluating any other third-party tools of this type.
Comments
Please log in to join the conversation