Optimizely Feature Experimentation Review
Use Cases and Deployment Scope
We utilize feature experimentation for our web experiments and our app experiments. Currently we have iOS and Android on there, and whenever we make any copy changes or new product features are rolled out, then we use it for our experimentation it addresses. It addresses the fact that we have this usually quite company wide and so having other users and other stakeholders that can access the charts to see the primary metrics, how they're performing secondary metrics is really helpful. And we also use the multi-arm bandit, for instance, whenever we're doing tester and peak periods like Mother's Day because we're a flower company. It addresses quite a few problems in that sense. It allows everyone to be unified with the experiments.
Pros
- I guess there are multi-on band, it's really helpful during peak periods. We also being able to feel flexible with the different metrics that we have. So probably not really meant to all the time, but sometimes we'll have a look at different primary metrics to optimize our new products that are out.
- It helps to bring us to statistical significance very accurately and quickly and we can trust and rely also on the results that it comes up with.
Cons
- I would probably say with apps, some of the things that we find a bit difficult is we obviously different app versions and so when we maybe start an experiment in one app version, it may be that they've had to do a bug fix or something like that, and so we end up having to roll out other versions during the experiment. I think really being able to just manage different versions and different user experiences on those versions would be really helpful.
- From an app's perspective, it's very difficult to be fair. Whilst with, I guess other features, more generically speaking, I think it'd be the results side of things. We'd really love to have that exported out into a way that we can have an overview of the experiments that we've run and the results and being able to understand our win rates and those sorts of things. I think it's very difficult to have a nice overview of the results and insights as well. Just having a single space for us to look at our insights is where we're struggling, I guess.
