Item: Optimizely Feature Experimentation
Rating: 9
Author: Marion Brooks-Bartlett

Overall Satisfaction with Optimizely Feature Experimentation

Use Cases and Deployment Scope

We utilize feature experimentation for our web experiments and our app experiments. Currently we have iOS and Android on there, and whenever we make any copy changes or new product features are rolled out, then we use it for our experimentation it addresses. It addresses the fact that we have this usually quite company wide and so having other users and other stakeholders that can access the charts to see the primary metrics, how they're performing secondary metrics is really helpful. And we also use the multi-arm bandit, for instance, whenever we're doing tester and peak periods like Mother's Day because we're a flower company. It addresses quite a few problems in that sense. It allows everyone to be unified with the experiments.

Pros and Cons

Pros

I guess there are multi-on band, it's really helpful during peak periods. We also being able to feel flexible with the different metrics that we have. So probably not really meant to all the time, but sometimes we'll have a look at different primary metrics to optimize our new products that are out.
It helps to bring us to statistical significance very accurately and quickly and we can trust and rely also on the results that it comes up with.

Cons

I would probably say with apps, some of the things that we find a bit difficult is we obviously different app versions and so when we maybe start an experiment in one app version, it may be that they've had to do a bug fix or something like that, and so we end up having to roll out other versions during the experiment. I think really being able to just manage different versions and different user experiences on those versions would be really helpful.
From an app's perspective, it's very difficult to be fair. Whilst with, I guess other features, more generically speaking, I think it'd be the results side of things. We'd really love to have that exported out into a way that we can have an overview of the experiments that we've run and the results and being able to understand our win rates and those sorts of things. I think it's very difficult to have a nice overview of the results and insights as well. Just having a single space for us to look at our insights is where we're struggling, I guess.

Usability

Key Insights

Do you think Optimizely Feature Experimentation delivers good value for the price?

Yes

Are you happy with Optimizely Feature Experimentation's feature set?

Yes

Did Optimizely Feature Experimentation live up to sales and marketing promises?

Yes

Did implementation of Optimizely Feature Experimentation go as expected?

Would you buy Optimizely Feature Experimentation again?

Yes

Likelihood to Recommend

I think it's good. Like I said, I think it's very good at unifying teams and everybody being on the same page. I think it's also, I love just seeing the charts and being able to see how a metric is performing throughout, so that's just super helpful for us, especially when we were trying to make quick decisions. Sometimes for some markets it's very difficult to reach significance or depending on the type of feature that you're rolling out or trying to experiment, it's quite difficult and so you want to see how is the metrics performing over time, so it's really helpful

Comments

Please log in to join the conversation

Optimizely Feature Experimentation Review