Get on the Sauce! Sauce Labs will make your automation life A LOT easier!
Updated July 25, 2018
Get on the Sauce! Sauce Labs will make your automation life A LOT easier!
Score 10 out of 10
Overall Satisfaction with Sauce Labs
We use Sauce Labs when we're running our regression test suite against a multitude of OS and browser combinations for our core web product. We are able to easily spin up virtual machines that contain a specific OS along with either the latest, beta or older version of the browsers we support with our app. This saves us a lot of maintenance and support issues with our test environments.
- Greatest strength of Sauce Labs to me is the ease of spinning up a clean virtual machine with the desired OS and browser combination.
- Sauce Labs support has been great! When I've run into issues whether it's from my own code issue or something on their side, they have been very responsive and helpful.
- The ability to scale up or down our needs from Sauce Labs has been great. As we build out more automation, we have a larger need of Sauce Labs services and it's been so simple to do. Need more parallel VMs? No problem. Need more automation time each month? No problem. It's all done almost instantly.
- My main area of concern that could be improved is the cost of the real mobile device farm offered. I have tried it out and it's great! I want to use it for most of my mobile automation, but can't justify the cost.....yet. Once more and more tests are created on my side along with a need to cover more devices then maybe it will work.
- A minor area that could be improved would be the responsiveness of the video stream of active tests. Sometimes I need to refresh the browser to get the video to come on or update.
- Another area of improvement would be the speed that tests can run. This has improved since I first started using Sauce Labs, but don't we all want everything, right now, immediately? I'm asking for the world, but you asked!
- Accelerate the pace of new feature releases
- Eliminate testing as a bottleneck
- Improve quality by increasing browser/OS/device coverage
- Achieve continuous delivery
A big area that would slow us down would be during regression testing and Sauce Labs has definitely helped us in this area. We are now able to run many parallel tests to get through our testing much faster than if we were to try to maintain our own environments. The constant setting up and breaking down tests and having the right configurations would have been much slower if we tried to use and maintain our own environment.
We have been able to test on different OS and browser combinations that we were not able to before. Yes, we could have eventually built virtual machine's with what we needed, but we didn't have the resources in place when we needed it and Sauce Labs gave us a quick way to get this started and still use today. This has given us better quality and definitely made us more efficient which in turn gets us to market quicker. It's a win all around!
A new, clean virtual machine is spun up each time for every test. You cannot access a previously used virtual machine. Any test remnants or access behind our firewall is strictly guarded within each virtual machine and the use of a secured tunnel. When testing mobile apps, the apps are only kept in storage for a minimal amount of time and then deleted. Access is guarded with our Sauce Labs credentials, as well.
Our product is supported on a number of OS and browser combinations and the ability to have clean working test environments is critical to our success. Sauce Labs has given us the ability to spin up a great deal of different combinations and have them ready within seconds. This frees up our time to spend on more useful tasks such as testing.
A company can build out the infrastructure to house a test grid and have it be fairly straightforward. The issue comes in when you need to keep that test grid maintained. When tests are run, they leave behind artifacts from doing installations, downloading files, and other test related stuff. This may cause an issue for future tests if they are not cleaned up properly after each test. Also, there are OS and browser upgrades and patches that need to be installed on a regular basis. Maintaining a sizable test grid is a full time job, so hiring someone to keep the test grid maintained plus any costs of paying for operating systems clearly outweighs the cost of subscribing to a service such as Sauce Labs. Sauce Labs spins up a new virtual machine with the desired browser and browser version (try maintaining different browser versions on your test grid and the number of machines quickly grows) and gives you a clean environment for your testing. Sauce Labs is a bargain when you figure in all the costs of doing it yourself. Why reinvent the wheel?
We use the Sauce Connect secure tunnel to provide access to our test environment behind our firewall to Sauce Labs. This has been critical for us being a security company and maintaining strict access control to our environments. This allows for us to maintain a secure connection and still able to automate tests on Sauce Labs.
I evaluated Browser Stack at the same time I tried Sauce Labs and the ease of use of Sauce Labs made it an instant winner to me. I found Browser Stack (in early 2015) to be a little cumbersome to use and navigation wasn't always very intuitive to me where Sauce Labs excelled in these areas.
When there is a need to run tests against many different OS and browser combinations, but don't want to take resources away from other tasks to constantly maintain, build and support this type of infrastructure then Sauce Labs would be a perfect option. Sauce Labs has been less helpful when I've been doing mobile automation and I need to have device passcode and being able to push notifications. This may be possible now or in the future, but wasn't when I first looked into it.
Evaluating Sauce Labs and Competitors
- Product Features
- Product Usability
Product Usability was the clear cut reason why we chose Sauce Labs. It came down to Sauce Labs and Browser Stack and Sauce Labs clearly had the edge in usability. This was back in early 2015, so things may have changed, but Sauce Labs has always been easy and intuitive to use.
I don't think I would have changed much of anything. I tried out Sauce Labs and Browser Stack and attempted to do the same type of automation on both. Sauce Labs was clearly easier to use and I believe their price was even better than Browser Stack, so it was a no brainer.
Using Sauce Labs
3 - The three of us represent the automation team for our core web product and our mobile application that supports it.
3 - All three of us who use Sauce Labs equally support for each other and any other team members who may want to use it on occasion. While it doesn't take extra technical skills to use Sauce Labs, it's good to have an understanding the concept of firewalls and how to pass in desired capabilities for Selenium Webdriver. If your test environment is behind a firewall then you will need to make sure you are running Sauce Labs secured tunnel before starting, so your tests will be able to access your environment. Knowing the various desired capabilities will also help with running the correct configuration on Sauce Labs, too.
- Number one use case is being able to test on many different OS and browser combinations on a clean machine. Even if you're able to spin up multiple virtual machines with your choice of configuration, it still takes quite a bit of time to create and maintain them.
- Automatic reporting of testing. Running automation with Sauce Labs keeps all of your tests in one place with easy filtering to get quick access to test results.
- Running on mobile emulators and simulators. You will not always have every device supported to test on, so you will need to rely on emulators and simulators for some of your testing. Like not having to spin up virtual machines for every desktop OS and browser combo, you don't need to create every emulator or simulator either. Run directly on Sauce Labs!
- Using Sauce Labs has increased our test coverage which is the biggest win for us. Previously, we were limited by resources to create all the environments, but now we can test across many different types of configurations. This has been a huge boost to our testing!
- Running tests in parallel! Before, we mostly ran our tests one by one which took a long time especially as we developed more and more scripts. Being able to seamlessly run many tests in parallel has greatly sped up testing and keeping our coverage at the same time.
- The use of mobile emulators and simulators. Before we were limited by what devices we had on hand and only using a browser's emulator for our testing which is very limited and doesn't cover many features of a mobile device. Using Sauce Labs for emulator and simulator testing has helped us with greater coverage than we had before.
- Sauce Labs real device farm! I've only test driven their real device farm, but it worked great! I was able to run my automation on native android and ios apps on real devices that we didn't have on hand. How can you beat that?
- The plan is to run many tests in parallel, so I'm expecting to need many more virtual machines to run at the same time.
- Real device farm: I've tried it out and now I want to use it on a regular basis. As more mobile scripts are developed, there will be a need at some point to move away from managing our own inventory and use Sauce Labs real device farm.
Our test strategy uses Sauce Labs as an integral piece of our testing puzzle. We rely on the ability to run tests in parallel and having clean virtual machines spun up in the configuration that we require. Our need for Sauce Labs will only continue to increase, so I absolutely see renewing our Sauce Labs account each year.
Sauce Labs Implementation
If your automation framework is setup correctly and you create your webdriver separate from tests then it should be easy enough to implement Sauce Labs into your testing.
Sauce Labs Support
They have helped with issues even when the issue wasn't on their side. The success of my automation scripts was their top priority and they did whatever they had to do to help me get the job done and sometimes that meant helping me with stuff that had nothing to do with them.
Problems get solved
Kept well informed
No escalation required
Immediate help available
Support understands my problem
Support cares about my success
Quick Initial Response
No - The support we receive with the subscription we have has been more than adequate.
Yes - Yes. They have always been very responsive to support issues and just questions. I can't say enough good things about how great they've been in supporting us.
When I was first trying to use Appium for mobile testing, I struggled quite a bit getting it up and running. I was attempting to use Sauce Labs, as well, while trying to get going. I ran into many issues (self made), but many times I didn't know if it was on my side or Sauce Labs, so there were a number of times that I sent them my problem. Not once did they kick it back to me saying it was on my side and not theirs. They always helped and would even escalate to a senior member to offer assistance with Appium to help out. Like I've said before, I can't say enough great things about the support I've received at Sauce Labs. They truly make your success their success.
Using Sauce Labs
I think Sauce Labs is great and I've been using them since late 2014. With that said, I'm very used to how their interface works and how to get what I need. I think if a brand new user would come in and use it, it would still be easy, but maybe not as easy as it is for me. That's why I rate this area a 9 instead of a 10. Still great though!
Like to use
Easy to use
Technical support not required
Feel confident using
- The ease of spinning up a new virtual machine with your choice of OS and browser
- Being able to retrieve and review test results
- Adding team members to their own sub account to separate testing
- While spinning up a new virtual machine is easy, it can be tricky if you're not familiar with the need of creating a secured tunnel to access a test environment behind a firewall. Easy after you realize it though.
- Getting live video to update while a test is running can be cumbersome. I've had to refresh the browser sometimes to get the video to load.
- I wish tests would run faster especially with mobile emulators and simulators. I hate to put this down because I know how difficult this must be, but if I could magically make something better it would be the speed of tests.