The Best Continuous Delivery Testing Service That's Continuously Improving
February 22, 2018
The Best Continuous Delivery Testing Service That's Continuously Improving
Score 8 out of 10
Overall Satisfaction with Sauce Labs
As we began architecting a CI/CD pipeline in our SDLC, a need quickly arose for left-shifted, continuous testing of our web applications in real browsers. Specifically, a service that could integrate seamlessly with our Jenkins CI servers and executing tests continuously against our Dockerized applications and firewalled environments. Furthermore, our company does not have the resources to implement and maintain a proprietary Selenium Grid for test distribution. Sauce Labs addresses all of these concerns, and furthermore offers a platform for manually debugging applications in any combination of platforms, browsers, and devices needed, either from an automated test script or from a manual test session. The largest selling point for our needs is its relatively seamless integration with our Jenkins CI servers, including detailed playback and test reports per build job, as well as easy configuration of the proxy tunnel used to access our firewalled environments and desired browsers-under-test, right from the Jenkins UI via their Sauce OnDemand Jenkins plugin.
- Sauce Labs offers the single best solution for integrating deep automated browser testing in a CI/CD pipeline, and integrates best with Jenkins over their competition.
- The Sauce Connect Proxy is a necessary technology for companies who have firewalled testing environments, as well as localhost testing.
- The Jenkins UI plugin offers a way to easily configure browser settings for tests being executed continuously.
- Browser support is quick to be updated as new versions of browsers are released.
- The custom capabilities that can be provided to Sauce Labs VMs during automated testing sessions are a valuable option for experimental or niche testing.
- The Jenkins Sauce OnDemand plugin could have more options available to tap into more of the custom capabilities Sauce Labs actually supports. Currently, in order to tap into these capabilities, a programatic solution is required from test code, rather than being able to configure them via the Jenkins plugin UI. For example, things like desktop resolution, browser automation binary versions, remote Selenium version, and browser console logs.
- Some issues with the Sauce Connect Proxy on our Jenkins servers failing to start, which makes the optimal CI/CD pipeline come to a halt. Better documentation, logging, and more intuitive configuration around the Sauce Connect Proxy would take what is already an industry leading feature to the next level.
- User account management needs an overhauls, allowing for user groups, rather than just a hierarchy structure.
- Test job queuing is first-come-first-serve, meaning there is no in-built way to queue batches of jobs from, say, different Jenkins servers/accounts.
- Increase the number of software release cycles
- Accelerate the pace of new feature releases
- Eliminate testing as a bottleneck
- Improve quality by increasing browser/OS/device coverage
- Achieve continuous delivery
Prior to moving to a continuous delivery model, we were on a classic two week regression testing cycle, mostly conducted via manual testing and manually-executed automated tests scripts in a large integrated testing environment. Upon shifting our SDLC to be more continuous, and re-architecting our delivery pipeline, we were able to very quickly shift that long testing cycle to be integrated into the delivery pipeline itself by left-shifting these deep functional/regression tests to be executed during build time on our CI servers, and the implementation of these automated tests scripts being integrated into the development effort itself. Sauce Labs allowed all of this to happen by supplying the service needed to execute automated browser tests in from a CI server, without having to implement or maintain our own proprietary grid of virtual machines.
The purchasing requirements for our company thoroughly vets the way in which external services handle our data, and furthermore requires that companies sign our own Non-Disclosure Agreement. Sauce Labs was willing to work with us to ensure NDA compliance. Furthermore, their retention of test execution data is currently 30 days. Virtual Machines used for automated testing sessions are one-time use, and thrown away at the end of each session.
Sauce Labs offers a collection of browsers, platforms, and devices that our automated and manual tests can be executed against that our company simply does not have the resources to implement or maintain. This includes desktop browsers rendered at low or abnormal resolutions, as well as mobile simulators and emulators we can use to test the overall responsiveness of our user interfaces to ensure proper user experience of our applications. Furthermore, but providing us all the browsers our company is contractually obligated to support, we can ensure that users are functionally experiencing our web applications as expected in a way that meets our contractual obligations.
Creating and maintaining a network of virtual machines with the proper collections of desktop browsers, and all their versions, across all major operating systems -- as well as mobile device simulation -- is an extremely large effort to implement, and a solution that requires constant maintenance as new browser versions, operating systems, mobile platforms, and mobile devices are released into the market. This is maintenance that never goes away. Sauce Labs offers this unavoidable maintenance as intrinsic to its service. The internal resources that would be required to develop a solution such as this is something our company cannot sustain. Yet, a solution such as this is required to reach the vision of a continuous delivery pipeline that delivers quality products, and thus Sauce Labs was the most cost-effective choice to reach this vision.
BrowserStack captures more market share than Sauce Labs does as of this writing, and thus has a much larger support network. During testing, it was also a little faster to spin-up virtual machines, and substantially quicker spinning-up mobile simulators and emulators. Pricing models were very similar in the enterprise tier. However, while BrowserStack may have been a better choice for manual testing or manually-executed automated tests, Sauce Labs offered a better product and better vision for tests that were left-shifted in a continuous delivery pipeline, which is the architectural direction or company has gone. Their service was proven to better integrate with this pipeline than BrowserStack.
If your need is architecting a CI/CD pipeline with left-shifted continuous real browser testing, then there is simply no better service available than Sauce Labs to integrate with this desired model. They are a smaller company than their leading competitor, but they are rapidly innovating in the right direction, and lead the charge in continuous automated testing. If, however, your need is specific to manual test sessions, or manually-executed automated test scripts, then their competition becomes more appealing given price points, market solidification, and support.
Evaluating Sauce Labs and Competitors
- Product Features
- Product Usability
- Positive Sales Experience with the Vendor
- Third-party Reviews
The features of Sauce Labs over its primary competition are ultimately what drove the purchase decision. Sauce Labs offers a far-better solution for integrating deep user interface testing into a continuous delivery pipeline, that takes care to allow for testing of firewalled environments. The Jenkins support, specifically, informed our decision quite a bit, since we use Jenkins CI servers for our CI/CD model.
Reaching out to Sauce Labs representatives to get a timeline estimate on development of new features might have been an addition to the process we considered. Furthermore, a lengthier trial implementation to better determine some of the shortcoming of features we liked might have been beneficial, such as relatively restrictive user account management.
Using Sauce Labs
30 - Software Architects, Software Development Engineers, Business Analysts, and Software Test Engineers all leverage the Sauce Labs service in some way.
2 - Most support required for Sauce Labs involves coding test automation solutions that interface with Sauce Labs and Jenkins in the ways that are expected of our CI/CD model, and evolve along with it. There are also the administrative tasks surround user account, including creation of new users in the Sauce Labs service, as well as managing their privileges.
- Functional and Regression testing of our web applications via the user interface
- A VM grid supporting all necessary platforms, devices, and browsers that tests can be executed in from our CI servers that is maintenance-free
- Ability to debug browsers, platforms, or devices that are not readily available in-office
- Integration with our continuous delivery pipeline
- Detailed test reports and playback
- The ability to test the responsiveness of our user interfaces against smaller resolutions (either desktop or mobile)
- The ability to test multiple independent browser sessions within the execution of a single test, such as for testing session independence for Single Sign-On cases, or for testing concurrent usage of an application across multiple browser sessions
- The ability to execute automated or manual tests in Sauce Labs against localhost
- Continuous and left-shifted performance testing of the User Experience
- Real Device testing, rather than mobile simulators or emulators
- More extensive mobile testing
Sauce Labs Implementation
By far the largest deduction for satisfaction is the lack of complete capability support in the Jenkins Sauce OnDemand plugin's user interface, as well as the lack of being able to queue test jobs by batches. Our organization has multiple Jenkins servers, and each has its own Sauce Labs account for executing tests at built time. If multiple Jenkins servers kick off test batches at the same time, we have to develop a solution internally to manage those collisions.
- Ensuring that implemented automated test scrips could interface with Sauce Labs from either a local workstation or a Jenkins CI server
- Determining a solution for test job queuing. Sauce Labs does not support batch job queuing, but rather test jobs are queuing by first-come-first-server. What this means is that if your organization has multiple CI servers, each with their own Sauce Labs account and each wanting to execute a batch of test jobs, your organization must internally implement a solution to ensure that those concurrent batches of tests do not collide with each other.
Sauce Labs Training
Product is extremely easy to learn on one's own, and requires little technical knowledge unless you are attempting to interface with the service from an in-house test automation framework. The documentation is well done, and updated with an acceptable frequency. Furthermore, they have a knowledge base and community that can be used to research more niche issues or questions.
Configuring Sauce Labs
The configurability is quite extensive, but to leverage it in a CI/CD model from a CI server requires a programmatic solution to bridge the gap between source code configuration and Jenkins Sauce OnDemand plugin's UI configurations, since the Jenkins plugin's configurations are far more limited than what Sauce Labs actually supports (yet are a preferred way of configuration since they do not require code or configuration file changes).
Ensure that automated test source code and leverage all of Sauce Labs features from either a local workstation or a CI server, and that (if applicable) the Jenkins Sauce OnDemand plugin's configurations take priority over source code configurations, unless source code configs explicitly override the Jenkins plugin configs. This is the best way to ensure that specific test cases can have specific configurations without blanketing them with a configuration for the entire suite, as well as ensure that capabilities Sauce supports that aren't reflected in the Jenkins plugin can be leveraged.
An internal solution needs to be implemented that organized batches of automated tests from multiple CI servers, such that if batches are executing concurrently, their individual tests do not collide with one another in the Sauce Labs service and unexpectedly subtract from the available concurrency pool. Sauce Labs does not support such batch queuing natively.
Sauce Labs Support
On the enterprise license tier, support tickets are answered very quickly. Furthermore, when issues are detected in the service itself, status pages are updated frequently, and notifications sent out with updates on the status of those issues. It would be nice, however, to see issues that are the result of missing features be taken more seriously, or a more seamless integration between support tickets and feature requests, rather than having the consumers of the service have to submit their own feature requests.
Problems get solved
Kept well informed
No escalation required
Immediate help available
Support understands my problem
Quick Initial Response
Yes - Premium support comes with entry into the enterprise license tier, which our company is a part of for legal reasons.
Our trial period before we decided to purchase Sauce Labs was extensive. At this time, we were still entertaining the idea of their main competitor as well. We mostly needed to vet what service would best work with our continuous delivery pipeline, and we needed time to prove this out. The Sauce Labs representative we were working with at the time was more than helpful and went above and beyond to ensure that our trial period was extended to as long as we needed to make an informed decision.
Using Sauce Labs
It is an incredibly easy service to use for what its primary intention is. The only reason a point is deducted is because more feature enrichment can be done around the Sauce Connect Proxy utility and the Jenkins Sauce OnDemand plugin. User Account administration also needs more work, such as the addition of user groups, rather than a simple hierarchy of users.
Like to use
Easy to use
Technical support not required
Quick to learn
Feel confident using
- Manual test sessions are very easy to spin up, and customize your platform, device, and browser and its version
- Test playback and metadata viewing is incredibly intuitive, for both manual and automated testing sessions
- Sometimes the Sauce Connect Proxy on a Jenkins server can be inexplicable flaky and fail to start during a build job. When this occurs, debugging the cause is difficult, and it breaks the intention of the CI/CD model.
- To execute a test in Sauce Labs from a local workstation against an application/environment that is firewalled requires starting the Sauce Connect Proxy utility on the local machine. This is a command line utility that may be unwieldy for non-technical people.
- The Jenkins Sauce OnDemand plugin is, as of this writing, relatively restrictive in what capabilities it can set for test execution. If your automated tests wish to tap into more granular capabilities that Sauce Labs supports (such as desktop resolutions, Selenium version in remote VM, automated driver binary version, etc), then a programatic solution is required, as these capabilities cannot be configured from the Jenkins plugin user interface.
Sauce Labs Reliability
The product is as scalable as your company's wallet can afford, since you are charged per unit of concurrency, where each new unit of concurrency represents another concurrent virtual machine you can spin up.
Outages with the virtual machine services or the Sauce Connect Proxy service are more frequent than would be ideal, but updates to resolutions are communicated frequently, and resolutions usually come quick.
The execution of automated tests in Sauce Labs virtual machines is slightly slower than its main competition. Furthermore, the spin-up time of virtual machines (particularly device simulators or emulators) is significantly slower in our experience than its primary competition. The Sauce Connect Proxy further adds to performance decrease when used in our experience (around 10%-20% decrease when used), such that each call to the remote WebDriver instance in your test scripts can take upwards of 100-200 milliseconds.
Integrating Sauce Labs
Relationship with Sauce Labs
Our company required signing of our own Non-Disclosure Agreement, which Sauce Labs was initially unwilling to do given the entry-level enterprise tier we were purchasing. This required more back-and-forth than we would have liked and delayed our purchasing process (and implementation) more than was ideal.
During purchasing talks for additional services after initial purchase, the sales staff worked with was extremely rude and pushy, and seemed annoyed by inquiries and requests.