1. Home
  2. Functional Testing Tools
  3. Sauce Labs Reviews
  4. User Review of Sauce Labs
The Best Continuous Delivery Testing Service That's Continuously Improving
February 22, 2018

The Best Continuous Delivery Testing Service That's Continuously Improving

Brook Beam | TrustRadius Reviewer
Score 8 out of 10
Vetted Review
Verified User

Overall Satisfaction with Sauce Labs

As we began architecting a CI/CD pipeline in our SDLC, a need quickly arose for left-shifted, continuous testing of our web applications in real browsers. Specifically, a service that could integrate seamlessly with our Jenkins CI servers and executing tests continuously against our Dockerized applications and firewalled environments. Furthermore, our company does not have the resources to implement and maintain a proprietary Selenium Grid for test distribution. Sauce Labs addresses all of these concerns, and furthermore offers a platform for manually debugging applications in any combination of platforms, browsers, and devices needed, either from an automated test script or from a manual test session. The largest selling point for our needs is its relatively seamless integration with our Jenkins CI servers, including detailed playback and test reports per build job, as well as easy configuration of the proxy tunnel used to access our firewalled environments and desired browsers-under-test, right from the Jenkins UI via their Sauce OnDemand Jenkins plugin.
  • Sauce Labs offers the single best solution for integrating deep automated browser testing in a CI/CD pipeline, and integrates best with Jenkins over their competition.
  • The Sauce Connect Proxy is a necessary technology for companies who have firewalled testing environments, as well as localhost testing.
  • The Jenkins UI plugin offers a way to easily configure browser settings for tests being executed continuously.
  • Browser support is quick to be updated as new versions of browsers are released.
  • The custom capabilities that can be provided to Sauce Labs VMs during automated testing sessions are a valuable option for experimental or niche testing.
  • The Jenkins Sauce OnDemand plugin could have more options available to tap into more of the custom capabilities Sauce Labs actually supports. Currently, in order to tap into these capabilities, a programatic solution is required from test code, rather than being able to configure them via the Jenkins plugin UI. For example, things like desktop resolution, browser automation binary versions, remote Selenium version, and browser console logs.
  • Some issues with the Sauce Connect Proxy on our Jenkins servers failing to start, which makes the optimal CI/CD pipeline come to a halt. Better documentation, logging, and more intuitive configuration around the Sauce Connect Proxy would take what is already an industry leading feature to the next level.
  • User account management needs an overhauls, allowing for user groups, rather than just a hierarchy structure.
  • Test job queuing is first-come-first-serve, meaning there is no in-built way to queue batches of jobs from, say, different Jenkins servers/accounts.
BrowserStack captures more market share than Sauce Labs does as of this writing, and thus has a much larger support network. During testing, it was also a little faster to spin-up virtual machines, and substantially quicker spinning-up mobile simulators and emulators. Pricing models were very similar in the enterprise tier. However, while BrowserStack may have been a better choice for manual testing or manually-executed automated tests, Sauce Labs offered a better product and better vision for tests that were left-shifted in a continuous delivery pipeline, which is the architectural direction or company has gone. Their service was proven to better integrate with this pipeline than BrowserStack.
If your need is architecting a CI/CD pipeline with left-shifted continuous real browser testing, then there is simply no better service available than Sauce Labs to integrate with this desired model. They are a smaller company than their leading competitor, but they are rapidly innovating in the right direction, and lead the charge in continuous automated testing. If, however, your need is specific to manual test sessions, or manually-executed automated test scripts, then their competition becomes more appealing given price points, market solidification, and support.

Evaluating Sauce Labs and Competitors

  • Price
  • Product Features
  • Product Usability
  • Positive Sales Experience with the Vendor
  • Third-party Reviews
The features of Sauce Labs over its primary competition are ultimately what drove the purchase decision. Sauce Labs offers a far-better solution for integrating deep user interface testing into a continuous delivery pipeline, that takes care to allow for testing of firewalled environments. The Jenkins support, specifically, informed our decision quite a bit, since we use Jenkins CI servers for our CI/CD model.
Reaching out to Sauce Labs representatives to get a timeline estimate on development of new features might have been an addition to the process we considered. Furthermore, a lengthier trial implementation to better determine some of the shortcoming of features we liked might have been beneficial, such as relatively restrictive user account management.

Using Sauce Labs

30 - Software Architects, Software Development Engineers, Business Analysts, and Software Test Engineers all leverage the Sauce Labs service in some way.
2 - Most support required for Sauce Labs involves coding test automation solutions that interface with Sauce Labs and Jenkins in the ways that are expected of our CI/CD model, and evolve along with it. There are also the administrative tasks surround user account, including creation of new users in the Sauce Labs service, as well as managing their privileges.
  • Functional and Regression testing of our web applications via the user interface
  • A VM grid supporting all necessary platforms, devices, and browsers that tests can be executed in from our CI servers that is maintenance-free
  • Ability to debug browsers, platforms, or devices that are not readily available in-office
  • Integration with our continuous delivery pipeline
  • Detailed test reports and playback
  • The ability to test the responsiveness of our user interfaces against smaller resolutions (either desktop or mobile)
  • The ability to test multiple independent browser sessions within the execution of a single test, such as for testing session independence for Single Sign-On cases, or for testing concurrent usage of an application across multiple browser sessions
  • The ability to execute automated or manual tests in Sauce Labs against localhost
  • Continuous and left-shifted performance testing of the User Experience
  • Real Device testing, rather than mobile simulators or emulators
  • More extensive mobile testing
Sauce Labs has become an integral part of our entire continuous delivery pipeline. Our architectural vision for CI/CD simply would not be possible without Sauce Labs given the resource constraints our company has to develop and maintain a solution such as this internally.

Sauce Labs Implementation

By far the largest deduction for satisfaction is the lack of complete capability support in the Jenkins Sauce OnDemand plugin's user interface, as well as the lack of being able to queue test jobs by batches. Our organization has multiple Jenkins servers, and each has its own Sauce Labs account for executing tests at built time. If multiple Jenkins servers kick off test batches at the same time, we have to develop a solution internally to manage those collisions.
  • Implemented in-house
Change management was minimal - N/A
  • Ensuring that implemented automated test scrips could interface with Sauce Labs from either a local workstation or a Jenkins CI server
  • Implementing solutions that allowed for all of Sauce Labs capabilities to be configured either from a local workstation or a Jenkins CI server, since the Jenkins Sauce OnDemand plugin offers less than the service provides in its user interface configuration
  • Determining a solution for test job queuing. Sauce Labs does not support batch job queuing, but rather test jobs are queuing by first-come-first-server. What this means is that if your organization has multiple CI servers, each with their own Sauce Labs account and each wanting to execute a batch of test jobs, your organization must internally implement a solution to ensure that those concurrent batches of tests do not collide with each other.

Sauce Labs Training

  • Self-taught
Product is extremely easy to learn on one's own, and requires little technical knowledge unless you are attempting to interface with the service from an in-house test automation framework. The documentation is well done, and updated with an acceptable frequency. Furthermore, they have a knowledge base and community that can be used to research more niche issues or questions.

Configuring Sauce Labs

The configurability is quite extensive, but to leverage it in a CI/CD model from a CI server requires a programmatic solution to bridge the gap between source code configuration and Jenkins Sauce OnDemand plugin's UI configurations, since the Jenkins plugin's configurations are far more limited than what Sauce Labs actually supports (yet are a preferred way of configuration since they do not require code or configuration file changes).
Ensure that automated test source code and leverage all of Sauce Labs features from either a local workstation or a CI server, and that (if applicable) the Jenkins Sauce OnDemand plugin's configurations take priority over source code configurations, unless source code configs explicitly override the Jenkins plugin configs. This is the best way to ensure that specific test cases can have specific configurations without blanketing them with a configuration for the entire suite, as well as ensure that capabilities Sauce supports that aren't reflected in the Jenkins plugin can be leveraged.
No - there is no facility to customize the interface
No - the product does not support adding custom code
An internal solution needs to be implemented that organized batches of automated tests from multiple CI servers, such that if batches are executing concurrently, their individual tests do not collide with one another in the Sauce Labs service and unexpectedly subtract from the available concurrency pool. Sauce Labs does not support such batch queuing natively.

Sauce Labs Support

On the enterprise license tier, support tickets are answered very quickly. Furthermore, when issues are detected in the service itself, status pages are updated frequently, and notifications sent out with updates on the status of those issues. It would be nice, however, to see issues that are the result of missing features be taken more seriously, or a more seamless integration between support tickets and feature requests, rather than having the consumers of the service have to submit their own feature requests.
ProsCons
Good followup
Knowledgeable team
Problems get solved
Kept well informed
No escalation required
Immediate help available
Support understands my problem
Quick Initial Response
None
Yes - Premium support comes with entry into the enterprise license tier, which our company is a part of for legal reasons.
Our trial period before we decided to purchase Sauce Labs was extensive. At this time, we were still entertaining the idea of their main competitor as well. We mostly needed to vet what service would best work with our continuous delivery pipeline, and we needed time to prove this out. The Sauce Labs representative we were working with at the time was more than helpful and went above and beyond to ensure that our trial period was extended to as long as we needed to make an informed decision.

Using Sauce Labs

It is an incredibly easy service to use for what its primary intention is. The only reason a point is deducted is because more feature enrichment can be done around the Sauce Connect Proxy utility and the Jenkins Sauce OnDemand plugin. User Account administration also needs more work, such as the addition of user groups, rather than a simple hierarchy of users.
ProsCons
Like to use
Relatively simple
Easy to use
Technical support not required
Well integrated
Quick to learn
Convenient
Feel confident using
Familiar
None
  • Manual test sessions are very easy to spin up, and customize your platform, device, and browser and its version
  • Basic Jenkins integration is incredibly easy via the Sauce OnDemand Jenkins plugin. Configuration is done completely via its user interface
  • Test playback and metadata viewing is incredibly intuitive, for both manual and automated testing sessions
  • Sometimes the Sauce Connect Proxy on a Jenkins server can be inexplicable flaky and fail to start during a build job. When this occurs, debugging the cause is difficult, and it breaks the intention of the CI/CD model.
  • To execute a test in Sauce Labs from a local workstation against an application/environment that is firewalled requires starting the Sauce Connect Proxy utility on the local machine. This is a command line utility that may be unwieldy for non-technical people.
  • The Jenkins Sauce OnDemand plugin is, as of this writing, relatively restrictive in what capabilities it can set for test execution. If your automated tests wish to tap into more granular capabilities that Sauce Labs supports (such as desktop resolutions, Selenium version in remote VM, automated driver binary version, etc), then a programatic solution is required, as these capabilities cannot be configured from the Jenkins plugin user interface.
Yes, but I don't use it

Sauce Labs Reliability

The product is as scalable as your company's wallet can afford, since you are charged per unit of concurrency, where each new unit of concurrency represents another concurrent virtual machine you can spin up.
Outages with the virtual machine services or the Sauce Connect Proxy service are more frequent than would be ideal, but updates to resolutions are communicated frequently, and resolutions usually come quick.
The execution of automated tests in Sauce Labs virtual machines is slightly slower than its main competition. Furthermore, the spin-up time of virtual machines (particularly device simulators or emulators) is significantly slower in our experience than its primary competition. The Sauce Connect Proxy further adds to performance decrease when used in our experience (around 10%-20% decrease when used), such that each call to the remote WebDriver instance in your test scripts can take upwards of 100-200 milliseconds.

Integrating Sauce Labs

Basic Jenkins integration is simple. A big exception, however, is if leveraging all of Sauce Labs capabilities is desired, which requires a programmatic approach.
Slack integration is communication of Sauce Labs outages et al, and was easy to achieve via Slack configuration. Basic Jenkins configuration is very simple via Sauce OnDemand Jenkins plugin that offers a configuration user interface.
N/A
  • API (e.g. SOAP or REST)
N/A

Relationship with Sauce Labs

Our company required signing of our own Non-Disclosure Agreement, which Sauce Labs was initially unwilling to do given the entry-level enterprise tier we were purchasing. This required more back-and-forth than we would have liked and delayed our purchasing process (and implementation) more than was ideal.
During purchasing talks for additional services after initial purchase, the sales staff worked with was extremely rude and pushy, and seemed annoyed by inquiries and requests.
N/A
Know that the enterprise pricing is flexible and can be negotiated.

Upgrading Sauce Labs

  • N/A, cloud-based
  • Hopefully more configurations for user account administration
  • More features implemented for the Jenkins Sauce OnDemand plugin
  • More fleshed-out Sauce Connect Proxy utility