AWS Device Farm vs. SolarWinds Pingdom

AWS Device Farm is perfect for small teams who don't have enough resources to conduct large scale testing scenarios or even large teams who would like an easier time with testing on various devices. [AWS] Device Farm makes it very easy to interact with a multitude of devices right from the browser.

Incentivized

I believe the scenarios we used it for were quite well covered, from the executive perspective. The downtime alarms worked very well and were easy to setup, uptime monitoring tools were clear and easy to use, even for non-technical people (C-level) and the SLA management tools allowed us to spend less time, and have less friction, with our clients

Incentivized

• Integration with CI/CD Pipeline
• Diversity in Real Devices over the Cloud
• Pay as you go option
• Reproduce Issues and easily collect all kinds of logs

Incentivized

• The alerts are very responsive offering quite detailed timing information
• The UI for reports is very clear and set up of reports is very intuitive
• Customer service was good when there were problems - e.g. lapsing billing causing reporting to go down

Incentivized

• Interface (CLI) not very easy. Competition has better options.
• Integration with IDE for test case coding.
• Reporting for the test scripts executed.

Incentivized

• The PagerDuty integration could be a lot better. When you use the PagerDuty integration, it doesn't send any information about which check failed! It just sends a message like "Timeout (> 30s)" -- this isn't very helpful when we have hundreds of checks. We've worked around this by using both the PagerDuty and Slack integrations and having them both post to the same Slack channel. But this means that when an engineer is paged from PagerDuty, they have to go to Slack (or Pingdom) to find the details about the page; it's not available on the page itself.

Incentivized

Recently added features have made Pingdom less intuitive for our requirements. While Pingdom has a broad offering and remains a good value, it is becoming more than we need. Our customer base is becoming more and more global and Pingdom still lacks Asia-Pacific monitoring, which we will need within a year.

Pingdom is easy to use, very intuitive and has a very short learning curve. From the onset, we've been able to jump in and leverage the tool to accomplish our goals for page speed performance and discover the insights we need to make improvements. Its a well-designed tool and makes for a good user experience.

Incentivized

Interface is unnecessarily complicated and frequently responds slowly.

We had the enterprise support with AWS, so overall support experience was good with great engineers on the back providing answers. As you may know, overall AWS support is different and this is not different. Responses through the regular web support channel came easily, fast and accurate. We had questions/issues which were solved fast. Documentation is good as well, especially around the test automation pieces.

Incentivized

Support responded the same day to my query, as I was setting the product up but couldn't find the setting I needed. This was successfully resolved in a short time frame, so I was pleased with how quickly we were able to get this resolved. I haven't needed to contact support since.

Incentivized

I got more features and handy within AWS Device farm than AWS Cloud9

Incentivized

1. PRTG Network Monitor was a far more complicated tool to use and set up albeit it does both Internal and External monitoring. The setup wasn't intuitive and there are too many configuration options to complete to form an alert
2. Amazon CloudWatch is specific to AWS resources and cannot be easily use outside of the AWS Ecosystem

Incentivized

• Quicker testing leads to faster product to market times
• Can deploy updates much more regularly by testing across many devices quickly

Incentivized

• Honestly, we have 4 other products that overlap this functionality whose organizations provide far superior support. At this point it is an unnecessary expense.
• In my opinion, their lack of support responsiveness and commitment has impacted our IT agility.
• Its time to explore other alternatives.