AWS Device Farm vs. IBM DevOps Test Performance

AWS Device Farm is perfect for small teams who don't have enough resources to conduct large scale testing scenarios or even large teams who would like an easier time with testing on various devices. [AWS] Device Farm makes it very easy to interact with a multitude of devices right from the browser.

Incentivized

Go for IBM RPT if: 1. You're testing a Java-based Web application with HTTP protocol 2. You wanted to distribute the load across machines easily 3. Your team is in learning phase/not really introduced to a wide range of performance testing tools Do not go for IBM RPT if: 1. You wanted to test REST or any other advanced protocols 2. Your system under test demands a very high user load 3. Your application is written in .NET or any other platform except Java.

Incentivized

• Integration with CI/CD Pipeline
• Diversity in Real Devices over the Cloud
• Pay as you go option
• Reproduce Issues and easily collect all kinds of logs

Incentivized

• Live Monitoring of Performance Testing Metrics
• Easier Test Data Parameterization
• Performance Tests Scheduler
• Server Resource Monitoring
• Great Reports that can be Exported as HTML Reports

• Interface (CLI) not very easy. Competition has better options.
• Integration with IDE for test case coding.
• Reporting for the test scripts executed.

Incentivized

• Memory utilization could have been improved.(Eats up system's RAM)! It may crash if a test is conducted with the heavy load if adequate RAM is not available in the VM/host machine.
• Licensing could have been made simpler. IBM's licensing method is difficult to follow.
• Support for protocols other than HTTP. Not really up to the current trend.

Incentivized

We had the enterprise support with AWS, so overall support experience was good with great engineers on the back providing answers. As you may know, overall AWS support is different and this is not different. Responses through the regular web support channel came easily, fast and accurate. We had questions/issues which were solved fast. Documentation is good as well, especially around the test automation pieces.

Incentivized

I got more features and handy within AWS Device farm than AWS Cloud9

Incentivized

IBM Rational Performance Tester is better in many aspects when compared to Apache JMeter. For example, you have ready made scripts to assist you, You have better monitoring of different performance metrics and can generate better reports that can be exported as HTML. The only downside of IBM RPT is that it can be very costly at times while JMeter is for free which is a huge advantage. That's why we try using in our organization a combination of both as much as we can.

• Quicker testing leads to faster product to market times
• Can deploy updates much more regularly by testing across many devices quickly

Incentivized

• Accuracy in metrics, thus improving system's performance
• Costs less compared to competitor like HP LoadRunner
• Helped the team of beginners learn things quickly

Incentivized