Blackbird API Development vs. Katalon vs. mabl

Ambassador is really well suited for scenarios were you need to give power and freedom to your developers so they can take advantage of the self-service approach. One of the few scenarios I can not recommend using Ambassador is in the case you are planning to not using Kubernetes as it is a solution designed to work specifically on that platform

Incentivized

Katalon TestOps is well suited for an oraganization who want to organize their automation collections and have everything in one place. It provides good insight on test runs, failures and also helps in defect tracking. It is extremely easy to schedule and run test suites automatically. With the help of test history, it becomes easier to triage the report and identify if it is a product issue or a test flakiness. Also with the automation run scheduling features with custom options, user do not have to manually trigger the runs and just run the scheduler to do the job.

Incentivized

We haven't found a scenario yet where it hasn't been appropriate. We did have one function on our application that mabl couldn't do, but they solved it and got back to us very quickly. Our application is web-based and mabl is able to handle this very easily. We use the command line runner a lot. Being able to easily and quickly change from a cloud based run to a local run has been fantastic. Setting up flows and environments is a wonderful feature

Incentivized

• Simple syntax
• Easy to understand
• Easy to install

Incentivized

• Easy to setup and quick to get started: As Katalon comes with built-in features a new automation tester can automate easily. Script creation time is quick.
• Test Analytics: Katalon also provides analytics which tells you the pattern of your automated tests at a different instance of time.
• Third Party Integration: Katalon provides amazing integration with 3rd party tools like JIRA, ALM, GIT, JENKINS, etc.
• Image-based testing: Katalon also provides inbuilt image based test automation.

Incentivized

• mabl trainer - record and play back test cases
• Organization - labels and flows make this easy
• Customer support!

Incentivized

• Documentation is not updated
• Implementing external authentication doesn't always work as described in the documentation
• The Administration console is limited

Incentivized

• Integration with Azure is painstakingly slow as it takes too much time to upload reports to Azure. When raised with the support team i was advised to contact Microsoft who promptly and correctly replied that its an issue with the katalon plugin.
• Non Responsive Customer Support. Several mails were replied late.
• The Desktop app uses too much memory when booting up and running tests even on a 16gb RAM.

• Asserts could be easier to add, but they aren't terribly hard

Incentivized

It's great! it's amazingly simple to use and the best part is the self-service approach. I also like how easy it is to add a new route to a endpoint with the mappings definition.

Incentivized

Due to the slowness, there is an intermediate timeout. There is no other problem.

Incentivized

Overall the support has been great and quick to answer the requests I've submitted in the past. I was originally using the community/open source version and I can't say it was the best experience I had, it wasn't terrible but it wasn't great. I believe the biggest issue was that they refer to the documentation a lot but the documentation isn't updated regularly so I feel it's lagging behind the most recent versions.

Incentivized

I love the support. They have provided answers and solutions for most of our questions or doubts that could come up. They also have provided basic tutorials on how to get familiarized with the program.

Incentivized

They both offer about the same in terms of end goal and purpose of use and scenarios. However, Ambassador does way better in terms of simplifying the syntax and makes offer a little bit more of control by adding the concepts of hosts, mappings while staying away from a hard to read single file configuration.

Incentivized

Katalon Studio is above both the tools is due to its codeless test automation feature and having API, desktop, web, and mobile automation in one place but obviously there is room for improvement when desktop automation is of primary importance in the automation.

Incentivized

We ultimately selected mabl cause it most met our needs and our budget. We needed a low code UI automation test tool. We also have a suite of existing testing tools and other tools that it needed to be able to integrate with. Price was the only thing that ruled out some of the above tools, we are a small start up, and don't have a huge budget. Some tools listed didn't have the same functionality or ease of use with the record and playback. mabl works on our machines and integrates with our existing tools

Incentivized

• Reduced downtime by redirecting traffic between DC.
• Reduced engineering time, don't have to dedicate 1 person to manage it.
• Increased training costs in the short term.

Incentivized

• Support can be better: As mentioned before, katlon community is still growing but is currently a small community. If you are stuck, its hard to find help in the community and one has to reach out to the support for help. This takes a while before you can get solution to your problem.
• Result upload takes little longer time: It is been observed that uploading tests results takes little longer.
• Sometimes it gets hard to integrate: Since this is fairly a new tool, requires lot of learning curve to get the job done effectively.

Incentivized

• We are able to more quickly automate UI tests

Incentivized