Cypress.io vs. Katalon vs. WebdriverIO

Everything web-based should be testable. Even our highly dynamic content-providing web application works well. Hopefully, it will get a bigger user base in the future so that one can find even more support at StackOverflow or similar forums.

Incentivized

Katalon TestOps is well suited for an oraganization who want to organize their automation collections and have everything in one place. It provides good insight on test runs, failures and also helps in defect tracking. It is extremely easy to schedule and run test suites automatically. With the help of test history, it becomes easier to triage the report and identify if it is a product issue or a test flakiness. Also with the automation run scheduling features with custom options, user do not have to manually trigger the runs and just run the scheduler to do the job.

Incentivized

Best suited where frontend developments are in React and where BDD and TDD test frameworks are to be used. Its syntax is very easy to write and understand. Even the non-programmer can do the initial setup. Not suited when the language you are using is other than Javascript(or Typescript).

Incentivized

• Cypress automatically waits for DOM elements before interacting dev, this eliminates adding explicits waits in your code.
• Has built-in recording and debugging tool.
• Cypress has executions faster since you can update your code in the fly.
• Very easy to configure / setup

Incentivized

• Easy to setup and quick to get started: As Katalon comes with built-in features a new automation tester can automate easily. Script creation time is quick.
• Test Analytics: Katalon also provides analytics which tells you the pattern of your automated tests at a different instance of time.
• Third Party Integration: Katalon provides amazing integration with 3rd party tools like JIRA, ALM, GIT, JENKINS, etc.
• Image-based testing: Katalon also provides inbuilt image based test automation.

Incentivized

• numerous integrations to tools
• fast easy location of page elements
• tool setup is amazing - they have a script to walk you through everything with prompts!!

Incentivized

• There are a lot of plugins but several of them have many errors. So Cypress.io should work integrating more features so that people should not rely on these plugins. For example, many people use cypress-cucumber-preprocessor and Cypress.io can start by integrating Gherkin support by default so that reliance in plugins are decreased.

Incentivized

• Integration with Azure is painstakingly slow as it takes too much time to upload reports to Azure. When raised with the support team i was advised to contact Microsoft who promptly and correctly replied that its an issue with the katalon plugin.
• Non Responsive Customer Support. Several mails were replied late.
• The Desktop app uses too much memory when booting up and running tests even on a 16gb RAM.

• wdio.conf can contain too much where everything is encapsulated there, like the before and after functions
• A data provider-like testing would be useful. The only way to input different data into the same test is via a loop
• everything is needed to be done using the 'browser' object. Can be limiting where you don't have access to the browser depending on where you are in the code

Incentivized

Highly satisfied and [a] big fan.

Incentivized

Due to the slowness, there is an intermediate timeout. There is no other problem.

Incentivized

I love the support. They have provided answers and solutions for most of our questions or doubts that could come up. They also have provided basic tutorials on how to get familiarized with the program.

Incentivized

Execution speed and interactive debugging tool is very easy. you can make changes in the fly and test will run automatically, thus saving more time debugging your code. Can easily be intergrated to to CI/CD pipeline

Incentivized

Katalon Studio is above both the tools is due to its codeless test automation feature and having API, desktop, web, and mobile automation in one place but obviously there is room for improvement when desktop automation is of primary importance in the automation.

Incentivized

Other tools we selected don't work in our application, it does or doesn't support multiple frames or need more tweaking just to make it works. When testing out WebdriverIO it just works as expected, no need to do such walk around to make it works multiple frames. It also can handle multiple tabs and windows.

Incentivized

• Flaky tests can be caught easily
• Regration testing of application with Cypress.io helps to maintain quality of the software product.

Incentivized

• Support can be better: As mentioned before, katlon community is still growing but is currently a small community. If you are stuck, its hard to find help in the community and one has to reach out to the support for help. This takes a while before you can get solution to your problem.
• Result upload takes little longer time: It is been observed that uploading tests results takes little longer.
• Sometimes it gets hard to integrate: Since this is fairly a new tool, requires lot of learning curve to get the job done effectively.

Incentivized

• It helped in cross browser testing
• It helped in automating multiple browsers at a time.
• Code coverage is optimum as this supports Javascript(Typescript)

Incentivized