AWS CodePipeline vs. Katalon TestOps

I think AWS CodePipeline is a great tool for anyone wanted automated deployments in a multi-server/container AWS environment. AWS also offers services like Elastic Beanstalk that provide a more managed hosting & deployment experience. CodePipeline is a good middle ground with solid, built-in automation with enough customizability to not lock people into one deployment or architecture philosophy.

Incentivized

Katalon TestOps is well suited for an oraganization who want to organize their automation collections and have everything in one place. It provides good insight on test runs, failures and also helps in defect tracking. It is extremely easy to schedule and run test suites automatically. With the help of test history, it becomes easier to triage the report and identify if it is a product issue or a test flakiness. Also with the automation run scheduling features with custom options, user do not have to manually trigger the runs and just run the scheduler to do the job.

Incentivized

• It is reliable and works without errors
• It integrates well with our repository and all other AWS functions as well as our end database

Incentivized

• One place for all report history
• Get count of tests automated each release
• Easy way to schedule and run test suites automatically
• Track test flakiness easily

Incentivized

• Ease of use - things like CircleCI or other tools are a bit easier to learn.
• Ability to build from more sources.

Incentivized

• Very small community
• Requires learning curve
• Sometimes application crashes

Incentivized

Overall, I give AWS Codepipeline a 9 because it gets the job done and I can't complain much about the web interface as much of the action is taking place behind the scenes on the terminal locally or via Amazon's infrastructure anyway. It would be nicer to have a better flowing and visualizable web interface, however.

Incentivized

Our pipeline takes about 30 minutes to run through. Although this time depends on the applications you are using on either end, I feel that it is a reasonable time to make upgrades and updates to our system as it is not an every day push.

Incentivized

We didn't need a lot of support with AWS CodePipeline as it was pretty straightforward to configure and use, but where we ran into problems, the AWS community was able to help. AWS support agents were also helpful in resolving some of the minor issues we encountered, which we could not find a solution elsewhere.

Incentivized

CodeCommit and CodeDeploy can be used with CodePipeline so it’s not really fair to stack them against each other as they can be quite the compliment. The same goes for Beanstalk, which is often used as a deployment target in relation to CodePipeline.

CodePipeline fulfills the CI/CD duty, where the other services do not focus on that specific function. They are supplements, not replacements. CodePipeline will detect the updated code and handle deploying it to the actual instance via Beanstalk.

Jenkins is open source and not a native AWS service, that is its primary differentiator. Jenkins can also be used as a supplement to CodePipeline.

Incentivized

Initially, our oragnization had to integrate Katalon tests with the Jenkins to schedule the test runs. The reports were stored in company local database. To identify test flakiness or product issue one had to dig deep into the reports. This issue was all solved by Katalon TestOps. Katalon TestOps provides report history in one place and also helps in easy triage of the report by pointing out script flakiness

Incentivized

• CodePipeline has reduced ongoing devops costs for my clients, especially around deployment & testing.
• CodePipeline has sped up development workflow by making the deployment process automated off git pushes. Deployment takes very little coordination as the system will just trigger based on what is the latest commit in a branch.
• CodePipeline offered a lot of out-of-the-box functionality that was much simpler to setup than a dedicated CI server. It allowed the deployment process to built and put into production with much less and effort and cost compared to rolling the functionality manually.

Incentivized

• Support can be better: As mentioned before, katlon community is still growing but is currently a small community. If you are stuck, its hard to find help in the community and one has to reach out to the support for help. This takes a while before you can get solution to your problem.
• Result upload takes little longer time: It is been observed that uploading tests results takes little longer.
• Sometimes it gets hard to integrate: Since this is fairly a new tool, requires lot of learning curve to get the job done effectively.

Incentivized