Apache Airflow is an open source tool that can be used to programmatically author, schedule and monitor data pipelines using Python and SQL. Created at Airbnb as an open-source project in 2014, Airflow was brought into the Apache Software Foundation’s Incubator Program 2016 and announced as Top-Level Apache Project in 2019. It is used as a data orchestration solution, with over 140 integrations and community support.
N/A
JAMS
Score 8.1 out of 10
N/A
JAMS is a centralized workload automation and job scheduling solution that runs, monitors, and manages jobs and workflows. Reliably orchestrate the critical IT processes that run your business from a single pane of glass.
For a quick job scanning of status and deep-diving into job issues, details, and flows, AirFlow does a good job. No fuss, no muss. The low learning curve as the UI is very straightforward, and navigating it will be familiar after spending some time using it. Our requirements are pretty simple. Job scheduler, workflows, and monitoring. The jobs we run are >100, but still is a lot to review and troubleshoot when jobs don't run. So when managing large jobs, AirFlow dated UI can be a bit of a drawback.
JAMS is well suited for centralizing schedules. We have hundreds of databases, and JAMS has allowed us to centralize the scheduling of all of the individual jobs related to each individual database. This allows certain teams to follow the activity of specific jobs without having to access the individual servers on which the databases reside.
Being able to interface / run other programs as either dos window, Power shell window, Python window through configurable execution methods
Purging JAMS history by Parameters, like dates and # of copies to keep of a job
Being able to configure a hot backup to take over in the event a Primary JAMS Server goes down, the backup automatically kicks in after 3 minutes. - High Availability mode
Being able to set up variables that can be used in JOBs / Sequences to hold login information (encrypted), date ranges or other Job Parameter values. And be able to programmatically updated by Power shell scripts to track progress between runs of the same job: count of orders processed, API Skip values when retrieving orders, when the API call only returns X amount per call and you need to get all available orders with multiple calls
JOB logs can be put anywhere on network for other users to access - IT allowing of course. With the high availability option, this is Great, since you don't have to check each of the JAMS Servers for a log when a fail over occures
Prior to JAMS I organized various batch job runs with a user friendly naming process (job, machine, date, etc). I had to jump through hoops to get JAMS to replicate by having a post-completion job rename and copy the log.
Natively support JAMS commands in Powershell7.
Failsafe guard to prevent runaway jobs that submit other JAMS jobs that were improperly coded. i.e. A throttle setting for new JOBS. I once had production go down when hundreds of jobs used up our daily license allocation in under a minute.
I have similar (but different) Execution Methods that would benefit from a shared template. Current setup is awkward to change.
I'd love my job Definitions to be natively linked to github for version control.
JAMS is an essential application within our organization. There is no chance that at any point in the foreseeable future we would migrate off of the platform or stop our maintenance contract. HelpSystems (now rebranded Forta) releases frequent updates which always add value to the product. The system's reliability and extensibility has earned our trust that it will scale with our organization.
At first its not 100% obvious what each item is meant for. When you look at the list for the first time there are a lot of objects, its taken some time to figure out what each one does. Once you figure out what each one does it makes more sense.
JAMS has been super reliable in our implementation. The only reason it isn't a 10 is the web client isn't fully functional so the availability suffers very mildly there
We had a very aggressive timeline to look for a new batch job processor that is compatible with our new server that could help us maintain and improve our current data processing procedure. The JAMS support team was there to address our inquiries promptly. On average, their response times have been within 24 hours, ensuring that we receive assistance when we need it
I Was not part of the original Implementation, and the persons did that are no longer with the Organization. But I was part of the recent Upgrade process a year ago and I am the JAMS admin and was very pleased
There are a number of reasons to choose Apache Airflow over other similar platforms- Integrations—ready-to-use operators allow you to integrate Airflow with cloud platforms (Google, AWS, Azure, etc) Apache Airflow helps with backups and other DevOps tasks, such as submitting a Spark job and storing the resulting data on a Hadoop cluster It has machine learning model training, such as triggering a Sage maker job.
JAMS is easier to use, provides more features and was easier to manage form a central location. simple features in JAMS were missing from VisualCron such as a projected daily schedule. The install setup with a client and agents was better suited for us too
I can only speak in in regards to scalability in the volume of jobs we have created. Many of our jobs exist in multiple environments, with each environment having its own unique folder names, connection strings, etc. We incorporate parameters on the folder level that contain the unique environment information. The jobs reference these values from the folder they are contained in, so we can easily copy a job from DEV to TEST and the source is the same but the values passed from the parameters are not. This makes it very easy to create many new jobs and copy them across multiple environments and have them work
JAMS was not the most cost-effective solution to the original program we were implementing at my Agency; but we were able to expand on the services we do provide using it's power and flexibility to enhance other program areas.
The cost-model from MVPSI has been great in allowing us to deploy a DEVELOPMENT, STAGING, then PRODUCTION scheduler integrating the platform into our SDLC.
