Apache Spark vs. SQL Server Integration Services

Well suited: To most of the local run of datasets and non-prod systems - scalability is not a problem at all. Including data from multiple types of data sources is an added advantage. MLlib is a decently nice built-in library that can be used for most of the ML tasks. Less appropriate: We had to work on a RecSys where the music dataset that we used was around 300+Gb in size. We faced memory-based issues. Few times we also got memory errors. Also the MLlib library does not have support for advanced analytics and deep-learning frameworks support. Understanding the internals of the working of Apache Spark for beginners is highly not possible.

Incentivized

There are always alternative options available to meet the demand for integration. In my opinion, SQL Server Integration Services has a wide variety of capabilities that makes it a very versatile tool for developing dependable integration strategies. When determining which tools to utilize, vendor interfaces may play a significant role, and technologies like PowerShell have been used by colleagues to aid in this decision. For even more user-friendliness, our SSIS solution additionally makes use of a third-party plugin.

Incentivized

• Apache Spark makes processing very large data sets possible. It handles these data sets in a fairly quick manner.
• Apache Spark does a fairly good job implementing machine learning models for larger data sets.
• Apache Spark seems to be a rapidly advancing software, with the new features making the software ever more straight-forward to use.

Incentivized

• Standard ETL use cases for daily loads
• Loading incoming data from Vendors which is placed on FTP and adding them to the SQL Warehouse
• Creating outgoing data files and writing them to Vendor FTPs
• Easy Active Directory integration for seamless connections to SQL Server
• CI/CD by hosting the code on visualstudio.com

Incentivized

• Memory management. Very weak on that.
• PySpark not as robust as scala with spark.
• spark master HA is needed. Not as HA as it should be.
• Locality should not be a necessity, but does help improvement. But would prefer no locality

Incentivized

• Connection managers for online data sources can be tricky to configure.
• Performance tuning is an art form and trialing different data flow task options can be cumbersome. SSIS can do a better job of providing performance data including historical for monitoring.
• Mapping destination using OLE DB command is difficult as destination columns are unnamed.
• Excel or flat file connections are limited by version and type.

Incentivized

Capacity of computing data in cluster and fast speed.

Some features should be revised or improved, some tools (using it with Visual Studio) of the toolbox should be less schematic and somewhat more flexible. Using for example, the CSV data import is still very old-fashioned and if the data format changes it requires a bit of manual labor to accept the new data structure

Incentivized

The only thing I dislike about spark's usability is the learning curve, there are many actions and transformations, however, its wide-range of uses for ETL processing, facility to integrate and it's multi-language support make this library a powerhouse for your data science solutions. It has especially aided us with its lightning-fast processing times.

Incentivized

SSIS is a great tool for most ETL needs. It has the 90% (or more) use cases covered and even in many of the use cases where it is not ideal SSIS can be extended via a .NET language to do the job well in a supportable way for almost any performance workload.

Incentivized

SQL Server Integration Services performance is dependent directly upon the resources provided to the system. In our environment, we allocated 6 nodes of 4 CPUs, 64GB each, running in parallel. Unfortunately, we had to ramp-up to such a robust environment to get the performance to where we needed it. Most of the reports are completed in a reasonable timeframe. However, in the case of slow running reports, it is often difficult if not impossible to cancel the report without killing the report instance or stopping the service.

Incentivized

1. It integrates very well with scala or python. 2. It's very easy to understand SQL interoperability. 3. Apache is way faster than the other competitive technologies. 4. The support from the Apache community is very huge for Spark. 5. Execution times are faster as compared to others. 6. There are a large number of forums available for Apache Spark. 7. The code availability for Apache Spark is simpler and easy to gain access to. 8. Many organizations use Apache Spark, so many solutions are available for existing applications.

The support, when necessary, is excellent. But beyond that, it is very rarely necessary because the user community is so large, vibrant and knowledgable, a simple Google query or forum question can answer almost everything you want to know. You can also get prewritten script tasks with a variety of functionality that saves a lot of time.

Incentivized

The implementation may be different in each case, it is important to properly analyze all the existing infrastructure to understand the kind of work needed, the type of software used and the compatibility between these, the features that you want to exploit, to understand what is possible and which ones require integration with third-party tools

Incentivized

All the above systems work quite well on big data transformations whereas Spark really shines with its bigger API support and its ability to read from and write to multiple data sources. Using Spark one can easily switch between declarative versus imperative versus functional type programming easily based on the situation. Also it doesn't need special data ingestion or indexing pre-processing like Presto. Combining it with Jupyter Notebooks (https://github.com/jupyter-incubator/sparkmagic), one can develop the Spark code in an interactive manner in Scala or Python

Incentivized

I had nothing to do with the choice or install. I assume it was made because it's easy to integrate with our SQL Server environment and free. I'm not sure of any other enterprise level solution that would solve this problem, but I would likely have approached it with traditional scripting. Comparably free, but my own familiarity with trad scripts would be my final deciding factor. Perhaps with some further training on SSIS I would have a different answer.

Incentivized

• Faster turn around on feature development, we have seen a noticeable improvement in our agile development since using Spark.
• Easy adoption, having multiple departments use the same underlying technology even if the use cases are very different allows for more commonality amongst applications which definitely makes the operations team happy.
• Performance, we have been able to make some applications run over 20x faster since switching to Spark. This has saved us time, headaches, and operating costs.

Incentivized

• Data integrity across various products allows unify certain processes inside the organization and save funds by reducing human labour factor.
• Automated data unification allows us plan our inputs better and reduce over-warehousing by overbuying
• The employee number, responsible for data management was reduced from 4 to 1 person

Incentivized