Databricks in San Francisco offers the Databricks Lakehouse Platform (formerly the Unified Analytics Platform), a data science platform and Apache Spark cluster manager. The Databricks Unified Data Service aims to provide a reliable and scalable platform for data pipelines, data lakes, and data platforms. Users can manage full data journey, to ingest, process, store, and expose data throughout an organization. Its Data Science Workspace is a collaborative environment for practitioners to run…
$0.07
Per DBU
SSIS
Score 7.9 out of 10
N/A
Microsoft's SQL Server Integration Services (SSIS) is a data integration solution.
SSIS is similar to Alteryx and Informatica PowerCenter in a way because these are all drag-and-drop ETL tools with similar functionality. Alteryx is a step ahead because it has some advanced ETL functionalities including statistical calculations etc. and a better ability to set …
Medium to Large data throughput shops will benefit the most from Databricks Spark processing. Smaller use cases may find the barrier to entry a bit too high for casual use cases. Some of the overhead to kicking off a Spark compute job can actually lead to your workloads taking longer, but past a certain point the performance returns cannot be beat.
Ideal for daily standard ETL use cases whether the data is sourced from / transferred to the native connectors (like SQL Server) or FTP. Best if the company uses MS suite of tools. There are better options in the market for chaining tasks where you want a custom flow of executions depending on the outcome of each process or if you want advanced functionality like API connections, etc.
Connect my local code in Visual code to my Databricks Lakehouse Platform cluster so I can run the code on the cluster. The old databricks-connect approach has many bugs and is hard to set up. The new Databricks Lakehouse Platform extension on Visual Code, doesn't allow the developers to debug their code line by line (only we can run the code).
Maybe have a specific Databricks Lakehouse Platform IDE that can be used by Databricks Lakehouse Platform users to develop locally.
Visualization in MLFLOW experiment can be enhanced
SSIS has been a bit neglected by Microsoft and new features are slow in coming.
When importing data from flat files and Excel workbooks, changes in the data structure will cause the extracts to fail. Workarounds do exist but are not easily implemented. If your source data structure does not change or rarely changes, this negative is relatively insignificant.
While add-on third-party SSIS tools exist, there are only a small number of vendors actively supporting SSIS and license fees for production server use can be significant especially in highly-scaled environments.
Some features should be revised or improved, some tools (using it with Visual Studio) of the toolbox should be less schematic and somewhat more flexible. Using for example, the CSV data import is still very old-fashioned and if the data format changes it requires a bit of manual labor to accept the new data structure
Because it is an amazing platform for designing experiments and delivering a deep dive analysis that requires execution of highly complex queries, as well as it allows to share the information and insights across the company with their shared workspaces, while keeping it secured.
in terms of graph generation and interaction it could improve their UI and UX
SQL Server Integration Services is a relatively nice tool but is simply not the ETL for a global, large-scale organization. With developing requirements such as NoSQL data, cloud-based tools, and extraordinarily large databases, SSIS is no longer our tool of choice.
Raw performance is great. At times, depending on the machine you are using for development, the IDE can have issues. Deploying projects is very easy and the tool set they give you to monitor jobs out of the box is decent. If you do very much with it you will have to write into your projects performance tracking though.
One of the best customer and technology support that I have ever experienced in my career. You pay for what you get and you get the Rolls Royce. It reminds me of the customer support of SAS in the 2000s when the tools were reaching some limits and their engineer wanted to know more about what we were doing, long before "data science" was even a name. Databricks truly embraces the partnership with their customer and help them on any given challenge.
The support, when necessary, is excellent. But beyond that, it is very rarely necessary because the user community is so large, vibrant and knowledgable, a simple Google query or forum question can answer almost everything you want to know. You can also get prewritten script tasks with a variety of functionality that saves a lot of time.
The implementation may be different in each case, it is important to properly analyze all the existing infrastructure to understand the kind of work needed, the type of software used and the compatibility between these, the features that you want to exploit, to understand what is possible and which ones require integration with third-party tools
The most important differentiating factor for Databricks Lakehouse Platform from these other platforms is support for ACID transactions and the time travel feature. Also, native integration with managed MLflow is a plus. EMR, Cloudera, and Hortonworks are not as optimized when it comes to Spark Job Execution. Other platforms need to be self-managed, which is another huge hassle.
I had nothing to do with the choice or install. I assume it was made because it's easy to integrate with our SQL Server environment and free. I'm not sure of any other enterprise level solution that would solve this problem, but I would likely have approached it with traditional scripting. Comparably free, but my own familiarity with trad scripts would be my final deciding factor. Perhaps with some further training on SSIS I would have a different answer.
