Look for where you are going to load the data and from where are you extracting it. A lot of those who sell you the DB already have tools for ETL. I have heard good things at an enterprise level from great colleagues about Informatica, but honestly, I prefer SSIS.
BizTalk is well suited as middleware. Where you wish to translate an input file into an output file and send it to some endpoint. In our case, we used it to convert and send files to SAP. In many ways, it very flexible, and you can do almost anything you want with it. In many ways, it's a better solution than your SAP XI or PI as middleware, since it's much less expensive, and allows you do interface with non-SAP systems.
As I mentioned earlier SQL Server Integration Services is suitable if you want to manage data from different applications. It really helps in fetching the data and generating reports. Its automation make it very easy and time efficient. It works well with large database as well. But it doesn't work well with real time data, it will take some time to gather the real time data. I would not recommend using it in a real time/fast-paced environment.
It is very user friendly. Users can change rules during run time and change workflow.
Huge capacity for queueing messages. It supports all types of adapters like Oracle, Salesforce, SMTP, FTP, etc. Also users can built custom adaptors.
If users want to dynamically deploy their solution without any downtime, this is a perfect solution. BizTalk will be a good fit, especially for public-facing websites.
Well-proven in the market. I used it when developing a website for Virgin Trains, catering more than 800K user requests per day.
Connection managers for online data sources can be tricky to configure.
Performance tuning is an art form and trialing different data flow task options can be cumbersome. SSIS can do a better job of providing performance data including historical for monitoring.
Mapping destination using OLE DB command is difficult as destination columns are unnamed.
Excel or flat file connections are limited by version and type.
Some features should be revised or improved, some tools (using it with Visual Studio) of the toolbox should be less schematic and somewhat more flexible. Using for example, the CSV data import is still very old-fashioned and if the data format changes it requires a bit of manual labor to accept the new data structure
SSIS is a great tool for most ETL needs. It has the 90% (or more) use cases covered and even in many of the use cases where it is not ideal SSIS can be extended via a .NET language to do the job well in a supportable way for almost any performance workload.
SQL Server Integration Services performance is dependent directly upon the resources provided to the system. In our environment, we allocated 6 nodes of 4 CPUs, 64GB each, running in parallel. Unfortunately, we had to ramp-up to such a robust environment to get the performance to where we needed it. Most of the reports are completed in a reasonable timeframe. However, in the case of slow running reports, it is often difficult if not impossible to cancel the report without killing the report instance or stopping the service.
BizTalk Server has been supported for more than 15 years. It is well proven in the market. Microsoft has provided excellent support with technical issues.
The support, when necessary, is excellent. But beyond that, it is very rarely necessary because the user community is so large, vibrant and knowledgable, a simple Google query or forum question can answer almost everything you want to know. You can also get prewritten script tasks with a variety of functionality that saves a lot of time.
The implementation may be different in each case, it is important to properly analyze all the existing infrastructure to understand the kind of work needed, the type of software used and the compatibility between these, the features that you want to exploit, to understand what is possible and which ones require integration with third-party tools
BizTalk was selected here mainly because it is easy to integrate to a .NET application (most of them are Web Service, WCF SOAP, WCF REST and Web API) and many backend databases are Microsoft SQL Server. Another benefit is that the monitoring job is easy to set up and centralize with other .NET application monitoring jobs.
I think SQL Server Integration Services is better suited for on-premises data movement and ADF is more suited for the cloud. Though ADF has more connectors, SQL Server Integration Services is more robust and has better functionality just because it has been around much longer
A positive impact has been the quicker turnaround time of a part request and that part showing up in SAP using Biztalk as middleware.
A somewhat negative impact has been the somewhat insufficient error logging/message capture settings that Biztalk provide. This has caused occasional delays when attempted to create parts for the business.
A somewhat negative impact has been the need to have a specialized developer who understands Biztalk to troubleshoot issues with the Biztalk and SAP interaction when creating parts, and when adding new fields to the parts.
Without this, we would have to manually update a spreadsheet of our SQL Server inventory
We would also have poor alerting; if an instance was down we wouldn't know until it was reported by a user
We only have one other person who uses SQL Server Integration Services , he's the expert. It would fall to me without him and I would not enjoy being responsible for it.
