Apache Spark vs. TensorFlow

Apache Spark has rich APIs for regular data transformations or for ML workloads or for graph workloads, whereas other systems may not such a wide range of support. Choose it when you need to perform data transformations for big data as offline jobs, whereas use MongoDB-like distributed database systems for more realtime queries.

Incentivized

1. Whenever the problem has the demand for a neural networks based solution, Tensorflow (TF) is a great fit.
2. The tf.dataset API makes it really simple to create complex data pipelines in a few lines of code.
3. tf.estimators API abstracts all the complex computation graph creation logic making it very simple to get started.
4. Eager execution makes it simple to develop a TF graph as debugging the code would be like any other imperative Python program.
5. TF abstracts all the complexities of scaling it to multiple machines. It has various code and data distribution algorithms ready to use.
6. Projects like TensorBoard make monitoring the training process really easy. It also gives the ability to view embeddings without any extra code. Their What-If is extremely useful for poking and understanding a black box model. It also has tools to visualize data to quickly check for anomalies.
7. TF Autograph aims to covert any normal Python code into a distributed program which is quite handy to scale an existing code base.

Incentivized

• It performs a conventional disk-based process when the data sets are too large to fit into memory, which is very useful because, regardless of the size of the data, it is always possible to store them.
• It has great speed and ability to join multiple types of databases and run different types of analysis applications. This functionality is super useful as it reduces work times
• Apache Spark uses the data storage model of Hadoop and can be integrated with other big data frameworks such as HBase, MongoDB, and Cassandra. This is very useful because it is compatible with multiple frameworks that the company has, and thus allows us to unify all the processes.

Incentivized

• Data pipeline implementation is quite good, loading large amounts of data and pre-process it in an efficient way is no more issue for us
• It supports all major DL algorithms and network layouts such as ConvNets, RNN, LSTMs, Word2Vec, and even the latest transformer architecture
• The abstraction for the device is perfectly done and its support seamlessly for multiple GPU and even TPU will bring a lot of performance gain for enterprise scoped solution while still keep the flexibility
• The TensorBoard is amazing. I haven't seen a similar thing in other frameworks on the market. It allows us to quickly understand and debug the model with the info visualization which makes understanding much better
• A very supportive community, which is the key for sharing the ideas and find the quick and best solutions

Incentivized

• Memory management. Very weak on that.
• PySpark not as robust as scala with spark.
• spark master HA is needed. Not as HA as it should be.
• Locality should not be a necessity, but does help improvement. But would prefer no locality

Incentivized

• It would be much better if they could provide good documentation and easy ways to understand concepts.
• It is difficult to understand the concept behind for example, Tensor Graph, which takes a lot of time.
• As you have to write everything, it is time consuming to write the implementation of whole neural network. It would be better if they can provide some wrapper library to make things easier.

Incentivized

Capacity of computing data in cluster and fast speed.

If the team looking to use Apache Spark is not used to debug and tweak settings for jobs to ensure maximum optimizations, it can be frustrating. However, the documentation and the support of the community on the internet can help resolve most issues. Moreover, it is highly configurable and it integrates with different tools (eg: it can be used by dbt core), which increase the scenarios where it can be used

Incentivized

Support of multiple components and ease of development.

Incentivized

1. It integrates very well with scala or python. 2. It's very easy to understand SQL interoperability. 3. Apache is way faster than the other competitive technologies. 4. The support from the Apache community is very huge for Spark. 5. Execution times are faster as compared to others. 6. There are a large number of forums available for Apache Spark. 7. The code availability for Apache Spark is simpler and easy to gain access to. 8. Many organizations use Apache Spark, so many solutions are available for existing applications.

Community support for TensorFlow is great. There's a huge community that truly loves the platform and there are many examples of development in TensorFlow. Often, when a new good technique is published, there will be a TensorFlow implementation not long after. This makes it quick to ally the latest techniques from academia straight to production-grade systems. Tooling around TensorFlow is also good. TensorBoard has been such a useful tool, I can't imagine how hard it would be to debug a deep neural network gone wrong without TensorBoard.

Incentivized

Use of cloud for better execution power is recommended.

Incentivized

We used Surprise Kit for one of the other research works. It is more fine-tuned to Recommendation systems and their algorithms. Apache Spark has MLlib for majority of ML problems. Where as software like Surprse Kit - it suitable for a specific task of Recommendations only

Incentivized

Can't seem to choose any deep learning platform in the above, so I'll list it here: 1. Apache MXNet: this has been used for one of our main algorithms for search as an end-to-end pipeline. We chose this because of the Scala bindings, which makes it easier to integrate with out JVM backend. MXNet seems comparable to TensorFlow, although community support is not as good as TensorFlow, and there are issues with memory leaks that are being worked on. TensorFlow in general is easier to use, but MXNet isn't too far behind. 2. Keras: still a favorite. Often I use this when paired with TensorFlow. TensorFlow 2.0 will make it even easier. 3. PyTorch: only used it a little, so it's hard to provide a good opinion. 4. DL4J: used it initially in an early days project because it has good JVM support. Harder to used not because of poor API design, but because community support is lacking and features don't come out as fast as TensorFlow.

Incentivized

• Faster turn around on feature development, we have seen a noticeable improvement in our agile development since using Spark.
• Easy adoption, having multiple departments use the same underlying technology even if the use cases are very different allows for more commonality amongst applications which definitely makes the operations team happy.
• Performance, we have been able to make some applications run over 20x faster since switching to Spark. This has saved us time, headaches, and operating costs.

Incentivized

• Positive Impact- As I mentioned before its open source. Very easy to learn for average programmer/ developer. We were able to design a POC model for understanding the patient appointment cancellation snd reasons behind it in 3 week time frame.
• Negative Impact- If you are using tensor flow for small project it works fine. If you are trying to build a model for face recognition it will be hard to program and train the system. It needs data to be processed before hand cannot learn on the go.