The State of Big Data: A Guide for 2018

Big data is big capital in today’s tech markets. Technological advances that make it cheaper and easier than ever to store and process large quantities of digital information. Thanks to these advances, the analysis of big data, or data analytics, is a flourishing sector of the tech economy. Many products and services offered by tech companies rely on their collection and analysis of huge, complex data sets from increasingly diverse sources.

Big data is conventionally defined in terms of the three Vs: a greater variety of data arriving in increasing volumes with ever-higher velocity. It is important to understand these three Vs a little more in depth. Why? In order to understand what big data companies have to offer and which ones to watch in 2018.

1. Variety

Whereas traditional data was neatly structured, big data is unstructured or semi-structured, as from text, audio, and video sources. What this means is that these and other new data types require a preprocessing that traditional data did not. Data from sensors, tweets, encrypted packets, audio recordings, photographs, e-mail, books, presentations, and documents, and so on are all qualitatively different from one another. They do not all fit neatly or in the same way into database software or spreadsheets.

Capturing, organizing, and analyzing such a robust array of digital information means contending with a degree of diversity not seen before the rise of big data.

2.Volume

Another way that big data differs is simply in straightforward quantitative terms. Web page or mobile app clickstreams, Twitter data feeds, and similar sources generate a lot of unstructured, low-density digital information that needs to be processed to be made intelligible. “A lot” can mean upwards of hundreds of petabytes.

Facebook provides a suitably face-melting illustration of just how big big data can get. To put it in context, the social media mega-site’s users outnumber the population of China. It stores roughly a quarter of a trillion images, and in the old days of 2016, it had 2.5 trillion posts. That’s big even by astronomical standards. And as a result, with every passing year, the numbers get bigger as our mobile world of increasingly connected devices and apps grows by leaps and bounds.

3. Velocity

Lastly, velocity means the rate at which data is coming in. To return to the case of Facebook, its users upload more than 900 million photos per day. The torrent of sensor data coursing through the internet daily provides another example of what velocity means in the context of big data. The flow of data from sensors keeps pace with growth in the number of them out there in the world.

Big Data Companies to Watch

The IT industry is full of companies who specialize in deriving value from the bigger, faster, and more diverse data that is out there. These services are referred to as data management or data analytics, and the competitive field is pretty big in itself. Last year the exploding global market for big data services and tech reached $21.19 billion and will grow to $77.58 billion by 2023.

After all, big data affects organizations across many industries, from banking and education to healthcare, government, manufacture, and retail. Given this bewildering complexity, it can be difficult to remain aware of who is doing what. More importantly, its crucial to know who does what best. Here (in no particular order) is a list of big data companies to keep on your radar in 2018. We have organized them into four categories according to the services provided.

1. Business Analytics

Business analytics is the use of statistical analysis, predictive modeling, online analytical processing (OLAP), and other iterative, data-analytical techniques to understand past business performance in order to drive future business planning. It is an essential tool for companies committed to data-driven decision-making.

Data analytics is really an extension of business intelligence for the big data age. There has been an explosion of new technology in this area in the last few years from in-memory processing, to Hadoop data lakes, to NoSQL databases.

The tools of business analytics are foremost in the arsenal needed to cope with and derive value from big data. Computer Reseller News’ 2018 Big Data 100 list of companies is a reliable, independent source of guidance and perspective for the IT market. Their picks for the best companies in the area of business analytics include:

• Alteryx, whose platform integrates predictive models into business apps and combines data discovery, analysis, and management.
• Arcadia Data, a London-based source of business intelligence and unified visual analytics software that can capture and integrate big data from the cloud and data lakes.
• ClearStory Data offers a data discovery system in the cloud that uses the Apache Spark engine and (as of March 2018) an AI-based system for decomposing and handling complex, blended data.
• Cooladata is a behavioral analytics platform for mobile and web applications that offers users flexible access and unifies data sources from websites, online marketing campaigns, in-house databases, CRM (customer relationship management) systems, and mobile apps. Overall, Cooladata’s products and services are of special interest to businesses who want to increase sales by better understanding their customer base in order to extend their reach and enhance their model.

2. Data Management and Integration

Lots of different tools are needed to make the big data world go ‘round. But it would grind to a halt without systems capable of hoovering and cleaning up, then integrating, transforming, and organizing big data in the first place. Here are some highlights from CRN’s top data management picks for 2018:

• Palo Alto, CA-based company Actian uses a proprietary combination of data analytics, integration, and management services to derive business value from big data. Users have raved about the ability of its products to solve problems around data mining and crunching. Actian Matrix, for instance, aggregates and analyzes data from multiple sources in a shared repository.
• For cataloging an automated inventory of your company’s data assets, and rapidly querying them for actionable business intelligence, Redwood City, CA-based Alation is highly rated.
• Attunity is a company whose strengths revolve around three core products: Replicate, designed for data ingest and replication; Compose, used to facilitate data warehouse construction; and Visibility, for usage and visibility of your big data.
• Israeli startup Iguazio recently unveiled a unified data turnkey system, the Continuous Data Platform, that turbo-charges the development of data-driven apps. They do this by collecting and analyzing big data from many sources. Iguazio touts its in-memory performance at high density and lower costs.

3. Big Data Platforms

The foundation of big data tech consists of the platforms and systems that live in the cloud or on-premise and on which big data tools, applications, and initiatives operate. Here are a handful of companies supplying these basic workhorses of big data that are turning heads in 2018:

• Hortonworks is best know as a Hadoop distribution vendor, but is a leading innovator with a range of open-source software designed to manage data and processing for things such as IOT and other big data applications.  If your company wants to collect and analyze data in motion, consider their real-time analytics software Hortonworks DataFlow.
• If your data analysis needs include enterprise text, speech, and video analytics, the IDOL software by vendor Micro Focus, may be a good fit.  
• Teradata, a pioneer in data warehouse systems, is competitive in 2018 as a provider of database and analytics products and services. Their product portfolio includes the Teradata Analytics Platform, Database, IntelliCloud analytic software which lives on the cloud. Another product is their Intelliflex integrated data warehouse system. Teradata’s database application has been praised for its scalability, adaptability, and performance in easily processing large amounts of data across entire organizations.  

4. Data Science and Machine Learning

Data science tools are capable of handling data volumes that are too big for traditional databases or statistical tools.

The skill sets required for true data science are in short supply. This puts pressure on tool developers to reduce complexity to increase the potential user pool. In general, data scientists have advanced analytics skills like actuaries, who calculate insurance risks and premiums. They are quantitative researchers who typically have strong programming skills.

A number of emerging companies have already made some noise in this space:

• Boston-based data science company DataRobot specializes in products that automate machine learning to enable companies to build strong predictive analytics without having to employ a team of data scientists.
• An open-source option is available from H2O.ai, whose machine learning tools include the recently-launched Driverless AI, which facilitates data modeling through visualization features while automating ML.
• Bay Area company Splunk, which acquired a number of ML companies such as SignalSense and Rocana in 2018, has revealed its first product designed for applications around industrial Internet of Things.
• Corporate data science training and placement services from The Data Incubator help organizations to hire top data science talent along with upskilling their current teams in the latest data science tools and techniques.
• Finally, data scientists, executives, and IT managers might consider Domino Data Lab’s product portfolio. Domino’s predictive analytics software offers publishing tools and a collaboration hub suitable for use by data science teams.

The world of big data keeps getting bigger and will continue to do so beyond 2018. With some of these leads in your back pocket, your company will enjoy a head start in the search for smarter big data solutions.