Item: Splunk Enterprise
Rating: 10
Author: Kenneth Taitingfong

Use Cases and Deployment Scope

Splunk is currently the SIEM for IT operations and IT security providing log aggregation and security event correlation for multiple departments. The IT operation groups use Splunk to trend operational data, trouble shoot issues, and send automated alerts when certain triggers are met. The security department utilizes Splunk for investigations and event management, leveraging automated alerts and dashboards. For our organization, Splunk provides the "single pane of glass" for users across several IT departments while also serving as our compliance tool for PCI-DSS and SOX.

Pros and Cons

Splunk is flexible and extensible, able to ingest logs from disparate systems using disparate formats and disparate file types. If the ability exists to make the logs human readable (either natively or via a script), Splunk can ingest it.
Splunk's flexibility in how you parse, format, and enhance your data is amazingly deep. When you start event typing, tagging, aliasing, and creating data models, you start to really open up Splunk's capabilities.
Splunk scales very well in large environments. Adding additional indexers as your environment grows is pretty trivial and its ability to do multi-site clustering and search head clustering provides load balancing and redundancy that's inherent to the product.

Splunk's search language goes very deep. To do some of the more advanced formatting or statistical analysis, there's a bit of a learning curve. Splunk training for learning the search language and manipulating your data can cost anywhere from $500.00 to $1500.00 (although a good number of free training exists).
Splunk's dashboard capabilities are pretty decent but to do more exciting visualizations requires a bit of development using simple XML, Java script, and CSS.
Splunk releases minor revisions very quickly but because of the sheer number of bugs we've run into, we've upgraded our environment four times in nine months.

Return on Investment

Splunk provided immediate results when an Active Directory change was made and our Windows AD team was unable to determine when or who had made the change. We were also able to provide information back to our CIRT for multiple security incidents and correlate what some thought was a DOS attack back to a massive scheduled data download occurring off hours.
Because of Splunk's role in our PCI-DSS compliance requirements, the compliance office is expanding Splunk's role into SOX compliance as well. We're also being asked by multiple departments to be their official system of record for their system logs.
Unfortunately, the decision to virtualize our environment means we're tied to some expensive storage solutions. We are currently facing difficult decisions with regards to data retention due to the cost.
According to our database team, showcasing Splunk's capabilities saved their department $75,000 (USD) when they were able to meet their monitoring needs without buying an additional tool. Our mainframe team is doing a proof of concept with a tool called IronStream that integrates directly with Splunk to provide mainframe monitoring, essentially the only tool in existence to do so. Splunk is also replacing both some end-of-life SCOM tools as well as the soon-to-be EOL Symantec SIEM.

Alternatives Considered

HP Arcsight and IBM Security QRadar

Splunk is certainly much more versatile than either of these three products. Unless ArcSight makes a "connector" for your product, you will be required to use Flex Connectors which is an additional license and apparently requires some serious development. Without Logger, you can't perform free form searches so you must know how your data is being normalized before you can find it.
McAfee Nitro uses Flash which presents a number of challenges itself. During our POC, it also misidentified McAfee Virus Scan Enterprise updates as malware traffic.
QRadar neither excelled in any one place and performed poorly during our POC, but it was unable to be as flexible as Splunk with custom data sources.

Other Software Used

Google Drive

Likelihood to Recommend

Splunk is well suited in both small and very large environments almost regardless of the types of devices. However, depending on how Splunk is architected, it can require a number of devoted engineers to onboard, normalize, and present the data. So for organizations that are unable to-provide dedicated resources, the day-to-day operations and backend duties can be overwhelming. Since Splunk is so flexible, it's easy to overwhelm its available resources when a large number of inefficient searches are running. Splunk users need to be trained to not run "sloppy" searches. The community help forums are a wealth of information but in some cases, without professional support, you're going to be lost. The Splunk licensing can also be costly and in some situations, Splunk virtual environments don't perform well.

Products Replaced

Yes - We replaced Symantec SIEM because it was going EOL.

Key Differentiators

Product Features
Product Usability
Product Reputation
Positive Sales Experience with the Vendor
Analyst Reports
Third-party Reviews

The wide native support for the various products in the environment and the ability to craft technical add-ons to ingest the data sources for which there were no existing technical add-ons.

Evaluation Lessons Learned

A head-to-head between the products using the same data looking for the same event may have been helpful, but not practical.

Usability

You can literally throw in a single word into Splunk and it will pull back all instances of that word across all of your logs for the time span you select (provided you have permission to see that data). We have several users who have taken a few of the free courses from Splunk that are able to pull data out of it everyday with little help at all.

Usability Pros and Cons

Pros	Cons
Like to use Easy to use Technical support not required Well integrated Consistent Quick to learn Convenient Feel confident using	None

Easy Tasks

Searching indexed data is pretty straight forward. You can do it without even really knowing the Splunk Search Language (SPL). Becoming intimately familiar with the SPL means you have a lot of flexibility in presenting and carving up your data how you want it.
Splunk's ability to ingest data using a variety of methods makes Splunk stand out among its competitors. You can stream it directly to Splunk, install a forwarder on a system, used scripted inputs, or even just use WMI for Windows environments.
All of Splunk's configuration files are flat text files which makes editing on the fly a breeze. The individual file specifications are well documented and the community support forum is extremely helpful.

Difficult Tasks

In large environments, you almost need dedicated Splunk engineers that have formal training to administer, onboard data, normalize data, and perform day-to-day operational tasks.
The configuration files can be intimidating. Splunk's flexibility can be a double edged sword. Sometimes finding the right way and the best way to do a specific task isn't very easy.
Sometimes, getting backend performance metrics out of Splunk can be like pulling teeth. While there are a number of Splunk Apps that can provide this information easily, it's not always in the format you want, so learning the SPL is a must.

Splunk - the most flexible SIEM tool on the market.

Overall Satisfaction with Splunk