Splunk ITSI in Practice

We are using Splunk IT Service Intelligence (ITSI) as the centerpiece of our Observability strategy for multiple product lines that provide interactive television services. It helps us to ensure the proper functionality of our services and the surrounding ecosystem as well as reducing mean time to service restoration when outages occur. Our Splunk ITSI system observes telemetry from our data center and clouds infrastructures as well as telemetry collected from our customer media consumption endpoint software on set-top boxes, IPTV streamers, mobile devices and web browsers.

Splunk ITSI's Event Analytics system has the ability to bring together many, disparate sources of information that all relate to the same service or functionality in our ecosystem. Using all of this data and the multiple perspectives that it provides allows Splunk ITSI to more accurately detect real issues from false positives and to provide better context when issues arise. These centralized, combined views reduce alert fatigue and aid in the understanding of our components and their state. Additionally, by building our operations practice around standard methods of data collection, incident detection and reporting, and common incident response tools and processes, we have improved our service delivery. Our teams share and action with the same data and tools, whether an incident is a low-level network issue or a high-level application bug. These standardized tools and processes improve our communication and actioning at just the time that such is most valuable...in the heat of battle while we are mitigating incidents.

We have liberally replaced older monitoring and alerting based on engineer-selected fixed thresholds with Splunk ITSI's adaptive thresholds and anomaly detection. No other single action in our monitoring and observability work has had a more significant impact on our success, in multiple ways, including: fewer false positives, less alert fatigue, and less operational maintenance and upkeep. We do not even use Splunk ITSI's Predictive Analytics yet, but we hope to on-board the feature soon!

Splunk ITSI summarizes numerous disparate data sources about the operation of our product lines. This information is useful minute by minute in our operation of our services, but we also use it in a more longitudinal fashion, in monthly reports, to identify components and services that deserve additional investment due to lower SLO (Service Level Objective) scores.