Incident management with Netdata

Increasingly, organizations rely on complex IT infrastructure for their operations. This dependency requires them to consider IT operations as part of their overall business continuity and resiliency planning processes. To ensure that organizations can actively prevent, manage, and recover from unforeseen outages, they must have real-time visibility into incidents as they occur.

Incident management is the process of identifying, analyzing, responding to, and preventing unplanned events or service disruptions with the goal of returning business to its normal, operational state. DevOps and IT operations teams often manage this function for their organizations and rely on ITIL or ITSM best practices. To consider an incident resolved, IT teams must create workflows to mitigate the impact of the anomaly and restore operations. The time needed to detect and resolve the incident are often key metrics by which operations teams are measured (mean time to detection or MTTD, and mean time to resolution or MTTR).

In today’s complex, dynamic environments that span on-premises to cloud deployments consisting of myriad systems and applications, IT teams are faced with unprecedented challenges in gaining visibility into and control over their infrastructure. The need to quickly identify and troubleshoot incidents when they occur has never been greater.

Incident management is a process that relies on best practices. At its core, incident management and response relies on:

• Detection – Monitoring and alerting is available to detect incidents before end users do.
• Mitigation – Teams put a plan in action to contain the service disruption while a resolution is identified.
• Recovery – A resolution has been found and put into place, with or without a root cause.
• Post mortem – The team learns from the incident and changes course (ideally, if a root cause has been determined, with a plan to mitigate this type or class of incident from occurring again).

Netdata can play the lead role in your detection, mitigation, and root cause analysis processes. Netdata is built to provide per-second metrics from hundreds of systems and applications in real time, key to detecting anomalies and incidents as they are happening. Unlike other tools, Netdata requires zero configuration, eliminating the need to predetermine metrics that you want to collect or visualize before an incident actually happens. This means that you have unlimited visibility into every possible metric from every system or application across your infrastructure instantly, making it much easier to detect unusual events or performance impacts. And since Netdata doesn’t backhaul or centralize metrics to a complex data lake, it’s infinitely scalable, putting no limits on what you can monitor or troubleshoot.

Netdata’s built-in health watchdog simultaneously analyzes metrics every second for anomalous behavior using preconfigured alarms. There’s no wasted time coding or worrying about thresholds and hysteresis. It’s easy to view active health alarm status for each node across the infrastructure to take appropriate action by drilling down into the related chart.

Netdata’s health watchdog is highly configurable, with turnkey support for dynamic thresholds, hysteresis, alarm templates, and more. You can tweak any of the existing alarms based on your infrastructure’s topology or specific monitoring needs, or create new entities. And with dozens of integrations to popular notification platforms like Opsgenie, PagerDuty, Slack, and Twilio, Netdata makes it easy to work within your existing toolchain.

Netdata Cloud brings teams and real-time, granular data together in one place to make troubleshooting faster and easier than ever before. By automatically categorizing and organizing thousands of collected metrics, then presenting them in hundreds of composite charts on a single screen, Netdata gives you everything you need to identify and resolve incidents quickly.

Key features that work out-of-the-box with Netdata means that you can spend time monitoring and troubleshooting your infrastructure, not pre-planning, configuring, and deploying tooling and platforms.

• Metric Correlations help you automatically identify all key metrics related to an incident to help you identify the root cause more quickly and easily.
• Customize your dashboard view with composite charts to understand how metrics relate to each group of nodes.
• Get a better understanding of how changes or deployments impact system and application performance to proactively prevent future incidents.
• Drill down on active alarms to relevant charts to view real-time and historical metrics to understand anomalous behavior. Compare impacted services with other healthy nodes to assist in root cause analysis.
• Invite team members to collaborate in your Spaces and War Rooms. Teams can work independently in parallel with their own data and workflows for easy organization.
• Click on the Pin button in any dashboard to put those charts into a separate panel at the bottom of the screen. You can now navigate through Netdata Cloud freely—individual Cloud dashboards, the Nodes view, different War Rooms, or even different Spaces—and have those valuable metrics follow you.
• It’s also easy to add text annotations to your dashboards as well, making it easier for team members to understand what’s going on quickly. The Add Text button creates a new card with user-defined text, which you can use to describe or document a particular dashboard’s meaning and purpose.

The key to driving down incident management time to resolution is helping teams collaborate with the information they need at hand to troubleshoot faster. Netdata makes it straightforward to pursue a philosophy of continuous improvement with custom dashboards, alarms and notifications; collaboration spaces designed for workflow organization; and granular, real-time data meaningfully visualized and correlated.