Threshold Alert Escalation

Normally, an alert is triggered whenever the threshold is breached. In cases where system metrics produce one-time spikes, even though there is no actual system impact, this can result in false alarms.

To improve alert accuracy and reduce false positives:

• The threshold override can be updated to generate alerts only if the issue persists.

• An escalation rule can be configured to trigger a Critical Alert only when the alert count exceeds a defined value within a specific time window.

Thresholds in IT-Conductor have dedicated escalation configurations that allow for complex rule definitions, such as considering prior values, frequency, and persistence of conditions.

This flexibility makes it possible to:

• Differentiate between one-off and recurring issues.

• Ensure only sustained problems trigger escalations.

What are escalation rules in IT-Conductor?

Escalation rules define how alerts progress in severity and priority while a warning or alarm condition persists.

When the threshold override Escalate Alerts checkbox is enabled:

• The Escalation Rules tool becomes visible in the threshold override toolbar.

• Defined escalation rules are applied as long as the warning or alarm condition persists.

• These rules determine how alerts are generated, determining alert escalation level and priority.

Note:

• Alerts with Priority ≥ High and Escalation Level > 0 are not auto-resolved.

• Escalation attributes influence notification priority and incident workflow.

• Subscriptions can target a certain level of alert escalation, preventing notifications on alerts with a lower level of escalation.

Use Case: PagerDuty Integration

In many IT operations environments, IT-Conductor alerts are integrated with PagerDuty to manage on-call notifications.

Scenario

PagerDuty is configured to trigger a call to the SAP Basis team whenever an alert is raised in production systems.

Expectations

• The team wants to be notified only when a critical production issue persists.

• A one-time occurrence should raise only a Warning Alert, not a Critical Alert or PagerDuty incident.

Solution

With IT-Conductor’s escalation rules capability:

• A Warning Alert is generated for one-time events.

• A Critical Alert (which triggers PagerDuty) is only raised if the issue repeats multiple times over a set duration.

Configure Threshold Alert Escalation

To configure threshold alert escalation, start by enabling the Escalation Rules tool, then create a new escalation rule.

Enable Escalation Rules

1. Navigate to the service grid and select the metric to configure with a threshold alert escalation. Click Threshold Overrides, then select Overrides.

Figure 1: Navigating to Page Out Overrides from Service Grid

1. Select the threshold override you want to modify, click Threshold Default Settings, then select Modify.

Figure 2: Selecting Modify to Edit the Page Out Rate Override

1. Tick the Escalate checkbox, then click Save to enable escalation rules.

Figure 3: Enabling Escalation Rules

Once enabled, Escalation Rules will appear in the threshold toolbar.

Note: Make sure to define the threshold Warning Value, and select the desired option in the Alert On dropdown menu. See Alerts for more information.

Create New Threshold Alert Escalation Rule

1. Click Escalation Rules to access the list of existing escalation rules.

Figure 4: Accessing Escalation Rules

1. Click Create New Object to start adding a new escalation rule.

Figure 5: List of Escalation Rules

1. Fill out all the necessary information in the Create Threshold Alert Escalation wizard. Once completed, click to save the configuration.

Figure 6: Create Threshold Alert Escalation Wizard

• Description – refers to a short description for the escalation rule.

• Level – refers to the escalation level applied when the rule is triggered.

• Intervals – refers to the look-back aggregation interval count used for monitoring alert conditions.

• Alert Count – refers to the number of alerts within the defined look-back interval that will trigger escalation.

• Escalation Priority – refers to the priority assigned to the escalation alert.

• Escalate Severity – refers to the severity level set for the escalation alert.

• Escalation Message – refers to the message content displayed when an escalation alert is generated.

Once created, the Escalation Rule is applied to the associated monitor.

Best Practices:

• Review escalation thresholds periodically based on alert trends or environmental patterns.

• Use historical alert data to fine-tune frequency count, time interval, and escalation priority.

• Document and communicate escalation logic with both monitoring and incident response teams.