How to Fine Tune Your IncidentHub Alerts

Introduction

IncidentHub can send outage alerts to many external systems. You can choose from Slack, Webhook, Email, Discord, PagerDuty, and more. Alerts are effective only when they are relevant and actionable. In this article, we will explore how to fine-tune your IncidentHub alerts to receive only the relevant ones for your third-party services.

• Introduction
• What Are Relevant and Actionable Alerts?
• Fine-Tuning Your IncidentHub Alerts
  • Component Filtering
  • Component Auto-Detection
  • Alert Notifications – Beginning, End, or Everything?
• Best Practices
• Conclusion
• References
• FAQ

What Are Relevant and Actionable Alerts?

For third-party services, an alert is relevant if it directly affects your business in some way. The third-party could be a cloud service, a SaaS application, or a payment gateway. An actionable alert is one that you can do something about. To keep your alerts relevant and actionable, you can use this checklist:

• Your applications/business should directly use the service or product that the alert originates from. For example, if you use Google Kubernetes Engine (GKE) in the us-central1 region, an alert from GKE in the us-central1 region is relevant. Alerts from GKE in other regions, or from other Google Cloud products, are not relevant.
• Some third-party services are more critical than others for your business. You would want to closely follow every outage update until it is resolved in a business-critical service. For not so critical services, it is okay to receive alerts only when the outage start and ends.

Irrelevant alerts just add to your already crowded list of notifications and cause alert fatigue. Let's see how to use IncidentHub's fine-tuning features to keep your alerts relevant.

Fine-Tuning Your IncidentHub Alerts

Component Filtering

Global services like Google Cloud and Microsoft Azure have many constituent services or components, and they are spread across different regions and zones. Your business most likely uses only a subset of these components. Here is how to filter alerts from the components you are interested in.

When adding a new service, you can select "Monitor Specific Components" and choose the ones you need. Some components might have sub-components.

In the example above, if you choose "App Platform" -> Amsterdam, London, and New York, you will receive alerts only when there is an outage in Amsterdam, London, or New York for App Platform.

You can add change the monitored components for a service at any time by clicking on "Edit" next to the service.

Component Auto-Detection

Some services have a large number of components. Choosing the components manually is a chore, and there is also the risk of missing something that you forgot about. To mitigate this, IncidentHub can auto-detect the components for you using the invoice for your cloud provider uploaded as a CSV file.

We don't store your billing data. It is only used during the auto-detection process. You can also remove the financial information from the CSV file before uploading it - IncidentHub needs only the service names.

As of this writing, this feature is in beta and available for Google Cloud Platform only. We are rolling it out to other services soon.

Alert Notifications – Beginning, End, or Everything?

An outage goes through multiple stages - beginning, in progress, and resolved. The middle stage has one or more, sometimes many, updates - sent out by the service provider's team as they investigate and mitigate the issue. Depending on how critical the service is for your business, you might want to be notified only when the outage starts, or when it ends, or for everything.

You can fine tune this behavior for each monitored service.

This setting is per-service, and the default is to notify for everything.

Let me give you an example from IncidentHub itself. We use Paddle for our subscription billing. It's business-critical for us. We want to be notified for every update from Paddle outages and maintenance events. This helps us to be on top of issues and respond to customer queries quickly.

In contrast, we use Grafana Cloud as an external store for our logs. It is not business-critical and we are okay to miss the intermediate notifications as long we know an outage is ongoing or has been resolved. A Grafana Cloud outage does make it difficult to debug production issues, but we can temporarily fallback to our cloud hosting provider's log viewer.

Best Practices

To get the most out of IncidentHub's alerting, you can follow these recommendations:

• Revisit your components for your services periodically to ensure they are up to date. Add or remove component filters as needed.
• Check your alert notification configuration once in a while - this might change depending on your business needs.
• If you or your team find yourself ignoring certain alerts, it's a sign that it's too noisy, and you need to reconfigure them.

Conclusion

IncidentHub's alert fine-tuning features are designed to make alerts more meaningful and prevent alert fatigue. Use them to focus on what matters to your business.

You can sign up for a free (forever) account at incidentHub.cloud.

References

• Component Filtering documentation
• Component Auto-Detection documentation
• Alert Notifications documentation
• More about component filtering

FAQ

What makes a third-party service alert relevant?

A third-party service alert is relevant if it directly impacts your business, such as an outage in a specific service or region your applications rely on.

What makes a third-party service alert actionable?

A third-party service alert is actionable if you can take steps to address it, helping you respond effectively to the outage.

How can I filter IncidentHub alerts by components?

When adding or editing a service in IncidentHub, select "Monitor Specific Components" and choose the components you want alerts for.

What is Component Auto-Detection?

This beta feature (currently for Google Cloud Platform) auto-detects components from your cloud provider’s invoice (CSV file) to simplify setup. Billing data isn’t stored.

Can I choose for which updates in an outage IncidentHub alerts me?

Yes, per-service settings let you choose notifications for outage start, end, or all updates, based on the service’s criticality to your business.

How can I avoid alert fatigue?

Filter out irrelevant alerts by selecting specific components and adjusting notification settings to focus on what matters.

What are some best practices for using IncidentHub alerts?

1. Periodically review monitored components.
2. Adjust notification settings as business needs evolve.
3. Reconfigure alerts if you're ignoring them due to noise.

IncidentHub is not affiliated with any of the services and vendors mentioned in this article.