6 posts tagged with "alerting"

Introduction​

This article is part of a series on setting up an end-to-end monitoring and alerting stack using Prometheus.

Continuing our series on setting up Prometheus in a container, this article provides a step-by-step guide for how to configure alerts in Prometheus. We will add alerting rules and deploy Prometheus Alertmanager with Slack integration.

If you follow the steps in this article, you will end up with a containerized setup for:

1. A Prometheus instance with alerting rules.
2. An Alertmanager instance which can send alerts originating from those rules to a Slack channel.

Let's get started.

Introduction​

Staying on top of your third-party Cloud and SaaS service outages is crucial to maintain the reliability of your own applications. Like many modern teams, Slack might be your communication tool of choice. You can keep up with such incidents by pushing these events to a Slack channel.

There are different ways of pushing incident events to Slack. In this article we will explore how to integrate IncidentHub incident lifecycle events using an incoming webhook. An incoming webhook can be used to send incident trigger, update, and resolve events to a specific Slack channel.

Note that IncidentHub also has an option to integrate with custom webhooks, which is different from Slack's webhooks. If you are using Slack, choose the Slack option. For a custom webhook server, choose the Webhook option. The format of the Slack webhook payload is different from that of the Slack webhook.

Slack Incoming Webhook Configuration​

You must have the correct permissions on your Slack workspace to be able to do this.

Follow these steps to configure an incoming webhook in your Slack workspace.

Introduction​

Staying on top of your third-party Cloud and SaaS service outages is crucial to maintain the reliability of your own applications. If Discord is your communication tool of choice, you can keep up with such incidents by pushing these events to a Discord channel.

Discord webhooks allow external applications to send messages to specific channels within a Discord server. This article describes how to integrate Discord as a channel in your IncidentHub account using webhooks.

Note that IncidentHub also has an option to integrate with custom webhooks, which is different from Discord's webhooks. If you are using Discord, choose the Discord option. For a custom webhook server, choose the Webhook option.

Discord Server Webhook Configuration​

You must have the correct permissions on your Discord server to be able to do this.

Introduction​

Modern businesses depend heavily on Software as a Service (SaaS). SaaS is not limited to being used by software development teams.
Almost all aspects of business operations - accounting, HR, payroll, marketing, IT, sales, support - depend on one or more SaaS applications. Given this dependency on SaaS applications, their uptime becomes tightly tied to a business's uptime. Any SaaS downtime can affect both a business's daily operations as well as the user experience.

How to check if a SaaS is experiencing downtime? Follow the steps below:

• Introduction
• Visit the SaaS Provider's Status Page
  • Locate the SaaS provider's Status Page
  • Understanding Status Pages
• Use External Monitoring Services
  • Setting Up a Monitoring Tool
• Check Social Media
• Run Manual Tests
• Incident Communication
• Conclusion
• FAQ
• Popular SaaS Service Statuses

Visit the SaaS Provider's Status Page​

The SaaS provider's status page will have first-hand information about ongoing issues.

Introduction​

A recent question in an SRE forum triggered this train of thought.

How do I deal with alerts that are triggered by internal patching/release activities but don't actually cause a downtime? If we react to these alerts we might not have time to react to actual alerts that are affecting customers.

I've paraphrased the question to reflect its essence. There is plenty to unravel here.

My first reaction to this question was that the SRE who posted this is in a difficult place with systemic issues.

Systemic Issues​

Without knowing more about the org and their alerting policies, let's look at what we can dig out based on this question alone

Chances are, most of your third-party cloud and SaaS dependencies are globally distributed and have many regions of operation. Chances are, your applications use a subset of a cloud or SaaS service. If you are monitoring such a service, why should you receive alerts for all regions or every single component in the service?

E.g. if you use Digital Ocean, you might be using Kubernetes in their US locations (NYC and SFO). You would want to know only when there is an outage in one of these locations. Digital Ocean's status page gives you the option to subscribe to outages across the board - it’s all or nothing. This is the case with most services with a few exceptions.

Choosing Specific Components to Monitor​

You can now choose which components/regions you wish to monitor in IncidentHub. Let us continue with our Digital Ocean example.

You can choose to monitor all components: