Hrishikesh Barua

Introduction​

This article is part of a series on setting up an end-to-end monitoring and alerting stack using Prometheus.

Continuing our series on setting Prometheus in a Docker container, we will add a Grafana instance to our Prometheus setup.

Please refer to the previous article where we use docker compose to run Prometheus and Alertmanager together as that forms the basis to run multiple related containers. We will add a container to run Grafana to the same compose file in this article.

Introduction​

A status page forms a key part of your incident communication strategy. When it comes to setting up a status page, you have two options:

• Host your own - using either an open source project or a custom solution.
• Use a managed status page provider.

We will examine the pros and cons of each option along these dimensions:

1. Feature Set
2. Service Related

For 1, if you choose a self-managed, open-source or custom solution, it's in your control. For a managed solution, you are limited by the provider's feature set.

For 2, if you choose a self-managed solution, your team is responsible for the quality of the service. For a managed solution, you are dependent on the provider's service quality.

In most cases, you are better off using a managed solution from a reputed provider, unless you have:

• Specific requirements that are not met by the vendor.
• Budget constraints.

Introduction​

If you manage applications running on cloud platforms, you likely depend on multiple cloud vendors and services. These could be infrastructure providers like AWS, GCP or Azure. A vulnerability in any of these services could potentially impact your applications and your users.

A cloud platform has many moving parts, many of which are dependent on other third-party providers. For example:

• Operating system images for VMs which are maintained by third-party vendors.
• Container images which are hosted on external repositories.
• Software stacks which are maintained by other vendors but available for deployment on the cloud provider.
• Libraries used by the cloud provider's internal software which are maintained by other developers or organizations.
• Control plane software like Kubernetes.
• Hardware, like processors, which are provided by the manufacturer.
• Hypervisors which are developed and maintained by third-party vendors.
• Networking hardware manufactured by other vendors.

Introduction​

This article is part of a series on setting up an end-to-end monitoring and alerting stack using Prometheus.

Continuing our series on setting up Prometheus in a container, this article provides a step-by-step guide for how to configure alerts in Prometheus. We will add alerting rules and deploy Prometheus Alertmanager with Slack integration.

If you follow the steps in this article, you will end up with a containerized setup for:

1. A Prometheus instance with alerting rules.
2. An Alertmanager instance which can send alerts originating from those rules to a Slack channel.

Let's get started.

Introduction​

This article is part of a series on setting up an end-to-end monitoring and alerting stack using Prometheus.

There are different ways you can use to deploy the Prometheus monitoring tool in your environment. One of the fastest ways to get started is to deploy it as a Docker container. This guide shows you how to quickly set up a minimal Prometheus on your laptop. You can then extend that setup to add a monitoring dashboard, alerting, and authentication.

Introduction​

This article is an attempt to list the best incident management material and guides available for free on the internet. If I've missed something you think should be here, do let me know and I'll be happy to add it.

Introduction​

This article is part of a series on setting up an end-to-end monitoring and alerting stack using Prometheus.

The Prometheus monitoring tool can store its metrics either locally or remotely. You can configure a remote data store using the remote_write configuration. This article describes the various data store options available as well as how to set up a remote store.

Overview of Remote Storage​

By default, Prometheus stores data locally wherever it is installed. The data directory can be configured by using the --storage.tsdb.path command line option when starting Prometheus. In practice you can use a separate disk for higher performance attached to the machine where Prometheus is running.

However, this may not be possible or optimal in all situations as you might want a data store that is more suited for time series data, and has larger storage capabilities for higher data retention. Prometheus would usually run in a standalone VM or a Kubernetes pod or a Docker container, and it would not have access to such data stores by default.

A remote store can add these capabilities to Prometheus. The remote storage option can be set by using the remote_write key in the Prometheus configuration YAML file.

Introduction​

This article is part of a series on setting up an end-to-end monitoring and alerting stack using Prometheus.

Service discovery (SD) is a mechanism by which the Prometheus monitoring tool can discover monitorable targets automatically. Instead of listing down each and every target to be scraped in the Prometheus configuration, service discovery acts as a source of targets that Prometheus can query at runtime.

Service discovery becomes crucial when there are dynamically changing hosts, especially in microservices architectures and environments like Kubernetes. In Prometheus parlance, service discovery is a way of discovering "scrape targets".

For example, pods are created dynamically in Kubernetes as a result of new services being deployed and undeployed, autoscaling events, and errors causing pods to crash and go away. If you are using Prometheus for scraping pods in such an environment, Prometheus has to know which pods are running and scrapable at any given point in time. The Kubernetes service discovery pluging enables this. Similarly, there are SD plugins for other common environments.

You can use service discovery in Prometheus with the predefined plugins, or write your own custom ones using file or HTTP, depending on the situation.

Introduction​

Last updated on August 11, 2025

Runbooks are a key part of incident management and preserve institutional knowledge. They can be used for both incident response as well as routine tasks like db maintenance and generating a complex report. We are mostly focused on incident response runbooks here.

This article is outdated. Please refer to The Ultimate Guide to Incident Management Tools in 2025 for the latest guide.

Introduction​

Incident management tools are important for organizations to effectively handle service outages. With so many incident management tools around with different feature sets, it's often difficult to find the one that is right for your needs. In this article, we attempt to make a list of incident management software available in 2024 with their features to help you arrive at the right one.

We have focused mostly on tools that offer incident management capabilities - which include at least incident lifecycle management, on-call scheduling, and third-party integrations.

There are many good tools which are focused only on incident response, or on monitoring and generating alerts, or on the ticketing aspect of incidents. We have not included those to avoid cluttering this article.

Introduction​

Why is Slack becoming so popular in incident management?

Slack is one of the most popular communication tools used in companies. If you're part of a remote team, your team is probably on Slack or something similar like MS Teams. Although IM tools lack the communication nuances that are taken for granted in face to face interactions, they provide many other advantages:

• Access to historical data
• Asynchronous communication
• The ability to share links and documents easily
• Adding anybody in the organization to a conversation

Slack in Incident Management​

One of the trends I've noticed in incident management is the growing rise of Slack in incident response and management tools. I think this is tied to the increase in remote work after COVID-19.

Introduction​

There is a newer version of this article for 2025.

Maintaining transparent communication about service availability is crucial for businesses of all sizes. Status pages are an important part of your communication strategy during times of outages and maintenance events.

You can choose to go with a fully managed status page provider, or host an open-source one yourself.

Open source status page providers offer a cost-effective and customizable solution. However, they can come with their own drawbacks. This guide explores open source status page providers in 2024 to help you choose the right tool for your needs.

Introduction​

Last updated on August 11, 2025

Downtime is inevitable but what sets successful businesses apart is how they handle it. A key part of incident management is incident communication with both internal and external stakeholders. A status page is a crucial tool for maintaining clear communication with users during outages or service interruptions. There are numerous status page providers available with different features. This article will guide you through best practices for selecting a provider that suits your needs.

Introduction​

Updated Mar 26, 2025

Staying on top of your third-party Cloud and SaaS service outages is crucial to maintaining the reliability of your own applications. Like many modern teams, Slack might be your communication tool of choice. You can keep up with such incidents by pushing these events to a Slack channel.

IncidentHub has its own Slack app which can be used to push incident lifecycle events to the Slack channel of your choice. It can be used to send incident trigger, update, and resolve events.

Installing IncidentHub's Slack App​

You must have the correct permissions on your Slack workspace to be able to do this.

Follow these steps to configure the Slack app in your Slack workspace.

Introduction​

A newer and expanded version of this article is available here.

Last updated on August 8, 2025.

Incident updates on the public status pages of your SaaS vendors and cloud providers are often the first indication that they might have an outage. Providers also post updates about upcoming and ongoing maintenance on their status pages. Monitoring your SaaS and cloud status pages to detect downtime becomes crucial to your business operations. This article will guide you through the process of effectively monitoring such status pages.

There are two ways to monitor multiple status pages:

• The manual process.
• Using a status page aggregator like IncidentHub.

If you are using the second option, which is the recommended approach, you can skip directly to the section on Use a Status Page Aggregator Tool.

In either case you will need to identify your cloud providers and locate their public status pages first.