Introduction​

There are plenty of good SRE/Ops related podcasts out there. I follow a few of them and listen to episodes whose titles sound interesting. The problem with podcasts is that some episodes focus on one topic, and other episodes deal with a host of topics. In between there is filler and things that are not relevant to the topic but are necessary to carry on a conversation. Spending 30-60 minutes listening to podcasts is not always a great use of time.

A while ago I decided to create a tool that summarizes podcasts for me using an LLM. If I find the summary interesting enough or there is something that I want to learn about, I go and listen to the entire episode. Such a tool might be useful to others also, so I made a website for it - https://www.srenews.info.

I encourage you to listen to the complete episodes if you find a summary interesting - they are linked from each summary page.

Introduction​

Continuing our series on setting up Prometheus in a container, this article provides a step-by-step guide for how to configure alerts in Prometheus. We will add alerting rules and deploy Prometheus Alertmanager with Slack integration.

If you follow the steps in this article, you will end up with a containerized setup for:

1. A Prometheus instance with alerting rules.
2. An Alertmanager instance which can send alerts originating from those rules to a Slack channel.

Let's get started.

Introduction​

There are different ways you can use to deploy the Prometheus monitoring tool in your environment. One of the fastest ways to get started is to deploy it as a Docker container. This guide shows you how to quickly set up a minimal Prometheus on your laptop. You can then extend that setup to add a monitoring dashboard, alerting, and authentication.

• Introduction
• Deploying Prometheus in a Docker Container
  • Basic Setup
    • Useful Tip
  • Separating the Configuration
  • Making the Data Storage Persistent
  • Further Configuration
• Conclusion
• References

Deploying Prometheus in a Docker Container​

Basic Setup​

Running Prometheus in Docker is very simple using this command

docker run -p 9000:9090 prometheus prom/prometheus:v3.0.0

This will pull and run the latest version (as of this writing) of Prometheus. You can access the Prometheus UI at localhost:9000. Note that the container port 9090 is forwarded to the localhost port 9000.

Introduction​

This article is an attempt to list the best incident management material and guides available for free on the internet. If I've missed something you think should be here, do let me know and I'll be happy to add it.

• Introduction
• Reading Material (Articles, Guides, Slides, Books)
• Templates and Real-World Examples
• Conference Talks
• Podcast Episodes

Introduction​

The Prometheus monitoring tool can store its metrics either locally or remotely. You can configure a remote data store using the remote_write configuration. This article describes the various data store options available as well as how to set up a remote store.

Overview of Remote Storage​

By default, Prometheus stores data locally wherever it is installed. The data directory can be configured by using the --storage.tsdb.path command line option when starting Prometheus. In practice you can use a separate disk for higher performance attached to the machine where Prometheus is running.

However, this may not be possible or optimal in all situations as you might want a data store that is more suited for time series data, and has larger storage capabilities for higher data retention. Prometheus would usually run in a standalone VM or a Kubernetes pod or a Docker container, and it would not have access to such data stores by default.

A remote store can add these capabilities to Prometheus. The remote storage option can be set by using the remote_write key in the Prometheus configuration YAML file.

Introduction​

Service discovery (SD) is a mechanism by which the Prometheus monitoring tool can discover monitorable targets automatically. Instead of listing down each and every target to be scraped in the Prometheus configuration, service discovery acts as a source of targets that Prometheus can query at runtime.

Service discovery becomes crucial when there are dynamically changing hosts, especially in microservices architectures and environments like Kubernetes. In Prometheus parlance, service discovery is a way of discovering "scrape targets".

For example, pods are created dynamically in Kubernetes as a result of new services being deployed and undeployed, autoscaling events, and errors causing pods to crash and go away. If you are using Prometheus for scraping pods in such an environment, Prometheus has to know which pods are running and scrapable at any given point in time. The Kubernetes service discovery pluging enables this. Similarly, there are SD plugins for other common environments.

You can use service discovery in Prometheus with the predefined plugins, or write your own custom ones using file or HTTP, depending on the situation.

Introduction​

Runbooks are a key part of incident management and preserve institutional knowledge. They can be used for both incident response as well as routine tasks like db maintenance and generating a complex report. We are mostly focused on incident response runbooks here.

• Introduction
• Best Practices
  • 1. Runbook Structure
  • 2. Runbook Content
  • 3. Updating and Maintaining Runbooks
  • 4. Testing Your Runbooks
  • 5. Locating Runbooks
  • 6. Runbook Ownership
  • 7. What Not To Do
  • 8. Dealing With the Unexpected
    • 1. What to do when there is no runbook for an incident or a situation?
    • 2. The runbook steps are wrong, or they don't work, or I don't have access, or the link is broken
• Conclusion
• FAQs

Best Practices​

1. Runbook Structure​

• Establish a standard format that will be used across your organization. This will ensure consistency and help on-call folks to quickly figure out the steps even for runbooks they may not have seen before. It will also help in editing and maintaining the runbooks.
• Get buy-in from your team on the decided format. If you don't have buy-in people might not want to maintain or use them.
• Create the runbooks as decision trees. You don't need a visual guide here but include it if it's easy to create. Don't have too many branches in the tree - that will cause confusion. If you find yourself adding

Introduction​

Incident management tools are important for organizations to effectively handle service outages. With so many incident management tools around with different feature sets, it's often difficult to find the one that is right for your needs. In this article, we attempt to make a list of incident management software available in 2024 with their features to help you arrive at the right one.

We have focused mostly on tools that offer incident management capabilities - which include at least incident lifecycle management, on-call scheduling, and third-party integrations.

There are many good tools which are focused only on incident response, or on monitoring and generating alerts, or on the ticketing aspect of incidents. We have not included those to avoid cluttering this article.

• Introduction
• Benefits of Using an Incident Management Tool
• List of Incident Management Tools in 2024
  • 1. PagerDuty
  • 2. Opsgenie
  • 3. Splunk On-Call
  • 4. Grafana OnCall
  • 5. ServiceNow
  • 6. iLert
  • 7. incident.io
  • 8. FireHydrant
  • 9. Squadcast
  • 10. Better Stack
  • 11. Rootly
• Conclusion

Benefits of Using an Incident Management Tool​

• An incident management tool streamlines the incident management process by helping to define and automate workflows. It can help you create runbooks, alerting and escalation policies, and define and manage on-call schedules.
• Incident Management software often come with integrations with your observability stack. Your observability stack is a key source of incidents. They can also integrate with your existing communication and

Introduction​

Why is Slack becoming so popular in incident management?

Slack is one of the most popular communication tools used in companies. If you're part of a remote team, your team is probably on Slack or something similar like MS Teams. Although IM tools lack the communication nuances that are taken for granted in face to face interactions, they provide many other advantages:

• Access to historical data
• Asynchronous communication
• The ability to share links and documents easily
• Adding anybody in the organization to a conversation

Slack in Incident Management​

One of the trends I've noticed in incident management is the growing rise of Slack in incident response and management tools. I think this is tied to the increase in remote work after COVID-19.

Introduction​

Maintaining transparent communication about service availability is crucial for businesses of all sizes. Status pages are an important part of your communication strategy during times of outages and maintenance events.

You can choose to go with a fully managed status page provider, or host an open-source one yourself.

Open source status page providers offer a cost-effective and customizable solution. However, then can come with their own drawbacks. This guide explores open source status page providers in 2024 to help you choose the right tool for your needs.

• Introduction
• List of Open Source Status Page Providers
  • 1. Cachet
  • 2. Statping-ng
  • 3. Cstate
  • 4. Upptime
  • 5. Staytus
  • 6. Vigil
  • 7. Gatus
  • 8. Statuspal
  • 9. Uptime Kuma
  • 10. OneUptime
  • 11. Kener
  • 12. OpenStatus
• Choosing the Right Open Source Status Page Provider
• Conclusion

List of Open Source Status Page Providers​

1. Cachet​

Cachet is a popular open source status page system built with PHP and Laravel. It offers a clean, minimalist design and a robust feature set.