How to visualize Azure Stack Hub Admin API Metrics in Grafana Dashboard – Part 1

Introduction

In this multi-part blogpost I will explain how to setup a Grafana Dashboard to visualize Azure Stack Hub Admin API metrics. My goal is a global view over multiple Azure Stack Hub’s in one Dashboard with specific alerting on defined thresholds. As a service provider I don’t want to overprovision quotas. For that prupose I have the need for a global view of how much resources I already offered my customers. I also want to know how much resources are left on my stamps. At some point I may need to consider adding new nodes to my Azure Stack Hub or maybe I even need to buy a new one. Maybe I only have a few public IP’s left, so I need to add a new IP pool.

Dashboard Example:

Documentation

Prerequisites

  • Grafana
  • Prometheus
    • Prometheus Server
    • Alertmanager
    • Pushgateway
  • Windows Server VM for running powershell scripts
  • Service User with read only rights to Azure Stack Admin Portal

Steps

  1. Install Grafana and Prometheus, if you do not already have a installation
  2. Setup Windows Server VM
  3. Setup service principal for Azure Stack Admin Portal
  4. Create powershell script to get data
  5. Create Scheduled Task for running the script in a reasonable interval
  6. Create Dashboard
  7. Create Alerts in Dashboard
  8. Create Alerts in Prometheues Alertmanager

Install Grafana and Prometheus

You should set up grafana and prometheus in a way which suits your environment best. There are tons of blogposts on how to set up grafana and prometheus.

Grafana and Prometheus both have a very good documentation:
GitHub Grafana
GitHub Prometheus

The easiest way for sure is to host both as a container in a linux environment. I myself chose to install both as a Windows Service on a Windows Server 2019. I always wanted to see how it works and how it performs. The main benefit is that you do not need Linux know-how to support your installation in case you need to add more storage etc.

Setup service principal for login

There is a very good Microsoft documentation for setting up service principals in Azure AD:
powershell
portal

Create powershell script to get data

To visualize data in Grafana you need to extract Azure Stack Admin API metrics and push them to prometheus.

For that purpose I set up a powershell script with the following features:

  • Login to the Azure Stack Admin API with a specific Service User
  • Get Azure Stack Admin metrics needed
  • Build prometheues exposition format around the metrics
  • Push metrics to prometheues

Prometheus exposition format

The difficulty is defining the correct metrics and meta data. You need to consider by which properties you will need to filter or group.

Prometheus exposition format

General example for the exposition format:

# HELP http_requests_total The total number of HTTP requests.
# TYPE http_requests_total counter
http_requests_total{method="post",code="200"} 1027 1395066363000
http_requests_total{method="post",code="400"}    3 1395066363000

Azure Stack Hub metric example

azs_physical_storage{instance="yourInstance",job="azs-admin-metrics",location="yourLocation",unit="TB",valueName="Available"}	115.77878887951374
azs_physical_storage{instance="yourInstance",job="azs-admin-metrics",location="yourLocation",unit="TB",valueName="Used"}	9.78657977283001

You may want to label things like AzureStackLocation, ID’s, resource provider names, etc.

I will publish my script in the next part of this blog series