Author Archives: juw

How to visualize Azure Stack Hub Admin API Metrics in Grafana Dashboard – Part 3

Introduction

In the first two parts of this blogpost series I talked about the prerequisites, installation, details to consider and the script logic.
The 3rd part will be about the the script to generate basic data and to visualize the data in a Dashboard.

Script to get basic parameters

#####################################
# Input Parameters                  #
#####################################
$date = Get-Date -Format "yyMMdd-HHmmss"
$logPath = "E:\Prometheus-powershell-Export\logs"
$location = "yourLocation"
Start-Transcript -Path $logPath\${date}-Get-AzureStack-$location-Admin.ps1.log -Force
$ArmEndpoint   = 'https://adminmanagement.$location.contoso.com'
$client_id     = 'your client id'
$client_secret = 'your client secret'
$AADTenantName = "yourAADTenant.onmicrosoft.com"
 
$EnvironmentName = "AzureStackAdmin$location"
$jobname = "azs-admin-metrics"
$stamp = $location
$adminSubscription = "your admin subscription id"

#####################################
# Clean old logfiles                #
#####################################
$logFile = "*-Get-AzureStack-$location-Admin.ps1.log"
$timeLimit = (Get-Date).AddDays(-7) 
if(Test-Path "$logPath\$logFile"){Get-ChildItem "$logPath\$logFile" -File | where { $_.LastWriteTime -lt $timeLimit } | Remove-Item -Force}


############################################################
# Set variables to get the endpoint Information needed     #
# Get Tenant Id and create access token                    #
############################################################
  
$Environment = Add-AzureRmEnvironment -Name $EnvironmentName -ARMEndpoint $ArmEndpoint
$ActiveDirectoryServiceEndpointResourceId = $Environment.ActiveDirectoryServiceEndpointResourceId.TrimEnd('/')
$AuthEndpoint = (Get-AzureRmEnvironment -Name $EnvironmentName).ActiveDirectoryAuthority.TrimEnd('/')
$TenantId = (invoke-restmethod "$($AuthEndpoint)/$($AADTenantName)/.well-known/openid-configuration").issuer.TrimEnd('/').Split('/')[-1]
$AccessTokenUri = (invoke-restmethod "$($AuthEndpoint)/$($AADTenantName)/.well-known/openid-configuration").token_endpoint

#####################################
# Request Bearer Token              #
#####################################
 
# Request Bearer Token
$body = "grant_type=client_credentials&client_id=$client_id&client_secret=$client_secret&resource=$ActiveDirectoryServiceEndpointResourceId"
$Token = Invoke-RestMethod -Method Post -Uri $AccessTokenUri -Body $body -ContentType 'application/x-www-form-urlencoded'
 
# Create Rest API Headers
$Headers = @{}
$Headers.Add("Authorization","$($Token.token_type)" + " " + "$($Token.access_token)")
$Headers.Add("Accept","application/json")
$Headers.Add("x-ms-effective-locale","en.en-us")

# Create Rest API Headers
$Headers = @{}
$Headers.Add("Authorization","$($Token.token_type)" + " " + "$($Token.access_token)")
$Headers.Add("Accept","application/json")
$Headers.Add("x-ms-effective-locale","en.en-us")

#region List Region Health through API
$ListRegionHealth = "$ArmEndPoint/subscriptions/$adminSubscription/resourcegroups/system.local/providers/Microsoft.InfrastructureInsights.Admin/regionHealths?api-version=2016-05-01"
$RegionHealth = (Invoke-RestMethod -Uri $ListRegionHealth -ContentType "application/json" -Headers $Headers -Method Get -Debug -Verbose).value

$results = $null
foreach($x in $RegionHealth)
    {
    foreach($metric in $x)
        {
        foreach($m in $metric.properties.usageMetrics)
            {

            $name = $m.name.Replace(" ","_").ToLower()
            $location = $x.location

            $result1 = "#HELP {0}`n" -f $name
            $result2 = "#TYPE {0} gauge`n" -f $name
            $results += $result1
            $results += $result2
            
            foreach($mValue in $m.metricsValue)
                {
                $valueName = $mValue.name
                $unit = $mValue.unit
                $value = $mValue.value

                $result3 = "azs_{0}{{location=`"{1}`",valueName=`"{2}`",unit=`"{3}`"}} {4}`n" -f @($name, $location, $valueName, $unit, $value)
                $results += $result3

                }

            }
        }
    }

$responseRegionHealth = Invoke-WebRequest -Uri "http://localhost:9091/metrics/job/$jobname/instance/$stamp" -Method Post -Body $results
#endregion

#region List configured Quotas through API
$ListConfiguredQuotas = "$ArmEndPoint/subscriptions/$adminSubscription/providers/Microsoft.Compute.Admin/locations/$stamp/quotas?api-version=2015-12-01-preview"
$ConfiguredQuotas = (Invoke-RestMethod -Uri $ListConfiguredQuotas -ContentType "application/json" -Headers $Headers -Method Get -Debug -Verbose).value

$results = $null
$name = $null
$name = "quotas"

$result1 = "#HELP {0}`n" -f $name
$result2 = "#TYPE {0} gauge`n" -f $name
$results += $result1
$results += $result2

foreach($ConfiguredQuota in $ConfiguredQuotas)
    {

    $quotaName = $null
    $type = $null


    $quotaName = $ConfiguredQuota.name.Replace(" ","_").ToLower()
    $type = $ConfiguredQuota.type
    
    foreach($propertyQuota in $ConfiguredQuota)
        {  
        $location = $null
        $helperPropertyQuota = $null

        $location = $propertyQuota.location 
        $helperPropertyQuota = $propertyQuota.properties | Get-member -MemberType NoteProperty

        foreach($p in $helperPropertyQuota)
            {
            $valueName = $null
            $value = $null            
            
            $valueName = $p.name
            $value = $propertyQuota.properties
            $value = $value.$valueName

            $result3 = "azs_{0}{{location=`"{1}`",quotaName=`"{2}`",valueName=`"{3}`",type=`"{4}`"}} {5}`n" -f @($name, $location, $quotaName, $valueName, $type, $value)
            $results += $result3
            }
        
        
        }
    }

$responseConfiguredQuotas = Invoke-WebRequest -Uri "http://localhost:9091/metrics/job/$jobname/instance/$stamp" -Method Post -Body $results
#endregion

#region List Service Health through API
$ListServiceHealths = "$ArmEndPoint/subscriptions/$adminSubscription/resourceGroups/System.$stamp/providers/Microsoft.InfrastructureInsights.Admin/regionHealths/$stamp/serviceHealths?api-version=2016-05-01"
$ServiceHealths = (Invoke-RestMethod -Uri $ListServiceHealths -ContentType "application/json" -Headers $Headers -Method Get -Debug -Verbose).value

$results = $null
$name = $null
$name = "service_health"

$result1 = "#HELP {0} 0=healthy, 1=warning, 2=critical, 3=unknownm, 4=something else happened`n" -f $name
$result2 = "#TYPE {0} gauge`n" -f $name
$results += $result1
$results += $result2

foreach($ServiceHealth in $ServiceHealths)
    {

    $serviceName = $null
    $type = $null


    $serviceName = $ServiceHealth.name.Replace(" ","_").ToLower()
    $type = $ServiceHealth.type
    
    foreach($propertyHealth in $ServiceHealth)
        {  
        $location = $null
        $helperpropertyHealth = $null
        $valueName = $null
        $value = $null            

        $location = $propertyHealth.location 
        $helperpropertyHealth = $propertyHealth.properties | Get-member -MemberType NoteProperty
            
        $displayName = $serviceHealth.properties.displayName
        $value = $propertyHealth.properties.healthstate

        switch ( $value.ToLower() )
            {
            healthy { $value = '0' }
            warning { $value = '1' }
            critical { $value = '2' }
            unknown { $value = '3' }
            default { $value = '4' }
            }


        $result3 = "azs_{0}{{location=`"{1}`",serviceHealthName=`"{2}`",displayName=`"{3}`",type=`"{4}`"}} {5}`n" -f @($name, $location, $serviceName, $displayName, $type, $value)
        $results += $result3
        
        }
    }

$responseServiceHealths = Invoke-WebRequest -Uri "http://localhost:9091/metrics/job/$jobname/instance/$stamp" -Method Post -Body $results
#endregion

#region List of Subscriptions through API
$ListSubscriptions = "$ArmEndPoint/subscriptions/$adminSubscription/providers/Microsoft.Subscriptions.Admin/subscriptions?api-version=2015-11-01"

$Subscriptions = (Invoke-RestMethod -Uri $ListSubscriptions -ContentType "application/json" -Headers $Headers -Method Get -Debug -Verbose).value

$results = $null
$name = $null
$name = "user_subscriptions"
$location = $stamp

$result1 = "#HELP {0} Get all User Subscriptions that exist and get the state of the user subscription. 0=disabled, 1=enabled, 4=something else happened`n" -f $name
$result2 = "#TYPE {0} gauge`n" -f $name
$results += $result1
$results += $result2

foreach($subscription in $Subscriptions)
    {

    $id = $null
    $subscriptionId = $null
    $delegatedProviderSubscriptionId = $null
    $displayName = $null
    $owner = $null
    $tenantId = $null
    $routingResourceManagerType = $null
    $offerId = $null
    $state = $null

    $id = $subscription.id
    $subscriptionId = $subscription.subscriptionId
    $delegatedProviderSubscriptionId = $subscription.delegatedProviderSubscriptionId
    $displayName = $subscription.displayName
    $owner = $subscription.owner
    $tenantId = $subscription.tenantId
    $routingResourceManagerType = $subscription.routingResourceManagerType
    $offerId = $subscription.offerId
    $state = $subscription.state

        switch ( $state.ToLower() )
            {
            disabled { $state = '0' }
            enabled { $state = '1' }
            default { $state = '4' }
            }

    $result3 = "azs_{0}{{location=`"{1}`",id=`"{2}`",subscriptionId=`"{3}`",delegatedProviderSubscriptionId=`"{4}`",displayName=`"{5}`",owner=`"{6}`",tenantId=`"{7}`",routingResourceManagerType=`"{8}`",offerId=`"{9}`"}} {10}`n" -f @($name, $location, $id, $subscriptionId, $delegatedProviderSubscriptionId, $displayName, $owner, $tenantId, $routingResourceManagerType, $offerId, $state)
    $results += $result3
    

    }

$responseListSubscriptions = Invoke-WebRequest -Uri "http://localhost:9091/metrics/job/$jobname/instance/$stamp" -Method Post -Body $results
#endregion

Stop-Transcript

Be aware that you might need to change the URL to the prometheus server in case you are not running everything on one host.

Feel free to get more parameter you need to monitor! Have a look at the Microsoft documentation: https://docs.microsoft.com/en-us/rest/api/azure-stack/

You can also search for REST API calls in the Adminportal UI:

Run script with the Task Scheduler

Now we have a script which delivers basic health parameters. I will implement a scheduled task which runs every 5 minutes for the PoC (Proof of Concept).
For that purpose I implemented one task per stamp. You can also think about getting the data with only one script….it’s up to you!

Create Dashboard in Grafana

You can use my custon created Dashboard….but feel free to change the dasboard! Save the Code as JSON and import the file to you Grafana installation. For security reasons I uploaded the file as a .txt file.
In the JSON-file you need to change “yourLocation1” and “yourLocation2” to your real locations to make the dashboard work.

Checkout the Dashboard

In my scenario I need to have a overview over two Azure Stack Hub stamps. If you only have one stamp you need to delete the singlestat panel’s from the second stamp. If you need a Dashboard to have have an overview over more then two stamp’s you should consider to get rid of all singlestats, because it’ll get confusing. You’ll be better off with a bar gauge or something else that fits your needs.

Check the Dashboard if you get data! Now your Dashboard should look like this:

I had to anonymize some of the customer data

ToDo:
Therea are many more parameters left to visualize. After I finished the next final part I will work on visualizing more data in grafana…..for example:

  • the capacity of the volume shares
  • the vNet health
  • the health of all resource providers in detail
  • the health of the scale units

Thanks for reading so far!!! Next Part will be about alerting and notifications in Grafana. At this point you should be able to have your own awesome Dashboard for Azure Stack Hub in Grafana!

How to visualize Azure Stack Hub Admin API Metrics in Grafana Dashboard – Part 2

Script logic

The 1st part of the blog post was about the prerequisites, the steps for the installation and some details you need to consider before you start with the project.
Now I will take a look into the script logic and how to get data from the Azure Stack Hub Admin API with powershell.

Input Parameters

First I define all input parameters needed. Normally I get the date, then I define the logpath and then I define every other parameter needed in the script.

#####################################
# Input Parameters                  #
#####################################
$date = Get-Date -Format "yyMMdd-HHmmss"
$logPath = "E:\Prometheus-powershell-Export\logs"
$location = "yourLocation"
Start-Transcript -Path $logPath\${date}-Get-AzureStack-$location-Admin.ps1.log -Force
$ArmEndpoint   = 'https://adminmanagement.$location.contoso.com'
$client_id     = 'your client id'
$client_secret = 'your client secret'
$AADTenantName = "yourAADTenant.onmicrosoft.com"

$EnvironmentName = "AzureStackAdmin$location"
$jobname = "azs-admin-metrics"
$stamp = $location 
$adminSubscription = "your admin subscription id"

Clean Logfiles

As my next step I like to delete old logfiles. I am a fan of a script execution history, but I also like to be organized and no one needs thousands of logfiles. For that reason I always clean the file system. In this example I clean all logfiles older than 7 days.

#####################################
# Clean old logfiles                #
#####################################
$logFile = "*-Get-AzureStack-$location-Admin.ps1.log"
$timeLimit = (Get-Date).AddDays(-7) 
if(Test-Path "$logPath\$logFile"){Get-ChildItem "$logPath\$logFile" -File | where { $_.LastWriteTime -lt $timeLimit } | Remove-Item -Force}

Set variables for Login

There is an awesome blog post from Robert van Vugt where he explains how to login to the Azure Stack Hub API and how to use different methods for the login.
http://robertvanvugt.com/how-to-call-azure-stack-rest-api-using-adfs-identity-provider/
Microsoft also has a very good documentation on how to connect with the API.


Here I choose to use a service principal in Azure AD.

############################################################
# Set variables to get the endpoint Information needed     #
# Get Tenant Id and create access token                    #
############################################################
 
$Environment = Add-AzureRmEnvironment -Name $EnvironmentName -ARMEndpoint $ArmEndpoint
$ActiveDirectoryServiceEndpointResourceId = $Environment.ActiveDirectoryServiceEndpointResourceId.TrimEnd('/')
$AuthEndpoint = (Get-AzureRmEnvironment -Name $EnvironmentName).ActiveDirectoryAuthority.TrimEnd('/')
$TenantId = (invoke-restmethod "$($AuthEndpoint)/$($AADTenantName)/.well-known/openid-configuration").issuer.TrimEnd('/').Split('/')[-1]
$AccessTokenUri = (invoke-restmethod "$($AuthEndpoint)/$($AADTenantName)/.well-known/openid-configuration").token_endpoint

Create Token to authenticate to the Web API

#####################################
# Request Bearer Token              #
#####################################

# Request Bearer Token
$body = "grant_type=client_credentials&client_id=$client_id&client_secret=$client_secret&resource=$ActiveDirectoryServiceEndpointResourceId"
$Token = Invoke-RestMethod -Method Post -Uri $AccessTokenUri -Body $body -ContentType 'application/x-www-form-urlencoded'

# Create Rest API Headers
$Headers = @{}
$Headers.Add("Authorization","$($Token.token_type)" + " " + "$($Token.access_token)")
$Headers.Add("Accept","application/json")
$Headers.Add("x-ms-effective-locale","en.en-us")

Get metrics from the API

Now everything is set to get the data from the Azure Stack Hub API!

# List Region Health through API
# Build the web request
$ListRegionHealth = "$ArmEndPoint/subscriptions/$adminSubscription/resourcegroups/system.local/providers/Microsoft.InfrastructureInsights.Admin/regionHealths?api-version=2016-05-01"
# Invoke the web request
$RegionHealth = (Invoke-RestMethod -Uri $ListRegionHealth -ContentType "application/json" -Headers $Headers -Method Get -Debug -Verbose).value

The variable “$RegionHealth” holds the data from the web request. Remember, that the data now needs to be transformed to the prometheus exposition format.

$results = $null
foreach($x in $RegionHealth)
    {
    foreach($metric in $x)
        {
        foreach($m in $metric.properties.usageMetrics)
            {

            $name = $m.name.Replace(" ","_").ToLower()
            $location = $x.location

            $result1 = "#HELP {0}`n" -f $name
            $result2 = "#TYPE {0} gauge`n" -f $name
            $results += $result1
            $results += $result2
            
            foreach($mValue in $m.metricsValue)
                {
                $valueName = $mValue.name
                $unit = $mValue.unit
                $value = $mValue.value

                $result3 = "azs_{0}{{location=`"{1}`",valueName=`"{2}`",unit=`"{3}`"}} {4}`n" -f @($name, $location, $valueName, $unit, $value)
                $results += $result3

                }

            }
        }
    }

# push the variable "$result" to the prometheus pushgateway
$responseRegionHealth = Invoke-WebRequest -Uri "http://localhost:9091/metrics/job/$jobname/instance/$stamp" -Method Post -Body $results

Output: prometheus exposition format

#HELP physical_memory
#TYPE physical_memory gauge
azs_physical_memory{location="yourLocation",valueName="Used",unit="GB"} 287
azs_physical_memory{location="yourLocation",valueName="Available",unit="GB"} 631
#HELP physical_storage
#TYPE physical_storage gauge
azs_physical_storage{location="yourLocation",valueName="Used",unit="TB"} 14.34
azs_physical_storage{location="yourLocation",valueName="Available",unit="TB"} 20
#HELP public_ip_address_pools
#TYPE public_ip_address_pools gauge
azs_public_ip_address_pools{location="yourLocation",valueName="Used",unit="One"} 39.0
azs_public_ip_address_pools{location="yourLocation",valueName="Available",unit="One"} 24.0

Get all the data you need!

Now you can have a look at the Azure Stack Hub Admin API reference documentation:
https://docs.microsoft.com/en-us/rest/api/azure-stack/

You can push any result you can get from the API to prometheus.

Next part will be my Dashboard and the API requests to fill it with data!!!

How to visualize Azure Stack Hub Admin API Metrics in Grafana Dashboard – Part 1

Introduction

In this multi-part blogpost I will explain how to setup a Grafana Dashboard to visualize Azure Stack Hub Admin API metrics. My goal is a global view over multiple Azure Stack Hub’s in one Dashboard with specific alerting on defined thresholds. As a service provider I don’t want to overprovision quotas. For that prupose I have the need for a global view of how much resources I already offered my customers. I also want to know how much resources are left on my stamps. At some point I may need to consider adding new nodes to my Azure Stack Hub or maybe I even need to buy a new one. Maybe I only have a few public IP’s left, so I need to add a new IP pool.

Dashboard Example:

Documentation

Prerequisites

  • Grafana
  • Prometheus
    • Prometheus Server
    • Alertmanager
    • Pushgateway
  • Windows Server VM for running powershell scripts
  • Service User with read only rights to Azure Stack Admin Portal

Steps

  1. Install Grafana and Prometheus, if you do not already have a installation
  2. Setup Windows Server VM
  3. Setup service principal for Azure Stack Admin Portal
  4. Create powershell script to get data
  5. Create Scheduled Task for running the script in a reasonable interval
  6. Create Dashboard
  7. Create Alerts in Dashboard
  8. Create Alerts in Prometheues Alertmanager

Install Grafana and Prometheus

You should set up grafana and prometheus in a way which suits your environment best. There are tons of blogposts on how to set up grafana and prometheus.

Grafana and Prometheus both have a very good documentation:
GitHub Grafana
GitHub Prometheus

The easiest way for sure is to host both as a container in a linux environment. I myself chose to install both as a Windows Service on a Windows Server 2019. I always wanted to see how it works and how it performs. The main benefit is that you do not need Linux know-how to support your installation in case you need to add more storage etc.

Setup service principal for login

There is a very good Microsoft documentation for setting up service principals in Azure AD:
powershell
portal

Create powershell script to get data

To visualize data in Grafana you need to extract Azure Stack Admin API metrics and push them to prometheus.

For that purpose I set up a powershell script with the following features:

  • Login to the Azure Stack Admin API with a specific Service User
  • Get Azure Stack Admin metrics needed
  • Build prometheues exposition format around the metrics
  • Push metrics to prometheues

Prometheus exposition format

The difficulty is defining the correct metrics and meta data. You need to consider by which properties you will need to filter or group.

Prometheus exposition format

General example for the exposition format:

# HELP http_requests_total The total number of HTTP requests.
# TYPE http_requests_total counter
http_requests_total{method="post",code="200"} 1027 1395066363000
http_requests_total{method="post",code="400"}    3 1395066363000

Azure Stack Hub metric example

azs_physical_storage{instance="yourInstance",job="azs-admin-metrics",location="yourLocation",unit="TB",valueName="Available"}	115.77878887951374
azs_physical_storage{instance="yourInstance",job="azs-admin-metrics",location="yourLocation",unit="TB",valueName="Used"}	9.78657977283001

You may want to label things like AzureStackLocation, ID’s, resource provider names, etc.

I will publish my script in the next part of this blog series