• How to visualize Azure Stack Hub Admin API Metrics in Grafana Dashboard – Part 1
• How to visualize Azure Stack Hub Admin API Metrics in Grafana Dashboard – Part 3

Introduction

In the first two parts of this blogpost series I talked about the prerequisites, installation, details to consider and the script logic.
The 3rd part will be about the the script to generate basic data and to visualize the data in a Dashboard.

Script to get basic parameters

#####################################
# Input Parameters                  #
#####################################
$date = Get-Date -Format "yyMMdd-HHmmss"
$logPath = "E:\Prometheus-powershell-Export\logs"
$location = "yourLocation"
Start-Transcript -Path $logPath\${date}-Get-AzureStack-$location-Admin.ps1.log -Force
$ArmEndpoint   = 'https://adminmanagement.$location.contoso.com'
$client_id     = 'your client id'
$client_secret = 'your client secret'
$AADTenantName = "yourAADTenant.onmicrosoft.com"
 
$EnvironmentName = "AzureStackAdmin$location"
$jobname = "azs-admin-metrics"
$stamp = $location
$adminSubscription = "your admin subscription id"

#####################################
# Clean old logfiles                #
#####################################
$logFile = "*-Get-AzureStack-$location-Admin.ps1.log"
$timeLimit = (Get-Date).AddDays(-7) 
if(Test-Path "$logPath\$logFile"){Get-ChildItem "$logPath\$logFile" -File | where { $_.LastWriteTime -lt $timeLimit } | Remove-Item -Force}


############################################################
# Set variables to get the endpoint Information needed     #
# Get Tenant Id and create access token                    #
############################################################
  
$Environment = Add-AzureRmEnvironment -Name $EnvironmentName -ARMEndpoint $ArmEndpoint
$ActiveDirectoryServiceEndpointResourceId = $Environment.ActiveDirectoryServiceEndpointResourceId.TrimEnd('/')
$AuthEndpoint = (Get-AzureRmEnvironment -Name $EnvironmentName).ActiveDirectoryAuthority.TrimEnd('/')
$TenantId = (invoke-restmethod "$($AuthEndpoint)/$($AADTenantName)/.well-known/openid-configuration").issuer.TrimEnd('/').Split('/')[-1]
$AccessTokenUri = (invoke-restmethod "$($AuthEndpoint)/$($AADTenantName)/.well-known/openid-configuration").token_endpoint

#####################################
# Request Bearer Token              #
#####################################
 
# Request Bearer Token
$body = "grant_type=client_credentials&client_id=$client_id&client_secret=$client_secret&resource=$ActiveDirectoryServiceEndpointResourceId"
$Token = Invoke-RestMethod -Method Post -Uri $AccessTokenUri -Body $body -ContentType 'application/x-www-form-urlencoded'
 
# Create Rest API Headers
$Headers = @{}
$Headers.Add("Authorization","$($Token.token_type)" + " " + "$($Token.access_token)")
$Headers.Add("Accept","application/json")
$Headers.Add("x-ms-effective-locale","en.en-us")

# Create Rest API Headers
$Headers = @{}
$Headers.Add("Authorization","$($Token.token_type)" + " " + "$($Token.access_token)")
$Headers.Add("Accept","application/json")
$Headers.Add("x-ms-effective-locale","en.en-us")

#region List Region Health through API
$ListRegionHealth = "$ArmEndPoint/subscriptions/$adminSubscription/resourcegroups/system.local/providers/Microsoft.InfrastructureInsights.Admin/regionHealths?api-version=2016-05-01"
$RegionHealth = (Invoke-RestMethod -Uri $ListRegionHealth -ContentType "application/json" -Headers $Headers -Method Get -Debug -Verbose).value

$results = $null
foreach($x in $RegionHealth)
    {
    foreach($metric in $x)
        {
        foreach($m in $metric.properties.usageMetrics)
            {

            $name = $m.name.Replace(" ","_").ToLower()
            $location = $x.location

            $result1 = "#HELP {0}`n" -f $name
            $result2 = "#TYPE {0} gauge`n" -f $name
            $results += $result1
            $results += $result2
            
            foreach($mValue in $m.metricsValue)
                {
                $valueName = $mValue.name
                $unit = $mValue.unit
                $value = $mValue.value

                $result3 = "azs_{0}{{location=`"{1}`",valueName=`"{2}`",unit=`"{3}`"}} {4}`n" -f @($name, $location, $valueName, $unit, $value)
                $results += $result3

                }

            }
        }
    }

$responseRegionHealth = Invoke-WebRequest -Uri "http://localhost:9091/metrics/job/$jobname/instance/$stamp" -Method Post -Body $results
#endregion

#region List configured Quotas through API
$ListConfiguredQuotas = "$ArmEndPoint/subscriptions/$adminSubscription/providers/Microsoft.Compute.Admin/locations/$stamp/quotas?api-version=2015-12-01-preview"
$ConfiguredQuotas = (Invoke-RestMethod -Uri $ListConfiguredQuotas -ContentType "application/json" -Headers $Headers -Method Get -Debug -Verbose).value

$results = $null
$name = $null
$name = "quotas"

$result1 = "#HELP {0}`n" -f $name
$result2 = "#TYPE {0} gauge`n" -f $name
$results += $result1
$results += $result2

foreach($ConfiguredQuota in $ConfiguredQuotas)
    {

    $quotaName = $null
    $type = $null


    $quotaName = $ConfiguredQuota.name.Replace(" ","_").ToLower()
    $type = $ConfiguredQuota.type
    
    foreach($propertyQuota in $ConfiguredQuota)
        {  
        $location = $null
        $helperPropertyQuota = $null

        $location = $propertyQuota.location 
        $helperPropertyQuota = $propertyQuota.properties | Get-member -MemberType NoteProperty

        foreach($p in $helperPropertyQuota)
            {
            $valueName = $null
            $value = $null            
            
            $valueName = $p.name
            $value = $propertyQuota.properties
            $value = $value.$valueName

            $result3 = "azs_{0}{{location=`"{1}`",quotaName=`"{2}`",valueName=`"{3}`",type=`"{4}`"}} {5}`n" -f @($name, $location, $quotaName, $valueName, $type, $value)
            $results += $result3
            }
        
        
        }
    }

$responseConfiguredQuotas = Invoke-WebRequest -Uri "http://localhost:9091/metrics/job/$jobname/instance/$stamp" -Method Post -Body $results
#endregion

#region List Service Health through API
$ListServiceHealths = "$ArmEndPoint/subscriptions/$adminSubscription/resourceGroups/System.$stamp/providers/Microsoft.InfrastructureInsights.Admin/regionHealths/$stamp/serviceHealths?api-version=2016-05-01"
$ServiceHealths = (Invoke-RestMethod -Uri $ListServiceHealths -ContentType "application/json" -Headers $Headers -Method Get -Debug -Verbose).value

$results = $null
$name = $null
$name = "service_health"

$result1 = "#HELP {0} 0=healthy, 1=warning, 2=critical, 3=unknownm, 4=something else happened`n" -f $name
$result2 = "#TYPE {0} gauge`n" -f $name
$results += $result1
$results += $result2

foreach($ServiceHealth in $ServiceHealths)
    {

    $serviceName = $null
    $type = $null


    $serviceName = $ServiceHealth.name.Replace(" ","_").ToLower()
    $type = $ServiceHealth.type
    
    foreach($propertyHealth in $ServiceHealth)
        {  
        $location = $null
        $helperpropertyHealth = $null
        $valueName = $null
        $value = $null            

        $location = $propertyHealth.location 
        $helperpropertyHealth = $propertyHealth.properties | Get-member -MemberType NoteProperty
            
        $displayName = $serviceHealth.properties.displayName
        $value = $propertyHealth.properties.healthstate

        switch ( $value.ToLower() )
            {
            healthy { $value = '0' }
            warning { $value = '1' }
            critical { $value = '2' }
            unknown { $value = '3' }
            default { $value = '4' }
            }


        $result3 = "azs_{0}{{location=`"{1}`",serviceHealthName=`"{2}`",displayName=`"{3}`",type=`"{4}`"}} {5}`n" -f @($name, $location, $serviceName, $displayName, $type, $value)
        $results += $result3
        
        }
    }

$responseServiceHealths = Invoke-WebRequest -Uri "http://localhost:9091/metrics/job/$jobname/instance/$stamp" -Method Post -Body $results
#endregion

#region List of Subscriptions through API
$ListSubscriptions = "$ArmEndPoint/subscriptions/$adminSubscription/providers/Microsoft.Subscriptions.Admin/subscriptions?api-version=2015-11-01"

$Subscriptions = (Invoke-RestMethod -Uri $ListSubscriptions -ContentType "application/json" -Headers $Headers -Method Get -Debug -Verbose).value

$results = $null
$name = $null
$name = "user_subscriptions"
$location = $stamp

$result1 = "#HELP {0} Get all User Subscriptions that exist and get the state of the user subscription. 0=disabled, 1=enabled, 4=something else happened`n" -f $name
$result2 = "#TYPE {0} gauge`n" -f $name
$results += $result1
$results += $result2

foreach($subscription in $Subscriptions)
    {

    $id = $null
    $subscriptionId = $null
    $delegatedProviderSubscriptionId = $null
    $displayName = $null
    $owner = $null
    $tenantId = $null
    $routingResourceManagerType = $null
    $offerId = $null
    $state = $null

    $id = $subscription.id
    $subscriptionId = $subscription.subscriptionId
    $delegatedProviderSubscriptionId = $subscription.delegatedProviderSubscriptionId
    $displayName = $subscription.displayName
    $owner = $subscription.owner
    $tenantId = $subscription.tenantId
    $routingResourceManagerType = $subscription.routingResourceManagerType
    $offerId = $subscription.offerId
    $state = $subscription.state

        switch ( $state.ToLower() )
            {
            disabled { $state = '0' }
            enabled { $state = '1' }
            default { $state = '4' }
            }

    $result3 = "azs_{0}{{location=`"{1}`",id=`"{2}`",subscriptionId=`"{3}`",delegatedProviderSubscriptionId=`"{4}`",displayName=`"{5}`",owner=`"{6}`",tenantId=`"{7}`",routingResourceManagerType=`"{8}`",offerId=`"{9}`"}} {10}`n" -f @($name, $location, $id, $subscriptionId, $delegatedProviderSubscriptionId, $displayName, $owner, $tenantId, $routingResourceManagerType, $offerId, $state)
    $results += $result3
    

    }

$responseListSubscriptions = Invoke-WebRequest -Uri "http://localhost:9091/metrics/job/$jobname/instance/$stamp" -Method Post -Body $results
#endregion

Stop-Transcript

Be aware that you might need to change the URL to the prometheus server in case you are not running everything on one host.

Feel free to get more parameter you need to monitor! Have a look at the Microsoft documentation: https://docs.microsoft.com/en-us/rest/api/azure-stack/

You can also search for REST API calls in the Adminportal UI:

Run script with the Task Scheduler

Now we have a script which delivers basic health parameters. I will implement a scheduled task which runs every 5 minutes for the PoC (Proof of Concept).
For that purpose I implemented one task per stamp. You can also think about getting the data with only one script….it’s up to you!

Create Dashboard in Grafana

You can use my custon created Dashboard….but feel free to change the dasboard! Save the Code as JSON and import the file to you Grafana installation. For security reasons I uploaded the file as a .txt file.
In the JSON-file you need to change “yourLocation1” and “yourLocation2” to your real locations to make the dashboard work.

Checkout the Dashboard

In my scenario I need to have a overview over two Azure Stack Hub stamps. If you only have one stamp you need to delete the singlestat panel’s from the second stamp. If you need a Dashboard to have have an overview over more then two stamp’s you should consider to get rid of all singlestats, because it’ll get confusing. You’ll be better off with a bar gauge or something else that fits your needs.

Check the Dashboard if you get data! Now your Dashboard should look like this:

ToDo:
Therea are many more parameters left to visualize. After I finished the next final part I will work on visualizing more data in grafana…..for example:

• the capacity of the volume shares
• the vNet health
• the health of all resource providers in detail
• the health of the scale units
• …

Thanks for reading so far!!! Next Part will be about alerting and notifications in Grafana. At this point you should be able to have your own awesome Dashboard for Azure Stack Hub in Grafana!