What would it take for managed grafana (quick to set up!) + managed kubernetes (also quick), plus Loki + promtail inside the cluster, but with all of the grafana -> Loki traffic happening on the azure backbone? More effort than using the integrated monitoring for your managed cluster, but 😎 cooler 😎.

Our Goals

  • Where possible, use Azure Managed Identities to set up access between services, as this is much easier to manage

Implementation

Loki & Storage Account

grant managed identity access created via script

configure storage account with right blobs https://github.com/grafana/loki/blob/main/production/helm/loki/values.yaml#L312 in bicep

https://grafana.com/docs/loki/latest/configure/#azure_storage_config It's fairly unclear what the docs are asking for.

disable auth_enabled - https://github.com/grafana/loki/issues/10873 - claims it was fixed in april. The fix was merged https://github.com/grafana/loki/pull/12411 and is in 3.1.0. The current helm chart references 3.0.0 as the most recent version.

All this to get an error. Refresh request failed. Status Code = '400'. Response body: {"error":"invalid_request","error_description":"Multiple user assigned identities exist, please specify the clientId / resourceId of the identity in the token request"} Endpoint http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fplaytechniquelokilogs.blob.core.windows.net Do we let this stop us? Do we fuck.

Downloading the loki source code; makefile has a build target for loki-image, can override the destination of make with IMAGE_PREFIX=playtechnique make loki-image Plan:

    • Build custom image with better logging or something
    • How do we figure out where Loki needs tweaking? Started by searching for 'azure storage multiple identities.' Found nice article https://learn.microsoft.com/en-us/azure/storage/common/migrate-azure-credentials?tabs=roles-azure-portal%2Csign-in-azure-cli%2Cgo%2Cazure-portal-create%2Cazure-portal-associate%2Capp-service-identity%2Capp-service-connector%2Cassign-role-azure-portal which says part way through that if I were writing a custom app in go (Loki's written in go) then go get -u github.com/Azure/azure-sdk-for-go/sdk/azidentity and cred, err := azidentity.NewDefaultAzureCredential(nil) should look familiar.