Monitoring & Observability with Prometheus, Grafana, Loki & Alloy

Project Overview

I built and validated an observability platform around the Kubernetes application workloads running on Amazon EKS. Prometheus collected application and Kubernetes metrics, Grafana provided dashboards and investigation workflows, Alertmanager handled monitoring alerts, Alloy collected pod logs and Loki stored those logs for querying through Grafana.

The monitoring and logging configuration was maintained in Git and reconciled through Argo CD. Dashboards, data sources, Helm values and logging configuration were treated as version-controlled platform configuration rather than one-off UI changes.

Engineering Problem

The Kubernetes platform needed more than pod-status checks. I needed a way to observe application availability, request volume, response latency, server errors, workload capacity and pod logs while keeping the observability configuration reproducible across the portfolio environments.

• Expose and collect application metrics from the API.
• Discover application targets through Kubernetes-native configuration rather than static scrape lists.
• Visualize availability, traffic, latency, errors and API capacity.
• Keep Grafana dashboards and data sources version-controlled.
• Collect Kubernetes pod logs across development, staging and production-style namespaces.
• Provide a centralized Grafana investigation surface for both metrics and logs.
• Persist Grafana state across pod recreation.
• Keep monitoring and logging configuration under Argo CD reconciliation.

Observability Architecture

The platform used separate telemetry paths for metrics and logs. Prometheus handled numeric time-series data, while Alloy collected pod logs and forwarded them to Loki. Grafana provided the common visualization and investigation layer.

Applications → ServiceMonitor → Prometheus → Grafana
                               ↘ Alertmanager

Application Pods → Alloy → Loki → Grafana

GitOps Repository → Argo CD
                  ├── monitoring
                  └── logging

What I Built

• Prometheus-compatible API instrumentation through prometheus-fastapi-instrumentator.
• ServiceMonitor resources for Kubernetes-native application target discovery.
• Prometheus collection across the development, staging and production-style namespaces.
• A six-panel Grafana API observability dashboard provisioned from Git.
• Alertmanager as the alert-processing layer of the Prometheus stack.
• Alloy-based Kubernetes pod-log discovery and collection.
• Loki centralized log storage and query integration.
• A read-only Loki Grafana data source provisioned through GitOps.
• Persistent Grafana storage backed by a 5 GiB Kubernetes PVC.
• Separate Argo CD applications for monitoring and logging configuration.

Metrics & Dashboards

Application Metrics

The API exposed a Prometheus-compatible /metrics endpoint. ServiceMonitor resources in platform, platform-staging and platform-prod allowed Prometheus to discover the application endpoints through Kubernetes-native configuration.

During validation, Prometheus successfully scraped all six API pods with UP=1. Request-count evidence was also captured from each environment, confirming that the monitoring path was receiving real application traffic.

Namespaces monitored:
  platform
  platform-staging
  platform-prod

API targets UP = 6

Verified request counts:
  production-style = 1391
  development      = 1885
  staging          = 1646

Grafana Dashboard

I provisioned the Cloud Platform API Observability dashboard from Git rather than relying on manual dashboard construction. The dashboard definition was stored at monitoring/dashboards/cloud-platform-api-observability.json and its provisioning was verified through the Grafana API.

The dashboard contained six operational panels:

• API Request Rate by Environment
• API Availability
• P95 API Response Time
• API 5xx Error Rate
• Total API Requests
• API Pods Running

The dashboard moved from service-level signals such as availability, traffic, latency and errors to workload capacity by also showing the number of available API pods.

Alert Handling

Alertmanager was deployed with the Prometheus stack as the alert-processing layer. The validated project included Alertmanager in the running monitoring architecture, but external notification integrations were not claimed as implemented and remain a production extension.

Centralized Logging

I extended the platform beyond metrics by implementing centralized Kubernetes log collection. Alloy discovered pod logs from the three application namespaces, attached operational labels and forwarded the streams to Loki.

• Logs were collected from platform, platform-staging and platform-prod.
• Streams included cluster=portfolio-dev plus application, container, namespace, pod, job and service labels.
• Loki was configured in single-binary mode with filesystem storage.
• Loki retention was configured for seven days.
• The Loki Grafana data source was provisioned through GitOps rather than manually in the UI.

Application Pods
      ↓
    Alloy
      ↓
 Loki Gateway
      ↓
     Loki
      ↓
Grafana Data Source
      ↓
Operational Log Queries

During runtime validation, Grafana successfully queried Loki and returned 15 log streams. Application logs from development, staging and the production-style namespace were confirmed through the full pod → Alloy → Loki → Grafana path.

Challenges & How I Solved Them

Persistent Grafana Storage Changed Deployment Behavior

Grafana needed persistent state, so the deployment used a bound 5 GiB ReadWriteOnce volume backed by AWS storage. With one RWO volume and constrained worker capacity, a rolling deployment could leave the replacement Grafana pod waiting while the existing pod still held the volume.

I changed the Grafana deployment strategy to Recreate so the existing pod released the volume before the replacement pod started. This aligned the deployment strategy with the actual storage attachment constraint instead of relying on the default rolling behavior.

Persistent Grafana State Created Credential Drift

Grafana's persistent SQLite database retained the previously configured admin password. A redeployment could generate a different Kubernetes Secret while the database still expected the original credential, causing authentication inconsistency.

I introduced a stable grafana-admin-credentials Secret and configured the Helm deployment to reference it for Grafana administration and sidecar authentication. The credential value itself remained outside repository content and portfolio evidence.

Healthy Pods Did Not Prove the Logging Pipeline Worked

Seeing Alloy and Loki pods in a Running state would only prove that the processes started. I validated the full data path by querying Loki from Grafana and confirming that real application log streams were returned.

Observability Configuration Needed to Survive UI Changes

Manually configured dashboards and data sources could become undocumented state. I stored the Grafana dashboard and Loki data-source configuration in Git and allowed Argo CD to reconcile that desired state into the monitoring platform.

GitOps & Stateful Operations

The observability platform was managed through two Argo CD applications:

• monitoring — Prometheus, Grafana, Alertmanager, dashboard provisioning and Grafana data-source configuration.
• logging — Loki and Alloy configuration.

Both applications were verified Synced / Healthy during validation.

Grafana Persistence

Grafana used a verified bound 5 GiB PVC for persistent state. EBS CSI support and the AWS identity integration required by persistent storage were managed at the infrastructure layer through Terraform.

GitOps-Provisioned Loki Data Source

name      : Loki
type      : loki
uid       : loki
url       : http://loki-gateway.monitoring.svc.cluster.local
access    : proxy
isDefault : False
readOnly  : True

Helm rendering was checked before deployment, Argo CD synchronization completed successfully and the Grafana API confirmed that the Loki data source was provisioned as read-only.

Verified Evidence

Monitoring application   Synced / Healthy
Logging application      Synced / Healthy

Prometheus               6 API targets UP
Grafana dashboard        6 panels provisioned
Alertmanager             verified in monitoring stack
Grafana persistence      5 GiB PVC Bound
Loki / Alloy             runtime verified
Dev logs                 queried successfully
Staging logs             queried successfully
Production-style logs    queried successfully
Grafana → Loki           success, 15 streams

The validation covered collection, storage, visualization and querying rather than relying only on healthy Kubernetes pod status.

The temporary AWS environment was later intentionally decommissioned after implementation evidence was captured, so these results describe the validated observability state rather than claiming that Prometheus, Grafana, Loki or Alloy remain permanently online.

Production Considerations

The portfolio implementation demonstrates GitOps-managed metrics, dashboards, alert handling, persistent visualization state and centralized logs. For a larger production platform I would extend it with:

• Highly available Prometheus and Alertmanager topology based on service objectives and scale.
• Object storage for longer-term metrics and Loki retention.
• Formal recording rules and SLO-based alerting.
• Alert routing into incident-management and escalation channels.
• Authentication, authorization and network controls for externally exposed observability endpoints.
• Capacity planning and retention policies based on measured telemetry volume.
• Backup and restore testing for stateful observability components where required.

Source Code & Evidence

The implementation is version-controlled across the application and GitOps repositories. Application instrumentation lives with the application source, while monitoring, logging, dashboards, data-source configuration and Argo CD definitions live with Kubernetes desired state.

• Application Repository — API instrumentation and application source.
• GitOps Repository — Prometheus/Grafana configuration, dashboard provisioning, Loki/Alloy configuration and Argo CD application definitions.

Implemented Configuration

monitoring/
├── dashboards/
│   └── cloud-platform-api-observability.json
├── logging/
│   ├── loki-values.yaml
│   └── alloy-values.yaml
└── values.yaml

argocd/
└── logging.yaml

Engineering Value

This project demonstrates how I approach observability as an operating capability rather than a dashboard exercise. I connected application instrumentation, Kubernetes-native target discovery, metrics collection, dashboards, alert processing, centralized logging, persistent visualization state and GitOps management into one validated platform.

It also demonstrates practical troubleshooting across observability and infrastructure layers: adapting Grafana deployment behavior to ReadWriteOnce storage, resolving persistent credential-state inconsistency, proving log ingestion through end-to-end queries and keeping dashboards and data sources under version-controlled reconciliation.