Adding Prometheus to on premises Kubernetes cluster via Helm
Published: 4 September 2019
Posted in:
The following is part of a series of posts called "Repurposing old equipment by building a Kubernetes cluster".
While old equipment by itself is in general not very useful unless you find a particular use case, by combining a number of old devices you can build a more powerful system that can span perhaps a number of use cases. Kubernetes is a perfect candidate to be able to do this. I had a number of old laptops laying about and decided to test this theory out.
- Creating a bare-bones on premises Kubernetes cluster from old hardware - 30 August 2019
- Adding persistance to on premises Kubernetes cluster - 1 September 2019
- Adding Helm to on premises Kubernetes cluster - 2 September 2019
- Adding an Ingress to on premises Kubernetes cluster without load balancer - 2 September 2019
- Adding Jenkins to on premises Kubernetes cluster via Helm - 4 September 2019
- Adding Prometheus to on premises Kubernetes cluster via Helm - 4 September 2019
Installing Prometheus is pretty simple via Helm:
$ helm install stable/prometheus-operator --name prometheus --namespace monitoring
NAME: prometheus
LAST DEPLOYED: Sun Sep 8 00:46:13 2019
NAMESPACE: monitoring
STATUS: DEPLOYED
RESOURCES:
==> v1/Alertmanager
NAME AGE
prometheus-prometheus-oper-alertmanager 43s
==> v1/ClusterRole
NAME AGE
prometheus-grafana-clusterrole 45s
prometheus-prometheus-oper-alertmanager 45s
prometheus-prometheus-oper-operator 45s
prometheus-prometheus-oper-operator-psp 45s
prometheus-prometheus-oper-prometheus 44s
prometheus-prometheus-oper-prometheus-psp 44s
psp-prometheus-kube-state-metrics 45s
==> v1/ClusterRoleBinding
NAME AGE
prometheus-grafana-clusterrolebinding 44s
prometheus-prometheus-oper-alertmanager 44s
prometheus-prometheus-oper-operator 44s
prometheus-prometheus-oper-operator-psp 44s
prometheus-prometheus-oper-prometheus 44s
prometheus-prometheus-oper-prometheus-psp 44s
psp-prometheus-kube-state-metrics 44s
==> v1/ConfigMap
NAME DATA AGE
prometheus-grafana 1 46s
prometheus-grafana-config-dashboards 1 46s
prometheus-grafana-test 1 46s
prometheus-prometheus-oper-apiserver 1 46s
prometheus-prometheus-oper-controller-manager 1 46s
prometheus-prometheus-oper-etcd 1 46s
prometheus-prometheus-oper-grafana-datasource 1 46s
prometheus-prometheus-oper-k8s-cluster-rsrc-use 1 46s
prometheus-prometheus-oper-k8s-coredns 1 46s
prometheus-prometheus-oper-k8s-node-rsrc-use 1 46s
prometheus-prometheus-oper-k8s-resources-cluster 1 46s
prometheus-prometheus-oper-k8s-resources-namespace 1 46s
prometheus-prometheus-oper-k8s-resources-pod 1 46s
prometheus-prometheus-oper-k8s-resources-workload 1 46s
prometheus-prometheus-oper-k8s-resources-workloads-namespace 1 46s
prometheus-prometheus-oper-kubelet 1 46s
prometheus-prometheus-oper-nodes 1 45s
prometheus-prometheus-oper-persistentvolumesusage 1 45s
prometheus-prometheus-oper-pods 1 45s
prometheus-prometheus-oper-prometheus 1 45s
prometheus-prometheus-oper-prometheus-remote-write 1 45s
prometheus-prometheus-oper-proxy 1 45s
prometheus-prometheus-oper-scheduler 1 45s
prometheus-prometheus-oper-statefulset 1 45s
==> v1/Deployment
NAME READY UP-TO-DATE AVAILABLE AGE
prometheus-kube-state-metrics 1/1 1 1 43s
prometheus-prometheus-oper-operator 1/1 1 1 43s
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
prometheus-grafana-58dbcdc499-5lr8b 2/2 Running 0 43s
prometheus-kube-state-metrics-57f9c94c59-l2s7s 1/1 Running 0 43s
prometheus-prometheus-node-exporter-2f8n6 1/1 Running 0 43s
prometheus-prometheus-node-exporter-2ls55 1/1 Running 0 43s
prometheus-prometheus-node-exporter-4hhp6 1/1 Running 0 43s
prometheus-prometheus-node-exporter-5fppk 1/1 Running 0 43s
prometheus-prometheus-node-exporter-64k2d 1/1 Running 0 43s
prometheus-prometheus-node-exporter-6tfsz 1/1 Running 0 43s
prometheus-prometheus-node-exporter-b95q9 1/1 Running 0 43s
prometheus-prometheus-node-exporter-pftsx 1/1 Running 0 43s
prometheus-prometheus-node-exporter-pvq66 1/1 Running 0 43s
prometheus-prometheus-node-exporter-r8str 1/1 Running 0 43s
prometheus-prometheus-oper-operator-6988ccdddd-f8gf9 2/2 Running 0 42s
==> v1/Prometheus
NAME AGE
prometheus-prometheus-oper-prometheus 42s
==> v1/PrometheusRule
NAME AGE
prometheus-prometheus-oper-alertmanager.rules 41s
prometheus-prometheus-oper-etcd 40s
prometheus-prometheus-oper-general.rules 39s
prometheus-prometheus-oper-k8s.rules 38s
prometheus-prometheus-oper-kube-apiserver.rules 38s
prometheus-prometheus-oper-kube-prometheus-node-alerting.rules 38s
prometheus-prometheus-oper-kube-prometheus-node-recording.rules 38s
prometheus-prometheus-oper-kube-scheduler.rules 38s
prometheus-prometheus-oper-kubernetes-absent 38s
prometheus-prometheus-oper-kubernetes-apps 38s
prometheus-prometheus-oper-kubernetes-resources 38s
prometheus-prometheus-oper-kubernetes-storage 38s
prometheus-prometheus-oper-kubernetes-system 38s
prometheus-prometheus-oper-node-network 38s
prometheus-prometheus-oper-node-time 37s
prometheus-prometheus-oper-node.rules 37s
prometheus-prometheus-oper-prometheus 37s
prometheus-prometheus-oper-prometheus-operator 37s
==> v1/Role
NAME AGE
prometheus-grafana-test 44s
==> v1/RoleBinding
NAME AGE
prometheus-grafana-test 44s
==> v1/Secret
NAME TYPE DATA AGE
alertmanager-prometheus-prometheus-oper-alertmanager Opaque 1 46s
prometheus-grafana Opaque 3 46s
==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
prometheus-grafana ClusterIP 10.102.35.148 <none> 80/TCP 44s
prometheus-kube-state-metrics ClusterIP 10.104.125.111 <none> 8080/TCP 44s
prometheus-prometheus-node-exporter ClusterIP 10.105.33.230 <none> 9100/TCP 43s
prometheus-prometheus-oper-alertmanager ClusterIP 10.98.80.7 <none> 9093/TCP 43s
prometheus-prometheus-oper-coredns ClusterIP None <none> 9153/TCP 43s
prometheus-prometheus-oper-kube-controller-manager ClusterIP None <none> 10252/TCP 43s
prometheus-prometheus-oper-kube-etcd ClusterIP None <none> 2379/TCP 43s
prometheus-prometheus-oper-kube-proxy ClusterIP None <none> 10249/TCP 43s
prometheus-prometheus-oper-kube-scheduler ClusterIP None <none> 10251/TCP 43s
prometheus-prometheus-oper-operator ClusterIP 10.102.170.247 <none> 8080/TCP,443/TCP 43s
prometheus-prometheus-oper-prometheus ClusterIP 10.109.153.116 <none> 9090/TCP 43s
==> v1/ServiceAccount
NAME SECRETS AGE
prometheus-grafana 1 45s
prometheus-grafana-test 1 45s
prometheus-kube-state-metrics 1 45s
prometheus-prometheus-node-exporter 1 45s
prometheus-prometheus-oper-alertmanager 1 45s
prometheus-prometheus-oper-operator 1 45s
prometheus-prometheus-oper-prometheus 1 45s
==> v1/ServiceMonitor
NAME AGE
prometheus-prometheus-oper-alertmanager 37s
prometheus-prometheus-oper-apiserver 37s
prometheus-prometheus-oper-coredns 37s
prometheus-prometheus-oper-grafana 37s
prometheus-prometheus-oper-kube-controller-manager 37s
prometheus-prometheus-oper-kube-etcd 37s
prometheus-prometheus-oper-kube-proxy 37s
prometheus-prometheus-oper-kube-scheduler 37s
prometheus-prometheus-oper-kube-state-metrics 37s
prometheus-prometheus-oper-kubelet 37s
prometheus-prometheus-oper-node-exporter 37s
prometheus-prometheus-oper-operator 37s
prometheus-prometheus-oper-prometheus 37s
==> v1beta1/ClusterRole
NAME AGE
prometheus-kube-state-metrics 45s
psp-prometheus-prometheus-node-exporter 45s
==> v1beta1/ClusterRoleBinding
NAME AGE
prometheus-kube-state-metrics 44s
psp-prometheus-prometheus-node-exporter 44s
==> v1beta1/DaemonSet
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
prometheus-prometheus-node-exporter 10 10 10 10 10 <none> 43s
==> v1beta1/MutatingWebhookConfiguration
NAME AGE
prometheus-prometheus-oper-admission 42s
==> v1beta1/PodSecurityPolicy
NAME PRIV CAPS SELINUX RUNASUSER FSGROUP SUPGROUP READONLYROOTFS VOLUMES
prometheus-grafana false RunAsAny RunAsAny RunAsAny RunAsAny false configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim
prometheus-grafana-test false RunAsAny RunAsAny RunAsAny RunAsAny false configMap,downwardAPI,emptyDir,projected,secret
prometheus-kube-state-metrics false RunAsAny MustRunAsNonRoot MustRunAs MustRunAs false secret
prometheus-prometheus-node-exporter false RunAsAny RunAsAny MustRunAs MustRunAs false configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim,hostPath
prometheus-prometheus-oper-alertmanager false RunAsAny RunAsAny MustRunAs MustRunAs false configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim
prometheus-prometheus-oper-operator false RunAsAny RunAsAny MustRunAs MustRunAs false configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim
prometheus-prometheus-oper-prometheus false RunAsAny RunAsAny MustRunAs MustRunAs false configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim
==> v1beta1/Role
NAME AGE
prometheus-grafana 44s
==> v1beta1/RoleBinding
NAME AGE
prometheus-grafana 44s
==> v1beta1/ValidatingWebhookConfiguration
NAME AGE
prometheus-prometheus-oper-admission 37s
==> v1beta2/Deployment
NAME READY UP-TO-DATE AVAILABLE AGE
prometheus-grafana 1/1 1 1 43s
NOTES:
The Prometheus Operator has been installed. Check its status by running:
kubectl --namespace monitoring get pods -l "release=prometheus"
Visit https://github.com/coreos/prometheus-operator for instructions on how
to create & configure Alertmanager and Prometheus instances using the Operator.Operator also comes with Grafana installed. To access it I just run the following:
$ kubectl port-forward $(kubectl get pods --selector=app=grafana -n monitoring --output=jsonpath="{.items..metadata.name}") -n monitoring 3000
Forwarding from 127.0.0.1:3000 -> 3000
Forwarding from [::1]:3000 -> 3000
Handling connection for 3000The dashboards that come are pre-populated in this Grafana instance are great. Wish there was a way of seeing all resources for the cluster immediately such as CPU and memory but I can likely build one later.
That is one of the great things about Grafana is that it allows you to take the stored data from the database that it is attached to and build out what you need.