Cover image for post

Adding Prometheus to on premises Kubernetes cluster via Helm

The following is part of a series of posts I wrote called "Repurposing old equipment by building a Kubernetes cluster".

While old equipment by itself is in general not very useful unless you find a particular use case, by combining a number of old devices you can build a more powerful system that can span perhaps a number of use cases. Kubernetes is a perfect candidate to be able to do this. I had a number of old laptops laying about and decided to test this theory out.

Installing Prometheus is pretty simple via Helm:

$ helm install stable/prometheus-operator --name prometheus --namespace monitoring
NAME:   prometheus
LAST DEPLOYED: Sun Sep  8 00:46:13 2019
NAMESPACE: monitoring
STATUS: DEPLOYED

RESOURCES:
==> v1/Alertmanager
NAME                                     AGE
prometheus-prometheus-oper-alertmanager  43s

==> v1/ClusterRole
NAME                                       AGE
prometheus-grafana-clusterrole             45s
prometheus-prometheus-oper-alertmanager    45s
prometheus-prometheus-oper-operator        45s
prometheus-prometheus-oper-operator-psp    45s
prometheus-prometheus-oper-prometheus      44s
prometheus-prometheus-oper-prometheus-psp  44s
psp-prometheus-kube-state-metrics          45s

==> v1/ClusterRoleBinding
NAME                                       AGE
prometheus-grafana-clusterrolebinding      44s
prometheus-prometheus-oper-alertmanager    44s
prometheus-prometheus-oper-operator        44s
prometheus-prometheus-oper-operator-psp    44s
prometheus-prometheus-oper-prometheus      44s
prometheus-prometheus-oper-prometheus-psp  44s
psp-prometheus-kube-state-metrics          44s

==> v1/ConfigMap
NAME                                                          DATA  AGE
prometheus-grafana                                            1     46s
prometheus-grafana-config-dashboards                          1     46s
prometheus-grafana-test                                       1     46s
prometheus-prometheus-oper-apiserver                          1     46s
prometheus-prometheus-oper-controller-manager                 1     46s
prometheus-prometheus-oper-etcd                               1     46s
prometheus-prometheus-oper-grafana-datasource                 1     46s
prometheus-prometheus-oper-k8s-cluster-rsrc-use               1     46s
prometheus-prometheus-oper-k8s-coredns                        1     46s
prometheus-prometheus-oper-k8s-node-rsrc-use                  1     46s
prometheus-prometheus-oper-k8s-resources-cluster              1     46s
prometheus-prometheus-oper-k8s-resources-namespace            1     46s
prometheus-prometheus-oper-k8s-resources-pod                  1     46s
prometheus-prometheus-oper-k8s-resources-workload             1     46s
prometheus-prometheus-oper-k8s-resources-workloads-namespace  1     46s
prometheus-prometheus-oper-kubelet                            1     46s
prometheus-prometheus-oper-nodes                              1     45s
prometheus-prometheus-oper-persistentvolumesusage             1     45s
prometheus-prometheus-oper-pods                               1     45s
prometheus-prometheus-oper-prometheus                         1     45s
prometheus-prometheus-oper-prometheus-remote-write            1     45s
prometheus-prometheus-oper-proxy                              1     45s
prometheus-prometheus-oper-scheduler                          1     45s
prometheus-prometheus-oper-statefulset                        1     45s

==> v1/Deployment
NAME                                 READY  UP-TO-DATE  AVAILABLE  AGE
prometheus-kube-state-metrics        1/1    1           1          43s
prometheus-prometheus-oper-operator  1/1    1           1          43s

==> v1/Pod(related)
NAME                                                  READY  STATUS   RESTARTS  AGE
prometheus-grafana-58dbcdc499-5lr8b                   2/2    Running  0         43s
prometheus-kube-state-metrics-57f9c94c59-l2s7s        1/1    Running  0         43s
prometheus-prometheus-node-exporter-2f8n6             1/1    Running  0         43s
prometheus-prometheus-node-exporter-2ls55             1/1    Running  0         43s
prometheus-prometheus-node-exporter-4hhp6             1/1    Running  0         43s
prometheus-prometheus-node-exporter-5fppk             1/1    Running  0         43s
prometheus-prometheus-node-exporter-64k2d             1/1    Running  0         43s
prometheus-prometheus-node-exporter-6tfsz             1/1    Running  0         43s
prometheus-prometheus-node-exporter-b95q9             1/1    Running  0         43s
prometheus-prometheus-node-exporter-pftsx             1/1    Running  0         43s
prometheus-prometheus-node-exporter-pvq66             1/1    Running  0         43s
prometheus-prometheus-node-exporter-r8str             1/1    Running  0         43s
prometheus-prometheus-oper-operator-6988ccdddd-f8gf9  2/2    Running  0         42s

==> v1/Prometheus
NAME                                   AGE
prometheus-prometheus-oper-prometheus  42s

==> v1/PrometheusRule
NAME                                                             AGE
prometheus-prometheus-oper-alertmanager.rules                    41s
prometheus-prometheus-oper-etcd                                  40s
prometheus-prometheus-oper-general.rules                         39s
prometheus-prometheus-oper-k8s.rules                             38s
prometheus-prometheus-oper-kube-apiserver.rules                  38s
prometheus-prometheus-oper-kube-prometheus-node-alerting.rules   38s
prometheus-prometheus-oper-kube-prometheus-node-recording.rules  38s
prometheus-prometheus-oper-kube-scheduler.rules                  38s
prometheus-prometheus-oper-kubernetes-absent                     38s
prometheus-prometheus-oper-kubernetes-apps                       38s
prometheus-prometheus-oper-kubernetes-resources                  38s
prometheus-prometheus-oper-kubernetes-storage                    38s
prometheus-prometheus-oper-kubernetes-system                     38s
prometheus-prometheus-oper-node-network                          38s
prometheus-prometheus-oper-node-time                             37s
prometheus-prometheus-oper-node.rules                            37s
prometheus-prometheus-oper-prometheus                            37s
prometheus-prometheus-oper-prometheus-operator                   37s

==> v1/Role
NAME                     AGE
prometheus-grafana-test  44s

==> v1/RoleBinding
NAME                     AGE
prometheus-grafana-test  44s

==> v1/Secret
NAME                                                  TYPE    DATA  AGE
alertmanager-prometheus-prometheus-oper-alertmanager  Opaque  1     46s
prometheus-grafana                                    Opaque  3     46s

==> v1/Service
NAME                                                TYPE       CLUSTER-IP      EXTERNAL-IP  PORT(S)           AGE
prometheus-grafana                                  ClusterIP  10.102.35.148   <none>       80/TCP            44s
prometheus-kube-state-metrics                       ClusterIP  10.104.125.111  <none>       8080/TCP          44s
prometheus-prometheus-node-exporter                 ClusterIP  10.105.33.230   <none>       9100/TCP          43s
prometheus-prometheus-oper-alertmanager             ClusterIP  10.98.80.7      <none>       9093/TCP          43s
prometheus-prometheus-oper-coredns                  ClusterIP  None            <none>       9153/TCP          43s
prometheus-prometheus-oper-kube-controller-manager  ClusterIP  None            <none>       10252/TCP         43s
prometheus-prometheus-oper-kube-etcd                ClusterIP  None            <none>       2379/TCP          43s
prometheus-prometheus-oper-kube-proxy               ClusterIP  None            <none>       10249/TCP         43s
prometheus-prometheus-oper-kube-scheduler           ClusterIP  None            <none>       10251/TCP         43s
prometheus-prometheus-oper-operator                 ClusterIP  10.102.170.247  <none>       8080/TCP,443/TCP  43s
prometheus-prometheus-oper-prometheus               ClusterIP  10.109.153.116  <none>       9090/TCP          43s

==> v1/ServiceAccount
NAME                                     SECRETS  AGE
prometheus-grafana                       1        45s
prometheus-grafana-test                  1        45s
prometheus-kube-state-metrics            1        45s
prometheus-prometheus-node-exporter      1        45s
prometheus-prometheus-oper-alertmanager  1        45s
prometheus-prometheus-oper-operator      1        45s
prometheus-prometheus-oper-prometheus    1        45s

==> v1/ServiceMonitor
NAME                                                AGE
prometheus-prometheus-oper-alertmanager             37s
prometheus-prometheus-oper-apiserver                37s
prometheus-prometheus-oper-coredns                  37s
prometheus-prometheus-oper-grafana                  37s
prometheus-prometheus-oper-kube-controller-manager  37s
prometheus-prometheus-oper-kube-etcd                37s
prometheus-prometheus-oper-kube-proxy               37s
prometheus-prometheus-oper-kube-scheduler           37s
prometheus-prometheus-oper-kube-state-metrics       37s
prometheus-prometheus-oper-kubelet                  37s
prometheus-prometheus-oper-node-exporter            37s
prometheus-prometheus-oper-operator                 37s
prometheus-prometheus-oper-prometheus               37s

==> v1beta1/ClusterRole
NAME                                     AGE
prometheus-kube-state-metrics            45s
psp-prometheus-prometheus-node-exporter  45s

==> v1beta1/ClusterRoleBinding
NAME                                     AGE
prometheus-kube-state-metrics            44s
psp-prometheus-prometheus-node-exporter  44s

==> v1beta1/DaemonSet
NAME                                 DESIRED  CURRENT  READY  UP-TO-DATE  AVAILABLE  NODE SELECTOR  AGE
prometheus-prometheus-node-exporter  10       10       10     10          10         <none>         43s

==> v1beta1/MutatingWebhookConfiguration
NAME                                  AGE
prometheus-prometheus-oper-admission  42s

==> v1beta1/PodSecurityPolicy
NAME                                     PRIV   CAPS      SELINUX           RUNASUSER  FSGROUP    SUPGROUP  READONLYROOTFS  VOLUMES
prometheus-grafana                       false  RunAsAny  RunAsAny          RunAsAny   RunAsAny   false     configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim
prometheus-grafana-test                  false  RunAsAny  RunAsAny          RunAsAny   RunAsAny   false     configMap,downwardAPI,emptyDir,projected,secret
prometheus-kube-state-metrics            false  RunAsAny  MustRunAsNonRoot  MustRunAs  MustRunAs  false     secret
prometheus-prometheus-node-exporter      false  RunAsAny  RunAsAny          MustRunAs  MustRunAs  false     configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim,hostPath
prometheus-prometheus-oper-alertmanager  false  RunAsAny  RunAsAny          MustRunAs  MustRunAs  false     configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim
prometheus-prometheus-oper-operator      false  RunAsAny  RunAsAny          MustRunAs  MustRunAs  false     configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim
prometheus-prometheus-oper-prometheus    false  RunAsAny  RunAsAny          MustRunAs  MustRunAs  false     configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim

==> v1beta1/Role
NAME                AGE
prometheus-grafana  44s

==> v1beta1/RoleBinding
NAME                AGE
prometheus-grafana  44s

==> v1beta1/ValidatingWebhookConfiguration
NAME                                  AGE
prometheus-prometheus-oper-admission  37s

==> v1beta2/Deployment
NAME                READY  UP-TO-DATE  AVAILABLE  AGE
prometheus-grafana  1/1    1           1          43s


NOTES:
The Prometheus Operator has been installed. Check its status by running:
  kubectl --namespace monitoring get pods -l "release=prometheus"

Visit https://github.com/coreos/prometheus-operator for instructions on how
to create & configure Alertmanager and Prometheus instances using the Operator.

Operator also comes with Grafana installed. To access it I just run the following:

$ kubectl port-forward $(kubectl get  pods --selector=app=grafana -n  monitoring --output=jsonpath="{.items..metadata.name}") -n monitoring  3000
Forwarding from 127.0.0.1:3000 -> 3000
Forwarding from [::1]:3000 -> 3000
Handling connection for 3000

The dashboards that come are pre-populated in this Grafana instance are great. Wish there was a way of seeing all resources for the cluster immediately such as CPU and memory but I can likely build one later.

Prometheus Node View

That is one of the great things about Grafana is that it allows you to take the stored data from the database that it is attached to and build out what you need.

Comments