diff --git a/doc/user/project/integrations/prometheus_library/index.md b/doc/user/project/integrations/prometheus_library/index.md index 9b9b4f6c8ca05b5fa1a14848ed8e8db21458ab0f..a79bc2bce06645af88a35d5912298c7c289f4735 100644 --- a/doc/user/project/integrations/prometheus_library/index.md +++ b/doc/user/project/integrations/prometheus_library/index.md @@ -10,7 +10,8 @@ Currently supported exporters are: - [Kubernetes](kubernetes.md) - [NGINX](nginx.md) -- [NGINX Ingress Controller](nginx_ingress.md) +- [NGINX Ingress Controller 0.9.0-0.15.x](nginx_ingress_vts.md) +- [NGINX Ingress Controller 0.16.0+](nginx_ingress.md) - [HAProxy](haproxy.md) - [Amazon Cloud Watch](cloudwatch.md) diff --git a/doc/user/project/integrations/prometheus_library/nginx_ingress.md b/doc/user/project/integrations/prometheus_library/nginx_ingress.md index d5f77d622be91e783789f2825d75c32c64046541..b7601f268022410caa5a91c6c1d3a09c06181ab3 100644 --- a/doc/user/project/integrations/prometheus_library/nginx_ingress.md +++ b/doc/user/project/integrations/prometheus_library/nginx_ingress.md @@ -1,8 +1,10 @@ # Monitoring NGINX Ingress Controller -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/13438) in GitLab 9.5. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22133) in GitLab 11.7. -GitLab has support for automatically detecting and monitoring the Kubernetes NGINX ingress controller. This is provided by leveraging the built in Prometheus metrics included in [version 0.9.0](https://github.com/kubernetes/ingress-nginx/blob/master/Changelog.md#09-beta1) and above of the ingress. +NOTE: **Note:** NGINX Ingress versions prior to 0.16.0 offer an included [VTS Prometheus metrics exporter](nginx_ingress_vts.md), which exports metrics different than the built-in metrics. + +GitLab has support for automatically detecting and monitoring the Kubernetes NGINX ingress controller. This is provided by leveraging the built-in Prometheus metrics included starting with [version 0.16.0](https://github.com/kubernetes/ingress-nginx/blob/master/Changelog.md#0160). ## Requirements @@ -12,9 +14,9 @@ GitLab has support for automatically detecting and monitoring the Kubernetes NGI | Name | Query | | ---- | ----- | -| Throughput (req/sec) | sum(rate(nginx_upstream_responses_total{upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"}[2m])) by (status_code) | -| Latency (ms) | avg(nginx_upstream_response_msecs_avg{upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"}) | -| HTTP Error Rate (%) | sum(rate(nginx_upstream_responses_total{status_code="5xx", upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"}[2m])) / sum(rate(nginx_upstream_responses_total{upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"}[2m])) * 100 | +| Throughput (req/sec) | sum(label_replace(rate(nginx_ingress_controller_requests{namespace="%{kube_namespace}",ingress=~".*%{ci_environment_slug}.*"}[2m]), "status_code", "${1}xx", "status", "(.)..")) by (status_code) | +| Latency (ms) | sum(rate(nginx_ingress_controller_ingress_upstream_latency_seconds_sum{namespace="%{kube_namespace}",ingress=~".*%{ci_environment_slug}.*"}[2m])) / sum(rate(nginx_ingress_controller_ingress_upstream_latency_seconds_count{namespace="%{kube_namespace}",ingress=~".*%{ci_environment_slug}.*"}[2m])) * 1000 | +| HTTP Error Rate (%) | sum(rate(nginx_ingress_controller_requests{status=~"5.*",namespace="%{kube_namespace}",ingress=~".*%{ci_environment_slug}.*"}[2m])) / sum(rate(nginx_ingress_controller_requests{namespace="%{kube_namespace}",ingress=~".*%{ci_environment_slug}.*"}[2m])) * 100 | ## Configuring NGINX ingress monitoring @@ -22,9 +24,9 @@ If you have deployed NGINX Ingress using GitLab's [Kubernetes cluster integratio For other deployments, there is [some configuration](#manually-setting-up-nginx-ingress-for-prometheus-monitoring) required depending on your installation: -- NGINX Ingress should be version 0.9.0 or above, with metrics enabled -- NGINX Ingress should be annotated for Prometheus monitoring -- Prometheus should be configured to monitor annotated pods +- NGINX Ingress should be version 0.16.0 or above, with metrics enabled. +- NGINX Ingress should be annotated for Prometheus monitoring. +- Prometheus should be configured to monitor annotated pods. ### About managed NGINX Ingress deployments @@ -32,9 +34,9 @@ NGINX Ingress is deployed into the `gitlab-managed-apps` namespace, using the [o NGINX is configured for Prometheus monitoring, by setting: -- `enable-vts-status: "true"`, to export Prometheus metrics -- `prometheus.io/scrape: "true"`, to enable automatic discovery -- `prometheus.io/port: "10254"`, to specify the metrics port +- `enable-vts-status: "true"`, to export Prometheus metrics. +- `prometheus.io/scrape: "true"`, to enable automatic discovery. +- `prometheus.io/port: "10254"`, to specify the metrics port. When used in conjunction with the GitLab deployed Prometheus service, response metrics will be automatically collected. @@ -51,6 +53,6 @@ Managing these settings depends on how NGINX ingress has been deployed. If you h ## Specifying the Environment label -In order to isolate and only display relevant metrics for a given environment, GitLab needs a method to detect which labels are associated. To do this, GitLab will search for metrics with appropriate labels. In this case, the `upstream` label must be of the form `<KUBE_NAMESPACE>-<CI_ENVIRONMENT_SLUG>-*`. +In order to isolate and only display relevant metrics for a given environment, GitLab needs a method to detect which labels are associated. To do this, GitLab will search for metrics with appropriate labels. In this case, the `ingress` label must `<CI_ENVIRONMENT_SLUG>`. If you have used [Auto Deploy](../../../../topics/autodevops/index.md#auto-deploy) to deploy your app, this format will be used automatically and metrics will be detected with no action on your part. diff --git a/doc/user/project/integrations/prometheus_library/nginx_ingress_vts.md b/doc/user/project/integrations/prometheus_library/nginx_ingress_vts.md new file mode 100644 index 0000000000000000000000000000000000000000..081eb8732ad3343a3c1cbe0adf3297a3690e8aca --- /dev/null +++ b/doc/user/project/integrations/prometheus_library/nginx_ingress_vts.md @@ -0,0 +1,58 @@ +# Monitoring NGINX Ingress Controller with VTS metrics + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/13438) in GitLab 9.5. + +NOTE: **Note:** [NGINX Ingress version 0.16](nginx_ingress.md) and above have built-in Prometheus metrics, which are different than the VTS based metrics. + +GitLab has support for automatically detecting and monitoring the Kubernetes NGINX ingress controller. This is provided by leveraging the included VTS Prometheus metrics exporter in [version 0.9.0](https://github.com/kubernetes/ingress-nginx/blob/master/Changelog.md#09-beta1) through [0.15.x](https://github.com/kubernetes/ingress-nginx/blob/master/Changelog.md#0150). + +## Requirements + +[Prometheus integration](../prometheus.md) must be active. + +## Metrics supported + +| Name | Query | +| ---- | ----- | +| Throughput (req/sec) | sum(rate(nginx_upstream_responses_total{upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"}[2m])) by (status_code) | +| Latency (ms) | avg(nginx_upstream_response_msecs_avg{upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"}) | +| HTTP Error Rate (%) | sum(rate(nginx_upstream_responses_total{status_code="5xx", upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"}[2m])) / sum(rate(nginx_upstream_responses_total{upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"}[2m])) * 100 | + +## Configuring NGINX ingress monitoring + +If you have deployed NGINX Ingress using GitLab's [Kubernetes cluster integration](../../clusters/index.md#installing-applications), it will [automatically be monitored](#about-managed-nginx-ingress-deployments) by Prometheus. + +For other deployments, there is [some configuration](#manually-setting-up-nginx-ingress-for-prometheus-monitoring) required depending on your installation: + +- NGINX Ingress should be version 0.9.0 or above, with metrics enabled. +- NGINX Ingress should be annotated for Prometheus monitoring. +- Prometheus should be configured to monitor annotated pods. + +### About managed NGINX Ingress deployments + +NGINX Ingress is deployed into the `gitlab-managed-apps` namespace, using the [official Helm chart](https://github.com/kubernetes/charts/tree/master/stable/nginx-ingress). NGINX Ingress will be [externally reachable via the Load Balancer's IP](../../clusters/index.md#getting-the-external-ip-address). + +NGINX is configured for Prometheus monitoring, by setting: + +- `enable-vts-status: "true"`, to export Prometheus metrics. +- `prometheus.io/scrape: "true"`, to enable automatic discovery. +- `prometheus.io/port: "10254"`, to specify the metrics port. + +When used in conjunction with the GitLab deployed Prometheus service, response metrics will be automatically collected. + +### Manually setting up NGINX Ingress for Prometheus monitoring + +Version 0.9.0 and above of [NGINX ingress](https://github.com/kubernetes/ingress-nginx) have built-in support for exporting Prometheus metrics. To enable, a ConfigMap setting must be passed: `enable-vts-status: "true"`. Once enabled, a Prometheus metrics endpoint will start running on port 10254. + +Next, the ingress needs to be annotated for Prometheus monitoring. Two new annotations need to be added: + +- `prometheus.io/scrape: "true"` +- `prometheus.io/port: "10254"` + +Managing these settings depends on how NGINX ingress has been deployed. If you have deployed via the [official Helm chart](https://github.com/kubernetes/charts/tree/master/stable/nginx-ingress), metrics can be enabled with `controller.stats.enabled` along with the required annotations. Alternatively it is possible edit the NGINX ingress YML directly in the [Kubernetes dashboard](https://github.com/kubernetes/dashboard). + +## Specifying the Environment label + +In order to isolate and only display relevant metrics for a given environment, GitLab needs a method to detect which labels are associated. To do this, GitLab will search for metrics with appropriate labels. In this case, the `upstream` label must be of the form `<KUBE_NAMESPACE>-<CI_ENVIRONMENT_SLUG>-*`. + +If you have used [Auto Deploy](../../../../topics/autodevops/index.md#auto-deploy) to deploy your app, this format will be used automatically and metrics will be detected with no action on your part.