It has been a long-standing issue that for Sidekiq,
serving metrics and health-checks was coupled into
a single server.

We had already separated settings keys for these two
concerns previously, in anticipation of splitting
these endpoints apart, which is a breaking change.

We can now do this with 15.0.

Changelog: changed

This change:

- Adds a rake task to install gitlab-metrics-exporter
  into a given directory.
- Allows to run it in place of the existing Ruby based
  exporters; this is guarded by an environment variable.
- Extends the existing end-to-end test to test both
  Ruby and Go implementations.

We were accidentally deleting metrics the exporter
server exported about itself. Actual sidekiq worker
metrics were not affected.

This happened because we would wipe the metrics dir
whenever sidekiq_0 starts up, but this happens after
the metrics server starts, so the worker was deleting
those existing metrics.

Changelog: fixed

This moves us away from a process-internal server thread
and toward an external process that we supervise instead.

This provides better memory use for Puma, and we
will require this for running a non-Ruby server
in the future.

This does not actually start the server for Puma
yet.

So far we had only tested this with Sidekiq.
This change makes sure that WebExporter and
the puma metrics directory can be used.

This is a prerequisite for extracting WebExporter
into a separate process.

This prevents metric names from clashing in cases
where multiple processes emit the same metrics
to the same scraper job and which do not have
a pid label.

This cuts down on long-term memory use with
a small increase in request latency.

This adds new Prometheus metrics such as
request count for the metrics server itself,
so we can observe scraper requests to
metrics server instances.

We move override_rails_constants.rb to another location so that it's
closer to its related code in metrics_server.
We also remove the usage in setup-test-env as they differ in usage, and
reusing the code does not make a lot of sense.