Fixed:
- Do not determine `alive` status from the pidset returned
  by caller, since this can lead to "bouncing" state.
- Swap loop-check with sleep call, since sleep suspends
  the calling thread, which may lead to the callback
  being invoked if `alive` is invalidated by another
  thread.
- Do not trap INT and TERM by default; this was swallowing
  these signals for the Puma master.

Changelog: fixed

This change:

- Adds a rake task to install gitlab-metrics-exporter
  into a given directory.
- Allows to run it in place of the existing Ruby based
  exporters; this is guarded by an environment variable.
- Extends the existing end-to-end test to test both
  Ruby and Go implementations.

There was a slight chance that the metrics directory got
wiped after workers had started up, which would result
in their metrics missing.

We were accidentally deleting metrics the exporter
server exported about itself. Actual sidekiq worker
metrics were not affected.

This happened because we would wipe the metrics dir
whenever sidekiq_0 starts up, but this happens after
the metrics server starts, so the worker was deleting
those existing metrics.

Changelog: fixed

This moves us away from a process-internal server thread
and toward an external process that we supervise instead.

This extracts the process supervision logic
from sidekiq-cluster into a helper module.

We will re-use this for Puma soon.

This provides better memory use for Puma, and we
will require this for running a non-Ruby server
in the future.

This does not actually start the server for Puma
yet.

These methods are not directly related to SidekiqCluster process
management. They are currently only used by sidekiq cluster,
but in the future they may be used in other places as well.

This prevents name clashes with library code that
is auto-loaded in workers from lib/

When using Helm charts, which are YAML files, it's relatively easy to
accidentally include a newline in the Sidekiq arguments. This will
typically be a trailing newline, which shouldn't be a problem, except
that in our cloud-native deployment we launch Sidekiq using the command
given by `sidekiq-cluster --dryrun` (to avoid having the wrapper
process), so we effectively do:

    $(bin/sidekiq-cluster --dryrun $ARGS)

Ruby's `Shellwords.escape` quotes newlines as `'\n'`:
https://github.com/ruby/shellwords/blob/v0.1.0/lib/shellwords.rb#L161-L163

When those are parsed back in the arguments array we get `foo'\n'` for
an original argument of `foo\n`. Then, when we apply splitting, we get
`foo'` and `'` as two separate arguments.

The quoting here is quite complicated but the below demonstrates (in
Bash) what's happening:

    $ ruby -e 'p ARGV; p ARGV.first.split' "foo'"$'\n'"'"
    ["foo'\n'"]
    ["foo'", "'"]

If we did care about newlines in this argument array, the solution would
be more involved. But we don't, because they are never valid, so we can
just error when we encounter them.

Changelog: fixed

The --dryrun option shows the full command sidekiq-cluster will run.
This new option just lists the matched queues (mostly useful in
conjunction with --queue-selector), one per line.

This is now just called `--queue-selector`.

Changelog: removed

We still support either the `--queue-selector` or
`--experimental-queue-selector` flag (not both at the same time,
though). We'll remove the 'experimental' variant on or after 14.0.

Makes the necessary changes to KILL the orphaned
Sidekiq process if the TERM can't make it within
5 seconds.

Considering we're moving towards using Sidekiq Cluster
only and customers using Omnibus are allowed to configure
a Sidekiq timeout through -t flag, here we support a
-t flag for `bin/sidekiq-cluster`.

Here's how it works:

Internally, we'll pass the given value to Sidekiq (`-t`) flag.
For whatever value given, Sidekiq Cluster will wait for 5
extra seconds in order to hard stop all remaining alive processes.

So for instance, if `10` is given, Sidekiq will try to gracefully
terminate within this time. If it gets stuck for some reason, the
processes will be sigkilled within `15` seconds (total wait time).

This is useful if you want to expand queues using the query selector,
but don't want to actually start the process (perhaps because you have
your own process supervisor).

The decision of moving the script back to Core
has the goal of making sidekiq-cluster the new
default (instead sidekiq). A series of steps
will be required for doing so, and these can
be seen at:

https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/181

Because it was designed for selecting specific queues, sidekiq-cluster
didn't have a convenient method to select all queues. As we want this to
become the default sidekiq entry point for all GitLab installations, and
the default is to handle all queues in a single process, we need that
option.

(This was possible before: using `--experimental-queue-selector`, you
could ask for something like `has_external_dependencies=true,false`, but
that's not obvious.)

Instead of making `*` a wildcard anywhere, which would be more general,
this commit just allows a single `*` in a queue group to represent all
queues. While this is less general, it's also simpler to implement, and
we can assume that YAGNI for the wildcards.

This avoids the error messages caused by loading
`Sidekiq::JSONFormatter` twice:

warning: already initialized constant
Gitlab::SidekiqLogging::JSONFormatter::TIMESTAMP_FIELDS

We can just re-use the Sidekiq JSON formatter to ensure this happens
since sidekiq-cli doesn't use the Sidekiq initializer.

Part of https://gitlab.com/gitlab-org/gitlab/issues/208922

We want the ability to react quickly to changes, so while this is
documented, we need to make sure that people are aware that using this
while it's still experimental is a significant risk of having to deal
with breaking changes.

queue-query-syntax is really hard to type!

This ensures that --queue-query-syntax works correctly when invoked from
the command line, without the rest of the Rails environment, as well as
adding documentation and unit-level tests.

Allow setting a minimum concurrency for sidekiq-cluster, as well as the
existing maximum. These can be set to the same value to enforce
concurrency at a particular number of threads, regardless of the number
of queues in the group.

These methods can't use any Rails dependencies, which also means the
SidekiqConfig module definition - i.e. constants - can't use Rails
dependencies either.

Move those methods to their own module to make this clearer. Also switch
to defining methods on the singleton class in both modules to make
private methods more convenient.

Per feedback on MR

Rather than stubbing out some internal methods of the test subject,
use known good values for pid (the current process, and negative values)
Much better code coverage test that way

In https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/8511
and https://gitlab.com/gitlab-com/gl-infra/production/issues/1309
we have seen cases where sidekiq-cluster has been asked to restart,
but it has left one or more sidekiq worker processes in a stuck/hung
state burning CPU.

With this, sidekiq-cluster now waits for the worker processes to
terminate themselves cleanly (and push jobs back onto the queue) and
another few seconds beyond, then if any processes remain, kills them
hard (:KILL signal) before exiting as usual (allowing whater process
monitor is in place to restart sidekiq-cluster)

We currently only measure the global (node-level) business in terms
of jobs being processed. We would instead like to be able to know
for each sidekiq process how saturated it is in terms of actual
concurrency vs requested/max concurrency.

This patch does the following:

- add a worker/process dimension to all existing metrics
- introduce a new metric to observe concurrency per process

by relating "actual job count per process" to "max job count per
process" we can then obtain some sort of saturation metric.

Enables frozen string in the following:

* ee/lib/gitlab/**/*.rb
* ee/lib/omni_auth/**/*.rb
* ee/lib/pseudonymizer/**/*.rb
* ee/lib/system_check/**/*.rb

Partially addresses https://gitlab.com/gitlab-org/gitlab-ce/issues/47424.

By default, sidekiq-cluster will allocate N threads for N queues specified
on the command line, which in turn will require N + 5 Redis connections.
Especially when used with the `--negate` flag, this can lead to using hundreds
of Redis connections, which can lead to performance issues.

This change also adds a `-m N` option to cap the number of threads. For each
queue group, the concurrency factor will be set to min(number of queues, N).

Closes #7374

Signed-off-by: Rémy Coutable <remy@rymai.me>