sidekiq-cluster handles process supervision for its child Sidekiq
processes, and terminates itself and all child processes if any child
Sidekiq process exits.

Previously, it always exited with a 0 status code (i.e. success), no
matter how the child process had terminated.

Now it exits with 1 if any child process had a non-zero exit code. This
allows a process supervisor one level up (like systemd) to detect
failures and restart.

Changelog: changed

The previous default of 50 was far too high. We don't use this on
GitLab.com and it's mostly inertia that has left it at this value.

Changelog: changed

This moves us away from a process-internal server thread
and toward an external process that we supervise instead.

This extracts the process supervision logic
from sidekiq-cluster into a helper module.

We will re-use this for Puma soon.

These methods are not directly related to SidekiqCluster process
management. They are currently only used by sidekiq cluster,
but in the future they may be used in other places as well.

This prevents name clashes with library code that
is auto-loaded in workers from lib/

`tally` is introduced since Ruby 2.7 and also adds small performance
gain compared to the old approach (~ 1.6x).

- enforce standards...

Considering we're moving towards using Sidekiq Cluster
only and customers using Omnibus are allowed to configure
a Sidekiq timeout through -t flag, here we support a
-t flag for `bin/sidekiq-cluster`.

Here's how it works:

Internally, we'll pass the given value to Sidekiq (`-t`) flag.
For whatever value given, Sidekiq Cluster will wait for 5
extra seconds in order to hard stop all remaining alive processes.

So for instance, if `10` is given, Sidekiq will try to gracefully
terminate within this time. If it gets stuck for some reason, the
processes will be sigkilled within `15` seconds (total wait time).

This is useful if you want to expand queues using the query selector,
but don't want to actually start the process (perhaps because you have
your own process supervisor).

The decision of moving the script back to Core
has the goal of making sidekiq-cluster the new
default (instead sidekiq). A series of steps
will be required for doing so, and these can
be seen at:

https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/181

Because it was designed for selecting specific queues, sidekiq-cluster
didn't have a convenient method to select all queues. As we want this to
become the default sidekiq entry point for all GitLab installations, and
the default is to handle all queues in a single process, we need that
option.

(This was possible before: using `--experimental-queue-selector`, you
could ask for something like `has_external_dependencies=true,false`, but
that's not obvious.)

Instead of making `*` a wildcard anywhere, which would be more general,
this commit just allows a single `*` in a queue group to represent all
queues. While this is less general, it's also simpler to implement, and
we can assume that YAGNI for the wildcards.

Allow setting a minimum concurrency for sidekiq-cluster, as well as the
existing maximum. These can be set to the same value to enforce
concurrency at a particular number of threads, regardless of the number
of queues in the group.

Adds critical clarifying comments about the meaning of signal 0, which
is effectively a liveness/existence (and permissions) check
See http://man7.org/linux/man-pages/man2/kill.2.html for a discussion
of that

In https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/8511
and https://gitlab.com/gitlab-com/gl-infra/production/issues/1309
we have seen cases where sidekiq-cluster has been asked to restart,
but it has left one or more sidekiq worker processes in a stuck/hung
state burning CPU.

With this, sidekiq-cluster now waits for the worker processes to
terminate themselves cleanly (and push jobs back onto the queue) and
another few seconds beyond, then if any processes remain, kills them
hard (:KILL signal) before exiting as usual (allowing whater process
monitor is in place to restart sidekiq-cluster)

We currently only measure the global (node-level) business in terms
of jobs being processed. We would instead like to be able to know
for each sidekiq process how saturated it is in terms of actual
concurrency vs requested/max concurrency.

This patch does the following:

- add a worker/process dimension to all existing metrics
- introduce a new metric to observe concurrency per process

by relating "actual job count per process" to "max job count per
process" we can then obtain some sort of saturation metric.

We now kill the whole process group for termination signals

Enables frozen string in the following:

* ee/lib/gitlab/**/*.rb
* ee/lib/omni_auth/**/*.rb
* ee/lib/pseudonymizer/**/*.rb
* ee/lib/system_check/**/*.rb

Partially addresses https://gitlab.com/gitlab-org/gitlab-ce/issues/47424.

Sidekiq 5.2.2 introduced a change
(https://github.com/mperham/sidekiq/pull/3936/files) that prevents queue
names from being duplicated. This breaks our current GitLab.com
config because we have some sidekiq-cluster processes configured
as the following:

```ruby
sidekiq_cluster['queue_groups'] = ["process_commit,process_commit"]
```

To keep things backwards compatible, we now only pass unique queue names
but scale the weights accordingly by the number of queues mentioned.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/54253

By default, sidekiq-cluster will allocate N threads for N queues specified
on the command line, which in turn will require N + 5 Redis connections.
Especially when used with the `--negate` flag, this can lead to using hundreds
of Redis connections, which can lead to performance issues.

This change also adds a `-m N` option to cap the number of threads. For each
queue group, the concurrency factor will be set to min(number of queues, N).

Closes #7374

Signed-off-by: Rémy Coutable <remy@rymai.me>

sidekiq-cluster now supports the same -r/--require option as Sidekiq
itself does. This is needed as Omnibus changes the working directory,
preventing us from properly starting the sidekiq processes.

This redirects STDERR and STDOUT of the various Sidekiq processes to the
sidekiq-cluster process. The logging output of sidekiq-cluster itself
has been changed so it matches Sidekiq's output more closely, making
parsing/grepping easier.

Fixes #1438

These processes are managed by ./bin/sidekiq-cluster. This script can
start multiple Sidekiq processes processing a dedicated list of queues.
Signals sent to the wrapper are forwarded to the underlying Sidekiq
processes.