Deprecated in https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/2760
in favor of the new `concurrency` option.

Changelog: removed

Removal of queue selector and negate options for 17.0

https://gitlab.com/gitlab-org/gitlab/-/merge_requests/149370

Changelog: removed

This MR lays groundwork for chosen relation import
(https://gitlab.com/gitlab-org/gitlab/-/issues/425798) by updating
the relation tree restorer to allow importing of a single relation.

It also adds a service and class which will be called by an API endpoint
in a follow-up MR.

Changelog: changed

When routing rules is not specified, all jobs go to `default` and
`mailers` queue by default. We can safely override
sidekiq queue_groups to default,mailers queues only so that Redis don't
incur overhead of BRPOPing 600+ queues.

Changelog: changed

The new concurrency option is a fixed value that doesnt depend on the
queues length. It is means to be a simple replacement over the
min-concurrency and max-concurrency.
https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/2760

Changelog: added

Changelog: other

This moves two spec helpers into `gitlab-rspec`.

- spec/[b-f]*/**/*.rb

Changelog: other

Add migration to remove workers from queue

Update sidekiq queues yml

Changelog: removed
MR: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/124430

Update default initializer value for Sidekiq routing_rules

See merge request gitlab-org/gitlab!97908

- For better backward compatibility use string keys on
  GitlabSettings::Options
- Remove all the Settingslogic references from code (not comments)
- Ensure to use GitlabSettings::Options when pushing new options to
  Array values
- Instead of inheriting from Hash, wrap a hash in
  GitlabSettings::Options and make it respond to all the requirements.
    * PROS: This provides more autonomy on what needs to be done
    * CONS: Some places were checking by Hash, we need to tackle these
    scenarios
- Ensure to deep_merge return nested Options
- Use the _raw_ version of options when passing to Fog::Storage
- Ensure to load required active_support libraries
- Show the option name in the GitlabSettings::MissingSetting error
- Ensure to have an option `source` for the path of the `gitlab.yml`
- Fix GitlabSettings::Options#deep_merge and
  GitlabSettings::Options#merge to return a new GitlabSettings::Options
  object with the options deep_merged/merged
- Add GitlabSettings#reload! for backward compatibility
- Make file/namespace initializer arguments
- Avoid mutating Settings in sidekiq_cluster/cli tests
- Fix configuration stubbing

Related to: https://gitlab.com/gitlab-org/gitlab/-/issues/286873

Changelog: other

There were an additional number of specs failing
that passed once spec_helper was required instead
of fast_spec_helper

Changelog: fixed

We want both processes here to listen to queues, so pick a fairly
lightly-used feature category with few queues to replace the missing
`chatops` category.

sidekiq-cluster handles process supervision for its child Sidekiq
processes, and terminates itself and all child processes if any child
Sidekiq process exits.

Previously, it always exited with a 0 status code (i.e. success), no
matter how the child process had terminated.

Now it exits with 1 if any child process had a non-zero exit code. This
allows a process supervisor one level up (like systemd) to detect
failures and restart.

Changelog: changed

The previous default of 50 was far too high. We don't use this on
GitLab.com and it's mostly inertia that has left it at this value.

Changelog: changed

This change is part of series of changes that will allow projects to
be exported using multiple jobs

Changelog: other

- Creates a new service that exports project relations separately and
stores the exported relation files in object-store.
- Creates a worker to trigger the generation of the export relation
in the background.

Only `feature_category` was correctly inherited, in addition to
`sidekiq_options` which comes from Sidekiq.

For the other attributes, we need to switch to `get_class_attribute`
in order to also lookup values on parents.

Changelog: changed

Fixed:
- Do not determine `alive` status from the pidset returned
  by caller, since this can lead to "bouncing" state.
- Swap loop-check with sleep call, since sleep suspends
  the calling thread, which may lead to the callback
  being invoked if `alive` is invalidated by another
  thread.
- Do not trap INT and TERM by default; this was swallowing
  these signals for the Puma master.

Changelog: fixed

It has been a long-standing issue that for Sidekiq,
serving metrics and health-checks was coupled into
a single server.

We had already separated settings keys for these two
concerns previously, in anticipation of splitting
these endpoints apart, which is a breaking change.

We can now do this with 15.0.

Changelog: changed

This change:

- Adds a rake task to install gitlab-metrics-exporter
  into a given directory.
- Allows to run it in place of the existing Ruby based
  exporters; this is guarded by an environment variable.
- Extends the existing end-to-end test to test both
  Ruby and Go implementations.

There was a slight chance that the metrics directory got
wiped after workers had started up, which would result
in their metrics missing.

We were accidentally deleting metrics the exporter
server exported about itself. Actual sidekiq worker
metrics were not affected.

This happened because we would wipe the metrics dir
whenever sidekiq_0 starts up, but this happens after
the metrics server starts, so the worker was deleting
those existing metrics.

Changelog: fixed

This moves us away from a process-internal server thread
and toward an external process that we supervise instead.

This extracts the process supervision logic
from sidekiq-cluster into a helper module.

We will re-use this for Puma soon.

This provides better memory use for Puma, and we
will require this for running a non-Ruby server
in the future.

This does not actually start the server for Puma
yet.

Rename `stubbing_settings_source` to `stub_settings_source` to keep
consistency across other support files.

Require this support file in spec where `stub_settings_source: true` is
used.

This avoids order dependent flaky specs like:

  bin/rspec \
    spec/commands/sidekiq_cluster/cli_spec.rb \
    spec/lib/gitlab/pagination/offset_header_builder_spec.rb

Note, both specs about are run with `fast_spec_helper` so previously
they would fail because RSpec trait `stubbing_settings_source` was not
available.

These methods are not directly related to SidekiqCluster process
management. They are currently only used by sidekiq cluster,
but in the future they may be used in other places as well.

This prevents name clashes with library code that
is auto-loaded in workers from lib/

When using Helm charts, which are YAML files, it's relatively easy to
accidentally include a newline in the Sidekiq arguments. This will
typically be a trailing newline, which shouldn't be a problem, except
that in our cloud-native deployment we launch Sidekiq using the command
given by `sidekiq-cluster --dryrun` (to avoid having the wrapper
process), so we effectively do:

    $(bin/sidekiq-cluster --dryrun $ARGS)

Ruby's `Shellwords.escape` quotes newlines as `'\n'`:
https://github.com/ruby/shellwords/blob/v0.1.0/lib/shellwords.rb#L161-L163

When those are parsed back in the arguments array we get `foo'\n'` for
an original argument of `foo\n`. Then, when we apply splitting, we get
`foo'` and `'` as two separate arguments.

The quoting here is quite complicated but the below demonstrates (in
Bash) what's happening:

    $ ruby -e 'p ARGV; p ARGV.first.split' "foo'"$'\n'"'"
    ["foo'\n'"]
    ["foo'", "'"]

If we did care about newlines in this argument array, the solution would
be more involved. But we don't, because they are never valid, so we can
just error when we encounter them.

Changelog: fixed

The --dryrun option shows the full command sidekiq-cluster will run.
This new option just lists the matched queues (mostly useful in
conjunction with --queue-selector), one per line.

This is now just called `--queue-selector`.

Changelog: removed