Gitaly has recently enhanced its configuration checks to detect more
invalid configurations. One of these checks asserts that the secret
required to connect to Rails actually exists on disk -- if it doesn't,
Gitaly will refuse to start.

This has in fact broken the "gitlab:setup" CI job because we do not
ensure that the secret actually exists. Let's fix that.

Changelog: fixed

The purpose of this Gem is to move all `Gitlab::Utils` code
into the Gem, and use this for all common functions that
are not monolith specific.

The purpose of this Gem is to move all `Gitlab::Utils` code
into the Gem, and use this for all common functions that
are not monolith specific.

Gitaly removed it's gitaly-ruby configuration in
gitaly@e361ab48 (config: Remove ruby configuration, 2023-03-16). So we
also can remove writing that section to the configuration toml.

Next to the regular tests where Praefect uses the in-memory election
strategy, this changes adds tests where Praefect uses a PostgreSQL
database for a proper, production-like, election strategy.

At the moment these test fail in CI, so we set them to allow to fail and
only run them when the ~"pipeline:run-praefect-with-db" label is applied
to the MR.

The workhorse integration tests for Gitaly need to have a Gitaly process
running. It uses scripts/setup-test-env to build Gitaly, but it used a
home-grown solution to start the process.

This change switches to the "standard way" of starting Gitaly in test
for workhorse, so it has the same environment variables when set up and
started. This has an added benefit as now the Gitaly socket is tested
before the workhorse tests start.

GitalySetup was shelling out to scripts/gitaly-test-spawn to spawn
Gitaly processes, which was quite complicated to understand.

This change simplifies the back-and-forth between the script and the
Ruby file. It avoids writing the PID of the Gitaly processes to file, to
pass them back to the test environment.

With the previous changes included, now scripts/gitaly-test-build &
scripts/gitaly-test-spawn are *only* used in CI. When running tests in a
local GDK, the Ruby code to build and spawn Gitaly in test is directly
called from TestEnv.init.

Setting up and spawning Gitaly in tests needs to have some Bundler
settings configured. Until now this was done by writing to the local
config file in the ruby directory inside the gitaly dir used in test.

This method didn't seem to be robust for all our user and
use-cases (i.e. CI), because this configuration _could_ be done from
different places.

To make the situation more predictable, this change injects all Bundler
configuration through environment variables instead.

This reverts merge request !76558

GitalySetup was shelling out to scripts/gitaly-test-spawn to spawn
Gitaly processes, which was quite complicated to understand.

This change simplifies the back-and-forth between the script and the
Ruby file. It avoids writing the PID of the Gitaly processes to file, to
pass them back to the test environment.

With the previous changes included, now scripts/gitaly-test-build &
scripts/gitaly-test-spawn are *only* used in CI. When running tests in a
local GDK, the Ruby code to build and spawn Gitaly in test is directly
called from TestEnv.init.

Setting up and spawning Gitaly in tests needs to have some Bundler
settings configured. Until now this was done by writing to the local
config file in the ruby directory inside the gitaly dir used in test.

This method didn't seem to be robust for all our user and
use-cases (i.e. CI), because this configuration _could_ be done from
different places.

To make the situation more predictable, this change injects all Bundler
configuration through environment variables instead.

This reverts merge request !72328

Setting up and spawning Gitaly in tests needs to have some Bundler
settings configured. Until now this was done by writing to the local
config file in the ruby directory inside the gitaly dir used in test.

This method didn't seem to be robust for all our user and
use-cases (i.e. CI), because this configuration _could_ be done from
different places.

To make the situation more predictable, this change injects all Bundler
configuration through environment variables instead. It sets
BUNDLE_IGNORE_CONFIG to make sure any existing configuration written to
file is ignored, this includes the .bundle/config inside the project and
possibly in the user's $HOME directory.

This makes sure unit and integration tests run when this file is
touched.

Signed-off-by: Rémy Coutable <remy@rymai.me>

Signed-off-by: Rémy Coutable <remy@rymai.me>

Auto-corrected with Rubocop

The _build dir is 211MB (uncompressed) but the executables are moved to
the top gitaly folder after they're compiled, so there's no need to save
the _build dir in the artifacts of the setup-test-env job.

That should speed up the artifacts upload/download phases for
setup-test-env and all the jobs that depend on it.

Signed-off-by: Rémy Coutable <remy@rymai.me>

In the test environment, we go through the trouble of spinning up Gitaly
and Praefect, only to bypass Praefect entirely and go directly to the
Gitaly socket. This renders the Praefect step useless and causes us to
miss errors in proxying.

To fix this, we now proxy all calls through Praefect and add a second
Gitaly shard.

This was discovered in
https://gitlab.com/gitlab-org/gitaly/-/issues/3379.

Praefect functions as a proxy, but there are a few RPCs that are only in
praefect and do not get proxied to gitaly. To test functionality that
hits these RPCs, we need a way to hit a praefect instance that talks to
a gitaly instance.

This MR sets up praefect in front of the existing gitaly that is used in
tests.

Without this patch, I would end up with:

```
An error occurred in a `before(:suite)` hook.
Failure/Error: raise "could not connect to gitaly at #{socket.inspect} after #{sleep_time} seconds"

RuntimeError:
  could not connect to gitaly at "tmp/tests/gitaly/gitaly.socket" after 10 seconds
```

Digging into it, it's because `scripts/gitaly-test-spawn` could not
spawn the process, because it cannot find the installed gems.

I personally installed all my gems under $HOME, namely with:

* `gem install rake --user-install` or:
* `bundle install --path ~/.gem`

The gems would be installed to `~/.gem/ruby/2.4.0/gems`, where
the version is Ruby ABI version.

Now we're changing $HOME, making RubyGems think that the gems
would be installed to `tmp/tests/ruby/2.4.0/gems` which is
apparently not the case.

In order to workaround this, we could preserve $GEM_PATH
populated by RubyGems, ignoring the default path based on $HOME.