Rename EachDatabase.each_database_connection method

The purpose of this change is to provide a better and more structured
way to define `gitlab_schemas` and `database_connections`.

This helps to define applied behavior for connections,
and what schemas the connection holds, and how those
are configured (load balancing, etc.).

- Each DB and schema is named, and contains description.
- Remove code duplication in `gitlab_schema` EE specs
- Make `Database.db_config_names` to require passing `gitlab_schema`
  to ensure that we return configs only having this schema
- Ensure that database connection order is the same
  as in `config/database.yml`

Changelog: added

Rather than using transactional tests, migration testing migrates the
database up after each test, and manually cleans each table.

This could pollute rows into the postgres_async_foreign_key_validations
table during a test run, since migrating up happened after cleaning all
tables.

Instead, clean tables after migrating to delete these rows.

Many of our specs are not considering the case when we
have two connections pointing to the same database.

Addressing: https://gitlab.com/gitlab-org/gitlab/-/issues/391646
Changelog: other

Changelog: fixed
Issue: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/112731

Adds a new `main_clusterwide` connection option that can be configured
in `config/database.yml`. This connection is part of the
[Pods](https://gitlab.com/groups/gitlab-org/-/epics/7582) initiative
(soon to be renamed to "Silos").

This connection won't be configured in production or even GDK any time
soon and as such we've opted to have these code paths untested in CI.
Trying to test this new connection in CI would mean having a separate
`config/database.yml` that ran all RSpec tests again which is just too
expensive for this early stage of the project.

The `main_clusterwide` connection will eventually be used to have a
shared Postgres database that is used by many separate GitLab pods
(silos/instances). We need to share some data like settings, users,
licenses across all these pods while separating pod-specific data like
projects, notes, merge_requests . This allows us to horizontally scale
most of the data but still users can login and access data in any pod.

When this connection is not configured then the `main` connection will
be used (as before) so this change should be a no-op for anyone that
doesn't explicitly go in and add `main_clusterwide` to their
`config/database.yml`.

There is a lot more detail on the architecture at
https://docs.gitlab.com/ee/architecture/blueprints/pods/ but the main
point is that this connection is for "shared" stuff and we're starting
with `application_settings` and `licenses` as these are the simplest
tables to decouple.

Changelog: other

Adds defense against accidental mis-configuration. We need this because
the configuration for two connections to one database is so similar to
two databases.

Signed-off-by: Rémy Coutable <remy@rymai.me>

https://gitlab.com/gitlab-org/gitlab/-/merge_requests/101613
consolidated some of the logic for calling `schema_migrate_down!` to
`spec/support/multiple_databases.rb`, but it introduced some issues
with database cleanup. It appears calling
`with_reestablished_active_record_base` disconnects connections and
clears attribute methods. We now need to:

1. Ensure that tables are deleted before we migrate.
2. Force a refresh of attribute methods after ensuring migrations are
up.

Relates to https://gitlab.com/gitlab-org/gitlab/-/issues/378582

If a CI migration leaves the database behind the current database
schema, we need to ensure that the schema is brought up-to-date after
the migrations run to prevent other tests from starting from an
inconsistent state.

This commit consolidates the database cleanup for multiple databases
by ensuring:

1. After a migration context is complete, either recreate the
databases (if the number of columns approaches the max) or migrate all
the schemas back to the latest and drop all data from both databases.

2. After a migration example runs, delete any leftover data for that
specific database.  If a CI migration leaves the database behind the
current database schema, we need to ensure that the schema is brought
up-to-date after the migrations run to prevent putting other tests in
an inconsistent state.

Related MR:
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/100233

Closes https://gitlab.com/gitlab-org/gitlab/-/issues/377796



Changelog: fixed
Signed-off-by: Rémy Coutable <remy@rymai.me>

This reverts merge request !100455

If a CI migration leaves the database behind the current database
schema, we need to ensure that the schema is brought up-to-date after
the migrations run to prevent other tests from starting from an
inconsistent state.

This commit consolidates the database cleanup for multiple databases
by ensuring:

1. After a migration context is complete, either recreate the
databases (if the number of columns approaches the max) or migrate all
the schemas back to the latest and drop all data from both databases.

2. After a migration example runs, delete any leftover data for that
specific database.  If a CI migration leaves the database behind the
current database schema, we need to ensure that the schema is brought
up-to-date after the migrations run to prevent putting other tests in
an inconsistent state.

Related MR:
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/100233

Closes https://gitlab.com/gitlab-org/gitlab/-/issues/377796

Changelog: fixed

Some GitLab classes, such as `Gitlab::Database::AsWithMaterialized`,
need to check the PostgreSQL version to determine whether certain
keywords can be used. We have some specs that specify exact query
types or check for N+1 queries that might be thrown off by an extra
`SELECT VERSION()` call. To ensure this doesn't happen, retrieve the
version at the beginning of the test.

Note that the initializer `config/initializers/1_postgresql_only.rb`
already attempts to check the PostgreSQL version, but the result isn't
memoized properly for callers that use
`ApplicationRecord.database.version`.

This came up in
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/99097#note_1118427731.

Changelog: changed

Currently the `:migration` helper will run all
migrations in context of `:main`. In some cases
when using `restrict_gitlab_migration gitlab_schema: :gitlab_main`
such migrations will be skipped.

Change `mock_ci_connection` to only setup a mocked
`Ci::ApplicationRecord` connection when running in a single database
mode. When running with multiple databases, we can use the actual
`Ci::ApplicationRecord` connection.

Since LB is initialised globally in a non-decomposed
run we will receive connection of `ActiveRecord::Base`.

Ensure that `GitlabSchemasMetrics` can use FF

This analyzers output is Prometheus metrics observing connections
vs gitlab schemas of executed queries.

It will present well in a single metric how many connections
do cross-join or are misplaced once additional databases
are configured.

The `db_config_name` describes a type of connection being
used by a given query.

Changelog: added