This commit removes old indexes and foreign keys which are no
longer being used.

Changelog: performance

 Remove package_registry duplicated indexes that already covered by other indexes.

 Changelog: changed

This code was merged into TestProf gem so we can now rely
on the gem

Changelog: changed

This commit adds new indexes including the partition_id key in
order to partition ci_job_artifacts table.

Changelog: added

Changelog: other

EE: true
Changelog: changed

This is to step to remove the trigger and old integer column
See https://docs.gitlab.com/ee/development/database/avoiding_downtime_in_migrations.html#remove-the-trigger-and-old-integer-columns-release-n--2

Changelog: changed

This is a duplicated index.

Changelog: performance

This is a duplicated index.

Changelog: performance

This is a duplicated index.

Changelog: performance

This is a duplicated index.

Changelog: performance

This is a duplicated index.

Changelog: performance

This is a duplicated index.

Changelog: performance

This is a duplicated index.

Changelog: performance
Relates to: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/135373

This is a duplicated index.

Changelog: performance
Relates to: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/135360

Removes index_merge_requests_on_author_id as it's
already covered by another index

Changelog: changed

This is a duplicated index.

Changelog: performance
Relates to: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/135370

Remove duplicated index.

Relates to: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/135366
Changelog: performance

Removes index_merge_request_assignees_on_merge_request_id
as is already covered by another index

Changelog: changed

Remove constraint, index and campaign column from
in_product_marketing_emails table

Changelog: removed

This change removes 2 redundant database indexes wihtin the
features of the Optimize team.

Changelog: removed

Changelog: removed

This is the second step to synchronously create indexes for ci_stages

See https://docs.gitlab.com/ee/development/database/adding_database_indexes.html#add-a-migration-to-create-the-index-synchronously

Changelog: added

Currently we keep pages deployments for about 30m after creating a new
one. This is to avoid racing conditions between the time the new
deployment is created and GitLab Pages is updates its caches with the
latest deployment.

Therefore, while we use this mechanism, we cannot enforce uniqueness on
pages_deployments for project_id and path_prefix, as more than one
deploy for given path_prefix might exist.

Related to:

- Issue that introduced the index:
  - https://gitlab.com/gitlab-org/gitlab/-/issues/416488
- Problem was found while working on
  - https://gitlab.com/gitlab-org/gitlab/-/merge_requests/129534
  - https://gitlab.com/gitlab-org/gitlab/-/issues/416492
Changelog: added

Adding tests to detect duplicate btree indexes. This will
help us avoid adding unnecessary indexes

Changelog: other

Rename EachDatabase.each_database_connection method

The purpose of this change is to provide a better and more structured
way to define `gitlab_schemas` and `database_connections`.

This helps to define applied behavior for connections,
and what schemas the connection holds, and how those
are configured (load balancing, etc.).

- Each DB and schema is named, and contains description.
- Remove code duplication in `gitlab_schema` EE specs
- Make `Database.db_config_names` to require passing `gitlab_schema`
  to ensure that we return configs only having this schema
- Ensure that database connection order is the same
  as in `config/database.yml`

Changelog: added

Rather than using transactional tests, migration testing migrates the
database up after each test, and manually cleans each table.

This could pollute rows into the postgres_async_foreign_key_validations
table during a test run, since migrating up happened after cleaning all
tables.

Instead, clean tables after migrating to delete these rows.

Database::GitlabSchema.table_schema to return nil

Moves the ConvertTableToFirstListPartition class to a List::ConvertTable
class, to prepare for adding additional functionality in the new module.

Many of our specs are not considering the case when we
have two connections pointing to the same database.

Addressing: https://gitlab.com/gitlab-org/gitlab/-/issues/391646
Changelog: other

Changelog: fixed
Issue: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/112731

Adds a new `main_clusterwide` connection option that can be configured
in `config/database.yml`. This connection is part of the
[Pods](https://gitlab.com/groups/gitlab-org/-/epics/7582) initiative
(soon to be renamed to "Silos").

This connection won't be configured in production or even GDK any time
soon and as such we've opted to have these code paths untested in CI.
Trying to test this new connection in CI would mean having a separate
`config/database.yml` that ran all RSpec tests again which is just too
expensive for this early stage of the project.

The `main_clusterwide` connection will eventually be used to have a
shared Postgres database that is used by many separate GitLab pods
(silos/instances). We need to share some data like settings, users,
licenses across all these pods while separating pod-specific data like
projects, notes, merge_requests . This allows us to horizontally scale
most of the data but still users can login and access data in any pod.

When this connection is not configured then the `main` connection will
be used (as before) so this change should be a no-op for anyone that
doesn't explicitly go in and add `main_clusterwide` to their
`config/database.yml`.

There is a lot more detail on the architecture at
https://docs.gitlab.com/ee/architecture/blueprints/pods/ but the main
point is that this connection is for "shared" stuff and we're starting
with `application_settings` and `licenses` as these are the simplest
tables to decouple.

Changelog: other

Adds defense against accidental mis-configuration. We need this because
the configuration for two connections to one database is so similar to
two databases.

We still have many ignored Rubocop Offenses for
Database/MultipleDatabases in specs. This is a MR
to address validate and fix all of them.

Addressing Issue: https://gitlab.com/gitlab-org/gitlab/-/issues/362031

Changelog: other

Signed-off-by: Rémy Coutable <remy@rymai.me>