This commit creates the routing table for ci_stages.

Changelog: other

Changelog: changed

This commit introduces a FF which will allow us to read
Ci::JobArtifact from our new partitioned table.

This table belonged to an old version of product
analytics that is no longer used.

No production has been stored here for years.

Changelog: removed

This commit creates the routing table for ci_job_artifacts.

Changelog: other

Adds an FK from p_ci_builds to ci_stages now that the partitiones have the foreign key

This commit also introduces synchronous validations for the FKs in the partitions, for self-managed

Changelog: other

It adds a new partitioned foreign key between the
partitions of `p_ci_builds` and `ci_stages` that
will be used for partitioning this table.

https://docs.gitlab.com/ee/development/database/partitioning/list.html#step-4---enforce-foreign-key-constraint

Changelog: other

EE: true
Changelog: added

Creates the partitioned table for ci_pipeline_variables
and attaches the table as the first partition.

Changelog: other

This MR cascades partition_id from pipeline to
Ci::PipelineConfig. We need to add this new column in order
to redefine the FK association between ci_pipeline_artifacts
and ci_pipelines.

Changelog: added

This MR cascades partition_id from pipeline to
Ci::PipelineArtifact. We need to add this new column in order
to redefine the FK association between ci_pipeline_artifacts
and ci_pipelines.

Changelog: added

This MR cascades partition_id from pipeline to
Ci::PipelineMetadata. We need to add this new column in order
to redefine the FK association between ci_pipeline_metadata
and ci_pipelines.

Changelog: added

This change adds a cache table for storing namespace descendants and
project ids so we can retrieve them efficiently.

Changelog: added

As part of the
[Cells](https://docs.gitlab.com/ee/architecture/blueprints/cells/)
effort we need to classify all tables as `gitlab_main_cell` or
`gitlab_main_clusterwide`. In the future these tables will be split into
different databases. For now when we classify these tables we will start
to get errors in CI for
[cross-joins](https://docs.gitlab.com/ee/development/database/multiple_databases.html#removing-joins-between-ci-and-non-ci-tables),
[cross-transactions](https://docs.gitlab.com/ee/development/database/multiple_databases.html#removing-cross-database-transactions)
and
[cross-database foreign keys](https://docs.gitlab.com/ee/development/database/multiple_databases.html#foreign-keys-that-cross-databases).

This MR adds an allowlist in the `db/docs` file per type to avoid seeing
these failures once we classify the table as `gitlab_main_cell`. This is
to to support
[automatically labelling hundreds of tables](https://gitlab.com/gitlab-org/gitlab/-/issues/429172)
in bulk with `gitlab_main_cell` (as part of the
[Organization
Isolation](https://docs.gitlab.com/ee/architecture/blueprints/organization/isolation.html)
effort)
. This will result in lots of pipeline failures
and we want to deal with these those cross-join (etc.) fixes later. For now
we just want to classify things.

You can see the example format of allowing in the
`db/docs/achievements.yml` change which was chosen as an arbitrary
example to prove that this works.

Changelog: added
EE: true

This commit cascades partition_id from the pipeline, we need
to add this new column in order to redefine the FK association
between ci_pipeline_chat_data and ci_pipelines. As in order to
partition ci_pipelines all references FK must include the partition
key.

Changelog: added

As part of our Cells effort we are going to need to backfill a
`sharding_key` to many tables. The `sharding_key` can be any foreign key
that references `projects`, `namespaces` or `organizations`. You can
read more about this at
https://docs.gitlab.com/ee/development/database/multiple_databases.html#defining-a-sharding-key-for-all-cell-local-tables
.The enforcement of having a `sharding_key` for new tables was already
implemented in
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/136682 .

Additionally we have recently shown examples of how we plan to backfill
a sharding_key to a table that is missing it in
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/136800 .

This MR extends this work by allowing a very minimal way to declare in
the `db/docs` file a `desired_sharding_key` a strategy to backfill a new
column that will eventually become the `sharding_key`. An example has
been given in this MR which looks like:

```yml
desired_sharding_key:
  project_id:
    references: projects
    backfill_via:
      parent:
        foreign_key: scanner_id
        table: vulnerability_scanners
        sharding_key: project_id
        belongs_to: scanner
```

To understand best how this YML data will be used you can map it onto
the merge request we created manually in
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/136800 . The idea
will be to automatically create this. The content of the YML specifies
the parent table and it's sharding_key to backfill from in the batched
background migration. It also specifies a `belongs_to` relation which
will be added to the model to automatically populate the sharding_key in
the `before_save`.

The idea behind this is to save developers the effort of creating MRs
which add the new column and writing a batched background migration to
backfill this column and all the followup work related to that. Instead
we will be building automation that uses this data in the YML file to
automatically create all the merge requests for creating the column and
backfilling it.

Adds catalog_resource_sync_events table and model
to serve as a queue of catalog resources that need to be synchronized
with data changes to their associated `projects` records. PG triggers
add SyncEvent records when a catalog resource is created or
when a relevant column in an associated project is updated.
A worker processes the sync events.

Changelog: added

Changelog: performance

This change adds the calls partitionable method on 
job_artifact_state scoped under job_artifact

Changelog: added

Resolves https://gitlab.com/gitlab-org/gitlab/-/issues/429163

Resolves https://gitlab.com/gitlab-org/gitlab/-/issues/429181

Resolves https://gitlab.com/gitlab-org/gitlab/-/issues/429162

Changelog: other

This is causing a master broken in
https://gitlab.com/gitlab-org/quality/engineering-productivity/master-broken-incidents/-/issues/4019

This was merged in 
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/133566
and must have somehow conflicted with a rubocop rule we were adding.

JH expects to use a separate database, which referred to the solution
of adding Geo database. So it needs upstream to change some
code to support it.

The detail description is in the issue:
https://gitlab.com/gitlab-jh/gitlab-jh-enablement/-/issues/343

And the JH whole draft MR can be found here:
!1898

Drops the index index_namespaces_on_created_at which is not used on
GitLab.com.

See https://gitlab.com/gitlab-org/gitlab/-/issues/425974.

Changelog: other

Adds support code to start requiring milestones as well as including the
current milestone when generating a new migration.

Update some failing tests

Slight updates

Changelog: other

Adds restrictions for predefined work item types.

Changelog: added

Changelog: added
EE: true

This change extends the analytics_cycle_analytics_issue_stage_events
with the weight and spring_id columns.

Changelog: added

Changelog: other

The asynchronous index was added in
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/130007

related to https://gitlab.com/gitlab-org/gitlab/-/issues/418970
resolves https://gitlab.com/gitlab-org/gitlab/-/issues/423541

Changelog: changed
EE: true

Changelog: other

Changelog: added

There is an existing `has_many`/`belongs_to` association between the
`vulnerabilities` and `vulnerability_occurrences` tables.

We want to make this a legit 1-to-1 relationship eventually. As a step
towards that we are adding an `id` column that will eventually be used
for a foreign ey constraint.

after adding the column and updating the app code to write to the
column, we need to backfill the new column. [MR 130058][0] will handle
the backfill

resolves: https://gitlab.com/gitlab-org/gitlab/-/issues/418970

[0]:https://gitlab.com/gitlab-org/gitlab/-/merge_requests/130058

Changelog: changed
EE: true

Adding tests to detect duplicate btree indexes. This will
help us avoid adding unnecessary indexes

Changelog: other