It's a scaffold for product usage activity tracking
for GitLab early access program. Only for participants
of the program and only who agreed on optional tracking.

Changelog: added

Changelog: deprecated

We need to explicitly set an Organization when creating a new user

Adds a new partitioned table p_catalog_resource_component_usages.
This table will track when a CI component is included in a project's
pipeline. Only unique records of component-used_by_project-used_date
are inserted.

This data will be used to display component usage popularity
aggregated by component or catalog_resource per 30 day rolling window.

This table is partitioned by month. For now, the
table will drop partitions with data older than 12 months. This
duration will likely be reduced in a follow up issue.

Changelog: added

We want to drop the `vulnerability_occurrence_pipelines` table. In
order to do this, we still need the first and latest pipeline ID
stored somewhere to support the existing feature-set

We decided to store these values in `vulnerability_occurrences`, and
added columns, along with indexes, in previous commits.

With that work done, we are now unblocked to add foreign keys and
model associations, to prepare to actually backfill and use these
columns

Loose Foreign Keys
------------------------

As the pipelines table and the vulnerabilities table are in separate
databases, we need to use [loose foreign keys][0]. As part of that
process, I ran the script:

```sh
scripts/decomposition/generate-loose-foreign-key -c vulnerability_occurrences
```

This generated specs that were not passing. To make these specs pass I
needed to modify the factories from:

```ruby
let_it_be(:model) { create(:vulnerability_occurrences, initial_pipeline: parent) }
let_it_be(:model) { create(:vulnerability_occurrences, latest_pipeline: parent) }
```

to:

```ruby
let_it_be(:model) { create(:vulnerabilities_finding, initial_pipeline_id: parent.id) }
let_it_be(:model) { create(:vulnerabilities_finding, latest_pipeline_id: parent.id) }
```

[0]:https://docs.gitlab.com/ee/development/database/loose_foreign_keys.html

related to: https://gitlab.com/gitlab-org/gitlab/-/issues/422382
resolves: https://gitlab.com/gitlab-org/gitlab/-/work_items/443284
resolves: https://gitlab.com/gitlab-org/gitlab/-/work_items/443283

This is the first step of partitioning the merge_request_diff_commits
table by int range.

Epic: https://gitlab.com/groups/gitlab-org/-/epics/11271

Changelog: added

The `notes` records with NULL noteable_type are orphaned notes
(the cause(s) are unknown.)
They should be removed from the `notes` table.

To insure against inadvertent data loss,
`temp_notes_backup` table is created to store the removed records.

`temp_notes_backup` should be removed later.

Changelog: other

We want to drop the `vulnerability_occurrence_pipelines` table. In
order to do this, we still need the first and latest pipeline ID
stored somewhere to support the existing feature-set

Follow-up Work
---------------------------

We will need to add loose foreign keys and model association. Both of
these are blocked on index creation, which has to be done via the
[async process][0]

That work is tracked in the following tasks:

- https://gitlab.com/gitlab-org/gitlab/-/work_items/443282
- https://gitlab.com/gitlab-org/gitlab/-/work_items/443283
- https://gitlab.com/gitlab-org/gitlab/-/work_items/443284

---

Related to https://gitlab.com/gitlab-org/gitlab/-/issues/422382
Resolves https://gitlab.com/gitlab-org/gitlab/-/work_items/443281
Resolves https://gitlab.com/gitlab-org/gitlab/-/work_items/443280

[0]:https://docs.gitlab.com/ee/development/database/adding_database_indexes.html#create-indexes-asynchronously

Changelog: added

This commit introduces a FF which will allow us to read
Ci::Stage from our new partitioned table.

Add background database migration to migrate dark `color_mode_id` (`2`)
where users have selected dark mode `theme_id` (`11`)
Add ColorMode model

Changelog: changed

This commit creates the routing table for ci_stages.

Changelog: other

Changelog: changed

This commit introduces a FF which will allow us to read
Ci::JobArtifact from our new partitioned table.

This table belonged to an old version of product
analytics that is no longer used.

No production has been stored here for years.

Changelog: removed

This commit creates the routing table for ci_job_artifacts.

Changelog: other

Adds an FK from p_ci_builds to ci_stages now that the partitiones have the foreign key

This commit also introduces synchronous validations for the FKs in the partitions, for self-managed

Changelog: other

It adds a new partitioned foreign key between the
partitions of `p_ci_builds` and `ci_stages` that
will be used for partitioning this table.

https://docs.gitlab.com/ee/development/database/partitioning/list.html#step-4---enforce-foreign-key-constraint

Changelog: other

EE: true
Changelog: added

Creates the partitioned table for ci_pipeline_variables
and attaches the table as the first partition.

Changelog: other

This MR cascades partition_id from pipeline to
Ci::PipelineConfig. We need to add this new column in order
to redefine the FK association between ci_pipeline_artifacts
and ci_pipelines.

Changelog: added

This MR cascades partition_id from pipeline to
Ci::PipelineArtifact. We need to add this new column in order
to redefine the FK association between ci_pipeline_artifacts
and ci_pipelines.

Changelog: added

This MR cascades partition_id from pipeline to
Ci::PipelineMetadata. We need to add this new column in order
to redefine the FK association between ci_pipeline_metadata
and ci_pipelines.

Changelog: added

This change adds a cache table for storing namespace descendants and
project ids so we can retrieve them efficiently.

Changelog: added

As part of the
[Cells](https://docs.gitlab.com/ee/architecture/blueprints/cells/)
effort we need to classify all tables as `gitlab_main_cell` or
`gitlab_main_clusterwide`. In the future these tables will be split into
different databases. For now when we classify these tables we will start
to get errors in CI for
[cross-joins](https://docs.gitlab.com/ee/development/database/multiple_databases.html#removing-joins-between-ci-and-non-ci-tables),
[cross-transactions](https://docs.gitlab.com/ee/development/database/multiple_databases.html#removing-cross-database-transactions)
and
[cross-database foreign keys](https://docs.gitlab.com/ee/development/database/multiple_databases.html#foreign-keys-that-cross-databases).

This MR adds an allowlist in the `db/docs` file per type to avoid seeing
these failures once we classify the table as `gitlab_main_cell`. This is
to to support
[automatically labelling hundreds of tables](https://gitlab.com/gitlab-org/gitlab/-/issues/429172)
in bulk with `gitlab_main_cell` (as part of the
[Organization
Isolation](https://docs.gitlab.com/ee/architecture/blueprints/organization/isolation.html)
effort)
. This will result in lots of pipeline failures
and we want to deal with these those cross-join (etc.) fixes later. For now
we just want to classify things.

You can see the example format of allowing in the
`db/docs/achievements.yml` change which was chosen as an arbitrary
example to prove that this works.

Changelog: added
EE: true

This commit cascades partition_id from the pipeline, we need
to add this new column in order to redefine the FK association
between ci_pipeline_chat_data and ci_pipelines. As in order to
partition ci_pipelines all references FK must include the partition
key.

Changelog: added

As part of our Cells effort we are going to need to backfill a
`sharding_key` to many tables. The `sharding_key` can be any foreign key
that references `projects`, `namespaces` or `organizations`. You can
read more about this at
https://docs.gitlab.com/ee/development/database/multiple_databases.html#defining-a-sharding-key-for-all-cell-local-tables
.The enforcement of having a `sharding_key` for new tables was already
implemented in
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/136682 .

Additionally we have recently shown examples of how we plan to backfill
a sharding_key to a table that is missing it in
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/136800 .

This MR extends this work by allowing a very minimal way to declare in
the `db/docs` file a `desired_sharding_key` a strategy to backfill a new
column that will eventually become the `sharding_key`. An example has
been given in this MR which looks like:

```yml
desired_sharding_key:
  project_id:
    references: projects
    backfill_via:
      parent:
        foreign_key: scanner_id
        table: vulnerability_scanners
        sharding_key: project_id
        belongs_to: scanner
```

To understand best how this YML data will be used you can map it onto
the merge request we created manually in
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/136800 . The idea
will be to automatically create this. The content of the YML specifies
the parent table and it's sharding_key to backfill from in the batched
background migration. It also specifies a `belongs_to` relation which
will be added to the model to automatically populate the sharding_key in
the `before_save`.

The idea behind this is to save developers the effort of creating MRs
which add the new column and writing a batched background migration to
backfill this column and all the followup work related to that. Instead
we will be building automation that uses this data in the YML file to
automatically create all the merge requests for creating the column and
backfilling it.

Adds catalog_resource_sync_events table and model
to serve as a queue of catalog resources that need to be synchronized
with data changes to their associated `projects` records. PG triggers
add SyncEvent records when a catalog resource is created or
when a relevant column in an associated project is updated.
A worker processes the sync events.

Changelog: added

Changelog: performance

This change adds the calls partitionable method on 
job_artifact_state scoped under job_artifact

Changelog: added

Resolves https://gitlab.com/gitlab-org/gitlab/-/issues/429163

Resolves https://gitlab.com/gitlab-org/gitlab/-/issues/429181

Resolves https://gitlab.com/gitlab-org/gitlab/-/issues/429162

Changelog: other

This is causing a master broken in
https://gitlab.com/gitlab-org/quality/engineering-productivity/master-broken-incidents/-/issues/4019

This was merged in 
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/133566
and must have somehow conflicted with a rubocop rule we were adding.

JH expects to use a separate database, which referred to the solution
of adding Geo database. So it needs upstream to change some
code to support it.

The detail description is in the issue:
https://gitlab.com/gitlab-jh/gitlab-jh-enablement/-/issues/343

And the JH whole draft MR can be found here:
!1898