Add cached responses table and connect them to upstreams.

Changelog: added

Virtual Registries will first support the Maven packages.
Introduce a new bounded context.
Introduce 3 tables / models required by Maven virtual registries.
Maven Registry: the registry object
Maven Registry Upstream: the joined table between Registry and Upstream
Maven Upstream: the upstream object

Changelog: added

Changelog: added

via the NamespaceSettings table, we are adding a
last_dormant_member_review_at timestamp column and index for use with a limited
capacity worker

Changelog: added

Normally a sequence should be dropped when its associated table is
dropped. However, due to a bug in GitLab's `sequences_owned_by`
implementation (fixed via
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/160528),
sequences may have erroneously been assigned to columns in the
`ci_builds` or `p_ci_builds` tables. As a result, if the sequences
were assigned to those CI tables, they may not have been dropped when
their assocated tables were dropped.

This commit adds a migration to remove possible orphaned sequences to
ensure they don't cause problems in the future, as they did with
https://gitlab.com/gitlab-org/gitlab/-/issues/474293.

Relates to https://gitlab.com/gitlab-org/gitlab/-/issues/474305

Changelog: fixed

This commit removes deprecated StoreSecurityReportsWorker class file.
It also regenerates the Sidekiq queue configuration file.

Changelog: removed

Issue: https://gitlab.com/gitlab-org/gitlab/-/issues/467944

GitLab 17.2.1 included a migration to fix the wrong owners of
sequences
(https://gitlab.com/gitlab-org/gitlab/-/merge_requests/160139). Unfortunately
due to a bug in the `sequences_owned_by` query described in
https://gitlab.com/gitlab-org/gitlab/-/issues/468541#note_2013158432,
some sequences may have been assigned the wrong owner and not
automatically dropped when its associated table was dropped.

For example, for one customer `geo_reset_checksum_events_id_seq` was
assigned to the wrong owner (`p_ci_builds.partition_id`), so when
`geo_reset_checksum_events` was dropped in GitLab 17.0 via
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/151938, the
sequence remained in the database. We could remove these unused
sequences, but that should probably be done by a different migration.
For now, ignoring these sequences is the safest way to handle this.

Relates to https://gitlab.com/gitlab-org/gitlab/-/issues/474293

Changelog: fixed

- Remove Packages::Composer::CacheFile model.
- Remove composer cache file sidekiq workers.
- Remove composer cache file creator.
- Remove composer cache file database records.

Changelog: removed

Add and backfill namespace_id for epic_issues.

This table has a
[desired sharding key](https://docs.gitlab.com/ee/development/database/multiple_databases.html#define-a-desired_sharding_key-to-automatically-backfill-a-sharding_key)
configured ([view configuration](https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/docs/epic_issues.yml)).

This merge request is the first step towards transforming the desired sharding key into a
[sharding key](https://docs.gitlab.com/ee/development/database/multiple_databases.html#defining-a-sharding-key-for-all-cell-local-tables).

This involves three changes:

- Adding a new column that will serve as the sharding key (along with the relevant index and foreign key).
- Populating the sharding key when new records are created by adding a database function and trigger.
- Scheduling a [batched background migration](https://docs.gitlab.com/ee/development/database/batched_background_migrations.html)
  to set the sharding key for existing records.

Once the background migration has completed, a second merge request will be created to finalize the background
migration and validate the not null constraint.

We have assigned a random backend engineer from ~"group::product planning" to review these changes. Please review this merge
request from a ~backend perspective. The main thing we are looking to verify is that the added column and association
match the values specified by the [desired sharding key](https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/docs/epic_issues.yml)
configuration and that backfilling the column from this other table makes sense in the context of this feature.

When you are finished, please:

1. Trigger the [database testing pipeline](https://docs.gitlab.com/ee/development/database/database_migration_pipeline.html)
   as instructed by Danger.
1. Request a review from the ~backend maintainer and ~database reviewer suggested by Danger.

If you have any questions or concerns, reach out to `@tigerwnz` or @manojmj.

This merge request was generated by a once off keep implemented in
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/143774

This change was generated by
[gitlab-housekeeper](https://gitlab.com/gitlab-org/gitlab/-/tree/master/gems/gitlab-housekeeper)
using the Keeps::BackfillDesiredShardingKeySmallTable keep.

To provide feedback on your experience with `gitlab-housekeeper` please comment in
<https://gitlab.com/gitlab-org/gitlab/-/issues/442003>.

Changelog: other

This changes introduces a link table between
projects and approval_policy_rules for efficient
lookup of approval rules for a given project.

EE: true
Changelog: added

This change updates the security_policies table
schema to be more generic and handle the
logic to persist pipeline_execution_policy
and ci_component_sources_policy type along with
approval_policy and scan_execution_policy.

EE: true
Changelog: added

Changelog: other

Contributes to https://gitlab.com/gitlab-org/gitlab/-/issues/460828

**Problem**

After https://gitlab.com/gitlab-org/gitlab/-/merge_requests/157768/,
`organization_id` should be set by the application. Default value in
database is not necessary anymore.

**Solution**

Remove default value from `organization_id`

Changelog: changed

Synchronously removes the index on project_statistics for wiki
size and project id, only for GitLab.com

Changelog: removed

Synchronously removes the index on project_statistics for packages
size and project id, only for GitLab.com

Changelog: removed

DumpAllWriteBuffersCronWorker is the new
replacement

Changelog: removed

Old data has DF score as high even with
1 deployment per month. This data migration
recalculates all previous DF scores to
correct ones.

Changelog: added
EE: true

- merge_requests.head_pipeline_id_convert_to_bigint
- merge_request_metrics.pipeline_id_convert_to_bigint
- merge_trains.pipeline_id_convert_to_bigint
- vulnerability_feedback.pipeline_id_convert_to_bigint
- vulnerability_occurrence_pipelines.pipeline_id_convert_to_bigint

Changelog: changed

GitLab 17.2 shipped a migration
(https://gitlab.com/gitlab-org/gitlab/-/merge_requests/158711) that
dropped `p_ci_builds.id`. However, we received a number of reports of
users who had unrelated sequences owned by `p_ci_builds.id`, so the
migration failed because the objects depended on that column existing.

To allow migrations to go through, scan the system catalog for all
sequences owned by either `p_ci_builds` or `ci_builds`. Generate the
`ALTER SEQUENCE` statement to update the ownership of the sequence
using the technique described in
https://wiki.postgresql.org/wiki/Fixing_Sequences.

Note that we could run all 600+ `ALTER SEQUENCE` commands to bring the
database to a consistent state, but to minimize risk we target only on
the ones to allow the CI builds migration to proceed.

Changelog: fixed

Relates to https://gitlab.com/gitlab-org/gitlab/-/issues/468541

Merge branch 'security/limit-access-to-exports' into 'master'

See merge request gitlab-org/security/gitlab!4080

Changelog: security

Merge branch 'security-import-export-based-on-user' into 'master'

See merge request gitlab-org/security/gitlab!4211

Changelog: security

## What does this MR do and why?

Add and backfill project_id for evidences.

This table has a
[desired sharding key](https://docs.gitlab.com/ee/development/database/multiple_databases.html#define-a-desired_sharding_key-to-automatically-backfill-a-sharding_key)
configured ([view configuration](https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/docs/evidences.yml)).

This merge request is the first step towards transforming the desired sharding key into a
[sharding key](https://docs.gitlab.com/ee/development/database/multiple_databases.html#defining-a-sharding-key-for-all-cell-local-tables).

This involves three changes:

- Adding a new column that will serve as the sharding key (along with the relevant index and foreign key).
- Populating the sharding key when new records are created by adding a database function and trigger.
- Scheduling a [batched background migration](https://docs.gitlab.com/ee/development/database/batched_background_migrations.html)
  to set the sharding key for existing records.

Once the background migration has completed, a second merge request will be created to finalize the background
migration and validate the not null constraint.

## How to verify

We have assigned a random backend engineer from ~"group::compliance" to review these changes. Please review this merge
request from a ~backend perspective. The main thing we are looking to verify is that the added column and association
match the values specified by the [desired sharding key](https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/docs/evidences.yml)
configuration and that backfilling the column from this other table makes sense in the context of this feature.

When you are finished, please:

1. Trigger the [database testing pipeline](https://docs.gitlab.com/ee/development/database/database_migration_pipeline.html)
   as instructed by Danger.
1. Request a review from the ~backend maintainer and ~database reviewer suggested by Danger.

If you have any questions or concerns, reach out to `@tigerwnz` or @manojmj.

This merge request was generated by a once off keep implemented in
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/143774

This change was generated by
[gitlab-housekeeper](https://gitlab.com/gitlab-org/gitlab/-/tree/master/gems/gitlab-housekeeper)
using the Keeps::BackfillDesiredShardingKeySmallTable keep.

To provide feedback on your experience with `gitlab-housekeeper` please comment in
<https://gitlab.com/gitlab-org/gitlab/-/issues/442003>.

Changelog: other

- include concerns that can be reused
- implement methods and classes for Discussion
- add specs
- add adding type to notes migration

Changelog: other

- needed to remove lazy creation of user_details
- background migration to go over users table and
  find users w/o the needed entries and create them.

Changelog: other

Introduces the Ci::Tagging model to replace
the ActsAsTaggableOn::Tagging model.

Remove incorrect model class.

Introduces the Ci::tag model which implements
the API needed by ActsAsTaggableOn::Tag to tag
jobs.

New column to store HMAC secret and
initialization vector. These secrets
will be used for authentication of
external status checks

Changelog: added
EE: true

Replaces the non-partitioned FK with a partitioned one
Changelog: other

Add a migration to add maximum_storage_bytes bigint columns
to zoekt_indices table.

Changelog: other
MR: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/159382

Changelog: other

pm_affected_packages.

Changelog: added

Replaces the non-partitioned FK with a partitioned one
Changelog: other