This foreign key caused the incident
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19474 . It
was added in
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/177799 . The
problem is that we added a foreign key before adding an index. This
means that all statements like `DELETE FROM projects ...` will need to
scan the entire `merge_request_diff_files` table to check if the foreign
key is valid. This was causing timeouts.

In order to resolve the incident we've already manually deleted this
migration on GitLab.com . This MR additionally adds another migration to
remove the foreign key to clean up any environments where this migration
may have already run.

This migration was finished at `2024-10-21 02:55:04 UTC`, you can confirm
the status using our
[batched background migration chatops commands](https://docs.gitlab.com/ee/development/database/batched_background_migrations.html#monitor-the-progress-and-status-of-a-batched-background-migration).
  To confirm it is finished you can run:

  ```
/chatops run batched_background_migrations status 1000730 --database main
```

The last time this background migration was triggered was in [db/post_migrate/20240920083708_queue_backfill_members_request_accepted_at.rb](https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/post_migrate/20240920083708_queue_backfill_members_request_accepted_at.rb)

  You can read more about the process for finalizing batched background migrations in
https://docs.gitlab.com/ee/development/database/batched_background_migrations.html .

  As part of our process we want to ensure all batched background migrations have had at least one
[required stop](https://docs.gitlab.com/ee/development/database/required_stops.html)
to process the migration. Therefore we can finalize any batched background migration that was added before the
last required stop.

This change was generated by
[gitlab-housekeeper](https://gitlab.com/gitlab-org/gitlab/-/tree/master/gems/gitlab-housekeeper)
using the Keeps::OverdueFinalizeBackgroundMigration keep.

To provide feedback on your experience with `gitlab-housekeeper` please create an issue with the
label ~"GitLab Housekeeper" and consider pinging the author of this keep.

Changelog: other

This matches the schema for `instance_type_ci_runners`

This is meant to more clearly denote groups of tables that are
cell-local. These cell-local tables should not have references to
organization tables.

Adds a new, empty table to support an upcoming data migration. This
table will only be populated manually by an SRE via a CR, and will be
removed at the end of the project.

Changelog: added

Changelog: other

Add a callback to backfill project with pipeline variables table on new records
Address all the comments
Skip id primary key in favor of project_id
Add a trigger function instead of model callback
Add a trigger migration
Allow projects_with_pipeline_variables cross database

Changelog: added

This reverts merge request !183638

This table stores the CI pipeline inputs values configured when creating
a pipeline schedule. It belongs to a pipeline schedule, ensures that the
input keys are unique, and encrypts the input values. It is on the
gitlab_ci database schema because pipeline schedules are also stored on
the CI schema.

Changelog: added
Issue: https://gitlab.com/gitlab-org/gitlab/-/issues/519960

## What does this MR do and why?

Add and backfill project_id for deployment_merge_requests.

This table has a
[desired sharding key](https://docs.gitlab.com/ee/development/database/multiple_databases.html#define-a-desired_sharding_key-to-automatically-backfill-a-sharding_key)
configured ([view configuration](https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/docs/deployment_merge_requests.yml)).

This merge request is the first step towards transforming the desired sharding key into a
[sharding key](https://docs.gitlab.com/ee/development/database/multiple_databases.html#defining-a-sharding-key-for-all-cell-local-tables).

This involves the following changes:

- Adding a new column that will serve as the sharding key (along with the relevant asynchronous index).
- Populating the sharding key when new records are created by adding a database function and trigger.
- Scheduling a [batched background migration](https://docs.gitlab.com/ee/development/database/batched_background_migrations.html)
  to set the sharding key for existing records.

Once the background migration has completed, a second merge request will be created to finalize the background
migration, and add a foreign key and not null constraint.

## How to verify

We have assigned a random backend engineer from ~"group::environments" to review these changes. Please review this merge
request from a ~backend perspective. The main thing we are looking to verify is that the added column and association
match the values specified by the [desired sharding key](https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/docs/deployment_merge_requests.yml)
configuration and that backfilling the column from this other table makes sense in the context of this feature.

When you are finished, please:

1. Trigger the [database testing pipeline](https://docs.gitlab.com/ee/development/database/database_migration_pipeline.html)
   as instructed by Danger.
1. Request a review from the ~backend maintainer and ~database reviewer suggested by Danger.

If you have any questions or concerns, reach out to @tigerwnz or @shubhamkrai.

This merge request was generated by a once off keep implemented in
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/143774

This change was generated by
[gitlab-housekeeper](https://gitlab.com/gitlab-org/gitlab/-/tree/master/gems/gitlab-housekeeper)
using the Keeps::BackfillDesiredShardingKeyLargeTable keep.

To provide feedback on your experience with `gitlab-housekeeper` please create an issue with the
label ~"GitLab Housekeeper" and consider pinging the author of this keep.

Changelog: other

Changelog: other

This migration was finished at `2024-09-27 03:06:01 UTC`, you can confirm
the status using our
[batched background migration chatops commands](https://docs.gitlab.com/ee/development/database/batched_background_migrations.html#monitor-the-progress-and-status-of-a-batched-background-migration).
  To confirm it is finished you can run:

  ```
/chatops run batched_background_migrations status 1000708 --database cell
```

The last time this background migration was triggered was in [db/post_migrate/20240918102413_queue_backfill_incident_management_timeline_event_tag_links_project_id.rb](https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/post_migrate/20240918102413_queue_backfill_incident_management_timeline_event_tag_links_project_id.rb)

  You can read more about the process for finalizing batched background migrations in
https://docs.gitlab.com/ee/development/database/batched_background_migrations.html .

  As part of our process we want to ensure all batched background migrations have had at least one
[required stop](https://docs.gitlab.com/ee/development/database/required_stops.html)
to process the migration. Therefore we can finalize any batched background migration that was added before the
last required stop.

This change was generated by
[gitlab-housekeeper](https://gitlab.com/gitlab-org/gitlab/-/tree/master/gems/gitlab-housekeeper)
using the Keeps::OverdueFinalizeBackgroundMigration keep.

To provide feedback on your experience with `gitlab-housekeeper` please create an issue with the
label ~"GitLab Housekeeper" and consider pinging the author of this keep.

Changelog: other

This migration was finished at `2024-08-24 06:54:04 UTC`, you can confirm
the status using our
[batched background migration chatops commands](https://docs.gitlab.com/ee/development/database/batched_background_migrations.html#monitor-the-progress-and-status-of-a-batched-background-migration).
  To confirm it is finished you can run:

  ```
/chatops run batched_background_migrations status 1000656 --database main
```

The last time this background migration was triggered was in [db/post_migrate/20240816110844_queue_rerun_epic_dates_to_work_item_dates_sources_sync.rb](https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/post_migrate/20240816110844_queue_rerun_epic_dates_to_work_item_dates_sources_sync.rb)

  You can read more about the process for finalizing batched background migrations in
https://docs.gitlab.com/ee/development/database/batched_background_migrations.html .

  As part of our process we want to ensure all batched background migrations have had at least one
[required stop](https://docs.gitlab.com/ee/development/database/required_stops.html)
to process the migration. Therefore we can finalize any batched background migration that was added before the
last required stop.

This change was generated by
[gitlab-housekeeper](https://gitlab.com/gitlab-org/gitlab/-/tree/master/gems/gitlab-housekeeper)
using the Keeps::OverdueFinalizeBackgroundMigration keep.

To provide feedback on your experience with `gitlab-housekeeper` please create an issue with the
label ~"GitLab Housekeeper" and consider pinging the author of this keep.

Changelog: other

This reverts commit 450754f5, reversing
changes made to 485c4091.

This will store users' assigned member roles in groups through:

- Directly assignment - user is a member of a group with an assigned
member role
- Inherited - user inherits their assigned member role from a parent
group
- Group share - when a group is shared to another group with a member
role, users of the invited group get the member role assigned to the
invited group on the shared group

shared_with_group attribute makes it easier to keep track which member
role assignment in a group come from group sharing. It is nil for direct
and inherited member role assignments.

Changelog: added
EE: true

Changelog: other

`index_lfs_objects_projects_on_project_id_lfs_object_id_null_repo_type`
was already created asynchronously in [a prior post-deployment migration](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/177919)

This commit ensures that it is properly created synchronously
for all installations, updating the schema definition accordingly.
[Reference](https://docs.gitlab.com/development/database/adding_database_indexes/#schedule-index-creation-for-a-low-impact-time)

Changelog: other

Changelog: other

- group saml users should not be in onboarding.
- remove them if they are in it.
- see https://gitlab.com/gitlab-org/gitlab/-/issues/522011

Changelog: other

Adds new approval rule associations
to user and user group
for new approval rule architecture.

Changelog: changed
EE: true

Changelog: added

This adds a new work item widget for vulnerabilities. It contains
related vulnerabilities of the work item. It's only applicable for
the work item type issue.

Changelog: added
EE: true

## What does this MR do and why?

Add and backfill project_id for approval_merge_request_rules_approved_approvers.

This table has a
[desired sharding key](https://docs.gitlab.com/ee/development/database/multiple_databases.html#define-a-desired_sharding_key-to-automatically-backfill-a-sharding_key)
configured ([view configuration](https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/docs/approval_merge_request_rules_approved_approvers.yml)).

This merge request is the first step towards transforming the desired sharding key into a
[sharding key](https://docs.gitlab.com/ee/development/database/multiple_databases.html#defining-a-sharding-key-for-all-cell-local-tables).

This involves three changes:

- Adding a new column that will serve as the sharding key (along with the relevant index and foreign key).
- Populating the sharding key when new records are created by adding a database function and trigger.
- Scheduling a [batched background migration](https://docs.gitlab.com/ee/development/database/batched_background_migrations.html)
  to set the sharding key for existing records.

Once the background migration has completed, a second merge request will be created to finalize the background
migration and validate the not null constraint.

## How to verify

We have assigned a random backend engineer from ~"group::code review" to review these changes. Please review this merge
request from a ~backend perspective. The main thing we are looking to verify is that the added column and association
match the values specified by the [desired sharding key](https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/docs/approval_merge_request_rules_approved_approvers.yml)
configuration and that backfilling the column from this other table makes sense in the context of this feature.

When you are finished, please:

1. Trigger the [database testing pipeline](https://docs.gitlab.com/ee/development/database/database_migration_pipeline.html)
   as instructed by Danger.
1. Request a review from the ~backend maintainer and ~database reviewer suggested by Danger.

If you have any questions or concerns, reach out to `@tigerwnz` or @shubhamkrai.

This merge request was generated by a once off keep implemented in
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/143774

This change was generated by
[gitlab-housekeeper](https://gitlab.com/gitlab-org/gitlab/-/tree/master/gems/gitlab-housekeeper)
using the Keeps::BackfillDesiredShardingKeySmallTable keep.

To provide feedback on your experience with `gitlab-housekeeper` please create an issue with the
label ~"GitLab Housekeeper" and consider pinging the author of this keep.

Changelog: other

Add LFK on ci_running_builds/ci_runners

Changelog: other

Changelog: changed

## What does this MR do and why?

Add and backfill project_id for approval_merge_request_rules_users.

This table has a
[desired sharding key](https://docs.gitlab.com/ee/development/database/multiple_databases.html#define-a-desired_sharding_key-to-automatically-backfill-a-sharding_key)
configured ([view configuration](https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/docs/approval_merge_request_rules_users.yml)).

This merge request is the first step towards transforming the desired sharding key into a
[sharding key](https://docs.gitlab.com/ee/development/database/multiple_databases.html#defining-a-sharding-key-for-all-cell-local-tables).

This involves the following changes:

- Adding a new column that will serve as the sharding key (along with the relevant asynchronous index).
- Populating the sharding key when new records are created by adding a database function and trigger.
- Scheduling a [batched background migration](https://docs.gitlab.com/ee/development/database/batched_background_migrations.html)
  to set the sharding key for existing records.

Once the background migration has completed, a second merge request will be created to finalize the background
migration, and add a foreign key and not null constraint.

## How to verify

We have assigned a random backend engineer from ~"group::code review" to review these changes. Please review this merge
request from a ~backend perspective. The main thing we are looking to verify is that the added column and association
match the values specified by the [desired sharding key](https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/docs/approval_merge_request_rules_users.yml)
configuration and that backfilling the column from this other table makes sense in the context of this feature.

When you are finished, please:

1. Trigger the [database testing pipeline](https://docs.gitlab.com/ee/development/database/database_migration_pipeline.html)
   as instructed by Danger.
1. Request a review from the ~backend maintainer and ~database reviewer suggested by Danger.

If you have any questions or concerns, reach out to @tigerwnz or @shubhamkrai.

This merge request was generated by a once off keep implemented in
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/143774

This change was generated by
[gitlab-housekeeper](https://gitlab.com/gitlab-org/gitlab/-/tree/master/gems/gitlab-housekeeper)
using the Keeps::BackfillDesiredShardingKeyLargeTable keep.

To provide feedback on your experience with `gitlab-housekeeper` please create an issue with the
label ~"GitLab Housekeeper" and consider pinging the author of this keep.

Changelog: other

User will be able to store their Docker Hub credentials in the
dependency proxy for containers so that when GitLab authenticates
against Docker Hub, we will switch to an authenticated request instead
of using an unauthenticated request.

This is behind a feature flag.

This migration was finished at `2024-10-08 02:31:52 UTC`, you can confirm
the status using our
[batched background migration chatops commands](https://docs.gitlab.com/ee/development/database/batched_background_migrations.html#monitor-the-progress-and-status-of-a-batched-background-migration).
  To confirm it is finished you can run:

  ```
/chatops run batched_background_migrations status 1000342 --database ci
```

The last time this background migration was triggered was in [db/post_migrate/20240930125311_queue_backfill_ci_secure_file_states_project_id.rb](https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/post_migrate/20240930125311_queue_backfill_ci_secure_file_states_project_id.rb)

  You can read more about the process for finalizing batched background migrations in
https://docs.gitlab.com/ee/development/database/batched_background_migrations.html .

  As part of our process we want to ensure all batched background migrations have had at least one
[required stop](https://docs.gitlab.com/ee/development/database/required_stops.html)
to process the migration. Therefore we can finalize any batched background migration that was added before the
last required stop.

This change was generated by
[gitlab-housekeeper](https://gitlab.com/gitlab-org/gitlab/-/tree/master/gems/gitlab-housekeeper)
using the Keeps::OverdueFinalizeBackgroundMigration keep.

To provide feedback on your experience with `gitlab-housekeeper` please create an issue with the
label ~"GitLab Housekeeper" and consider pinging the author of this keep.

Changelog: other

This migration was finished at `2024-09-23 02:59:03 UTC`, you can confirm
the status using our
[batched background migration chatops commands](https://docs.gitlab.com/ee/development/database/batched_background_migrations.html#monitor-the-progress-and-status-of-a-batched-background-migration).
  To confirm it is finished you can run:

  ```
/chatops run batched_background_migrations status 1000705 --database cell
```

The last time this background migration was triggered was in [db/post_migrate/20240916132826_queue_backfill_incident_management_oncall_rotations_project_id.rb](https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/post_migrate/20240916132826_queue_backfill_incident_management_oncall_rotations_project_id.rb)

  You can read more about the process for finalizing batched background migrations in
https://docs.gitlab.com/ee/development/database/batched_background_migrations.html .

  As part of our process we want to ensure all batched background migrations have had at least one
[required stop](https://docs.gitlab.com/ee/development/database/required_stops.html)
to process the migration. Therefore we can finalize any batched background migration that was added before the
last required stop.

This change was generated by
[gitlab-housekeeper](https://gitlab.com/gitlab-org/gitlab/-/tree/master/gems/gitlab-housekeeper)
using the Keeps::OverdueFinalizeBackgroundMigration keep.

To provide feedback on your experience with `gitlab-housekeeper` please create an issue with the
label ~"GitLab Housekeeper" and consider pinging the author of this keep.

Changelog: other

This migration was finished at `2024-10-15 09:46:03 UTC`, you can confirm
the status using our
[batched background migration chatops commands](https://docs.gitlab.com/ee/development/database/batched_background_migrations.html#monitor-the-progress-and-status-of-a-batched-background-migration).
  To confirm it is finished you can run:

  ```
/chatops run batched_background_migrations status 1000728 --database sec
```

The last time this background migration was triggered was in [db/post_migrate/20241001122045_queue_backfill_dast_site_profiles_builds_project_id.rb](https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/post_migrate/20241001122045_queue_backfill_dast_site_profiles_builds_project_id.rb)

  You can read more about the process for finalizing batched background migrations in
https://docs.gitlab.com/ee/development/database/batched_background_migrations.html .

  As part of our process we want to ensure all batched background migrations have had at least one
[required stop](https://docs.gitlab.com/ee/development/database/required_stops.html)
to process the migration. Therefore we can finalize any batched background migration that was added before the
last required stop.

This change was generated by
[gitlab-housekeeper](https://gitlab.com/gitlab-org/gitlab/-/tree/master/gems/gitlab-housekeeper)
using the Keeps::OverdueFinalizeBackgroundMigration keep.

To provide feedback on your experience with `gitlab-housekeeper` please create an issue with the
label ~"GitLab Housekeeper" and consider pinging the author of this keep.

Changelog: other

Changelog: added
EE: true

This migration was finished at `2024-08-23 03:43:02 UTC`.

To confirm it is finished you can run:

```
/chatops run batched_background_migrations status 1000655
```

Changelog: other

* This field is used to store user input when Duo Workflow asks for
  follow-up information in the chat window.
* The previous limit was 255, which is much too constrained for user
  input.
* The new limit added here is 4096, which is somewhat arbitrary but we
  can always increase further later. Picked this to match the limit on
  the original workflow goal field.
* Fixes https://gitlab.com/gitlab-org/gitlab/-/issues/523868

Changelog: changed

## What does this MR do and why?

Add and backfill project_id for packages_debian_project_component_files.

This table has a
[desired sharding key](https://docs.gitlab.com/ee/development/database/multiple_databases.html#define-a-desired_sharding_key-to-automatically-backfill-a-sharding_key)
configured ([view configuration](https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/docs/packages_debian_project_component_files.yml)).

This merge request is the first step towards transforming the desired sharding key into a
[sharding key](https://docs.gitlab.com/ee/development/database/multiple_databases.html#defining-a-sharding-key-for-all-cell-local-tables).

This involves three changes:

- Adding a new column that will serve as the sharding key (along with the relevant index and foreign key).
- Populating the sharding key when new records are created by adding a database function and trigger.
- Scheduling a [batched background migration](https://docs.gitlab.com/ee/development/database/batched_background_migrations.html)
  to set the sharding key for existing records.

Once the background migration has completed, a second merge request will be created to finalize the background
migration and validate the not null constraint.

## How to verify

We have assigned a random backend engineer from ~"group::package registry" to review these changes. Please review this merge
request from a ~backend perspective. The main thing we are looking to verify is that the added column and association
match the values specified by the [desired sharding key](https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/docs/packages_debian_project_component_files.yml)
configuration and that backfilling the column from this other table makes sense in the context of this feature.

When you are finished, please:

1. Trigger the [database testing pipeline](https://docs.gitlab.com/ee/development/database/database_migration_pipeline.html)
   as instructed by Danger.
1. Request a review from the ~backend maintainer and ~database reviewer suggested by Danger.

If you have any questions or concerns, reach out to `@tigerwnz` or @shubhamkrai.

This merge request was generated by a once off keep implemented in
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/143774

This change was generated by
[gitlab-housekeeper](https://gitlab.com/gitlab-org/gitlab/-/tree/master/gems/gitlab-housekeeper)
using the Keeps::BackfillDesiredShardingKeySmallTable keep.

To provide feedback on your experience with `gitlab-housekeeper` please create an issue with the
label ~"GitLab Housekeeper" and consider pinging the author of this keep.

Changelog: other

Changelog: other
EE: true

Changelog: other