Add & backfill sharding keys for snippet_repositories

Add and backfill multiple sharding keys for snippet_repositories.

This table has a
[desired sharding key](https://docs.gitlab.com/ee/development/database/multiple_databases.html#define-a-desired_sharding_key-to-automatically-backfill-a-sharding_key)
configured ([view configuration](https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/docs/snippet_repositories.yml)).

This merge request is the first step towards transforming the desired sharding key into a
[sharding key](https://docs.gitlab.com/ee/development/database/multiple_databases.html#defining-a-sharding-key-for-all-cell-local-tables).

This involves three changes:

- Adding a new column that will serve as the sharding key (along with the relevant index and foreign key).
- Populating the sharding key when new records are created by adding a database function and trigger.
- Scheduling a [batched background migration](https://docs.gitlab.com/ee/development/database/batched_background_migrations.html)
  to set the sharding key for existing records.

Once the background migration has completed, a second merge request will be created to finalize the background
migration and validate the not null constraint.

Please review this merge
request from a ~backend perspective. The main thing we are looking to verify is that the added column and association
match the values specified by the [desired sharding key](https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/docs/snippet_repositories.yml)
configuration and that backfilling the column from this other table makes sense in the context of this feature.

When you are finished, please:

1. Trigger the [database testing pipeline](https://docs.gitlab.com/ee/development/database/database_migration_pipeline.html)
   as instructed by Danger.
1. Request a review from the ~backend maintainer and ~database reviewer suggested by Danger.

If you have any questions or concerns, reach out to `@tigerwnz` or @shubhamkrai.

This merge request was generated by a once off keep implemented in
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/143774

This change was generated by
[gitlab-housekeeper](https://gitlab.com/gitlab-org/gitlab/-/tree/master/gems/gitlab-housekeeper)
using the Keeps::BackfillMultipleDesiredShardingKeySmallTable keep.

To provide feedback on your experience with `gitlab-housekeeper` please create an issue with the
label ~"GitLab Housekeeper" and consider pinging the author of this keep.

Changelog: other

db/docs/batched_background_migrations/backfill_snippet_repositories_snippet_organization_id.yml

+---
+migration_job_name: BackfillSnippetRepositoriesSnippetOrganizationId
+description: Backfills sharding key `snippet_repositories.snippet_organization_id` from `snippets`.
+feature_category: source_code_management
+introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/175410
+milestone: '17.10'
+queued_migration_version: 20241211134715
+finalized_by: # version of the migration that finalized this BBM

db/docs/batched_background_migrations/backfill_snippet_repositories_snippet_project_id.yml

+---
+migration_job_name: BackfillSnippetRepositoriesSnippetProjectId
+description: Backfills sharding key `snippet_repositories.snippet_project_id` from `snippets`.
+feature_category: source_code_management
+introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/175410
+milestone: '17.10'
+queued_migration_version: 20241211134710
+finalized_by: # version of the migration that finalized this BBM

db/docs/snippet_repositories.yml

@@ -28,3 +28,6 @@ desired_sharding_key:
         table: snippets
         sharding_key: organization_id
         belongs_to: snippet
+desired_sharding_key_migration_job_name:
+- BackfillSnippetRepositoriesSnippetProjectId
+- BackfillSnippetRepositoriesSnippetOrganizationId

db/migrate/20241211134706_add_snippet_project_id_to_snippet_repositories.rb

+# frozen_string_literal: true
+
+class AddSnippetProjectIdToSnippetRepositories < Gitlab::Database::Migration[2.2]
+  milestone '17.10'
+
+  def change
+    add_column :snippet_repositories, :snippet_project_id, :bigint
+  end
+end

db/migrate/20241211134711_add_snippet_organization_id_to_snippet_repositories.rb

+# frozen_string_literal: true
+
+class AddSnippetOrganizationIdToSnippetRepositories < Gitlab::Database::Migration[2.2]
+  milestone '17.10'
+
+  def change
+    add_column :snippet_repositories, :snippet_organization_id, :bigint
+  end
+end

db/post_migrate/20241211134707_index_snippet_repositories_on_snippet_project_id.rb

+# frozen_string_literal: true
+
+class IndexSnippetRepositoriesOnSnippetProjectId < Gitlab::Database::Migration[2.2]
+  milestone '17.10'
+  disable_ddl_transaction!
+
+  INDEX_NAME = 'index_snippet_repositories_on_snippet_project_id'
+
+  def up
+    add_concurrent_index :snippet_repositories, :snippet_project_id, name: INDEX_NAME
+  end
+
+  def down
+    remove_concurrent_index_by_name :snippet_repositories, INDEX_NAME
+  end
+end

db/post_migrate/20241211134708_add_snippet_repositories_snippet_project_id_fk.rb

+# frozen_string_literal: true
+
+class AddSnippetRepositoriesSnippetProjectIdFk < Gitlab::Database::Migration[2.2]
+  milestone '17.10'
+  disable_ddl_transaction!
+
+  def up
+    add_concurrent_foreign_key :snippet_repositories, :projects, column: :snippet_project_id, on_delete: :cascade
+  end
+
+  def down
+    with_lock_retries do
+      remove_foreign_key :snippet_repositories, column: :snippet_project_id
+    end
+  end
+end

db/post_migrate/20241211134709_add_snippet_repositories_snippet_project_id_trigger.rb

+# frozen_string_literal: true
+
+class AddSnippetRepositoriesSnippetProjectIdTrigger < Gitlab::Database::Migration[2.2]
+  milestone '17.10'
+
+  def up
+    install_sharding_key_assignment_trigger(
+      table: :snippet_repositories,
+      sharding_key: :snippet_project_id,
+      parent_table: :snippets,
+      parent_sharding_key: :project_id,
+      foreign_key: :snippet_id
+    )
+  end
+
+  def down
+    remove_sharding_key_assignment_trigger(
+      table: :snippet_repositories,
+      sharding_key: :snippet_project_id,
+      parent_table: :snippets,
+      parent_sharding_key: :project_id,
+      foreign_key: :snippet_id
+    )
+  end
+end

db/post_migrate/20241211134710_queue_backfill_snippet_repositories_snippet_project_id.rb

+# frozen_string_literal: true
+
+class QueueBackfillSnippetRepositoriesSnippetProjectId < Gitlab::Database::Migration[2.2]
+  milestone '17.10'
+  restrict_gitlab_migration gitlab_schema: :gitlab_main_cell
+
+  MIGRATION = "BackfillSnippetRepositoriesSnippetProjectId"
+  DELAY_INTERVAL = 2.minutes
+  BATCH_SIZE = 1000
+  SUB_BATCH_SIZE = 100
+
+  def up
+    queue_batched_background_migration(
+      MIGRATION,
+      :snippet_repositories,
+      :snippet_id,
+      :snippet_project_id,
+      :snippets,
+      :project_id,
+      :snippet_id,
+      job_interval: DELAY_INTERVAL,
+      batch_size: BATCH_SIZE,
+      sub_batch_size: SUB_BATCH_SIZE
+    )
+  end
+
+  def down
+    delete_batched_background_migration(
+      MIGRATION,
+      :snippet_repositories,
+      :snippet_id,
+      [
+        :snippet_project_id,
+        :snippets,
+        :project_id,
+        :snippet_id
+      ]
+    )
+  end
+end

db/post_migrate/20241211134712_index_snippet_repositories_on_snippet_organization_id.rb

+# frozen_string_literal: true
+
+class IndexSnippetRepositoriesOnSnippetOrganizationId < Gitlab::Database::Migration[2.2]
+  milestone '17.10'
+  disable_ddl_transaction!
+
+  INDEX_NAME = 'index_snippet_repositories_on_snippet_organization_id'
+
+  def up
+    add_concurrent_index :snippet_repositories, :snippet_organization_id, name: INDEX_NAME
+  end
+
+  def down
+    remove_concurrent_index_by_name :snippet_repositories, INDEX_NAME
+  end
+end

db/post_migrate/20241211134713_add_snippet_repositories_snippet_organization_id_fk.rb

+# frozen_string_literal: true
+
+class AddSnippetRepositoriesSnippetOrganizationIdFk < Gitlab::Database::Migration[2.2]
+  milestone '17.10'
+  disable_ddl_transaction!
+
+  def up
+    add_concurrent_foreign_key :snippet_repositories, :organizations, column: :snippet_organization_id,
+      on_delete: :cascade
+  end
+
+  def down
+    with_lock_retries do
+      remove_foreign_key :snippet_repositories, column: :snippet_organization_id
+    end
+  end
+end

db/post_migrate/20241211134714_add_snippet_repositories_snippet_organization_id_trigger.rb

+# frozen_string_literal: true
+
+class AddSnippetRepositoriesSnippetOrganizationIdTrigger < Gitlab::Database::Migration[2.2]
+  milestone '17.10'
+
+  def up
+    install_sharding_key_assignment_trigger(
+      table: :snippet_repositories,
+      sharding_key: :snippet_organization_id,
+      parent_table: :snippets,
+      parent_sharding_key: :organization_id,
+      foreign_key: :snippet_id
+    )
+  end
+
+  def down
+    remove_sharding_key_assignment_trigger(
+      table: :snippet_repositories,
+      sharding_key: :snippet_organization_id,
+      parent_table: :snippets,
+      parent_sharding_key: :organization_id,
+      foreign_key: :snippet_id
+    )
+  end
+end

db/post_migrate/20241211134715_queue_backfill_snippet_repositories_snippet_organization_id.rb

+# frozen_string_literal: true
+
+class QueueBackfillSnippetRepositoriesSnippetOrganizationId < Gitlab::Database::Migration[2.2]
+  milestone '17.10'
+  restrict_gitlab_migration gitlab_schema: :gitlab_main_cell
+
+  MIGRATION = "BackfillSnippetRepositoriesSnippetOrganizationId"
+  DELAY_INTERVAL = 2.minutes
+  BATCH_SIZE = 1000
+  SUB_BATCH_SIZE = 100
+
+  def up
+    queue_batched_background_migration(
+      MIGRATION,
+      :snippet_repositories,
+      :snippet_id,
+      :snippet_organization_id,
+      :snippets,
+      :organization_id,
+      :snippet_id,
+      job_interval: DELAY_INTERVAL,
+      batch_size: BATCH_SIZE,
+      sub_batch_size: SUB_BATCH_SIZE
+    )
+  end
+
+  def down
+    delete_batched_background_migration(
+      MIGRATION,
+      :snippet_repositories,
+      :snippet_id,
+      [
+        :snippet_organization_id,
+        :snippets,
+        :organization_id,
+        :snippet_id
+      ]
+    )
+  end
+end

db/schema_migrations/20241211134706

+48f5f852380a7a3e179c913568219ae8799b32c9b59c0f3219be6dc31a45332b
\ No newline at end of file

db/schema_migrations/20241211134707

+4a3a2de5da48f40751d6b7bacb5a4081954eda13a31f535e0e65c17c821c2e3a
\ No newline at end of file

db/schema_migrations/20241211134708

+d3b3fd1a860be5e1fcf4b1ed8a645d350c9527601db2bf9d0cc5cf8872382026
\ No newline at end of file

db/schema_migrations/20241211134709

+035aea2280dca568860cf636727798b9f2a0d98669d8302d12ed8183f07035c7
\ No newline at end of file

db/schema_migrations/20241211134710

+38438dced134d7c7bf46dc0bb955f566c4edaebe6bd47b9a4b7d39027c2f0e98
\ No newline at end of file

db/schema_migrations/20241211134711

+7bcb615592724e6a9b499293e5e4eb48daab5a867499a707735fbe336c46b78e
\ No newline at end of file

db/schema_migrations/20241211134712

+e53745adeacf8ec8b3ca1f343927942093bdfbccb76d56ac90b4728f967bf7c8
\ No newline at end of file