From 28421ea1c55311d89a98add71bcc5fb0387a508f Mon Sep 17 00:00:00 2001
From: Stan Hu <stanhu@gmail.com>
Date: Thu, 30 Jun 2022 23:31:48 -0700
Subject: [PATCH] Add more docs about background upload removal
This follows up on a number of issues we found when direct uploads
became the default:
- Buckets may not have delete permissions
- Using a bucket prefix no longer works
We include a workaround to make background uploads work again.
Relates to items discussed in:
- https://gitlab.com/gitlab-org/gitlab/-/issues/365810
- https://gitlab.com/gitlab-org/gitlab-runner/-/issues/26869
---
...emove-background-upload-object-storage.yml | 31 ++++++++++++++++++-
doc/update/removals.md | 29 ++++++++++++++++-
2 files changed, 58 insertions(+), 2 deletions(-)
diff --git a/data/removals/15_0/15-0-remove-background-upload-object-storage.yml b/data/removals/15_0/15-0-remove-background-upload-object-storage.yml
index dcfd18cf809a6..16aab3d14c21b 100644
--- a/data/removals/15_0/15-0-remove-background-upload-object-storage.yml
+++ b/data/removals/15_0/15-0-remove-background-upload-object-storage.yml
@@ -7,17 +7,46 @@
reporter: fzimmer
body: | # Do not modify this line, instead modify the lines below.
To reduce the overall complexity and maintenance burden of GitLab's [object storage feature](https://docs.gitlab.com/ee/administration/object_storage.html), support for using `background_upload` has been removed in GitLab 15.0.
+ By default [direct upload](https://docs.gitlab.com/ee/development/uploads/index.html#direct-upload) will be used.
- This impacts a small subset of object storage providers, including but not limited to:
+ This impacts a subset of object storage providers, including but not limited to:
- **OpenStack** Customers using OpenStack need to change their configuration to use the S3 API instead of Swift.
- **RackSpace** Customers using RackSpace-based object storage need to migrate data to a different provider.
If your object storage provider does not support `background_upload`, please [migrate objects to a supported object storage provider](https://docs.gitlab.com/ee/administration/object_storage.html#migrate-objects-to-a-different-object-storage-provider).
+ #### Encrypted S3 buckets
+
Additionally, this also breaks the use of [encrypted S3 buckets](https://docs.gitlab.com/ee/administration/object_storage.html#encrypted-s3-buckets) with [storage-specific configuration form](https://docs.gitlab.com/ee/administration/object_storage.html#storage-specific-configuration).
If your S3 buckets have [SSE-S3 or SSE-KMS encryption enabled](https://docs.aws.amazon.com/kms/latest/developerguide/services-s3.html), please [migrate your configuration to use consolidated object storage form](https://docs.gitlab.com/ee/administration/object_storage.html#transition-to-consolidated-form) before upgrading to GitLab 15.0. Otherwise, you may start getting `ETag mismatch` errors during objects upload.
+
+ #### 403 errors
+
+ If you see 403 errors when uploading to object storage after
+ upgrading to GitLab 15.0, check that the [correct permissions](https://docs.gitlab.com/ee/administration/object_storage.html#iam-permissions)
+ are assigned to the bucket. Direct upload needs the ability to delete an
+ object (example: `s3:DeleteObject`), but background uploads do not.
+
+ #### `remote_directory` with a path prefix
+
+ If the object storage `remote_directory` configuration contains a slash (`/`) after the bucket (example: `gitlab/uploads`), be aware that this [was never officially supported](https://gitlab.com/gitlab-org/gitlab/-/issues/292958).
+ Some users found that they could specify a path prefix to the bucket. In direct upload mode, object storage uploads will fail if a slash is present in GitLab 15.0.
+
+ If you have set a prefix, you can use a workaround to revert to background uploads:
+
+ 1. Continue to use [storage-specific configuration](https://docs.gitlab.com/ee/administration/object_storage.html#storage-specific-configuration).
+ 1. In Omnibus GitLab, set the `GITLAB_LEGACY_BACKGROUND_UPLOADS` to re-enable background uploads:
+
+ ```ruby
+ gitlab_rails['env'] = { 'GITLAB_LEGACY_BACKGROUND_UPLOADS' => 'artifacts,external_diffs,lfs,uploads,packages,dependency_proxy,terraform_state,pages' }
+ ```
+
+ Prefixes will be supported officially in [GitLab 15.2](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91307).
+ This workaround will be dropped, so we encourage migrating to consolidated object storage.
+
+
stage: Enablement
tiers: [Core, Premium, Ultimate]
issue_url: https://gitlab.com/gitlab-org/gitlab/-/issues/26600
diff --git a/doc/update/removals.md b/doc/update/removals.md
index 32fbf85117696..dc4cf874a9386 100644
--- a/doc/update/removals.md
+++ b/doc/update/removals.md
@@ -66,18 +66,45 @@ This is a [breaking change](https://docs.gitlab.com/ee/development/contributing/
Review the details carefully before upgrading.
To reduce the overall complexity and maintenance burden of GitLab's [object storage feature](https://docs.gitlab.com/ee/administration/object_storage.html), support for using `background_upload` has been removed in GitLab 15.0.
+By default [direct upload](https://docs.gitlab.com/ee/development/uploads/index.html#direct-upload) will be used.
-This impacts a small subset of object storage providers, including but not limited to:
+This impacts a subset of object storage providers, including but not limited to:
- **OpenStack** Customers using OpenStack need to change their configuration to use the S3 API instead of Swift.
- **RackSpace** Customers using RackSpace-based object storage need to migrate data to a different provider.
If your object storage provider does not support `background_upload`, please [migrate objects to a supported object storage provider](https://docs.gitlab.com/ee/administration/object_storage.html#migrate-objects-to-a-different-object-storage-provider).
+#### Encrypted S3 buckets
+
Additionally, this also breaks the use of [encrypted S3 buckets](https://docs.gitlab.com/ee/administration/object_storage.html#encrypted-s3-buckets) with [storage-specific configuration form](https://docs.gitlab.com/ee/administration/object_storage.html#storage-specific-configuration).
If your S3 buckets have [SSE-S3 or SSE-KMS encryption enabled](https://docs.aws.amazon.com/kms/latest/developerguide/services-s3.html), please [migrate your configuration to use consolidated object storage form](https://docs.gitlab.com/ee/administration/object_storage.html#transition-to-consolidated-form) before upgrading to GitLab 15.0. Otherwise, you may start getting `ETag mismatch` errors during objects upload.
+#### 403 errors
+
+If you see 403 errors when uploading to object storage after
+upgrading to GitLab 15.0, check that the [correct permissions](https://docs.gitlab.com/ee/administration/object_storage.html#iam-permissions)
+are assigned to the bucket. Direct upload needs the ability to delete an
+object (example: `s3:DeleteObject`), but background uploads do not.
+
+#### `remote_directory` with a path prefix
+
+If the object storage `remote_directory` configuration contains a slash (`/`) after the bucket (example: `gitlab/uploads`), be aware that this [was never officially supported](https://gitlab.com/gitlab-org/gitlab/-/issues/292958).
+Some users found that they could specify a path prefix to the bucket. In direct upload mode, object storage uploads will fail if a slash is present in GitLab 15.0.
+
+If you have set a prefix, you can use a workaround to revert to background uploads:
+
+1. Continue to use [storage-specific configuration](https://docs.gitlab.com/ee/administration/object_storage.html#storage-specific-configuration).
+1. In Omnibus GitLab, set the `GITLAB_LEGACY_BACKGROUND_UPLOADS` to re-enable background uploads:
+
+ ```ruby
+ gitlab_rails['env'] = { 'GITLAB_LEGACY_BACKGROUND_UPLOADS' => 'artifacts,external_diffs,lfs,uploads,packages,dependency_proxy,terraform_state,pages' }
+ ```
+
+Prefixes will be supported officially in [GitLab 15.2](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91307).
+This workaround will be dropped, so we encourage migrating to consolidated object storage.
+
### Container Network and Host Security
WARNING:
--
GitLab