In order to give some room with a namespace for google cloud platform
integration, specify more accurate namespace for cloud seed files as
cloud_seed/google_cloud.

Changelog: changed

Before [serving pages directly from pages_deployments], we were creating
an `PagesDeployment` and in the commit callback, the ZIP file was
uploaded. Only after the `PagesDeployment` entry was committed to the
database, we would update the `ProjectPagesMetadatum` to indicate the
deployment was ready.

Now, [serving pages directly from pages_deployments], created a problem
where we're starting to serve a `PagesDeployment` before its upload is
ready.

To fix that, a new `upload_ready` column will be introduced, to
signalize when the new `PagesDeployment` upload is finished and we
start to serve it.

[serving pages directly from pages_deployments]: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/135993

Changelog: fixed

    In order to be able to consume symbol files from the server, each uploaded symbol package should undergo an indexing process. This indexing process involves extracting the symbol files (.pdb) from the compressed package file and indexing their information for efficient symbol lookup and retrieval. The indexing service should create a record in the packages_nuget_symbols table for each .pdb file, storing relevant details such as the symbol's signature and file.

Changelog: added

In order to support symbol server in GitLab NuGet repository, we need to create a DB table to store the symbol files in with the needed metadata such as file signature, file_path, & size.

Changelog: added

This fixes the geo replication bug wherein the default behavior of
Carrierwave is skipped when job artifact is uploaded to final location.

Changelog: fixed

This fixes the bug of pending direct uploads not being
properly cleared off the Redis hash even on successful upload.

The cause of the problem was that the redis key pattern being used
to delete the redis field still includes the bucket prefix.

Changelog: fixed

Fixes some `Naming/InclusiveLanguage` offenses.

Changelog: added

Job artifacts that are uploaded directly to object storage
by workhorse don't need to be copied anymore.

Changelog: added

- For better backward compatibility use string keys on
  GitlabSettings::Options
- Remove all the Settingslogic references from code (not comments)
- Ensure to use GitlabSettings::Options when pushing new options to
  Array values
- Instead of inheriting from Hash, wrap a hash in
  GitlabSettings::Options and make it respond to all the requirements.
    * PROS: This provides more autonomy on what needs to be done
    * CONS: Some places were checking by Hash, we need to tackle these
    scenarios
- Ensure to deep_merge return nested Options
- Use the _raw_ version of options when passing to Fog::Storage
- Ensure to load required active_support libraries
- Show the option name in the GitlabSettings::MissingSetting error
- Ensure to have an option `source` for the path of the `gitlab.yml`
- Fix GitlabSettings::Options#deep_merge and
  GitlabSettings::Options#merge to return a new GitlabSettings::Options
  object with the options deep_merged/merged
- Add GitlabSettings#reload! for backward compatibility
- Make file/namespace initializer arguments
- Avoid mutating Settings in sidekiq_cluster/cli tests
- Fix configuration stubbing

Related to: https://gitlab.com/gitlab-org/gitlab/-/issues/286873

Signed-off-by: Rémy Coutable <remy@rymai.me>

Fix Layout/ArgumentAlignment offenses in spec/uploaders

Previously if a job artifact were prefaced with "", those bytes were
stripped when rendering a Google CDN URL because the URL did not
properly encode the characters. This commit now uses
`Addressable::URI.encode_component` to do the
job. `CarrierWave::Utilities::Uri#encode_path` does the same thing.

Relates to https://gitlab.com/gitlab-org/gitlab/-/issues/396945

Changelog: fixed

This adds support for referencing storage config
via symbols.

This is needed so that the original filename is properly
shown when downloaded by users even for when there is no
CDN used and the uploaded object name is a random generated
hash.

Changelog: fixed

* Remove the `head_request_on_aws_file?` from API::Helpers since it's
  not much of an abstraction and its signature conflicts with a method
  of the same name in API::MavenPackages

Contributes to: https://gitlab.com/gitlab-org/gitlab/-/issues/389241
Changelog: fixed

Updates the feature categories to reflect that the
google CDN is only used by build artifacts currently.

On a FIPS kernel with a non-FIPS GitLab build, Maven package uploads
previously would fail with 422 Unprocessible Entity errors. This
occurred because Workhorse was generating an MD5 sum for the
accelerated upload, but the Rails API rejects any use of MD5 in FIPS
mode.

Workhorse was generating an MD5 sum because its on check for FIPS
involves more conditions:

1. The binary has been compiled with the `fips` build tag.
2. The platform is amd64 running on a Linux runtime.
3. The kernel has FIPS enabled (e.g. `/proc/sys/crypto/fips_enabled` is 1).
4. A system OpenSSL can be dynamically loaded via ldopen().

On a standard GitLab installation, 1 is not true, so Workhorse
believed it was working in a standard environment. However, GitLab
Rails only checks whether the OpenSSL library is FIPS-enabled.

To resolve this discrepancy, we now:

1. Pass `UploadHashFunctions` to indicate which hashes should be used
   in the Workhorse `/authorize` response.
2. Workhorse will use this list to determine which hashes to compute.
   If the list is empty, Workhorse will assume all hashes can be used.

Relates to https://gitlab.com/gitlab-org/gitlab/-/issues/380559

Changelog: fixed

This commit introduces a new finder class to be used while we are
transitioning from using artifacts to utilizing data stored in the DB.

This new finder class will mainly use the data stored in the database
except the remediations are still read from report artifacts by using
the `multi_read` method.

With this change, we will also no longer show the findings of purged
security scans.

Changelog: fixed
EE: true

Fix rubocop offenses

Changelog: other

This reverts commit 6496911bac582f474b43c057d50277eace2b01b9.

This removes the background_upload object storage configuration
setting. This setting has been hard-coded to false since
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/97170, so there
is no point in keeping the setting.

All code that depends on the setting has already been removed in
previous commits.

Changelog: other

This concern has been a no-op since
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/97170.

Part of https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1690.

Changelog: other

Since https://gitlab.com/gitlab-org/gitlab/-/merge_requests/97170, the
object storage background_upload setting is hard coded to false. This
makes ObjectStorage::BackgroundMoveWorker a no-op _and_ prevents jobs
for this worker to be scheduled in the first place.

This commit removes ObjectStorage::BackgroundMoveWorker. Part of
https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1690.

Changelog: other

Since https://gitlab.com/gitlab-org/gitlab/-/merge_requests/97170 it
is impossible to enable background_upload. This commit removes code
from Package uploaders that would only run when background_upload was
enabled.

Changelog: other

This is mostly safe anyway, as it doesn't pass any arguments, but we
should still check that the given `field_field` (mount point) is
actually an uploader of the correct type before we call it.

This is a follow-up merge request to
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/98010, which
enabled job artifacts to be served by Google Cloud CDN for specific
API endpoints. This extends this functionality to artifact downloads
from the Web UI via the `Download` artifact button.

This functionality is behind the
`use_cdn_with_job_artifacts_ui_downloads` feature flag. This feature
should not only make artifacts downloads faster for users, but it also
helps reduce network egress bandwidth costs.

Since UI downloads can have `response-content-disposition` and
`response-content-type` query parameters, we also need to modify the
Cloud CDN implementation to allow passing of query parameters to
Google Cloud Storage. Note that Cloud CDN requires that the `Expires`,
`KeyName`, and `Signature` query parameters appear at the end of the
URL, or the request will fail with a 403 error.

Relates to https://gitlab.com/gitlab-org/gitlab/-/issues/378473

Changelog: added

The feature flag has been enabled by default on GitLab.com for several
days now with no issues. We can now drop the flag and changes relating
to making it scoped to the project
(https://gitlab.com/gitlab-org/gitlab/-/merge_requests/98010).

This will make it easier to support using the CDN for Web UI downloads
(https://gitlab.com/gitlab-org/gitlab/-/issues/378473).

Relates to https://gitlab.com/gitlab-org/gitlab/-/issues/373860

Changelog: changed

Addresses such as 127.0.0.1 and 169.254.0.0 should be excluded when
deciding to use the Google CDN.

This wasn't being used in practice, but for completeness we should
exclude these addresses.

Changelog: changed

Specifying a URL without https:// is invalid. If the scheme is
omitted, mark the configuration invalid so the CDN will not be used.

This came up in
https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/16467.

Changelog: changed

Issue url: https://gitlab.com/gitlab-org/gitlab/-/issues/372286
Added Packages::Rpm::RepositoryFile model
for project-level files required by RPM yum/dnf repositories.
Added corresponding uploader and specs for model.

Changelog: added

This builds upon
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/96336 to provide
configuration support and hooks to serve CI artifacts via Google CDN
if it is configured in front of a Google Cloud Storage bucket.

This is currently behind the `ci_job_artifacts_cdn` feature flag.

Relates to https://gitlab.com/gitlab-org/gitlab/-/issues/360462

Changelog: added

Previously the initial merge request in the Google CDN code
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/96336 would
fetch the list of Google IPs within the same thread as the process
that attempts to serve out a Google CDN-frontend URL. However, there
are several issues with this:

1. If the HTTP endpoint is slow or down for some reason, this will
delay the response handling.

2. There is no throttling to prevent excessive requests to this
endpoint.

To improve the reliability, we move the fetching of this URL to a
background Sidekiq job that has an application rate limit to prevent
too many requests. The initial limit is 10 requests in a minute, which
should should be adequate since we only need one worker to populate
the cache for a whole day.

Relates to https://gitlab.com/gitlab-org/gitlab/-/issues/373115

Changelog: changed

This commit adds base support for using Google CDN in front of an
object storage bucket to save costs. Note that this commit does not
actually enable the use of the CDN yet; additional configuration and
changes to the callers must be made for this to work.

To enable a bucket to use the Google CDN:

1. A CarrierWave uploader should `include ObjectStorage::CDN::Concern`.
2. Instead of calling `#url`, a caller can now call `#use_cdn?` and
   `#cdn_signed_url`.

At the moment, transfers within the Google Cloud Platform (GCP)
network between Google Cloud Storage and virutal machines are free, so
serving out artifacts to GCP machines would waste money in the form of
egress traffic.

To avoid that, `#use_cdn?` will fetch the list of valid, public Google
IP ranges and exclude IPs that are private or within the public Google
network.

`#cdn_signed_url` will create a pre-signed URL valid for 10 minutes by
default. The Google CDN must be set up so that a Google Cloud
Storage bucket is behind a unique URL.

Relates to https://gitlab.com/gitlab-org/gitlab/-/issues/360462

Changelog: added

And use ::Packages::UpdatePackageFileService.

BREAKING CHANGE: existing Debian packages will be unreachable.

Changelog: changed