From 2f7dcc257d214b5586b5bb6cdd688ada7fc6efb0 Mon Sep 17 00:00:00 2001 From: Ben Bodenmiller <bbodenmiller@gmail.com> Date: Mon, 28 Jun 2021 20:56:42 +0000 Subject: [PATCH] Improve incremental logging architecture details --- doc/administration/job_logs.md | 8 ++++---- doc/administration/object_storage.md | 6 ------ doc/install/aws/index.md | 2 +- 3 files changed, 5 insertions(+), 11 deletions(-) diff --git a/doc/administration/job_logs.md b/doc/administration/job_logs.md index 510da68442c08..d42444b4ec2ef 100644 --- a/doc/administration/job_logs.md +++ b/doc/administration/job_logs.md @@ -108,7 +108,7 @@ See "Phase 4: uploading" in [Data flow](#data-flow) to learn about the process. If you want to avoid any local disk usage for job logs, you can do so using one of the following options: -- Enable the [beta incremental logging](#incremental-logging-architecture) feature. +- Enable the [incremental logging](#incremental-logging-architecture) feature. - Set the [job logs location](#changing-the-job-logs-local-location) to an NFS drive. @@ -140,7 +140,7 @@ For more information, see [delete references to missing artifacts](raketasks/che > - [Recommended for production use with AWS S3](https://gitlab.com/gitlab-org/gitlab/-/issues/273498) in GitLab 13.7. > - To use in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-incremental-logging). **(FREE SELF)** -Job logs are sent from the GitLab Runner in chunks and cached temporarily on disk +By default job logs are sent from the GitLab Runner in chunks and cached temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by Omnibus GitLab. After the job completes, a background job archives the job log. The log is moved to `/var/opt/gitlab/gitlab-rails/shared/artifacts/` by default, or to object storage if configured. @@ -150,7 +150,7 @@ server, these two locations on the filesystem have to be shared using NFS. To eliminate both filesystem requirements: -- Enable the incremental logging feature, which uses Redis instead of disk space for temporary caching of job logs. +- [Enable the incremental logging feature](#enable-or-disable-incremental-logging), which uses Redis instead of disk space for temporary caching of job logs. - Configure [object storage](job_artifacts.md#object-storage-settings) for storing archived job logs. ### Technical details @@ -185,7 +185,7 @@ Here is the detailed data flow: ### Enable or disable incremental logging **(FREE SELF)** -Incremental logging is under development, but ready for production use. It is +Incremental logging is under development, but [ready for production use as of GitLab 13.6](https://gitlab.com/groups/gitlab-org/-/epics/4275). It is deployed behind a feature flag that is **disabled by default**. [GitLab administrators with access to the GitLab Rails console](feature_flags.md) can enable it. diff --git a/doc/administration/object_storage.md b/doc/administration/object_storage.md index f1025bd18463f..5e1a2efbb4dc2 100644 --- a/doc/administration/object_storage.md +++ b/doc/administration/object_storage.md @@ -591,12 +591,6 @@ with the Fog library that GitLab uses. Symptoms include an error in `production. 411 Length Required ``` -### Incremental logging is required for CI to use object storage - -If you configure GitLab to use object storage for CI logs and artifacts, -you can avoid [local disk usage for job logs](job_logs.md#data-flow) by enabling -[beta incremental logging](job_logs.md#incremental-logging-architecture). - ### Proxy Download Clients can download files in object storage by receiving a pre-signed, time-limited URL, diff --git a/doc/install/aws/index.md b/doc/install/aws/index.md index df18c5255a4c1..5abc4bd312215 100644 --- a/doc/install/aws/index.md +++ b/doc/install/aws/index.md @@ -841,6 +841,6 @@ If you see this page when trying to set a password via the web interface, make s ### Some job logs are not uploaded to object storage -When the GitLab deployment is scaled up to more than one node, some job logs may not be uploaded to [object storage](../../administration/object_storage.md) properly. [Incremental logging is required](../../administration/object_storage.md#incremental-logging-is-required-for-ci-to-use-object-storage) for CI to use object storage. +When the GitLab deployment is scaled up to more than one node, some job logs may not be uploaded to [object storage](../../administration/object_storage.md) properly. [Incremental logging is required](../../administration/object_storage.md#other-alternatives-to-file-system-storage) for CI to use object storage. Enable [incremental logging](../../administration/job_logs.md#enable-or-disable-incremental-logging) if it has not already been enabled. -- GitLab