diff --git a/doc/administration/raketasks/_index.md b/doc/administration/raketasks/_index.md index d7394c8a6d0dc258c0e102b8d59348eba622c6ca..4129aa3e7a8f8f50bc2c7e98fdd2d6f0153411e3 100644 --- a/doc/administration/raketasks/_index.md +++ b/doc/administration/raketasks/_index.md @@ -53,14 +53,31 @@ The following Rake tasks are available for use with GitLab: To list all available Rake tasks: +{{< tabs >}} + +{{< tab title="Linux package (Omnibus)" >}} + ```shell -# Omnibus GitLab sudo gitlab-rake -vT +``` -# GitLab Helm chart +{{< /tab >}} + +{{< tab title="Helm chart (Kubernetes)" >}} + +```shell gitlab-rake -vT +``` + +{{< /tab >}} + +{{< tab title="Self-compiled (source)" >}} -# Installations from source +```shell cd /home/git/gitlab sudo -u git -H bundle exec rake -vT RAILS_ENV=production ``` + +{{< /tab >}} + +{{< /tabs >}} diff --git a/doc/administration/raketasks/check.md b/doc/administration/raketasks/check.md index 405f75878e3a3f828d35cf1dd7583c00a976f785..b0d68841345b3d62073dab9affee69ea6b112f10 100644 --- a/doc/administration/raketasks/check.md +++ b/doc/administration/raketasks/check.md @@ -48,17 +48,27 @@ This task loops through the project code repositories and runs the integrity che described previously. If a project uses a pool repository, that is also checked. Other types of Git repositories [are not checked](https://gitlab.com/gitlab-org/gitaly/-/issues/3643). -- Linux package installations: +To check project code repositories: - ```shell - sudo gitlab-rake gitlab:git:fsck - ``` +{{< tabs >}} -- Self-compiled installations: +{{< tab title="Linux package (Omnibus)" >}} - ```shell - sudo -u git -H bundle exec rake gitlab:git:fsck RAILS_ENV=production - ``` +```shell +sudo gitlab-rake gitlab:git:fsck +``` + +{{< /tab >}} + +{{< tab title="Self-compiled (source)" >}} + +```shell +sudo -u git -H bundle exec rake gitlab:git:fsck RAILS_ENV=production +``` + +{{< /tab >}} + +{{< /tabs >}} ## Checksum of repository refs @@ -79,17 +89,27 @@ checksums in the format `<PROJECT ID>,<CHECKSUM>`. - If a repository exists but is empty, the output checksum is `0000000000000000000000000000000000000000`. - Projects which don't exist are skipped. -- Linux package installations: +To check all GitLab repositories: + +{{< tabs >}} + +{{< tab title="Linux package (Omnibus)" >}} + +```shell +sudo gitlab-rake gitlab:git:checksum_projects +``` + +{{< /tab >}} - ```shell - sudo gitlab-rake gitlab:git:checksum_projects - ``` +{{< tab title="Self-compiled (source)" >}} -- Self-compiled installations: +```shell +sudo -u git -H bundle exec rake gitlab:git:checksum_projects RAILS_ENV=production +``` - ```shell - sudo -u git -H bundle exec rake gitlab:git:checksum_projects RAILS_ENV=production - ``` +{{< /tab >}} + +{{< /tabs >}} For example, if: @@ -130,23 +150,33 @@ Integrity checks are supported for the following types of file: - Project-level Secure Files (introduced in GitLab 16.1.0) - User uploads -- Linux package installations: +To check the integrity of uploaded files: + +{{< tabs >}} - ```shell - sudo gitlab-rake gitlab:artifacts:check - sudo gitlab-rake gitlab:ci_secure_files:check - sudo gitlab-rake gitlab:lfs:check - sudo gitlab-rake gitlab:uploads:check - ``` +{{< tab title="Linux package (Omnibus)" >}} -- Self-compiled installations: +```shell +sudo gitlab-rake gitlab:artifacts:check +sudo gitlab-rake gitlab:ci_secure_files:check +sudo gitlab-rake gitlab:lfs:check +sudo gitlab-rake gitlab:uploads:check +``` - ```shell - sudo -u git -H bundle exec rake gitlab:artifacts:check RAILS_ENV=production - sudo -u git -H bundle exec rake gitlab:ci_secure_files:check RAILS_ENV=production - sudo -u git -H bundle exec rake gitlab:lfs:check RAILS_ENV=production - sudo -u git -H bundle exec rake gitlab:uploads:check RAILS_ENV=production - ``` +{{< /tab >}} + +{{< tab title="Self-compiled (source)" >}} + +```shell +sudo -u git -H bundle exec rake gitlab:artifacts:check RAILS_ENV=production +sudo -u git -H bundle exec rake gitlab:ci_secure_files:check RAILS_ENV=production +sudo -u git -H bundle exec rake gitlab:lfs:check RAILS_ENV=production +sudo -u git -H bundle exec rake gitlab:uploads:check RAILS_ENV=production +``` + +{{< /tab >}} + +{{< /tabs >}} These tasks also accept some environment variables which you can use to override certain values: @@ -224,17 +254,27 @@ documentation on what to do [when the secrets file is lost](../backup_restore/tr This can take a very long time, depending on the size of your database, as it checks all rows in all tables. -- Linux package installations: +To verify database values can be decrypted using the current secrets: - ```shell - sudo gitlab-rake gitlab:doctor:secrets - ``` +{{< tabs >}} -- Self-compiled installations: +{{< tab title="Linux package (Omnibus)" >}} + +```shell +sudo gitlab-rake gitlab:doctor:secrets +``` - ```shell - bundle exec rake gitlab:doctor:secrets RAILS_ENV=production - ``` +{{< /tab >}} + +{{< tab title="Self-compiled (source)" >}} + +```shell +bundle exec rake gitlab:doctor:secrets RAILS_ENV=production +``` + +{{< /tab >}} + +{{< /tabs >}} **Example output** @@ -253,19 +293,29 @@ I, [2020-06-11T17:18:15.575711 #27148] INFO -- : Done! ### Verbose mode To get more detailed information about which rows and columns can't be -decrypted, you can pass a `VERBOSE` environment variable: +decrypted, you can pass a `VERBOSE` environment variable. + +To verify database values can be decrypted using the current secrets with detailed information: + +{{< tabs >}} -- Linux package installations: +{{< tab title="Linux package (Omnibus)" >}} - ```shell - sudo gitlab-rake gitlab:doctor:secrets VERBOSE=1 - ``` +```shell +sudo gitlab-rake gitlab:doctor:secrets VERBOSE=1 +``` + +{{< /tab >}} + +{{< tab title="Self-compiled (source)" >}} + +```shell +bundle exec rake gitlab:doctor:secrets RAILS_ENV=production VERBOSE=1 +``` -- Self-compiled installations: +{{< /tab >}} - ```shell - bundle exec rake gitlab:doctor:secrets RAILS_ENV=production VERBOSE=1 - ``` +{{< /tabs >}} **Example verbose output** @@ -311,18 +361,26 @@ To reset broken tokens: 1. Identify the broken tokens. For example `runners_token`. 1. To reset broken tokens, run `gitlab:doctor:reset_encrypted_tokens` with `VERBOSE=true MODEL_NAMES=Model1,Model2 TOKEN_NAMES=broken_token1,broken_token2`. For example: - - Linux package installations: - - ```shell - VERBOSE=true MODEL_NAMES=Project,Group TOKEN_NAMES=runners_token gitlab-rake gitlab:doctor:reset_encrypted_tokens - ``` + {{< tabs >}} - - Self-compiled installations: + {{< tab title="Linux package (Omnibus)" >}} + + ```shell + VERBOSE=true MODEL_NAMES=Project,Group TOKEN_NAMES=runners_token gitlab-rake gitlab:doctor:reset_encrypted_tokens + ``` + + {{< /tab >}} + + {{< tab title="Self-compiled (source)" >}} + + ```shell + bundle exec rake gitlab:doctor:reset_encrypted_tokens RAILS_ENV=production VERBOSE=true MODEL_NAMES=Project,Group TOKEN_NAMES=runners_token + ``` + + {{< /tab >}} + + {{< /tabs >}} - ```shell - bundle exec rake gitlab:doctor:reset_encrypted_tokens RAILS_ENV=production VERBOSE=true MODEL_NAMES=Project,Group TOKEN_NAMES=runners_token - ``` - You will see every action this task would try to perform: ```plain @@ -346,17 +404,25 @@ To reset broken tokens: 1. If you are confident that this operation resets the correct tokens, disable dry-run mode and run the operation again: - - Linux package installations: + {{< tabs >}} + + {{< tab title="Linux package (Omnibus)" >}} + + ```shell + DRY_RUN=false VERBOSE=true MODEL_NAMES=Project,Group TOKEN_NAMES=runners_token gitlab-rake gitlab:doctor:reset_encrypted_tokens + ``` - ```shell - DRY_RUN=false VERBOSE=true MODEL_NAMES=Project,Group TOKEN_NAMES=runners_token gitlab-rake gitlab:doctor:reset_encrypted_tokens - ``` + {{< /tab >}} + + {{< tab title="Self-compiled (source)" >}} + + ```shell + bundle exec rake gitlab:doctor:reset_encrypted_tokens RAILS_ENV=production DRY_RUN=false VERBOSE=true MODEL_NAMES=Project,Group TOKEN_NAMES=runners_token + ``` - - Self-compiled installations: + {{< /tab >}} - ```shell - bundle exec rake gitlab:doctor:reset_encrypted_tokens RAILS_ENV=production DRY_RUN=false VERBOSE=true MODEL_NAMES=Project,Group TOKEN_NAMES=runners_token - ``` + {{< /tabs >}} ## Troubleshooting diff --git a/doc/administration/raketasks/cleanup.md b/doc/administration/raketasks/cleanup.md index 01a868bca5846e2cc2ba06327d563bbc7a8a5f37..ba8d7edf118ea8a426606a037f88add6f0f11ac4 100644 --- a/doc/administration/raketasks/cleanup.md +++ b/doc/administration/raketasks/cleanup.md @@ -25,18 +25,28 @@ have finished, which otherwise may lead to data loss. When you remove LFS files from a repository's history, they become orphaned and continue to consume disk space. With this Rake task, you can remove invalid references from the database, which -allows garbage collection of LFS files. +allows garbage collection of LFS files. For example: -For example: +{{< tabs >}} + +{{< tab title="Linux package (Omnibus)" >}} ```shell -# omnibus-gitlab sudo gitlab-rake gitlab:cleanup:orphan_lfs_file_references PROJECT_PATH="gitlab-org/gitlab-foss" +``` -# installation from source +{{< /tab >}} + +{{< tab title="Self-compiled (source)" >}} + +```shell bundle exec rake gitlab:cleanup:orphan_lfs_file_references RAILS_ENV=production PROJECT_PATH="gitlab-org/gitlab-foss" ``` +{{< /tab >}} + +{{< /tabs >}} + You can also specify the project with `PROJECT_ID` instead of `PROJECT_PATH`. For example: @@ -59,16 +69,28 @@ later (once a day). If you need to garbage collect them immediately, run ### Remove unreferenced LFS files immediately Unreferenced LFS files are removed on a daily basis but you can remove them immediately if -you need to. For example: +you need to. To remove unreferenced LFS files immediately: + +{{< tabs >}} + +{{< tab title="Linux package (Omnibus)" >}} ```shell -# omnibus-gitlab sudo gitlab-rake gitlab:cleanup:orphan_lfs_files +``` -# installation from source +{{< /tab >}} + +{{< tab title="Self-compiled (source)" >}} + +```shell bundle exec rake gitlab:cleanup:orphan_lfs_files ``` +{{< /tab >}} + +{{< /tabs >}} + Example output: ```shell @@ -84,16 +106,28 @@ Clean up project upload files if they don't exist in GitLab database. Clean up local project upload files if they don't exist in GitLab database. The task attempts to fix the file if it can find its project, otherwise it moves the -file to a lost and found directory. +file to a lost and found directory. To clean up project upload files from file system: + +{{< tabs >}} + +{{< tab title="Linux package (Omnibus)" >}} ```shell -# omnibus-gitlab sudo gitlab-rake gitlab:cleanup:project_uploads +``` + +{{< /tab >}} + +{{< tab title="Self-compiled (source)" >}} -# installation from source +```shell bundle exec rake gitlab:cleanup:project_uploads RAILS_ENV=production ``` +{{< /tab >}} + +{{< /tabs >}} + Example output: ```shell @@ -114,20 +148,34 @@ I, [2018-07-27T12:08:33.755624 #89817] INFO -- : Did fix /opt/gitlab/embedded/s I, [2018-07-27T12:08:33.760257 #89817] INFO -- : Did move to lost and found /opt/gitlab/embedded/service/gitlab-rails/public/uploads/foo/bar/1dd6f0f7eefd2acc4c2233f89a0f7b0b/image.png -> /opt/gitlab/embedded/service/gitlab-rails/public/uploads/-/project-lost-found/foo/bar/1dd6f0f7eefd2acc4c2233f89a0f7b0b/image.png ``` -If using object storage, run the [All-in-one Rake task](../raketasks/uploads/migrate.md#all-in-one-rake-task) to ensure all uploads are migrated to object storage and there are no files on disk in the uploads folder. +If using object storage, run the [All-in-one Rake task](../raketasks/uploads/migrate.md#all-in-one-rake-task) to ensure +all uploads are migrated to object storage and there are no files on disk in the uploads folder. ### Clean up project upload files from object storage Move object store upload files to a lost and found directory if they don't exist in GitLab database. +To clean up project upload files from object storage: + +{{< tabs >}} + +{{< tab title="Linux package (Omnibus)" >}} ```shell -# omnibus-gitlab sudo gitlab-rake gitlab:cleanup:remote_upload_files +``` + +{{< /tab >}} + +{{< tab title="Self-compiled (source)" >}} -# installation from source +```shell bundle exec rake gitlab:cleanup:remote_upload_files RAILS_ENV=production ``` +{{< /tab >}} + +{{< /tabs >}} + Example output: ```shell @@ -201,14 +249,28 @@ level with `NICENESS`. Below are the valid levels, but consult ## Remove expired ActiveSession lookup keys +To remove expired ActiveSession lookup keys: + +{{< tabs >}} + +{{< tab title="Linux package (Omnibus)" >}} + ```shell -# omnibus-gitlab sudo gitlab-rake gitlab:cleanup:sessions:active_sessions_lookup_keys +``` -# installation from source +{{< /tab >}} + +{{< tab title="Self-compiled (source)" >}} + +```shell bundle exec rake gitlab:cleanup:sessions:active_sessions_lookup_keys RAILS_ENV=production ``` +{{< /tab >}} + +{{< /tabs >}} + ## Container registry garbage collection Container Registry can use considerable amounts of disk space. To clear up