From eb9d11bef59342d1e09de2cdb1d95c4880feea6a Mon Sep 17 00:00:00 2001
From: Ashraf Khamis <akhamis@gitlab.com>
Date: Mon, 9 Dec 2024 10:13:57 +0000
Subject: [PATCH] Refactor Elasticsearch troubleshooting docs (3)
---
.../sidekiq/sidekiq_troubleshooting.md | 2 +-
doc/development/advanced_search.md | 3 -
.../advanced_search/elasticsearch.md | 4 +
.../elasticsearch_troubleshooting.md | 196 +-----------------
.../elasticsearch/troubleshooting/index.md | 17 ++
.../troubleshooting/migrations.md | 153 ++++++++++++++
6 files changed, 183 insertions(+), 192 deletions(-)
create mode 100644 doc/integration/elasticsearch/troubleshooting/index.md
create mode 100644 doc/integration/elasticsearch/troubleshooting/migrations.md
diff --git a/doc/administration/sidekiq/sidekiq_troubleshooting.md b/doc/administration/sidekiq/sidekiq_troubleshooting.md
index 499b3ff1a422..393478f91472 100644
--- a/doc/administration/sidekiq/sidekiq_troubleshooting.md
+++ b/doc/administration/sidekiq/sidekiq_troubleshooting.md
@@ -633,4 +633,4 @@ This error means that the processes are unable to decrypt encrypted data that is
## Related topics
-- [Elasticsearch workers overload Sidekiq](../../integration/advanced_search/elasticsearch_troubleshooting.md#elasticsearch-workers-overload-sidekiq).
+- [Elasticsearch workers overload Sidekiq](../../integration/elasticsearch/troubleshooting/migrations.md#elasticsearch-workers-overload-sidekiq).
diff --git a/doc/development/advanced_search.md b/doc/development/advanced_search.md
index 1c9a095427f0..37f615ca6563 100644
--- a/doc/development/advanced_search.md
+++ b/doc/development/advanced_search.md
@@ -147,9 +147,6 @@ The `whitespace` tokenizer was selected to have more control over how tokens are
See the `code` filter for an explanation on how tokens are split.
-NOTE:
-The [Elasticsearch `code_analyzer` doesn't account for all code cases](../integration/advanced_search/elasticsearch_troubleshooting.md#elasticsearch-code_analyzer-doesnt-account-for-all-code-cases).
-
### Tokenizers
#### `sha_tokenizer`
diff --git a/doc/integration/advanced_search/elasticsearch.md b/doc/integration/advanced_search/elasticsearch.md
index e930f4f20b1b..02debb0b35f1 100644
--- a/doc/integration/advanced_search/elasticsearch.md
+++ b/doc/integration/advanced_search/elasticsearch.md
@@ -18,6 +18,10 @@ To enable advanced search, you must:
1. [Install an Elasticsearch or AWS OpenSearch cluster](#install-an-elasticsearch-or-aws-opensearch-cluster).
1. [Enable advanced search](#enable-advanced-search).
+NOTE:
+Advanced search stores all projects in the same Elasticsearch indices.
+However, private projects appear in search results only to users who have access.
+
## Elasticsearch glossary
This glossary provides definitions for terms related to Elasticsearch.
diff --git a/doc/integration/advanced_search/elasticsearch_troubleshooting.md b/doc/integration/advanced_search/elasticsearch_troubleshooting.md
index f42a48c135bc..49552b4ea9a7 100644
--- a/doc/integration/advanced_search/elasticsearch_troubleshooting.md
+++ b/doc/integration/advanced_search/elasticsearch_troubleshooting.md
@@ -1,193 +1,13 @@
---
-stage: Foundations
-group: Global Search
-info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
+redirect_to: '../elasticsearch/troubleshooting/index.md'
+remove_date: '2025-03-09'
---
-# Troubleshooting Elasticsearch
+<!-- markdownlint-disable -->
-DETAILS:
-**Tier:** Premium, Ultimate
-**Offering:** Self-managed, GitLab Dedicated
+This document was moved to [another location](../elasticsearch/troubleshooting/index.md).
-When working with Elasticsearch, you might encounter the following issues.
-
-## Issues with migrations
-
-Ensure you've read about [Elasticsearch Migrations](../advanced_search/elasticsearch.md#advanced-search-migrations).
-
-If there is a halted migration and your [`elasticsearch.log`](../../administration/logs/index.md#elasticsearchlog) file contain errors, this could potentially be a bug/issue. Escalate to GitLab support if retrying migrations does not succeed.
-
-## Error: `Can't specify parent if no parent field has been configured`
-
-If you enabled Elasticsearch before GitLab 8.12 and have not rebuilt indices, you get
-exceptions in lots of different cases:
-
-```plaintext
-Elasticsearch::Transport::Transport::Errors::BadRequest([400] {
- "error": {
- "root_cause": [{
- "type": "illegal_argument_exception",
- "reason": "Can't specify parent if no parent field has been configured"
- }],
- "type": "illegal_argument_exception",
- "reason": "Can't specify parent if no parent field has been configured"
- },
- "status": 400
-}):
-```
-
-This is because we changed the index mapping in GitLab 8.12 and the old indices should be removed and built from scratch again,
-see details in the [update guide](../../update/upgrading_from_source.md).
-
-## Error: `Elasticsearch::Transport::Transport::Errors::BadRequest`
-
-If you have this exception (just like in the case above but the actual message is different), check that you have the correct Elasticsearch version and you met the other [requirements](elasticsearch.md#system-requirements).
-There is also an easy way to check it automatically with `sudo gitlab-rake gitlab:check` command.
-
-## Error: `Elasticsearch::Transport::Transport::Errors::RequestEntityTooLarge`
-
-```plaintext
-[413] {"Message":"Request size exceeded 10485760 bytes"}
-```
-
-This exception is seen when your Elasticsearch cluster is configured to reject requests above a certain size (10 MiB in this case). This corresponds to the `http.max_content_length` setting in `elasticsearch.yml`. Increase it to a larger size and restart your Elasticsearch cluster.
-
-AWS has [network limits](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/limits.html#network-limits) on the maximum size of HTTP request payloads based on the size of the underlying instance. Set the maximum bulk request size to a value lower than 10 MiB.
-
-## Error: `Faraday::TimeoutError (execution expired)`
-
-When you use a proxy, set a custom `gitlab_rails['env']` environment variable
-named [`no_proxy`](https://docs.gitlab.com/omnibus/settings/environment-variables.html)
-with the IP address of your Elasticsearch host.
-
-## My single node Elasticsearch cluster status never goes from `yellow` to `green`
-
-**For a single node Elasticsearch cluster the functional cluster health status is yellow** (never green) because the primary shard is allocated but replicas cannot be as there is no other node to which Elasticsearch can assign a replica. This also applies if you are using the [Amazon OpenSearch](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/aes-handling-errors.html#aes-handling-errors-yellow-cluster-status) service.
-
-WARNING:
-Setting the number of replicas to `0` is discouraged (this is not allowed in the GitLab Elasticsearch Integration menu). If you are planning to add more Elasticsearch nodes (for a total of more than 1 Elasticsearch) the number of replicas needs to be set to an integer value larger than `0`. Failure to do so results in lack of redundancy (losing one node corrupts the index).
-
-If you have a **hard requirement to have a green status for your single node Elasticsearch cluster**, make sure you understand the risks outlined in the previous paragraph and then run the following query to set the number of replicas to `0`(the cluster no longer tries to create any shard replicas):
-
-```shell
-curl --request PUT localhost:9200/gitlab-production/_settings --header 'Content-Type: application/json' \
- --data '{
- "index" : {
- "number_of_replicas" : 0
- }
- }'
-```
-
-## Error: `health check timeout: no Elasticsearch node available`
-
-If you're getting a `health check timeout: no Elasticsearch node available` error in Sidekiq during the indexing process:
-
-```plaintext
-Gitlab::Elastic::Indexer::Error: time="2020-01-23T09:13:00Z" level=fatal msg="health check timeout: no Elasticsearch node available"
-```
-
-You probably have not used either `http://` or `https://` as part of your value in the **"URL"** field of the Elasticsearch Integration Menu. Make sure you are using either `http://` or `https://` in this field as the [Elasticsearch client for Go](https://github.com/olivere/elastic) that we are using [needs the prefix for the URL to be accepted as valid](https://github.com/olivere/elastic/commit/a80af35aa41856dc2c986204e2b64eab81ccac3a).
-After you have corrected the formatting of the URL, delete the index (via the [dedicated Rake task](elasticsearch.md#gitlab-advanced-search-rake-tasks)) and [reindex the content of your instance](elasticsearch.md#enable-advanced-search).
-
-## My Elasticsearch cluster has a plugin and the integration is not working
-
-Certain 3rd party plugins may introduce bugs in your cluster or for whatever
-reason may be incompatible with our integration. You should try disabling
-plugins so you can rule out the possibility that the plugin is causing the
-problem.
-
-## Elasticsearch `code_analyzer` doesn't account for all code cases
-
-The `code_analyzer` pattern and filter configuration is being evaluated for improvement. We have fixed [most edge cases](https://gitlab.com/groups/gitlab-org/-/epics/3621#note_363429094) that were not returning expected search results due to our pattern and filter configuration.
-
-Improvements to the `code_analyzer` pattern and filters are being discussed in [epic 3621](https://gitlab.com/groups/gitlab-org/-/epics/3621).
-
-## Some binary files may not be searchable by name
-
-In GitLab 13.9, a change was made where [binary filenames are being indexed](https://gitlab.com/gitlab-org/gitlab/-/issues/301083). However, without indexing all projects' data from scratch, only binary files that are added or updated after the GitLab 13.9 release are searchable.
-
-## How does advanced search handle private projects?
-
-Advanced search stores all the projects in the same Elasticsearch indices,
-however, searches only surface results that can be viewed by the user.
-Advanced search honors all permission checks in the application by
-filtering out projects that a user does not have access to at search time.
-
-## Elasticsearch workers overload Sidekiq
-
-In some cases, Elasticsearch cannot connect to GitLab anymore because:
-
-- The Elasticsearch password has been updated on one side only (`Unauthorized [401] ... unable to authenticate user` errors).
-- A firewall or network issue impairs connectivity (`Failed to open TCP connection to <ip>:9200` errors).
-
-These errors are logged in [`gitlab-rails/elasticsearch.log`](../../administration/logs/index.md#elasticsearchlog). To retrieve the errors, use [`jq`](../../administration/logs/log_parsing.md):
-
-```shell
-$ jq --raw-output 'select(.severity == "ERROR") | [.error_class, .error_message] | @tsv' \
- gitlab-rails/elasticsearch.log |
- sort | uniq -c
-```
-
-`Elastic` workers and [Sidekiq jobs](../../administration/admin_area.md#background-jobs) could also appear much more often
-because Elasticsearch frequently attempts to reindex if a previous job fails.
-You can use [`fast-stats`](https://gitlab.com/gitlab-com/support/toolbox/fast-stats#usage)
-or `jq` to count workers in the [Sidekiq logs](../../administration/logs/index.md#sidekiq-logs):
-
-```shell
-$ fast-stats --print-fields=count,score sidekiq/current
-WORKER COUNT SCORE
-ElasticIndexBulkCronWorker 234 123456
-ElasticIndexInitialBulkCronWorker 345 12345
-Some::OtherWorker 12 123
-...
-
-$ jq '.class' sidekiq/current | sort | uniq -c | sort -nr
- 234 "ElasticIndexInitialBulkCronWorker"
- 345 "ElasticIndexBulkCronWorker"
- 12 "Some::OtherWorker"
-...
-```
-
-In this case, `free -m` on the overloaded GitLab node would also show
-unexpectedly high `buff/cache` usage.
-
-## Error: `Couldn't load task status`
-
-When you reindex, you might get a `Couldn't load task status` error. A `sliceId must be greater than 0 but was [-1]` error might also appear on the Elasticsearch host. As a workaround, consider [reindexing from scratch](../elasticsearch/troubleshooting/indexing.md#last-resort-to-recreate-an-index) or upgrading to GitLab 16.3.
-
-For more information, see [issue 422938](https://gitlab.com/gitlab-org/gitlab/-/issues/422938).
-
-## Migration `BackfillProjectPermissionsInBlobs` has been halted in GitLab 15.11
-
-In GitLab 15.11, it is possible for the `BackfillProjectPermissionsInBlobs` migration to be halted with the following error message in the `elasticsearch.log`:
-
-```shell
-migration has failed with NoMethodError:undefined method `<<' for nil:NilClass, no retries left
-```
-
-If `BackfillProjectPermissionsInBlobs` is the only halted migration, you can upgrade to the latest patch version of GitLab 16.0, which includes [the fix](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/118494). Otherwise, you can ignore the error as it will not affect the current functionality of advanced search.
-
-## `ElasticIndexInitialBulkCronWorker` and `ElasticIndexBulkCronWorker` jobs stuck in deduplication
-
-In GitLab 16.5 and earlier, the `ElasticIndexInitialBulkCronWorker` and `ElasticIndexBulkCronWorker` jobs might get stuck in deduplication. This issue might prevent advanced search from properly indexing documents even after creating a new index. In GitLab 16.6, `idempotent!` was [removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/135817) for bulk cron workers that perform indexing.
-
-The Sidekiq log might have the following entries:
-
-```shell
-{"severity":"INFO","time":"2023-10-31T10:33:06.998Z","retry":0,"queue":"default","version":0,"queue_namespace":"cronjob","args":[],"class":"ElasticIndexInitialBulkCronWorker",
-...
-"idempotency_key":"resque:gitlab:duplicate:default:<value>","duplicate-of":"91e8673347d4dc84fbad5319","job_size_bytes":2,"pid":12047,"job_status":"deduplicated","message":"ElasticIndexInitialBulkCronWorker JID-5e1af9180d6e8f991fc773c6: deduplicated: until executing","deduplication.type":"until executing"}
-```
-
-To resolve this issue:
-
-1. In a [Rails console session](../../administration/operations/rails_console.md#starting-a-rails-console-session), run this command:
-
- ```shell
- idempotency_key = "<idempotency_key_from_log_entry>"
- duplicate_key = "resque:gitlab:#{idempotency_key}:cookie:v2"
- Gitlab::Redis::Queues.with { |c| c.del(duplicate_key) }
- ```
-
-1. Replace `<idempotency_key_from_log_entry>` with the actual entry in your log.
+<!-- This redirect file can be deleted after <2025-03-09>. -->
+<!-- Redirects that point to other docs in the same project expire in three months. -->
+<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. -->
+<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
diff --git a/doc/integration/elasticsearch/troubleshooting/index.md b/doc/integration/elasticsearch/troubleshooting/index.md
new file mode 100644
index 000000000000..93a1f7436aca
--- /dev/null
+++ b/doc/integration/elasticsearch/troubleshooting/index.md
@@ -0,0 +1,17 @@
+---
+stage: Foundations
+group: Global Search
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# Troubleshooting Elasticsearch
+
+DETAILS:
+**Tier:** Premium, Ultimate
+**Offering:** Self-managed, GitLab Dedicated
+
+When working with Elasticsearch, you might encounter issues with:
+
+- [Elasticsearch access](access.md)
+- [Elasticsearch indexing](indexing.md)
+- [Elasticsearch migrations](migrations.md)
diff --git a/doc/integration/elasticsearch/troubleshooting/migrations.md b/doc/integration/elasticsearch/troubleshooting/migrations.md
new file mode 100644
index 000000000000..9012448f970c
--- /dev/null
+++ b/doc/integration/elasticsearch/troubleshooting/migrations.md
@@ -0,0 +1,153 @@
+---
+stage: Foundations
+group: Global Search
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# Troubleshooting Elasticsearch migrations
+
+DETAILS:
+**Tier:** Premium, Ultimate
+**Offering:** Self-managed, GitLab Dedicated
+
+When working with Elasticsearch migrations, you might encounter the following issues.
+
+If [`elasticsearch.log`](../../../administration/logs/index.md#elasticsearchlog) contains errors
+and retrying failed migrations does not work, contact GitLab Support.
+For more information, see [advanced search migrations](../../advanced_search/elasticsearch.md#advanced-search-migrations).
+
+## Error: `Elasticsearch::Transport::Transport::Errors::BadRequest`
+
+If you have a similar exception, ensure you have the correct Elasticsearch version and you meet the [system requirements](../../advanced_search/elasticsearch.md#system-requirements).
+You can also check the version automatically by using the `sudo gitlab-rake gitlab:check` command.
+
+## Error: `Elasticsearch::Transport::Transport::Errors::RequestEntityTooLarge`
+
+```plaintext
+[413] {"Message":"Request size exceeded 10485760 bytes"}
+```
+
+This exception is seen when your Elasticsearch cluster is configured to reject requests above a certain size (10 MiB in this case). This corresponds to the `http.max_content_length` setting in `elasticsearch.yml`. Increase it to a larger size and restart your Elasticsearch cluster.
+
+AWS has [network limits](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/limits.html#network-limits) on the maximum size of HTTP request payloads based on the size of the underlying instance. Set the maximum bulk request size to a value lower than 10 MiB.
+
+## Error: `Faraday::TimeoutError (execution expired)`
+
+When you use a proxy, set a custom `gitlab_rails['env']` environment variable
+named [`no_proxy`](https://docs.gitlab.com/omnibus/settings/environment-variables.html)
+with the IP address of your Elasticsearch host.
+
+## Single-node Elasticsearch cluster status never goes from yellow to green
+
+For a single-node Elasticsearch cluster, the functional cluster health status is yellow (never green). The reason is that the primary shard is allocated, but replicas cannot be as no other node to which Elasticsearch can assign a replica exists. This also applies if you are using the [Amazon OpenSearch](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/aes-handling-errors.html#aes-handling-errors-yellow-cluster-status) service.
+
+WARNING:
+Setting the number of replicas to `0` is discouraged (this is not allowed in the GitLab Elasticsearch Integration menu). If you are planning to add more Elasticsearch nodes (for a total of more than 1 Elasticsearch) the number of replicas needs to be set to an integer value larger than `0`. Failure to do so results in lack of redundancy (losing one node corrupts the index).
+
+If you want to have a green status for your single-node Elasticsearch cluster, understand the risks and run the following query to set the number of replicas to `0`. The cluster no longer tries to create any shard replicas.
+
+```shell
+curl --request PUT localhost:9200/gitlab-production/_settings --header 'Content-Type: application/json' \
+ --data '{
+ "index" : {
+ "number_of_replicas" : 0
+ }
+ }'
+```
+
+## Error: `health check timeout: no Elasticsearch node available`
+
+If you're getting a `health check timeout: no Elasticsearch node available` error in Sidekiq during the indexing process:
+
+```plaintext
+Gitlab::Elastic::Indexer::Error: time="2020-01-23T09:13:00Z" level=fatal msg="health check timeout: no Elasticsearch node available"
+```
+
+You probably have not used either `http://` or `https://` as part of your value in the **"URL"** field of the Elasticsearch Integration Menu. Make sure you are using either `http://` or `https://` in this field as the [Elasticsearch client for Go](https://github.com/olivere/elastic) that we are using [needs the prefix for the URL to be accepted as valid](https://github.com/olivere/elastic/commit/a80af35aa41856dc2c986204e2b64eab81ccac3a).
+After you have corrected the formatting of the URL, [delete the index](../../advanced_search/elasticsearch.md#gitlab-advanced-search-rake-tasks) and [reindex the content of your instance](../../advanced_search/elasticsearch.md#enable-advanced-search).
+
+## Elasticsearch does not work with some third-party plugins
+
+Certain third-party plugins might introduce bugs in your cluster or
+be incompatible with the integration.
+
+If your Elasticsearch cluster has third-party plugins and the integration is not working,
+try to disable the plugins.
+
+## Elasticsearch workers overload Sidekiq
+
+In some cases, Elasticsearch cannot connect to GitLab anymore because:
+
+- The Elasticsearch password has been updated on one side only (`Unauthorized [401] ... unable to authenticate user` errors).
+- A firewall or network issue impairs connectivity (`Failed to open TCP connection to <ip>:9200` errors).
+
+These errors are logged in [`gitlab-rails/elasticsearch.log`](../../../administration/logs/index.md#elasticsearchlog). To retrieve the errors, use [`jq`](../../../administration/logs/log_parsing.md):
+
+```shell
+$ jq --raw-output 'select(.severity == "ERROR") | [.error_class, .error_message] | @tsv' \
+ gitlab-rails/elasticsearch.log |
+ sort | uniq -c
+```
+
+`Elastic` workers and [Sidekiq jobs](../../../administration/admin_area.md#background-jobs) could also appear much more often
+because Elasticsearch frequently attempts to reindex if a previous job fails.
+You can use [`fast-stats`](https://gitlab.com/gitlab-com/support/toolbox/fast-stats#usage)
+or `jq` to count workers in the [Sidekiq logs](../../../administration/logs/index.md#sidekiq-logs):
+
+```shell
+$ fast-stats --print-fields=count,score sidekiq/current
+WORKER COUNT SCORE
+ElasticIndexBulkCronWorker 234 123456
+ElasticIndexInitialBulkCronWorker 345 12345
+Some::OtherWorker 12 123
+...
+
+$ jq '.class' sidekiq/current | sort | uniq -c | sort -nr
+ 234 "ElasticIndexInitialBulkCronWorker"
+ 345 "ElasticIndexBulkCronWorker"
+ 12 "Some::OtherWorker"
+...
+```
+
+In this case, `free -m` on the overloaded GitLab node would also show
+unexpectedly high `buff/cache` usage.
+
+## Error: `Couldn't load task status`
+
+When you reindex, you might get a `Couldn't load task status` error. A `sliceId must be greater than 0 but was [-1]` error might also appear on the Elasticsearch host. As a workaround, consider [reindexing from scratch](indexing.md#last-resort-to-recreate-an-index) or upgrading to GitLab 16.3.
+
+For more information, see [issue 422938](https://gitlab.com/gitlab-org/gitlab/-/issues/422938).
+
+## Error: `migration has failed with NoMethodError:undefined method`
+
+In GitLab 15.11, the `BackfillProjectPermissionsInBlobs` migration might fail with the following error message in `elasticsearch.log`:
+
+```shell
+migration has failed with NoMethodError:undefined method `<<' for nil:NilClass, no retries left
+```
+
+If `BackfillProjectPermissionsInBlobs` is the only failed migration, you can upgrade to the latest patch version of GitLab 16.0, which includes [the fix](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/118494). Otherwise, you can ignore the error as it does not affect the functionality of advanced search.
+
+## `ElasticIndexInitialBulkCronWorker` and `ElasticIndexBulkCronWorker` jobs stuck in deduplication
+
+In GitLab 16.5 and earlier, the `ElasticIndexInitialBulkCronWorker` and `ElasticIndexBulkCronWorker` jobs might get stuck in deduplication. This issue might prevent advanced search from properly indexing documents even after creating a new index. In GitLab 16.6, `idempotent!` was [removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/135817) for bulk cron workers that perform indexing.
+
+The Sidekiq log might have the following entries:
+
+```shell
+{"severity":"INFO","time":"2023-10-31T10:33:06.998Z","retry":0,"queue":"default","version":0,"queue_namespace":"cronjob","args":[],"class":"ElasticIndexInitialBulkCronWorker",
+...
+"idempotency_key":"resque:gitlab:duplicate:default:<value>","duplicate-of":"91e8673347d4dc84fbad5319","job_size_bytes":2,"pid":12047,"job_status":"deduplicated","message":"ElasticIndexInitialBulkCronWorker JID-5e1af9180d6e8f991fc773c6: deduplicated: until executing","deduplication.type":"until executing"}
+```
+
+To resolve this issue:
+
+1. In a [Rails console session](../../../administration/operations/rails_console.md#starting-a-rails-console-session), run this command:
+
+ ```shell
+ idempotency_key = "<idempotency_key_from_log_entry>"
+ duplicate_key = "resque:gitlab:#{idempotency_key}:cookie:v2"
+ Gitlab::Redis::Queues.with { |c| c.del(duplicate_key) }
+ ```
+
+1. Replace `<idempotency_key_from_log_entry>` with the actual entry in your log.
--
GitLab