From 07215fe6526ce7c67faedacec4aa7e687ab65c56 Mon Sep 17 00:00:00 2001
From: Paul Okstad <pokstad@gitlab.com>
Date: Tue, 24 Mar 2020 08:09:48 +0000
Subject: [PATCH] Update Gitaly to 12.9.0-rc5

---
 GITALY_SERVER_VERSION                         |  2 +-
 ...okstad1-praefect-docs-reconcile-subcmd.yml |  5 ++++
 doc/administration/gitaly/praefect.md         | 23 +++++++++++++++++++
 3 files changed, 29 insertions(+), 1 deletion(-)
 create mode 100644 changelogs/unreleased/pokstad1-praefect-docs-reconcile-subcmd.yml

diff --git a/GITALY_SERVER_VERSION b/GITALY_SERVER_VERSION
index c75f0df204f2b..e1e97da12924d 100644
--- a/GITALY_SERVER_VERSION
+++ b/GITALY_SERVER_VERSION
@@ -1 +1 @@
-12.9.0-rc1
+12.9.0-rc5
diff --git a/changelogs/unreleased/pokstad1-praefect-docs-reconcile-subcmd.yml b/changelogs/unreleased/pokstad1-praefect-docs-reconcile-subcmd.yml
new file mode 100644
index 0000000000000..2c20f425d8092
--- /dev/null
+++ b/changelogs/unreleased/pokstad1-praefect-docs-reconcile-subcmd.yml
@@ -0,0 +1,5 @@
+---
+title: Update Gitaly to 12.9.0-rc5
+merge_request: 27631
+author:
+type: added
diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md
index 8c76677486849..5276e3a781608 100644
--- a/doc/administration/gitaly/praefect.md
+++ b/doc/administration/gitaly/praefect.md
@@ -682,6 +682,29 @@ for example behind a load balancer, `failover_enabled` should be disabled. The r
 is no coordination that currently happens across different Praefect instances, so there could be a situation where
 two Praefect instances think two different Gitaly nodes are the primary.
 
+## Backend Node Recovery
+
+When a Praefect backend node fails and is no longer able to
+replicate changes, the backend node will start to drift from the primary. If
+that node eventually recovers, it will need to be reconciled with the current
+primary. The primary node is considered the single source of truth for the
+state of a shard. The Praefect `reconcile` subcommand allows for the manual
+reconciliation between a backend node and the current primary.
+
+Run the following command on the Praefect server after all placeholders
+(`<virtual-storage>` and `<target-storage>`) have been replaced:
+
+```shell
+sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml reconcile -virtual <virtual-storage> -target <target-storage>
+```
+
+- Replace the placeholder `<virtual-storage>` with the virtual storage containing the backend node storage to be checked.
+- Replace the placeholder `<target-storage>` with the backend storage name.
+
+The command will return a list of repositories that were found to be
+inconsistent against the current primary. Each of these inconsistencies will
+also be logged with an accompanying replication job ID.
+
 ## Grafana
 
 Grafana is included with GitLab, and can be used to monitor your Praefect
-- 
GitLab