diff --git a/doc/administration/geo/index.md b/doc/administration/geo/index.md index dec6939176c1053acfa2320e7f9cabfa048a6c30..daafda890ed4bcdae126049a3e50fc552cdc399b 100644 --- a/doc/administration/geo/index.md +++ b/doc/administration/geo/index.md @@ -124,7 +124,9 @@ The following are required to run Geo: - Git 2.9 or later - Git-lfs 2.4.2 or later on the user side when using LFS - All sites must run [the same GitLab and PostgreSQL versions](setup/database.md#postgresql-replication). -- If using different operating system versions between Geo sites, [check OS locale data compatibility](replication/troubleshooting.md#check-os-locale-data-compatibility) across Geo sites. + - If using different operating system versions between Geo sites, + [check OS locale data compatibility](replication/troubleshooting.md#check-os-locale-data-compatibility) + across Geo sites to avoid silent corruption of database indexes. Additionally, check the GitLab [minimum requirements](../../install/requirements.md), and use the latest version of GitLab for a better experience. diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index 1dcced781ceb0b2efd8bd47b16183580e36f32a3..f6bb7491ac01fd173cec04a9183f47d6d4081d00 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -1535,9 +1535,12 @@ If the above steps are **not successful**, proceed through the next steps: ## Check OS locale data compatibility -If different operating systems or different operating system versions are deployed across Geo sites, we recommend that you perform a locale data compatibility check setting up Geo. +If different operating systems or different operating system versions are deployed across Geo sites, you should perform a locale data compatibility check before setting up Geo. -Geo uses PostgreSQL and Streaming Replication to replicate data across Geo sites. PostgreSQL uses locale data provided by the operating system’s C library for sorting text. If the locale data in the C library is incompatible across Geo sites, erroneous query results that lead to [incorrect behavior on secondary sites](https://gitlab.com/gitlab-org/gitlab/-/issues/360723). See [here](https://wiki.postgresql.org/wiki/Locale_data_changes) for more details. +Geo uses PostgreSQL and Streaming Replication to replicate data across Geo sites. PostgreSQL uses locale data provided by the operating system's C library for sorting text. If the locale data in the C library is incompatible across Geo sites, erroneous query results that lead to [incorrect behavior on secondary sites](https://gitlab.com/gitlab-org/gitlab/-/issues/360723). + +For example, Ubuntu 18.04 (and earlier) and RHEL/Centos7 (and earlier) are incompatible with their later releases. +See the [PostgreSQL wiki for more details](https://wiki.postgresql.org/wiki/Locale_data_changes). On all hosts running PostgreSQL, across all Geo sites, run the following shell command: @@ -1561,4 +1564,8 @@ or the reverse order: If the output is identical on all hosts, then they running compatible versions of locale data. -If the output differs on some hosts, then PostgreSQL replication will not work properly. We advise that you select operating system versions that are compatible. +If the output differs on some hosts, PostgreSQL replication does not work properly: indexes are corrupted on the database replicas. You should select operating system versions that are compatible. + +A full index rebuild is required if the on-disk data is transferred 'at rest' to an operating system with an incompatible locale, or through replication. + +This check is also required when using a mixture of GitLab deployments. The locale might be different between an Linux package install, a GitLab Docker container, a Helm chart deployment, or external database services. diff --git a/doc/install/requirements.md b/doc/install/requirements.md index 7fcb371c178a7027986eafc9683e39ec04524619..540f35c5b032f8f0197a0c243b479259bd0b83fc 100644 --- a/doc/install/requirements.md +++ b/doc/install/requirements.md @@ -120,6 +120,18 @@ test based on those. We try to be compatible with most external (not managed by Omnibus GitLab) databases (for example, [AWS Relational Database Service (RDS)](https://aws.amazon.com/rds/)), but we can't guarantee compatibility. +#### Operating system locale compatibility and silent index corruption + +Changes to locale data in `glibc` means that PostgreSQL database files are not fully compatible +between different OS releases. + +To avoid index corruption, [check for locale compatibility](../administration/geo/replication/troubleshooting.md#check-os-locale-data-compatibility) +when: + +- Moving binary PostgreSQL data between servers. +- Upgrading your Linux distribution. +- Updating or changing third party container images. + #### Gitaly Cluster database requirements [Read more in the Gitaly Cluster documentation](../administration/gitaly/praefect.md).