diff --git a/doc/administration/backup_restore/backup_gitlab.md b/doc/administration/backup_restore/backup_gitlab.md index 9133875ac345a41f17bf3b1d862ee901fe0b7769..10369d88927fca3fa20b23ad945cf078bdf53d25 100644 --- a/doc/administration/backup_restore/backup_gitlab.md +++ b/doc/administration/backup_restore/backup_gitlab.md @@ -55,8 +55,6 @@ on a machine: - With multiple disks mounted as a single mount-point (like with a RAID array). - Using LVM. -Gitaly may work with NFS or a mounted Storage Appliance, but it is [not officially supported](../nfs.md#gitaly-with-nfs-not-supported) because Git requires low latency. - Each project can have up to 3 different repositories: - A project repository, where the source code is stored. diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index 5e074e95bb01e7e5f0a59f16e676c5ba6bef1821..b25700ccd29c8b38ad9c0070cf7aae139415cd68 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -80,10 +80,8 @@ on a machine: - With multiple disks mounted as a single mount-point (like with a RAID array). - Using LVM. -GitLab does not require a special file system and can work with: - -- NFS. -- A mounted Storage Appliance (there may be performance limitations when using a remote file system). +GitLab does not require a special file system and can work with a mounted Storage Appliance. However, there can be +performance limitations and consistency issues when using a remote file system. Geo triggers garbage collection in Gitaly to [deduplicate forked repositories](../../../development/git_object_deduplication.md#git-object-deduplication-and-gitlab-geo) on Geo secondary sites. diff --git a/doc/administration/gitaly/configure_gitaly.md b/doc/administration/gitaly/configure_gitaly.md index 5c6fc3700529e0d169bab4e0a0f748803071148c..f346cd0abf6a7349c17ad0405e6b5896eca1d1aa 100644 --- a/doc/administration/gitaly/configure_gitaly.md +++ b/doc/administration/gitaly/configure_gitaly.md @@ -377,9 +377,6 @@ This can be risky because anything that prevents your Gitaly clients from reachi servers causes all Gitaly requests to fail. For example, any sort of network, firewall, or name resolution problems. -Additionally, you must [disable Rugged](../nfs.md#improving-nfs-performance-with-gitlab) -if previously enabled manually. - Gitaly makes the following assumptions: - Your `gitaly1.internal` Gitaly server can be reached at `gitaly1.internal:8075` from your Gitaly diff --git a/doc/administration/gitaly/gitaly_geo_capabilities.md b/doc/administration/gitaly/gitaly_geo_capabilities.md index e4147eec1621135732032e07907fdfe0c2eca322..a98477318f2404fc6e5ebc0d056e82b3dc8868fd 100644 --- a/doc/administration/gitaly/gitaly_geo_capabilities.md +++ b/doc/administration/gitaly/gitaly_geo_capabilities.md @@ -17,7 +17,6 @@ The following tables are intended to guide you to choose the right combination o |------------|--------------|----------------|-----------------|-------------|-----------------| |Gitaly Cluster | Very high - tolerant of node failures | RTO for a single node of 10 s with no manual intervention | Data is stored on multiple nodes | Good - While writes may take slightly longer due to voting, read distribution improves read speeds | **Trade-off** - Slight decrease in write speed for redundant, strongly-consistent storage solution. **Risks** - [Does not support snapshot backups](../gitaly/index.md#snapshot-backup-and-recovery-limitations), GitLab backup task can be slow for large data sets | |Gitaly Shards | Single storage location is a single point of failure | Would need to restore only shards which failed | Single point of failure | Good - can allocate repositories to shards to spread load | **Trade-off** - Need to manually configure repositories into different shards to balance loads / storage space **Risks** - Single point of failure relies on recovery process when single-node failure occurs | -|Gitaly + NFS | Single storage location is a single point of failure | Single node failure requires restoration from backup | Single point of failure | Average - NFS is not ideally suited to large quantities of small reads / writes which can have a detrimental impact on performance | **Trade-off** - Familiar administration though NFS is not ideally suited to Git demands **Risks** - Many instances of NFS compatibility issues which provide very poor customer experiences | ## Geo capabilities diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index b7a239f2e271ef60f1b959a814723a57fdfbeea1..0c7a7a70127994a3fd9596806626588430beb032 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -52,13 +52,10 @@ Before deploying Gitaly Cluster, review: - [Configuration guidance](configure_gitaly.md) and [Repository storage options](../repository_storage_paths.md) to make sure that Gitaly Cluster is the best setup for you. -If you have: +If you have not yet migrated to Gitaly Cluster, you have two options: -- Not yet migrated to Gitaly Cluster and want to continue using NFS, remain on the service you are using. However, NFS - is [no longer supported](../../update/deprecations.md#nfs-for-git-repository-storage). -- Not yet migrated to Gitaly Cluster but want to migrate away from NFS, you have two options: - - A sharded Gitaly instance. - - Gitaly Cluster. +- A sharded Gitaly instance. +- Gitaly Cluster. Contact your [Customer Success Manager](https://about.gitlab.com/job-families/sales/customer-success-management/) or customer support if you have any questions. diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index 5a44eb476c7e1cbba903e65ec2957bd25ced895b..eb42fddb00c7ee2e078662ec7416a5ca9e42cc51 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -1487,7 +1487,7 @@ repository storage redundancy. For a replication factor: -- Of `1`: NFS, Gitaly, and Gitaly Cluster have roughly the same storage requirements. +- Of `1`: Gitaly and Gitaly Cluster have roughly the same storage requirements. - More than `1`: The amount of required storage is `used space * replication factor`. `used space` should include any planned future growth. diff --git a/doc/administration/monitoring/performance/performance_bar.md b/doc/administration/monitoring/performance/performance_bar.md index 3fdd4c24177dec00f3f0bf2d4ae4eeba86a18e6f..04c7b74f1bdcd1a11ac51b2274b6a755b855392e 100644 --- a/doc/administration/monitoring/performance/performance_bar.md +++ b/doc/administration/monitoring/performance/performance_bar.md @@ -38,8 +38,7 @@ From left to right, the performance bar displays: [Gitaly](../../gitaly/index.md) calls. Select to display a modal window with more details. - **Rugged calls**: the time taken (in milliseconds) and the total number of - [Rugged](../../nfs.md#improving-nfs-performance-with-gitlab) calls. - Select to display a modal window with more details. + Rugged calls. Select to display a modal window with more details. - **Redis calls**: the time taken (in milliseconds) and the total number of Redis calls. Select to display a modal window with more details. - **Elasticsearch calls**: the time taken (in milliseconds) and the total number of diff --git a/doc/administration/nfs.md b/doc/administration/nfs.md index 9ea905b921aad1ef896d0a5305d92c1bf5b72d4f..0273c4b03b12f4d17a910bcf42befaa1bc464ad0 100644 --- a/doc/administration/nfs.md +++ b/doc/administration/nfs.md @@ -15,35 +15,10 @@ is recommended over NFS where possible, due to better performance. When eliminating the usage of NFS, there are [additional steps you need to take](object_storage.md#alternatives-to-file-system-storage) in addition to moving to Object Storage. -File system performance can impact overall GitLab performance, especially for -actions that read or write to Git repositories. For steps you can use to test -file system performance, see -[File System Performance Benchmarking](operations/filesystem_benchmarking.md). - -## Gitaly with NFS not supported - -Technical and engineering support for using NFS to store Git repository data is officially at end-of-life. No product -changes or troubleshooting is provided through engineering, security or paid support channels. - -If an issue is reproducible, or if it happens intermittently but regularly, GitLab Support can investigate providing the -issue can be reproduced without NFS. To reproduce without NFS, migrate the affected repositories to a different Gitaly -shard. For example, a Gitaly Cluster or a standalone Gitaly VM, backed with block storage. - -## Known kernel version incompatibilities - -RedHat Enterprise Linux (RHEL) and CentOS v7.7 and v7.8 ship with kernel -version `3.10.0-1127`, which [contains a bug](https://bugzilla.redhat.com/show_bug.cgi?id=1783554) that causes -[uploads to fail to copy over NFS](https://gitlab.com/gitlab-org/gitlab/-/issues/218999). The -following GitLab versions include a fix to work properly with that -kernel version: - -- [12.10.12](https://about.gitlab.com/releases/2020/06/25/gitlab-12-10-12-released/) -- [13.0.7](https://about.gitlab.com/releases/2020/06/25/gitlab-13-0-7-released/) -- [13.1.1](https://about.gitlab.com/releases/2020/06/24/gitlab-13-1-1-released/) -- 13.2 and up +NFS cannot be used for repository storage. -If you are using that kernel version, be sure to upgrade GitLab to avoid -errors. +For steps you can use to test file system performance, see +[File System Performance Benchmarking](operations/filesystem_benchmarking.md). ## Fast lookup of authorized SSH keys @@ -59,26 +34,6 @@ is moved to NFS. We are investigating the use of [fast lookup as the default](https://gitlab.com/groups/gitlab-org/-/epics/3104). -## Improving NFS performance with GitLab - -NFS performance with GitLab can in some cases be improved with -[direct Git access](gitaly/index.md#direct-access-to-git-in-gitlab) using [Rugged](https://github.com/libgit2/rugged). - -Depending on the GitLab version, GitLab [automatically detects](gitaly/index.md#automatic-detection) if Rugged can and should -be used per storage. - -If the Rugged feature flag is explicitly set to either `true` or `false`, GitLab uses the value explicitly set. If you -previously enabled Rugged using the feature flag and you want to use automatic detection instead, you must unset -the feature flag: - -```shell -sudo gitlab-rake gitlab:features:unset_rugged -``` - -From GitLab 12.7, Rugged is only automatically enabled for use with Puma if the -[Puma thread count is set to `1`](../install/requirements.md#puma-settings). To use Rugged with a Puma thread count of -more than `1`, enable Rugged using the [feature flag](../development/gitaly.md#legacy-rugged-code). - ## NFS server Installing the `nfs-kernel-server` package allows you to share directories with @@ -361,33 +316,6 @@ sudo ufw allow from <client_ip_address> to any port nfs ## Known issues -### Upgrade to Gitaly Cluster or disable caching if experiencing data loss - -WARNING: -NFS for Git repositories -[has been removed](../update/deprecations.md#nfs-for-git-repository-storage). - -Customers and users have reported data loss on high-traffic repositories when using NFS for Git repositories. -For example, we have seen: - -- [Inconsistent updates after a push](https://gitlab.com/gitlab-org/gitaly/-/issues/2589). -- `git ls-remote` [returning the wrong (or no branches)](https://gitlab.com/gitlab-org/gitaly/-/issues/3083) -causing Jenkins to intermittently re-run all pipelines for a repository. - -The problem may be partially mitigated by adjusting caching using the following NFS client mount options: - -| Setting | Description | -| ------- | ----------- | -| `lookupcache=positive` | Tells the NFS client to honor `positive` cache results but invalidates any `negative` cache results. Negative cache results cause problems with Git. Specifically, a `git push` can fail to register uniformly across all NFS clients. The negative cache causes the clients to 'remember' that the files did not exist previously. -| `actimeo=0` | Sets the time to zero that the NFS client caches files and directories before requesting fresh information from a server. | -| `noac` | Tells the NFS client not to cache file attributes and forces application writes to become synchronous so that local changes to a file become visible on the server immediately. | - -WARNING: -The `actimeo=0` and `noac` options both result in a significant reduction in performance, possibly leading to timeouts. -You may be able to avoid timeouts and data loss using `actimeo=0` and `lookupcache=positive` _without_ `noac`, however -we expect the performance reduction is still significant. Upgrade to -[Gitaly Cluster](gitaly/praefect.md) as soon as possible. - ### Avoid using cloud-based file systems GitLab strongly recommends against using cloud-based file systems such as: @@ -447,10 +375,3 @@ On Ubuntu 16.04, use: ```shell sudo perf trace --no-syscalls --event 'nfs4:*' -p $(pgrep -fd ',' puma) ``` - -### Warnings `garbage found: .../repositories/@hashed/...git/objects/pack/.nfs...` in Gitaly logs - -If you find any warnings like `garbage found: .../repositories/@hashed/...git/objects/pack/.nfs...` in [Gitaly logs](logs/index.md#gitaly-logs), -this problem occurs if `lookupcache=positive` is not set, which we recommend as an -[NFS mount option](#mount-options). -See [Gitaly issue #3175](https://gitlab.com/gitlab-org/gitaly/-/issues/3175) for more details. diff --git a/doc/administration/sidekiq/index.md b/doc/administration/sidekiq/index.md index bb517e21fd0dd9daf59db223aa6557b1d0dd2ebe..c3e1182cb05cb4fc8d2e7d87f74ed9456eed37b9 100644 --- a/doc/administration/sidekiq/index.md +++ b/doc/administration/sidekiq/index.md @@ -383,7 +383,7 @@ blocking all jobs on that worker from proceeding. If Rugged calls performed by S background task processing. By default, Rugged is used when Git repository data is stored on local storage or on an NFS mount. -[Using Rugged is recommended when using NFS](../nfs.md#improving-nfs-performance-with-gitlab), but if +Using Rugged is recommended when using NFS, but if you are using local storage, disabling Rugged can improve Sidekiq performance: ```shell