-
由 Achilleas Pipinellis 创作于
Update whitespace usage across admin, install, update docs See merge request https://gitlab.com/gitlab-org/gitlab/-/merge_requests/140967 Merged-by:
Achilleas Pipinellis <axil@gitlab.com> Approved-by:
Achilleas Pipinellis <axil@gitlab.com> Co-authored-by:
Marcel Amirault <mamirault@gitlab.com>
由 Achilleas Pipinellis 创作于Update whitespace usage across admin, install, update docs See merge request https://gitlab.com/gitlab-org/gitlab/-/merge_requests/140967 Merged-by:
Achilleas Pipinellis <axil@gitlab.com> Approved-by:
Achilleas Pipinellis <axil@gitlab.com> Co-authored-by:
Marcel Amirault <mamirault@gitlab.com>
stage: Systems
group: Gitaly
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
Configure Gitaly (FREE SELF)
Configure Gitaly in one of two ways:
::Tabs
:::TabTitle Linux package (Omnibus)
- Edit
/etc/gitlab/gitlab.rb
and add or change the Gitaly settings. - Save the file and reconfigure GitLab.
:::TabTitle Self-compiled (source)
- Edit
/home/git/gitaly/config.toml
and add or change the Gitaly settings. - Save the file and restart GitLab.
::EndTabs
The following configuration options are also available:
- Enabling TLS support.
- Limiting RPC concurrency.
- Limiting pack-objects concurrency.
About the Gitaly token
The token referred to throughout the Gitaly documentation is just an arbitrary password selected by the administrator. It is unrelated to tokens created for the GitLab API or other similar web API tokens.
Run Gitaly on its own server
By default, Gitaly is run on the same server as Gitaly clients and is configured as above. Single-server installations are best served by this default configuration used by:
However, Gitaly can be deployed to its own server, which can benefit GitLab installations that span multiple machines.
NOTE: When configured to run on their own servers, Gitaly servers must be upgraded before Gitaly clients in your cluster.
NOTE: Disk requirements apply to Gitaly nodes.
The process for setting up Gitaly on its own server is:
- Install Gitaly.
- Configure authentication.
- Configure Gitaly servers.
- Configure Gitaly clients.
- Disable Gitaly where not required (optional).
Network architecture
The following list depicts the network architecture of Gitaly:
- GitLab Rails shards repositories into repository storages.
-
/config/gitlab.yml
contains a map from storage names to(Gitaly address, Gitaly token)
pairs. - The
storage name
->(Gitaly address, Gitaly token)
map in/config/gitlab.yml
is the single source of truth for the Gitaly network topology. - A
(Gitaly address, Gitaly token)
corresponds to a Gitaly server. - A Gitaly server hosts one or more storages.
- A Gitaly client can use one or more Gitaly servers.
- Gitaly addresses must be specified in such a way that they resolve correctly for all Gitaly clients.
- Gitaly clients are:
- Puma.
- Sidekiq.
- GitLab Workhorse.
- GitLab Shell.
- Elasticsearch indexer.
- Gitaly itself.
- A Gitaly server must be able to make RPC calls to itself by using its own
(Gitaly address, Gitaly token)
pair as specified in/config/gitlab.yml
. - Authentication is done through a static token which is shared among the Gitaly and GitLab Rails nodes.
The following digraph illustrates communication between Gitaly servers and GitLab Rails showing the default ports for HTTP and HTTPs communication.
WARNING: Gitaly servers must not be exposed to the public internet as Gitaly network traffic is unencrypted by default. The use of firewall is highly recommended to restrict access to the Gitaly server. Another option is to use TLS.
In the following sections, we describe how to configure two Gitaly servers with secret token
abc123secret
:
-
gitaly1.internal
. -
gitaly2.internal
.
We assume your GitLab installation has three repository storages:
-
default
. -
storage1
. -
storage2
.
You can use as few as one server with one repository storage if desired.
Install Gitaly
Install Gitaly on each Gitaly server using either:
- A Linux package installation. Download and install the Linux package you want
but do not provide the
EXTERNAL_URL=
value. - A self-compiled installation. Follow the steps at Install Gitaly.
Configure Gitaly servers
To configure Gitaly servers, you must:
- Configure authentication.
- Configure storage paths.
- Enable the network listener.
The git
user must be able to read, write, and set permissions on the configured storage path.
To avoid downtime while rotating the Gitaly token, you can temporarily disable authentication using the gitaly['auth_transitioning']
setting. For more information, see the documentation on
enabling "auth transitioning mode".
Configure authentication
Gitaly and GitLab use two shared secrets for authentication:
- Gitaly token: used to authenticate gRPC requests to Gitaly
- GitLab Shell token: used for authentication callbacks from GitLab Shell to the GitLab internal API
Configure authentication in one of two ways:
::Tabs
:::TabTitle Linux package (Omnibus)
To configure the Gitaly token, edit /etc/gitlab/gitlab.rb
:
gitaly['configuration'] = {
# ...
auth: {
# ...
token: 'abc123secret',
},
}
Configure the GitLab Shell token in one of two ways.
Method 1 (recommended):
Copy /etc/gitlab/gitlab-secrets.json
from the Gitaly client to same path on the Gitaly servers
(and any other Gitaly clients).
Method 2:
Edit /etc/gitlab/gitlab.rb
:
gitlab_shell['secret_token'] = 'shellsecret'
:::TabTitle Self-compiled (source)
-
Copy
/home/git/gitlab/.gitlab_shell_secret
from the Gitaly client to the same path on the Gitaly servers (and any other Gitaly clients). -
On the Gitaly clients, edit
/home/git/gitlab/config/gitlab.yml
:gitlab: gitaly: token: 'abc123secret'
-
Save the file and restart GitLab.
-
On the Gitaly servers, edit
/home/git/gitaly/config.toml
:[auth] token = 'abc123secret'
-
Save the file and restart GitLab.
::EndTabs
Configure Gitaly server
Configure Gitaly server in one of two ways:
::Tabs
:::TabTitle Linux package (Omnibus)
-
Edit
/etc/gitlab/gitlab.rb
:# Avoid running unnecessary services on the Gitaly server postgresql['enable'] = false redis['enable'] = false nginx['enable'] = false puma['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false grafana['enable'] = false gitlab_exporter['enable'] = false gitlab_kas['enable'] = false # If you run a separate monitoring node you can disable these services prometheus['enable'] = false alertmanager['enable'] = false # If you don't run a separate monitoring node you can # enable Prometheus access & disable these extra services. # This makes Prometheus listen on all interfaces. You must use firewalls to restrict access to this address/port. # prometheus['listen_address'] = '0.0.0.0:9090' # prometheus['monitor_kubernetes'] = false # If you don't want to run monitoring services uncomment the following (not recommended) # node_exporter['enable'] = false # Prevent database connections during 'gitlab-ctl reconfigure' gitlab_rails['auto_migrate'] = false # Configure the gitlab-shell API callback URL. Without this, `git push` will # fail. This can be your 'front door' GitLab URL or an internal load # balancer. # Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from Gitaly client to Gitaly server. gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' gitaly['configuration'] = { # ... # # Make Gitaly accept connections on all network interfaces. You must use # firewalls to restrict access to this address/port. # Comment out following line if you only want to support TLS connections listen_addr: '0.0.0.0:8075', auth: { # ... # # Authentication token to ensure only authorized servers can communicate with # Gitaly server token: 'AUTH_TOKEN', }, }
-
Append the following to
/etc/gitlab/gitlab.rb
for each respective Gitaly server:On
gitaly1.internal
:gitaly['configuration'] = { # ... storage: [ { name: 'default', path: '/var/opt/gitlab/git-data/repositories', }, { name: 'storage1', path: '/mnt/gitlab/git-data/repositories', }, ], }
On
gitaly2.internal
:gitaly['configuration'] = { # ... storage: [ { name: 'storage2', path: '/srv/gitlab/git-data/repositories', }, ], }
-
Save the file and reconfigure GitLab.
-
Confirm that Gitaly can perform callbacks to the GitLab internal API:
- For GitLab 15.3 and later, run
sudo /opt/gitlab/embedded/bin/gitaly check /var/opt/gitlab/gitaly/config.toml
. - For GitLab 15.2 and earlier, run
sudo /opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml
.
- For GitLab 15.3 and later, run
:::TabTitle Self-compiled (source)
-
Edit
/home/git/gitaly/config.toml
:listen_addr = '0.0.0.0:8075' runtime_dir = '/var/opt/gitlab/gitaly' [logging] format = 'json' level = 'info' dir = '/var/log/gitaly'
For GitLab 14.9 and earlier, set
internal_socket_dir = '/var/opt/gitlab/gitaly'
instead ofruntime_dir
. -
Append the following to
/home/git/gitaly/config.toml
for each respective Gitaly server:On
gitaly1.internal
:[[storage]] name = 'default' path = '/var/opt/gitlab/git-data/repositories' [[storage]] name = 'storage1' path = '/mnt/gitlab/git-data/repositories'
On
gitaly2.internal
:[[storage]] name = 'storage2' path = '/srv/gitlab/git-data/repositories'
-
Edit
/home/git/gitlab-shell/config.yml
:gitlab_url: https://gitlab.example.com
-
Save the files and restart GitLab.
-
Confirm that Gitaly can perform callbacks to the GitLab internal API:
- For GitLab 15.3 and later, run
sudo /opt/gitlab/embedded/bin/gitaly check /var/opt/gitlab/gitaly/config.toml
. - For GitLab 15.2 and earlier, run
sudo /opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml
.
- For GitLab 15.3 and later, run
::EndTabs
WARNING:
If directly copying repository data from a GitLab server to Gitaly, ensure that the metadata file,
default path /var/opt/gitlab/git-data/repositories/.gitaly-metadata
, is not included in the transfer.
Copying this file causes GitLab to use the direct disk access to repositories hosted on the Gitaly server,
leading to Error creating pipeline
and Commit not found
errors, or stale data.
Configure Gitaly clients
As the final step, you must update Gitaly clients to switch from using local Gitaly service to use the Gitaly servers you just configured.
NOTE:
GitLab requires a default
repository storage to be configured.
Read more about this limitation.
This can be risky because anything that prevents your Gitaly clients from reaching the Gitaly servers causes all Gitaly requests to fail. For example, any sort of network, firewall, or name resolution problems.
Gitaly makes the following assumptions:
- Your
gitaly1.internal
Gitaly server can be reached atgitaly1.internal:8075
from your Gitaly clients, and that Gitaly server can read, write, and set permissions on/var/opt/gitlab/git-data
and/mnt/gitlab/git-data
. - Your
gitaly2.internal
Gitaly server can be reached atgitaly2.internal:8075
from your Gitaly clients, and that Gitaly server can read, write, and set permissions on/srv/gitlab/git-data
. - Your
gitaly1.internal
andgitaly2.internal
Gitaly servers can reach each other.
You can't define Gitaly servers with some as a local Gitaly server
(without gitaly_address
) and some as remote
server (with gitaly_address
) unless you use
mixed configuration.
Configure Gitaly clients in one of two ways:
::Tabs
:::TabTitle Linux package (Omnibus)
-
Edit
/etc/gitlab/gitlab.rb
:# Use the same token value configured on all Gitaly servers gitlab_rails['gitaly_token'] = '<AUTH_TOKEN>' git_data_dirs({ 'default' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, 'storage1' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, 'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' }, })
Alternatively, if each Gitaly server is configured to use a different authentication token:
git_data_dirs({ 'default' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075', 'gitaly_token' => '<AUTH_TOKEN_1>' }, 'storage1' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075', 'gitaly_token' => '<AUTH_TOKEN_1>' }, 'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075', 'gitaly_token' => '<AUTH_TOKEN_2>' }, })
-
Save the file and reconfigure GitLab.
-
Run
sudo gitlab-rake gitlab:gitaly:check
on the Gitaly client (for example, the Rails application) to confirm it can connect to Gitaly servers. -
Tail the logs to see the requests:
sudo gitlab-ctl tail gitaly
:::TabTitle Self-compiled (source)
-
Edit
/home/git/gitlab/config/gitlab.yml
:gitlab: repositories: storages: default: gitaly_address: tcp://gitaly1.internal:8075 gitaly_token: AUTH_TOKEN_1 storage1: gitaly_address: tcp://gitaly1.internal:8075 gitaly_token: AUTH_TOKEN_1 storage2: gitaly_address: tcp://gitaly2.internal:8075 gitaly_token: AUTH_TOKEN_2
-
Save the file and restart GitLab.
-
Run
sudo -u git -H bundle exec rake gitlab:gitaly:check RAILS_ENV=production
to confirm the Gitaly client can connect to Gitaly servers. -
Tail the logs to see the requests:
tail -f /home/git/gitlab/log/gitaly.log
::EndTabs
When you tail the Gitaly logs on your Gitaly server, you should see requests coming in. One sure way to trigger a Gitaly request is to clone a repository from GitLab over HTTP or HTTPS.
WARNING: If you have server hooks configured, either per repository or globally, you must move these to the Gitaly servers. If you have multiple Gitaly servers, copy your server hooks to all Gitaly servers.
Mixed configuration
GitLab can reside on the same server as one of many Gitaly servers, but doesn't support configuration that mixes local and remote configuration. The following setup is incorrect, because:
- All addresses must be reachable from the other Gitaly servers.
-
storage1
is assigned a Unix socket forgitaly_address
which is invalid for some of the Gitaly servers.
git_data_dirs({
'default' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' },
'storage1' => { 'path' => '/mnt/gitlab/git-data' },
'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' },
})
To combine local and remote Gitaly servers, use an external address for the local Gitaly server. For example:
git_data_dirs({
'default' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' },
# Address of the GitLab server that also has Gitaly running on it
'storage1' => { 'gitaly_address' => 'tcp://gitlab.internal:8075' },
'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' },
})
gitaly['configuration'] = {
# ...
#
# Make Gitaly accept connections on all network interfaces
listen_addr: '0.0.0.0:8075',
# Or for TLS
tls_listen_addr: '0.0.0.0:9999',
tls: {
certificate_path: '/etc/gitlab/ssl/cert.pem',
key_path: '/etc/gitlab/ssl/key.pem',
},
storage: [
{
name: 'storage1',
path: '/mnt/gitlab/git-data/repositories',
},
],
}
path
can be included only for storage shards on the local Gitaly server.
If it's excluded, default Git storage directory is used for that storage shard.