Consider bcache/zfs on gitaly node
Bcache
Bcache (block cache) allows one to use an SSD as a read/write cache (in writeback mode) or read cache (writethrough or writearound) for another blockdevice (generally a rotating HDD or array).
ZFS
or use zfsonlinux
ZFS is a combined file system and logical volume manager originally designed and implemented by a team at Sun Microsystems led by Jeff Bonwick and Matthew Ahrens. Features of ZFS include protection against data corruption, high storage capacity (256 ZiB), snapshots and copy-on-write clones and continuous integrity checking to name but a few. If you are dealing with large amounts of data, or providing a backing filesystem for virtualisation, ZFS is a great choice.
Using the above two techniques helps us to optimize the cost and improve the performance and availability of gitaly nodes.
Comparisons
- zfs is a more complete file system. zfs can override all of bcache's features.
- bcache still has serious bugs,Official ubuntu support for zfs.
- The inability of bcache to achieve consistency across multiple drives can be a barrier to our use of cloud drive snapshots, we are unable to use snapshots with a multi disk setup, while zfs provides filesystem-level snapshot capabilities.
- zfs can be performance enhanced or scaled with raid0/1, and bcache can also be disk scaled, but requires additional modifications to the file system
- zfs provides encryption, snapshot and compression features
Reference
- https://gitlab.com/gitlab-com/gl-infra/readiness/-/blob/master/library/gitaly-cache-storage/index.md
- https://wiki.archlinux.org/title/bcache
- https://ubuntu.com/tutorials/setup-zfs-storage-pool#1-overview
- https://gitlab.com/gitlab-com/www-gitlab-com/-/merge_requests/23785/diffs#00e31f3977e7830e0e6ba98ed0f9575b3b9491b3
- https://gitlab.com/gitlab-cookbooks/gitlab_disks/-/blob/master/attributes/default.rb
- custom_attributes: https://docs.gitlab.com/ee/api/custom_attributes.html
- https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/247
- https://vadosware.io/post/everything-ive-seen-on-optimizing-postgres-on-zfs-on-linux/