diff --git a/doc/architecture/blueprints/cells/rejected/deployment-architecture.md b/doc/architecture/blueprints/cells/rejected/deployment-architecture.md index aefe43ed63bf31d0b450e0c62d4378a04c0757be..fb9e43e8b511417069be0e5e744a08c8335d351f 100644 --- a/doc/architecture/blueprints/cells/rejected/deployment-architecture.md +++ b/doc/architecture/blueprints/cells/rejected/deployment-architecture.md @@ -14,7 +14,7 @@ of GitLab.com and contrasts it with the expected Cells architecture. ## 1. Before Cells - Monolithic architecture -<img src="diagrams/deployment-before-cells.drawio.png" width="800"> +<img src="diagrams/deployment-before-cells.drawio.png" width="800" alt=""> The diagram represents simplified GitLab.com deployment components before the introduction of a Cells architecture. This diagram intentionally omits some services that are not relevant for the architecture overview (Cloudflare, Consul, PgBouncers, ...). @@ -41,7 +41,7 @@ The dependent services: ## 2. Development Cells - Adapting application to Cellular architecture -<img src="diagrams/deployment-development-cells.drawio.png" width="800"> +<img src="diagrams/deployment-development-cells.drawio.png" width="800" alt=""> The purpose of **Development Cells** is to model a production-like architecture to test and validate the changes introduced. This could be achieved with testing Cells on top of the [Reference Architectures](../../../../administration/reference_architectures/index.md). @@ -57,7 +57,7 @@ The differences compared to [Before Cells](#1-before-cells---monolithic-architec ## 3. Initial Cells deployment - Transforming monolithic architecture to Cells architecture -<img src="diagrams/deployment-initial-cells.drawio.png" width="800"> +<img src="diagrams/deployment-initial-cells.drawio.png" width="800" alt=""> The differences compared to [Development Cells](#2-development-cells---adapting-application-to-cellular-architecture) are: @@ -83,7 +83,7 @@ The differences compared to [Development Cells](#2-development-cells---adapting- ## 4. Hybrid Cells deployment - Initial complete Cells architecture -<img src="diagrams/deployment-hybrid-cells.drawio.png" width="800"> +<img src="diagrams/deployment-hybrid-cells.drawio.png" width="800" alt=""> The differences compared to [Initial Cells deployment](#3-initial-cells-deployment---transforming-monolithic-architecture-to-cells-architecture) are: @@ -95,7 +95,7 @@ The differences compared to [Initial Cells deployment](#3-initial-cells-deployme ## 5. Target Cells - Fully isolated Cells architecture -<img src="diagrams/deployment-target-cells.drawio.png" width="800"> +<img src="diagrams/deployment-target-cells.drawio.png" width="800" alt=""> The differences compared to [Hybrid Cells deployment](#4-hybrid-cells-deployment---initial-complete-cells-architecture) are: @@ -110,10 +110,10 @@ with scaling the service, its location (cluster-wide or Cell-local), and impact ### Cluster-wide services -| Service | Type | Uses | Description | -| ------------------------------ | ------------ | ------------------------------- | --------------------------------------------------------------------------------------------------- | +| Service | Type | Uses | Description | +|--------------------------------|--------------|---------------------------------|-------------| | **Routing Service** | GitLab-built | Cluster-wide Data Provider | A general purpose routing service that can redirect requests from all GitLab SaaS domains to the Cell | -| **Cluster-wide Data Provider** | GitLab-built | PostgreSQL, Redis, Event Queue? | Provide user profile and routing information to all clustered services | +| **Cluster-wide Data Provider** | GitLab-built | PostgreSQL, Redis, Event Queue? | Provide user profile and routing information to all clustered services | As per the architecture, the above services are required to be run cluster-wide: @@ -121,36 +121,36 @@ As per the architecture, the above services are required to be run cluster-wide: ### Cell-local services -| Service | Type | Uses | Description | -| ------------------------------ | ------------ | ------------------------------- | --------------------------------------------------------------------------------------------------- | -| **Redis Cluster** | Managed service | Disk storage | No problem | Redis is used to hold user sessions, application caches, or Sidekiq queues. Most of that data is only applicable to Cells. | -| **GitLab Runners Manager** | Managed service | API, uses Google Cloud VM Instances | No problem | Significant changes required to API and execution of CI jobs | +| Service | Type | Uses | Migrate from cluster-wide to Cell | Description | +|----------------------------|-----------------|-------------------------------------|-----------------------------------|-------------| +| **Redis Cluster** | Managed service | Disk storage | No problem | Redis is used to hold user sessions, application caches, or Sidekiq queues. Most of that data is only applicable to Cells. | +| **GitLab Runners Manager** | Managed service | API, uses Google Cloud VM Instances | No problem | Significant changes required to API and execution of CI jobs | As per the architecture, the above services are required to be run Cell-local: - The consumer data held by the Cell-local services needs to be migratable to another Cell. -- The compute generated by the service is substatial, and it is strongly desired to reduce impact of [single Cell failure](../goals.md#high-resilience-to-a-single-cell-failure). +- The compute generated by the service is substantial, and it is strongly desired to reduce impact of [single Cell failure](../goals.md#high-resilience-to-a-single-cell-failure). - It is complex to run the service cluster-wide from the Cells architecture perspective. ### Hybrid Services -| Service | | Uses | Migrate from cluster-wide to Cell | Description | -| ------------------- | --------------- | ------------------------------- | ----------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- | -| **GitLab Pages** | GitLab-built | Routing Service, Rails API | No problem | Serving CI generated pages under `.gitlab.io` or custom domains | -| **GitLab Registry** | GitLab-built | Object Storage, PostgreSQL | Non-trivial data migration in case of split | Service to provide GitLab container registry | +| Service | Type | Uses | Migrate from cluster-wide to Cell | Description | +|---------------------|-----------------|---------------------------------|-------------------------------------------------------------------------|-------------| +| **GitLab Pages** | GitLab-built | Routing Service, Rails API | No problem | Serving CI generated pages under `.gitlab.io` or custom domains | +| **GitLab Registry** | GitLab-built | Object Storage, PostgreSQL | Non-trivial data migration in case of split | Service to provide GitLab container registry | | **Gitaly Cluster** | GitLab-built | Disk storage, PostgreSQL | No problem: Built-in migration routines to balance Gitaly nodes | Gitaly holds Git repository data. Many Gitaly clusters can be configured in application. | -| **Elasticsearch** | Managed service | Many nodes required by sharding | Time-consuming: Rebuild cluster from scratch | Search across all projects | -| **Object Storage** | Managed service | | Not straightforward: Rather hard to selectively migrate between buckets | Holds all user and CI uploaded files that is served by GitLab | +| **Elasticsearch** | Managed service | Many nodes required by sharding | Time-consuming: Rebuild cluster from scratch | Search across all projects | +| **Object Storage** | Managed service | | Not straightforward: Rather hard to selectively migrate between buckets | Holds all user and CI uploaded files that is served by GitLab | As per the architecture, the above services are allowed to be run either cluster-wide or Cell-local: - The ability to run hybrid services cluster-wide might reduce the amount of work to migrate data between Cells due to some services being shared. - The hybrid services that are run cluster-wide might negatively impact Cell availability and resiliency due to increased impact caused by [single Cell failure](../goals.md#high-resilience-to-a-single-cell-failure). -| Service | Type | Uses | Description | -| ------------------------------ | ------------ | ------------------------------- | --------------------------------------------------------------------------------------------------- | -| **Elasticsearch** | Managed service | Many nodes requires by sharding | Time-consuming: Rebuild cluster from scratch | Search across all projects | -| **Object Storage** | Managed service | | Not straightforward: Rather hard to selectively migrate between buckets | Holds all user and CI uploaded files that is served by GitLab | +| Service | Type | Uses | Migrate from cluster-wide to Cell | Description | +|--------------------|-----------------|---------------------------------|-------------------------------------------------------------------------|-------------| +| **Elasticsearch** | Managed service | Many nodes requires by sharding | Time-consuming: Rebuild cluster from scratch | Search across all projects | +| **Object Storage** | Managed service | | Not straightforward: Rather hard to selectively migrate between buckets | Holds all user and CI uploaded files that is served by GitLab | As per the architecture, the above services are allowed to be run either cluster-wide or Cell-local: diff --git a/doc/architecture/blueprints/code_search_with_zoekt/index.md b/doc/architecture/blueprints/code_search_with_zoekt/index.md index 1482676520841d501542a258d972e5208dee135d..d0d000a47359e9d9559a4fde89947698befce5c9 100644 --- a/doc/architecture/blueprints/code_search_with_zoekt/index.md +++ b/doc/architecture/blueprints/code_search_with_zoekt/index.md @@ -292,7 +292,7 @@ Some of those benefits could also be seen as downsides and maybe not the right c ##### High level proposal -<img src="diagrams/sharding_proposal_2023-08.drawio.png" height="600"> +<img src="diagrams/sharding_proposal_2023-08.drawio.png" height="600" alt=""> 1. Zoekt nodes are started with 3 additional arguments: its own address, shard name, and GitLab URL. 1. We'd like to keep shard name separate so that one will be able to migrate a shard to a different address. diff --git a/doc/architecture/blueprints/observability_tracing/index.md b/doc/architecture/blueprints/observability_tracing/index.md index 0b167400188ce08429191d1b0767c43c862a2118..980f9d9546e8a86985e117a1341cb260b376ad60 100644 --- a/doc/architecture/blueprints/observability_tracing/index.md +++ b/doc/architecture/blueprints/observability_tracing/index.md @@ -64,7 +64,7 @@ Much of the proposed architecture already exists and is in operation for GitLab. The following diagram outlines the architecture for GitLab Observability Backend and how clients, including the GitLab UI, will interact with it. -<img src="./arch.png"> +<img src="./arch.png" alt=""> ### Key Components diff --git a/doc/architecture/blueprints/organization/isolation.md b/doc/architecture/blueprints/organization/isolation.md index 467bd1932bd9d823e913182950a3f7d281b620f2..5cb6a5e3a88d4ee2c70c71291a8a2015165549be 100644 --- a/doc/architecture/blueprints/organization/isolation.md +++ b/doc/architecture/blueprints/organization/isolation.md @@ -18,7 +18,7 @@ Read more about what an Organization is in [Organization](index.md). ## What? -<img src="diagrams/organization-isolation.drawio.png" width="800"> +<img src="diagrams/organization-isolation.drawio.png" width="800" alt=""> All Cell-local data and functionality in GitLab (all data except the few things that need to exist on all Cells in the cluster) must be isolated. @@ -32,7 +32,7 @@ A few examples of things that would be disallowed by Organization Isolation are: ## Why? -<img src="diagrams/organization-isolation-broken.drawio.png" width="800"> +<img src="diagrams/organization-isolation-broken.drawio.png" width="800" alt=""> [GitLab Cells](../cells/index.md) depend on using the Organization as the sharding key, which will allow us to shard data between different Cells. Initially, when we start rolling out Organizations, we will be working with a single Cell `Cell 1`. diff --git a/doc/architecture/blueprints/runner_tokens/index.md b/doc/architecture/blueprints/runner_tokens/index.md index 3097eaa1f65904afc0a87decb56f6836456003cf..ac83559d3df7ecc2426887482852d700e8a64982 100644 --- a/doc/architecture/blueprints/runner_tokens/index.md +++ b/doc/architecture/blueprints/runner_tokens/index.md @@ -338,30 +338,30 @@ scope. ### Stage 1 - Deprecations -| Component | Milestone | Changes | -|------------------|----------:|---------| -| GitLab Rails app | `15.6` | Deprecate `POST /api/v4/runners` endpoint for `17.0`. This hinges on a [proposal](https://gitlab.com/gitlab-org/gitlab/-/issues/373774) to allow deprecating REST API endpoints for security reasons. | -| GitLab Runner | `15.6` | Add deprecation notice for `register` command for `17.0`. | -| GitLab Runner Helm Chart | `15.6` | Add deprecation notice for `runnerRegistrationToken` command for `17.0`. | -| GitLab Runner Operator | `15.6` | Add deprecation notice for `runner-registration-token` command for `17.0`. | -| GitLab Runner / GitLab Rails app | `15.7` | Add deprecation notice for registration token reset for `17.0`. | +| Component | Milestone | Changes | +|----------------------------------|----------:|---------| +| GitLab Rails app | `15.6` | Deprecate `POST /api/v4/runners` endpoint for `17.0`. This hinges on a [proposal](https://gitlab.com/gitlab-org/gitlab/-/issues/373774) to allow deprecating REST API endpoints for security reasons. | +| GitLab Runner | `15.6` | Add deprecation notice for `register` command for `17.0`. | +| GitLab Runner Helm Chart | `15.6` | Add deprecation notice for `runnerRegistrationToken` command for `17.0`. | +| GitLab Runner Operator | `15.6` | Add deprecation notice for `runner-registration-token` command for `17.0`. | +| GitLab Runner / GitLab Rails app | `15.7` | Add deprecation notice for registration token reset for `17.0`. | ### Stage 2 - Prepare `gitlab-runner` for `system_id` -| Component | Milestone | Changes | -|------------------|----------:|---------| -| GitLab Runner | `15.7` | Ensure a sidecar TOML file exists with a `system_id` value.<br/>Log new system ID values with `INFO` level as they get assigned. | -| GitLab Runner | `15.9` | Log unique system ID in the build logs. | -| GitLab Runner | `15.9` | Label Prometheus metrics with unique system ID. | -| GitLab Runner | `15.8` | Prepare `register` command to fail if runner server-side configuration options are passed together with a new `glrt-` token. | +| Component | Milestone | Changes | +|---------------|----------:|---------| +| GitLab Runner | `15.7` | Ensure a sidecar TOML file exists with a `system_id` value.<br/>Log new system ID values with `INFO` level as they get assigned. | +| GitLab Runner | `15.9` | Log unique system ID in the build logs. | +| GitLab Runner | `15.9` | Label Prometheus metrics with unique system ID. | +| GitLab Runner | `15.8` | Prepare `register` command to fail if runner server-side configuration options are passed together with a new `glrt-` token. | ### Stage 2a - Prepare GitLab Runner Helm Chart and GitLab Runner Operator -| Component | Milestone | Issue | Changes | -|------------------|----------:|-------|---------| -| GitLab Runner Helm Chart | `%15.10` | Update the Runner Helm Chart to support registration with the authentication token. | -| GitLab Runner Operator | `%15.10` | Update the Runner Operator to support registration with the authentication token. | -| GitLab Runner Helm Chart | `%16.2` | Add `systemID` to Runner Helm Chart. | +| Component | Milestone | Changes | +|--------------------------|----------:|---------| +| GitLab Runner Helm Chart | `%15.10` | Update the Runner Helm Chart to support registration with the authentication token. | +| GitLab Runner Operator | `%15.10` | Update the Runner Operator to support registration with the authentication token. | +| GitLab Runner Helm Chart | `%16.2` | Add `systemID` to Runner Helm Chart. | ### Stage 3 - Database changes