diff --git a/doc/ci/jobs/job_control.md b/doc/ci/jobs/job_control.md index 2cf74613b4299818dd32d6b5945d69462a191744..a03b7d92e150c2bc0ec65a2031e7af077f7f0b57 100644 --- a/doc/ci/jobs/job_control.md +++ b/doc/ci/jobs/job_control.md @@ -640,6 +640,106 @@ This job can no longer be scheduled to run automatically. You can, however, exec To start a delayed job immediately, select **Play** (**{play}**). Soon GitLab Runner starts the job. +## Parallelize large jobs + +To split a large job into multiple smaller jobs that run in parallel, use the +[`parallel`](../yaml/index.md#parallel) keyword in your `.gitlab-ci.yml` file. + +Different languages and test suites have different methods to enable parallelization. +For example, use [Semaphore Test Boosters](https://github.com/renderedtext/test-boosters) +and RSpec to run Ruby tests in parallel: + +```ruby +# Gemfile +source 'https://rubygems.org' + +gem 'rspec' +gem 'semaphore_test_boosters' +``` + +```yaml +test: + parallel: 3 + script: + - bundle + - bundle exec rspec_booster --job $CI_NODE_INDEX/$CI_NODE_TOTAL +``` + +You can then navigate to the **Jobs** tab of a new pipeline build and see your RSpec +job split into three separate jobs. + +WARNING: +Test Boosters reports usage statistics to the author. + +### Run a one-dimensional matrix of parallel jobs + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/26362) in GitLab 13.5. + +You can create a one-dimensional matrix of parallel jobs: + +```yaml +deploystacks: + stage: deploy + script: + - bin/deploy + parallel: + matrix: + - PROVIDER: [aws, ovh, gcp, vultr] +``` + +You can also [create a multi-dimensional matrix](../yaml/index.md#parallelmatrix). + +### Run a matrix of parallel trigger jobs + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/270957) in GitLab 13.10. + +You can run a [trigger](../yaml/index.md#trigger) job multiple times in parallel in a single pipeline, +but with different variable values for each instance of the job. + +```yaml +deploystacks: + stage: deploy + trigger: + include: path/to/child-pipeline.yml + parallel: + matrix: + - PROVIDER: aws + STACK: [monitoring, app1] + - PROVIDER: ovh + STACK: [monitoring, backup] + - PROVIDER: [gcp, vultr] + STACK: [data] +``` + +This example generates 6 parallel `deploystacks` trigger jobs, each with different values +for `PROVIDER` and `STACK`, and they create 6 different child pipelines with those variables. + +```plaintext +deploystacks: [aws, monitoring] +deploystacks: [aws, app1] +deploystacks: [ovh, monitoring] +deploystacks: [ovh, backup] +deploystacks: [gcp, data] +deploystacks: [vultr, data] +``` + +In [GitLab 14.1 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/239737), you can +use the variables defined in `parallel: matrix` with the [`tags`](../yaml/index.md#tags) keyword for +dynamic runner selection. + +```yaml +deploystacks: + stage: deploy + parallel: + matrix: + - PROVIDER: aws + STACK: [monitoring, app1] + - PROVIDER: gcp + STACK: [data] + tags: + - ${PROVIDER}-${STACK} +``` + ## Use predefined CI/CD variables to run jobs only in specific pipeline types You can use [predefined CI/CD variables](../variables/predefined_variables.md) to choose diff --git a/doc/ci/yaml/index.md b/doc/ci/yaml/index.md index 170ff25bd96939b711b189696562f7d68d5da483..4b7c9ca17cd6c9f700194b23961071cac20baf20 100644 --- a/doc/ci/yaml/index.md +++ b/doc/ci/yaml/index.md @@ -3679,11 +3679,17 @@ test: > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/21480) in GitLab 11.5. -Use `parallel` to configure how many instances of a job to run in parallel. -The value can be from 2 to 50. +Use `parallel` to run a job multiple times in parallel in a single pipeline. -The `parallel` keyword creates N instances of the same job that run in parallel. -They are named sequentially from `job_name 1/N` to `job_name N/N`: +Multiple runners must exist, or a single runner must be configured to run multiple jobs concurrently. + +Parallel jobs are named sequentially from `job_name 1/N` to `job_name N/N`. + +**Keyword type**: Job keyword. You can use it only as part of a job. + +**Possible inputs**: A numeric value from `2` to `50`. + +**Example of `parallel`**: ```yaml test: @@ -3691,47 +3697,32 @@ test: parallel: 5 ``` -Every parallel job has a `CI_NODE_INDEX` and `CI_NODE_TOTAL` -[predefined CI/CD variable](../variables/index.md#predefined-cicd-variables) set. +This example creates 5 jobs that run in parallel, named `test 1/5` to `test 5/5`. -Different languages and test suites have different methods to enable parallelization. -For example, use [Semaphore Test Boosters](https://github.com/renderedtext/test-boosters) -and RSpec to run Ruby tests in parallel: - -```ruby -# Gemfile -source 'https://rubygems.org' - -gem 'rspec' -gem 'semaphore_test_boosters' -``` +**Additional details**: -```yaml -test: - parallel: 3 - script: - - bundle - - bundle exec rspec_booster --job $CI_NODE_INDEX/$CI_NODE_TOTAL -``` +- Every parallel job has a `CI_NODE_INDEX` and `CI_NODE_TOTAL` + [predefined CI/CD variable](../variables/index.md#predefined-cicd-variables) set. -WARNING: -Test Boosters reports usage statistics to the author. +**Related topics**: -You can then navigate to the **Jobs** tab of a new pipeline build and see your RSpec -job split into three separate jobs. +- [Parallelize large jobs](../jobs/job_control.md#parallelize-large-jobs). -#### Parallel `matrix` jobs +#### `parallel:matrix` > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15356) in GitLab 13.3. +> - The job naming style was [improved in GitLab 13.4](https://gitlab.com/gitlab-org/gitlab/-/issues/230452). -Use `matrix:` to run a job multiple times in parallel in a single pipeline, +Use `parallel:matrix` to run a job multiple times in parallel in a single pipeline, but with different variable values for each instance of the job. -There can be from 2 to 50 jobs. -Jobs can only run in parallel if there are multiple runners, or a single runner is -configured to run multiple jobs concurrently. +Multiple runners must exist, or a single runner must be configured to run multiple jobs concurrently. + +**Keyword type**: Job keyword. You can use it only as part of a job. + +**Possible inputs**: A numeric value from `2` to `50`. -Every job gets the same `CI_NODE_TOTAL` [CI/CD variable](../variables/index.md#predefined-cicd-variables) value, and a unique `CI_NODE_INDEX` value. +**Example of `parallel:matrix`**: ```yaml deploystacks: @@ -3751,7 +3742,7 @@ deploystacks: STACK: [data, processing] ``` -The following example generates 10 parallel `deploystacks` jobs, each with different values +The example generates 10 parallel `deploystacks` jobs, each with different values for `PROVIDER` and `STACK`: ```plaintext @@ -3767,74 +3758,10 @@ deploystacks: [vultr, data] deploystacks: [vultr, processing] ``` -The job naming style was [improved in GitLab 13.4](https://gitlab.com/gitlab-org/gitlab/-/issues/230452). - -##### One-dimensional `matrix` jobs - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/26362) in GitLab 13.5. - -You can also have one-dimensional matrices with a single job: - -```yaml -deploystacks: - stage: deploy - script: - - bin/deploy - parallel: - matrix: - - PROVIDER: [aws, ovh, gcp, vultr] -``` - -##### Parallel `matrix` trigger jobs - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/270957) in GitLab 13.10. - -Use `matrix:` to run a [trigger](#trigger) job multiple times in parallel in a single pipeline, -but with different variable values for each instance of the job. - -```yaml -deploystacks: - stage: deploy - trigger: - include: path/to/child-pipeline.yml - parallel: - matrix: - - PROVIDER: aws - STACK: [monitoring, app1] - - PROVIDER: ovh - STACK: [monitoring, backup] - - PROVIDER: [gcp, vultr] - STACK: [data] -``` - -This example generates 6 parallel `deploystacks` trigger jobs, each with different values -for `PROVIDER` and `STACK`, and they create 6 different child pipelines with those variables. - -```plaintext -deploystacks: [aws, monitoring] -deploystacks: [aws, app1] -deploystacks: [ovh, monitoring] -deploystacks: [ovh, backup] -deploystacks: [gcp, data] -deploystacks: [vultr, data] -``` - -In [GitLab 14.1 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/239737), you can -use the variables defined in `parallel: matrix` with the [`tags`](#tags) keyword for -dynamic runner selection. +**Related topics**: -```yaml -deploystacks: - stage: deploy - parallel: - matrix: - - PROVIDER: aws - STACK: [monitoring, app1] - - PROVIDER: gcp - STACK: [data] - tags: - - ${PROVIDER}-${STACK} -``` +- [Run a one-dimensional matrix of parallel jobs](../jobs/job_control.md#run-a-one-dimensional-matrix-of-parallel-jobs). +- [Run a matrix of triggered parallel jobs](../jobs/job_control.md#run-a-matrix-of-parallel-trigger-jobs). ### `trigger` diff --git a/doc/development/ruby_upgrade.md b/doc/development/ruby_upgrade.md index ad6bff8499a5ddff2ce3ca82c8c7ea8c18e739e8..f9816986b2d6335269542c22cb4240a2e3e81b41 100644 --- a/doc/development/ruby_upgrade.md +++ b/doc/development/ruby_upgrade.md @@ -137,7 +137,7 @@ This is typically necessary, since gems or Ruby applications that we maintain ou update these repositories for the GitLab Rails application to work with a new Ruby, it is good practice to keep Ruby versions in lock-step across all our repositories. For minor and major upgrades, add new CI/CD jobs to these repositories using the new Ruby. -A [build matrix definition](../ci/yaml/index.md#parallel-matrix-jobs) can do this efficiently. +A [build matrix definition](../ci/yaml/index.md#parallelmatrix) can do this efficiently. #### Decide which repositories to update