diff --git a/doc/ci/steps/index.md b/doc/ci/steps/index.md index ffaf6e7cac334d596c004911f7e5d41f654f7edf..b4028f882457c237e405eb13a6bdc655f3d6b603 100644 --- a/doc/ci/steps/index.md +++ b/doc/ci/steps/index.md @@ -16,21 +16,13 @@ While you are not required to use steps, the reusability, composability, testabi of steps make it easier to understand and maintain CI/CD pipeline. To get started, you can try the [Set up steps tutorial](../../tutorials/setup_steps/index.md). -To start creating your own steps, see [Creating your own step](#create-your-own-step). +To start creating your own steps, see [Creating your own step](#create-your-own-step). To understand how pipelines can benefit +from using both CI/CD Components and CI/CD Steps, see [Combine CI/CD Components and CI/CD Steps](#combine-cicd-components-and-cicd-steps). This experimental feature is still in active development and might have breaking changes at any time. Review the [changelog](https://gitlab.com/gitlab-org/step-runner/-/blob/main/CHANGELOG.md) for full details on any breaking changes. -CI/CD steps are different than [CI/CD components](../components/index.md). Components -are reusable single pipeline configuration units. They are included in a pipeline when it is created, -adding jobs and configuration to the pipeline. Files such as common scripts or programs -from the component project cannot be referenced from a CI/CD job. - -CI/CD Steps are reusable units of a job. When the job runs, the referenced step is downloaded to -the execution environment or image, bringing along any extra files included with the step. -Execution of the step replaces the `script` in the job. - ## Step workflow A step either runs a sequence of steps or executes a command. Each step specifies inputs received and outputs returned, has @@ -227,16 +219,16 @@ just prior to step execution in the job environment and can be used in: Expressions can reference the following variables: -| Variable | Example | Description | -|:----------------------------|:-----------------------------------|:------------| -| `env` | `${{env.HOME}}` | Access to environment variables set on the execution environment or in previous steps. | -| `export_file` | `echo name=FRED >${{export_file}}` | The path to the export file. Write to this file to export environment variables for use by subsequent running steps. | -| `inputs` | `${{inputs.message}}` | Access inputs to the step. | -| `job` | `${{job.GITLAB_USER_NAME}}` | Access GitLab CI/CD job variables, limited to those starting with `CI_`, `DOCKER_` or `GITLAB_`. | -| `output_file` | `echo name=Fred >${{output_file}}` | The path to the output file. Write to this file to set output variables from the step. | -| `step_dir` | `work_dir: ${{step_dir}}` | The folder to where the step has been downloaded. Use to refer to files in the step, or to set the work directory of an executable step. | -| `steps.[step-name].outputs` | `${{steps.my-step.outputs.name}}` | Access to outputs from previously executed steps. Choose the specific step using the step name. | -| `work_dir` | `${{work_dir}}` | The work directory of an executing step. | +| Variable | Example | Description | +|:----------------------------|:--------------------------------------------------------------|:------------| +| `env` | `${{env.HOME}}` | Access environment variables set in the execution environment or in previous steps. | +| `export_file` | `echo '{"name":"NAME","value":"Fred"}' >${{export_file}}` | The path to the [export file](#export-an-environment-variable). Write to this file to export environment variables for use by subsequent running steps. | +| `inputs` | `${{inputs.message}}` | Access the step's inputs. | +| `job` | `${{job.GITLAB_USER_NAME}}` | Access GitLab CI/CD variables, limited to those starting with `CI_`, `DOCKER_` or `GITLAB_`. | +| `output_file` | `echo '{"name":"meaning_life","value":42}' >${{output_file}}` | The path to the [output file](#return-an-output). Write to this file to set output variables from the step. | +| `step_dir` | `work_dir: ${{step_dir}}` | The directory where the step has been downloaded. Use to refer to files in the step, or to set the working directory of an executable step. | +| `steps.[step_name].outputs` | `${{steps.my_step.outputs.name}}` | Access [outputs](#specify-outputs) from previously executed steps. Choose the specific step using the step name. | +| `work_dir` | `${{work_dir}}` | The working directory of an executing step. | Expressions are different from template interpolation which uses double square-brackets (`$[[ ]]`) and are evaluated during job generation. @@ -381,7 +373,7 @@ For example, the following step returns outputs returned by the `random-generato spec: outputs: delegate --- -steps: +run: - name: random_generator step: ./random-generator delegate: random-generator @@ -420,7 +412,7 @@ spec: env: FIRST_NAME: Sally LAST_NAME: Seashells -steps: +run: # omitted for brevity ``` @@ -453,11 +445,19 @@ For example, if a step calls `go`, it should first install it. ##### Return an output -Executable steps return an output by adding a line to the `${{output_file}}` in the format `name=value`, -where the value is a JSON representation of the output type. The type of value written by the step -must match the type of the output in the step specification. +Executable steps return an output by adding a line to the `${{output_file}}` in JSON Line format. +Each line is a JSON object with `name` and `value` key pairs. The `name` must be a string, +and the `value` must be a type that matches the output type in the step specification: + +| Step specification type | Expected JSONL value type | +|:------------------------|:--------------------------| +| `array` | `array` | +| `boolean` | `boolean` | +| `number` | `number` | +| `string` | `string` | +| `struct` | `object` | -For example, to return the output named `car` with `string` value `range rover`: +For example, to return the output named `car` with `string` value `Range Rover`: ```yaml spec: @@ -469,13 +469,13 @@ exec: command: - bash - -c - - echo car=\"range rover\" >>${{output_file}} + - echo '{"name":"car","value":"Range Rover"}' >${{output_file}} ``` ##### Export an environment variable -Executable steps export an environment variable by adding a line to the `${{export_file}}` -in the format `name=value`. Double quotation marks are not required around the value. +Executable steps export an environment variable by adding a line to the `${{export_file}}` in JSON Line format. +Each line is a JSON object with `name` and `value` key pairs. Both `name` and `value` must be strings. For example, to set the variable `GOPATH` to value `/go`: @@ -486,7 +486,7 @@ exec: command: - bash - -c - - echo GOPATH=/go >${{export_file}} + - echo '{"name":"GOPATH","value":"/go"}' >${{export_file}} ``` #### Run a sequence of steps @@ -502,7 +502,7 @@ have been installed: ```yaml spec: --- -steps: +run: - name: install_go step: ./go-steps/install-go inputs: @@ -527,7 +527,7 @@ spec: java_version: type: string --- -steps: +run: - name: install_java step: ./common/install-java outputs: @@ -541,8 +541,66 @@ For example: spec: outputs: delegate --- -steps: +run: - name: install_java step: ./common/install-java delegate: install_java ``` + +## Combine CI/CD Components and CI/CD Steps + +[CI/CD components](../components/index.md) are reusable single pipeline configuration units. They are included in a pipeline when it is +created, adding jobs and configuration to the pipeline. Files such as common scripts or programs +from the component project cannot be referenced from a CI/CD job. + +CI/CD Steps are reusable units of a job. When the job runs, the referenced step is downloaded to +the execution environment or image, bringing along any extra files included with the step. +Execution of the step replaces the `script` in the job. + +Components and steps work well together to create solutions for CI/CD pipelines. Steps handle the complexity of +how jobs are composed, and automatically retrieve the files necessary to run the job. Components provide +a method to import job configuration, but hide the underlying job composition from the user. + +Steps and components use different syntax for expressions to help differentiate the expression types. +Component expressions use square brackets `$[[ ]]` and are evaluated during pipeline creation. +Step expressions use braces `${{ }}` and are evaluated during job execution, just before executing the step. + +For example, a project could use a component that adds a job to format Go code: + +- In the project's `.gitlab-ci.yml` file: + + ```yaml + include: + - component: gitlab.com/my-components/go@main + inputs: + fmt_packages: "./..." + ``` + +- Internally, the component uses CI/CD steps to compose the job, which installs Go then runs + the formatter. In the component's `templates/go.yml` file: + + ```yaml + spec: + inputs: + fmt_packages: + description: The Go packages that will be formatted using the Go formatter. + go_version: + default: "1.22" + description: The version of Go to install before running go fmt. + --- + + format code: + run: + - name: install_go + step: ./languages/go/install + inputs: + version: $[[ inputs.go_version ]] # version set to the value of the component input go_version + - name: format_code + step: ./languages/go/go-fmt + inputs: + go_binary: ${{ steps.install_go.outputs.go_binary }} # go_binary set to the value of the go_binary output from the previous step + fmt_packages: $[[ inputs.fmt_packages ]] # fmt_packages set to the value of the component input fmt_packages + ``` + +In this example, the complexity of the steps the component author used to compose the job are hidden from the user +in the CI/CD component. diff --git a/doc/tutorials/setup_steps/index.md b/doc/tutorials/setup_steps/index.md index a1ba5dac9c93b4f03c7b8e2718b78aec2005082b..b8be539509f838e62da9ab5a509f580adfdf97b0 100644 --- a/doc/tutorials/setup_steps/index.md +++ b/doc/tutorials/setup_steps/index.md @@ -22,8 +22,6 @@ In this tutorial, use the GitLab CLI (`glab`) to: ## Before you begin - You must install and sign in to the [GitLab CLI](../../editor_extensions/gitlab_cli/index.md) (`glab`). -- On GitLab.com and self-managed in GitLab 17.3 and later, you must include the - `image: registry.gitlab.com/gitlab-org/step-runner:v0` runner image in the `.gitlab-ci.yml` file. ## Create a step @@ -56,8 +54,8 @@ First, create a step with: spec: inputs: who: - default: world type: string + default: world ``` - `spec` has one input called `who`. @@ -69,14 +67,14 @@ First, create a step with: spec: inputs: who: - default: world type: string + default: world --- exec: command: - bash - -c - - "echo hello ${{ inputs.who }}" + - echo 'hello ${{inputs.who}}' ``` The triple em dash (`---`) separates the file into two YAML documents: @@ -87,7 +85,7 @@ The triple em dash (`---`) separates the file into two YAML documents: The `bash` and `-c` arguments start a Bash shell and take the script input from the command line arguments. In addition to shell scripts, you can use `command` to execute programs like `docker` or `terraform`. -The `"echo hello ${{ input.name }}"` argument includes an expression inside `${{` and `}}`. +The `echo 'hello ${{input.name}}'` argument includes an expression inside `${{` and `}}`. Expressions are evaluated at the last possible moment and have access to the current execution context. This expression accesses `inputs` and reads the value of `who`: @@ -105,48 +103,38 @@ This expression accesses `inputs` and reads the value of `who`: 1. In the `.gitlab-ci.yml`, add the following job: ```yaml - hello-world: - variables: - STEPS: - expand: false - value: | - - name: hello_world - step: . - image: registry.gitlab.com/gitlab-org/step-runner:v0 - script: - - /step-runner ci + hello-world: + run: + - name: hello_world + step: . ``` - - The steps are given in an environment variable called `STEPS`. `STEPS` is a list of step invocations. + - The `run` keyword has a list of step invocations. - Each invocation is given a `name` so you can reference the outputs in later steps. - - Each invocations specifies a `step` to run. A local reference (`.`) points to the root of the repository. - - The job script invokes `step-runner ci` which is in the `step-runner:v0` image. + - Each invocation specifies a `step` to run. A local reference (`.`) points to the root of the repository. - For an example of how this code should look in your repository, see [this example](https://gitlab.com/josephburnett/zero-to-steps/-/tree/part-1?ref_type=tags). + For an example of how this code should look in your repository, see the [Steps tutorial, part 1](https://gitlab.com/gitlab-org/step-runner/-/tree/main/examples/tutorial_part_1). 1. Commit both files and push the project repository. This triggers a pipeline that runs the job: ```shell git add . git commit -m 'Part 1 complete' - git push --set-upstream origin master + git push --set-upstream origin main glab ci status ``` 1. Follow the job under "View Logs" until the pipeline completes. Here's an example of a successful job: ```shell - $ /step-runner ci + Step Runner version: a7c7c8fd + See https://gitlab.com/gitlab-org/step-runner/-/blob/main/CHANGELOG.md for changes. + ... hello world - trace written to step-results.json Cleaning up project directory and file based variables Job succeeded ``` -NOTE: -Usage of an environment variable is a temporary work-around until the `run` keyword is implemented. -See [the `run` keyword epic](https://gitlab.com/groups/gitlab-org/-/epics/11846). - You've now created and used your first step! ## Add multiple steps to a job @@ -157,24 +145,18 @@ You can have more than one step in a job. ```yaml hello-world: - variables: - STEPS: - expand: false - value: | - - name: hello_world - step: . - - name: hello_steps - step: . - inputs: - who: gitlab steps - image: registry.gitlab.com/gitlab-org/step-runner:v0 - script: - - /step-runner ci + run: + - name: hello_world + step: . + - name: hello_steps + step: . + inputs: + who: gitlab steps ``` This `hello_steps` step provides a non-default input `who` of `gitlab steps`. - For an example of how this code should look in your repository, see [this example](https://gitlab.com/josephburnett/zero-to-steps/-/tree/part-2-a?ref_type=tags). + For an example of how this code should look in your repository, see the [Steps tutorial, part 2a](https://gitlab.com/gitlab-org/step-runner/-/tree/main/examples/tutorial_part_2a). 1. Commit and push the changes: @@ -187,10 +169,11 @@ You can have more than one step in a job. 1. In the terminal, select **View Logs** and follow the pipeline until it completes. Here's an example of a successful output: ```shell - $ /step-runner ci + Step Runner version: a7c7c8fd + See https://gitlab.com/gitlab-org/step-runner/-/blob/main/CHANGELOG.md for changes. + ... hello world hello gitlab steps - trace written to step-results.json Cleaning up project directory and file based variables Job succeeded ``` @@ -216,9 +199,9 @@ To refactor your steps by moving them from CI Config into a dedicated file: 1. Add the following configuration to the new `step.yml`: ```yaml - spec: {} + spec: --- - steps: + run: - name: hello_world step: ./hello - name: hello_steps @@ -227,7 +210,7 @@ To refactor your steps by moving them from CI Config into a dedicated file: who: gitlab steps ``` - This new step has no inputs, so the `spec` is empty (`{}`). + This new step has no inputs, so the `spec` is empty. It is a `steps` type, which has the same syntax as steps in `.gitlab-ci.yml`. However, the local reference now points to your step in the `hello` directory. @@ -235,21 +218,15 @@ To refactor your steps by moving them from CI Config into a dedicated file: ```yaml hello-world: - variables: - STEPS: - expand: false - value: | - - name: hello_everybody - step: . - image: registry.gitlab.com/gitlab-org/step-runner:v0 - script: - - /step-runner ci + run: + - name: hello_everybody + step: . ``` Now your job invokes only the new step with no inputs. You've refactored the details of the job into a separate file. - For an example of how this code should look in your repository, see [this example](https://gitlab.com/josephburnett/zero-to-steps/-/tree/part-2-b?ref_type=tags). + For an example of how this code should look in your repository, see the [Steps tutorial, part 2b](https://gitlab.com/gitlab-org/step-runner/-/tree/main/examples/tutorial_part_2b). 1. Commit and push the changes: @@ -267,7 +244,6 @@ To refactor your steps by moving them from CI Config into a dedicated file: $ /step-runner ci hello world hello gitlab steps - trace written to step-results.json Cleaning up project directory and file based variables Job succeeded ``` @@ -282,8 +258,8 @@ Add an output to your `hello` step. spec: inputs: who: - default: world type: string + default: world outputs: greeting: type: string @@ -292,22 +268,25 @@ Add an output to your `hello` step. command: - bash - -c - - "echo greeting=\"hello ${{ inputs.who }}\" | tee ${{ output_file }}" + - echo '{"name":"greeting","value":"hello ${{inputs.who}}"}' | tee ${{output_file}} ``` - In this `spec`, you've defined a single output `greeting` without a default. Because there is no default, the output `greeting` is required. - - Outputs are written to a file `${{ output_file }}` (provided at run time) in the form `key=value`. - - This step runs `echo greeting=\"hello ${{ inputs.name }}\"` and sends the output to the logs and the output file (`tee ${{ output_file }}`). + - Outputs are written to the `${{output_file}}` file provided at run time in JSON Line format. Each line written to the + output file must be a JSON object with two keys, `name` and `value`. + - This step runs `echo '{"name":"greeting","value":"hello ${{inputs.who}}"}'` and sends the output to the job log and + the output file (`tee ${{output_file}}`). 1. In `step.yml`, add an output to the step: ```yaml spec: outputs: - all_greetings: {} + all_greetings: + type: string --- - steps: + run: - name: hello_world step: ./hello - name: hello_steps @@ -315,18 +294,19 @@ Add an output to your `hello` step. inputs: who: gitlab steps outputs: - all_greetings: "${{ steps.hello_world.outputs.greeting }} and ${{ steps.hello_steps.outputs.greeting }}" + all_greetings: "${{steps.hello_world.outputs.greeting}} and ${{steps.hello_steps.outputs.greeting}}" ``` -You've now added an output to this step called `all_greetings`. + You've now added an output to this step called `all_greetings`. -This output shows the use of a new expression syntax: `${{ steps.hello_world.outputs.greeting }}`. -This expression reads the `outputs` of the two sub-steps, `hello_world` and `hello_steps`. -Both sub-step outputs are concatenated into a single string output. + This output shows the expression syntax: `${{steps.hello_world.outputs.greeting}}`. + `all_greetings` reads the outputs of the two sub-steps, `hello_world` and `hello_steps`. + Both sub-step outputs are concatenated into a single string output. ## Use a remote step -Before you commit and run your code, add another step to your job to see the final `all_greetings` output of your main `step.yml`. +Before you commit and run your code, add another step to your job to see the final `all_greetings` output of your main +`step.yml`. This step invocation references a remote step named `echo-step`. The echo step takes a single input `echo`, prints it to the logs, and outputs it as `echo`. @@ -335,22 +315,16 @@ The echo step takes a single input `echo`, prints it to the logs, and outputs it ```yaml hello-world: - variables: - STEPS: - expand: false - value: | - - name: hello_everybody - step: . - - name: all_my_greetings - step: gitlab.com/gitlab-org/ci-cd/runner-tools/echo-step@main - inputs: - echo: "all my greetings say ${{ steps.hello_everybody.outputs.all_greetings }}" - image: registry.gitlab.com/gitlab-org/step-runner:v0 - script: - - /step-runner ci + run: + - name: hello_everybody + step: . + - name: all_my_greetings + step: gitlab.com/gitlab-org/ci-cd/runner-tools/echo-step@main + inputs: + echo: "all my greetings say ${{steps.hello_everybody.outputs.all_greetings}}" ``` - For an example of how this code should look in your repository, see [this example](https://gitlab.com/josephburnett/zero-to-steps/-/tree/part-2-c?ref_type=tags). + For an example of how this code should look in your repository, see the [Steps tutorial, part 2c](https://gitlab.com/gitlab-org/step-runner/-/tree/main/examples/tutorial_part_2c). 1. Commit and push the changes: @@ -363,11 +337,12 @@ The echo step takes a single input `echo`, prints it to the logs, and outputs it 1. Follow the job under "View Logs" until the pipeline completes. Here's an example of a successful output: ```shell - $ /step-runner ci - greeting=hello world - greeting=hello gitlab steps - echo=all my greetings say hello world and hello gitlab steps - trace written to step-results.json + Step Runner version: a7c7c8fd + See https://gitlab.com/gitlab-org/step-runner/-/blob/main/CHANGELOG.md for changes. + ... + {"name":"greeting","value":"hello world"} + {"name":"greeting","value":"hello gitlab steps"} + all my greetings say hello world and hello gitlab steps Cleaning up project directory and file based variables Job succeeded ```