Skip to content
代码片段 群组 项目
未验证 提交 9f3d03fe 编辑于 作者: Rajendra Kadam's avatar Rajendra Kadam 提交者： GitLab
浏览文件

Merge branch '440667-remove-ci-text-interpolation-2' into 'master' 

Remove CI text interpolation code

See merge request https://gitlab.com/gitlab-org/gitlab/-/merge_requests/147803



Merged-by: default avatarRajendra Kadam <rkadam@gitlab.com>
Approved-by: default avatarLaura Montemayor <lmontemayor@gitlab.com>
Approved-by: default avatarRajendra Kadam <rkadam@gitlab.com>
Co-authored-by: default avatarAvielle Wolfe <awolfe@gitlab.com>
上级 7b440ce1 96f86196
No related branches found
No related tags found
无相关合并请求
隐藏空白变更内容
行内 左右并排
显示
有 5 个添加 和 456 个删除
config/feature_flags/gitlab_com_derisk/ci_text_interpolation.yml 已删除 100644 → 0
+ 0
− 9
浏览文件 @ 7b440ce1
---
name: ci_text_interpolation
feature_issue_url: https://gitlab.com/gitlab-org/gitlab/-/issues/433002
introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/142009
rollout_issue_url: https://gitlab.com/gitlab-org/gitlab/-/issues/435177
milestone: '16.9'
group: group::pipeline authoring
type: gitlab_com_derisk
default_enabled: false
lib/gitlab/ci/config/interpolation/text_interpolator.rb 已删除 100644 → 0
+ 0
− 100
浏览文件 @ 7b440ce1
# frozen_string_literal: true
module Gitlab
module Ci
class Config
module Interpolation
##
# Performs CI config file interpolation and either returns the interpolated result or interpolation errors.
#
class TextInterpolator
attr_reader :errors
def initialize(yaml_documents, input_args, variables)
@yaml_documents = yaml_documents
@input_args = input_args.to_h
@variables = variables
@errors = []
@interpolated = false
end
def valid?
errors.none?
end
def to_result
@result
end
def error_message
# Interpolator can have multiple error messages, like: ["interpolation interrupted by errors", "unknown
# interpolation key: `abc`"] ?
#
# We are joining them together into a single one, because only one error can be surfaced when an external
# file gets included and is invalid. The limit to three error messages combined is more than required.
#
errors.first(3).join(', ')
end
def interpolate!
if inputs_without_header?
return errors.push(
_('Given inputs not defined in the `spec` section of the included configuration file'))
end
return @result ||= yaml_documents.content unless yaml_documents.header
return errors.concat(header.errors) unless header.valid?
return errors.concat(inputs.errors) unless inputs.valid?
return errors.concat(context.errors) unless context.valid?
return errors.concat(template.errors) unless template.valid?
@interpolated = true
@result ||= template.interpolated
end
def interpolated?
@interpolated
end
private
attr_reader :yaml_documents, :input_args, :variables
def inputs_without_header?
input_args.any? && !yaml_documents.header
end
def header
@header ||= Header::Root.new(yaml_documents.header).tap do |header|
header.key = 'header'
header.compose!
end
end
def content
@content ||= yaml_documents.content
end
def spec
@spec ||= header.inputs_value
end
def inputs
@inputs ||= Inputs.new(spec, input_args)
end
def context
@context ||= Context.new({ inputs: inputs.to_hash }, variables: variables)
end
def template
@template ||= TextTemplate.new(content, context)
end
end
end
end
end
end
lib/gitlab/ci/config/interpolation/text_template.rb 已删除 100644 → 0
+ 0
− 59
浏览文件 @ 7b440ce1
# frozen_string_literal: true
module Gitlab
module Ci
class Config
module Interpolation
class TextTemplate
MAX_BLOCKS = 10_000
def initialize(content, ctx)
@content = content
@ctx = Interpolation::Context.fabricate(ctx)
@errors = []
@blocks = {}
interpolate! if valid?
end
def valid?
errors.none?
end
def errors
@errors + ctx.errors + blocks.values.flat_map(&:errors)
end
def interpolated
@result if valid?
end
private
attr_reader :blocks, :content, :ctx
def interpolate!
return @errors.push('config too large') if content.bytesize > max_total_yaml_size_bytes
@result = Interpolation::Block.match(content) do |matched, data|
block = (blocks[matched] ||= Interpolation::Block.new(matched, data, ctx))
break @errors.push('too many interpolation blocks') if blocks.count > MAX_BLOCKS
break unless block.valid?
if block.value.is_a?(String)
block.value
else
block.value.to_json
end
end
end
def max_total_yaml_size_bytes
Gitlab::CurrentSettings.current_application_settings.ci_max_total_yaml_size_bytes
end
end
end
end
end
end
lib/gitlab/ci/config/yaml/loader.rb
+ 5
− 39
浏览文件 @ 9f3d03fe
......@@ -17,18 +17,16 @@ def initialize(content, inputs: {}, variables: [])
end
def load
if Feature.disabled?(:ci_text_interpolation, Feature.current_request, type: :gitlab_com_derisk)
return legacy_load
end
yaml_result = load_uninterpolated_yaml
return yaml_result unless yaml_result.valid?
interpolator = Interpolation::TextInterpolator.new(yaml_documents, inputs, variables)
interpolator = Interpolation::Interpolator.new(yaml_result, inputs, variables)
interpolator.interpolate!
if interpolator.valid?
loaded_yaml = yaml(interpolator.to_result).load!
Yaml::Result.new(config: loaded_yaml, error: nil, interpolated: interpolator.interpolated?)
Yaml::Result.new(config: interpolator.to_hash, error: nil, interpolated: interpolator.interpolated?)
else
Yaml::Result.new(error: interpolator.error_message, interpolated: interpolator.interpolated?)
end
......@@ -46,38 +44,6 @@ def load_uninterpolated_yaml
attr_reader :content, :inputs, :variables
def yaml(content)
ensure_custom_tags
::Gitlab::Config::Loader::Yaml.new(content, additional_permitted_classes: AVAILABLE_TAGS)
end
def yaml_documents
docs = content
.split(::Gitlab::Config::Loader::MultiDocYaml::MULTI_DOC_DIVIDER, MAX_DOCUMENTS + 1)
.map { |d| yaml(d) }
docs.reject!(&:blank?)
Yaml::Documents.new(docs)
end
def legacy_load
yaml_result = load_uninterpolated_yaml
return yaml_result unless yaml_result.valid?
interpolator = Interpolation::Interpolator.new(yaml_result, inputs, variables)
interpolator.interpolate!
if interpolator.valid?
Yaml::Result.new(config: interpolator.to_hash, error: nil, interpolated: interpolator.interpolated?)
else
Yaml::Result.new(error: interpolator.error_message, interpolated: interpolator.interpolated?)
end
end
def load_yaml!
ensure_custom_tags
......
spec/lib/gitlab/ci/config/interpolation/text_interpolator_spec.rb 已删除 100644 → 0
+ 0
− 139
浏览文件 @ 7b440ce1
# frozen_string_literal: true
require 'spec_helper'
RSpec.describe Gitlab::Ci::Config::Interpolation::TextInterpolator, feature_category: :pipeline_composition, quarantine: 'https://gitlab.com/gitlab-org/gitlab/-/issues/440667' do
let(:arguments) { { website: 'gitlab.com' } }
let(:content) { ::Gitlab::Config::Loader::Yaml.new("test: 'deploy $[[ inputs.website ]]'") }
let(:header) { ::Gitlab::Config::Loader::Yaml.new("spec:\n inputs:\n website: ") }
let(:documents) { ::Gitlab::Ci::Config::Yaml::Documents.new([header, content]) }
subject(:interpolator) { described_class.new(documents, arguments, []) }
context 'when input data is valid' do
it 'correctly interpolates the config' do
interpolator.interpolate!
expect(interpolator).to be_interpolated
expect(interpolator).to be_valid
expect(interpolator.to_result).to eq("test: 'deploy gitlab.com'")
end
end
context 'when interpolation is not used' do
let(:arguments) { nil }
let(:content) { ::Gitlab::Config::Loader::Yaml.new("test: 'deploy production'") }
let(:documents) { ::Gitlab::Ci::Config::Yaml::Documents.new([content]) }
it 'returns original content' do
interpolator.interpolate!
expect(interpolator).not_to be_interpolated
expect(interpolator).to be_valid
expect(interpolator.to_result).to eq("test: 'deploy production'")
end
end
context 'when spec header is missing but inputs are specified' do
let(:documents) { ::Gitlab::Ci::Config::Yaml::Documents.new([content]) }
it 'surfaces an error about invalid inputs' do
interpolator.interpolate!
expect(interpolator).not_to be_valid
expect(interpolator.error_message).to eq(
'Given inputs not defined in the `spec` section of the included configuration file'
)
end
end
context 'when spec header is invalid' do
let(:header) { ::Gitlab::Config::Loader::Yaml.new("spec:\n arguments:\n website:") }
it 'surfaces an error about invalid header' do
interpolator.interpolate!
expect(interpolator).not_to be_valid
expect(interpolator.error_message).to eq('header:spec config contains unknown keys: arguments')
end
end
context 'when provided interpolation argument is invalid' do
let(:arguments) { { website: ['gitlab.com'] } }
it 'returns an error about the invalid argument' do
interpolator.interpolate!
expect(interpolator).not_to be_valid
expect(interpolator.error_message).to eq('`website` input: provided value is not a string')
end
end
context 'when interpolation block is invalid' do
let(:content) { ::Gitlab::Config::Loader::Yaml.new("test: 'deploy $[[ inputs.abc ]]'") }
it 'returns an error about the invalid block' do
interpolator.interpolate!
expect(interpolator).not_to be_valid
expect(interpolator.error_message).to eq('unknown interpolation key: `abc`')
end
end
context 'when multiple interpolation blocks are invalid' do
let(:content) do
::Gitlab::Config::Loader::Yaml.new("test: 'deploy $[[ inputs.something.abc ]] $[[ inputs.cde ]] $[[ efg ]]'")
end
it 'stops execution after the first invalid block' do
interpolator.interpolate!
expect(interpolator).not_to be_valid
expect(interpolator.error_message).to eq('unknown interpolation key: `something`')
end
end
context 'when there are many invalid arguments' do
let(:header) do
::Gitlab::Config::Loader::Yaml.new(
<<~YAML
spec:
inputs:
allow_failure:
type: boolean
image:
parallel:
type: number
website:
YAML
)
end
let(:content) do
::Gitlab::Config::Loader::Yaml.new(
"test: 'deploy $[[ inputs.website ]] $[[ inputs.parallel ]] $[[ inputs.allow_failure ]] $[[ inputs.image ]]'"
)
end
let(:arguments) do
{ allow_failure: 'no', parallel: 'yes', website: 8 }
end
it 'reports a maximum of 3 errors in the error message' do
interpolator.interpolate!
expect(interpolator).not_to be_valid
expect(interpolator.error_message).to eq(
'`allow_failure` input: provided value is not a boolean, ' \
'`image` input: required value has not been provided, ' \
'`parallel` input: provided value is not a number'
)
expect(interpolator.errors).to contain_exactly(
'`allow_failure` input: provided value is not a boolean',
'`image` input: required value has not been provided',
'`parallel` input: provided value is not a number',
'`website` input: provided value is not a string'
)
end
end
end
spec/lib/gitlab/ci/config/interpolation/text_template_spec.rb 已删除 100644 → 0
+ 0
− 105
浏览文件 @ 7b440ce1
# frozen_string_literal: true
require 'spec_helper'
RSpec.describe Gitlab::Ci::Config::Interpolation::TextTemplate, feature_category: :pipeline_composition, quarantine: 'https://gitlab.com/gitlab-org/gitlab/-/issues/440667' do
subject(:template) { described_class.new(config, ctx) }
let(:config) do
<<~CFG
test:
spec:
env: $[[ inputs.env ]]
$[[ inputs.key ]]:
name: $[[ inputs.key ]]
script: my-value
parallel: $[[ inputs.parallel ]]
CFG
end
let(:ctx) do
{ inputs: { env: 'dev', key: 'abc', parallel: 6 } }
end
it 'interpolates the values properly' do
expect(template.interpolated).to eq <<~RESULT
test:
spec:
env: dev
abc:
name: abc
script: my-value
parallel: 6
RESULT
end
context 'when the config has an unknown interpolation key' do
let(:config) { '$[[ xxx.yyy ]]: abc' }
it 'does not interpolate the config' do
expect(template).not_to be_valid
expect(template.interpolated).to be_nil
expect(template.errors).to contain_exactly('unknown interpolation key: `xxx`')
end
end
context 'when template consists of nested arrays with hashes and values' do
let(:config) do
<<~CFG
test:
- a-$[[ inputs.key ]]-b
- c-$[[ inputs.key ]]-d:
d-$[[ inputs.key ]]-e
val: 1
CFG
end
it 'performs a valid interpolation' do
result = <<~RESULT
test:
- a-abc-b
- c-abc-d:
d-abc-e
val: 1
RESULT
expect(template).to be_valid
expect(template.interpolated).to eq result
end
end
context 'when template contains symbols that need interpolation' do
subject(:template) do
described_class.new("'$[[ inputs.key ]]': 'cde'", ctx)
end
it 'performs a valid interpolation' do
expect(template).to be_valid
expect(template.interpolated).to eq("'abc': 'cde'")
end
end
context 'when template is too large' do
before do
stub_application_setting(ci_max_total_yaml_size_bytes: 1)
end
it 'returns an error' do
expect(template.interpolated).to be_nil
expect(template.errors).to contain_exactly('config too large')
end
end
context 'when there are too many interpolation blocks' do
before do
stub_const("#{described_class}::MAX_BLOCKS", 1)
end
it 'returns an error' do
expect(template.interpolated).to be_nil
expect(template.errors).to contain_exactly('too many interpolation blocks')
end
end
end
spec/spec_helper.rb
+ 0
− 5
浏览文件 @ 9f3d03fe
......@@ -321,11 +321,6 @@
# Postgres is the primary data source, and ClickHouse only when enabled in certain cases.
stub_feature_flags(clickhouse_data_collection: false)
# The code under this flag will be removed soon
# We are temporarily keeping it in place while we confirm some assumptions
# https://gitlab.com/gitlab-org/gitlab/-/issues/440667
stub_feature_flags(ci_text_interpolation: false)
# The Vue version of the merge request list app is missing a lot of information
# disabling this for now whilst we work on it across multiple merge requests
stub_feature_flags(vue_merge_request_list: false)
......
0% 或 .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑！
想要评论请 注册 或