BEAM-13939: Restructure Protos to fix namespace conflicts (#16961)
* Updates the Beam model protos to be housed in a namespaced directory structure. This is so that the proto files require usage of a org/apache/beam/model namespace in their imports and so that the generated files also include this namespace in their source file metadata. * Updates session.proto to use the new namespaced import * Adds a generic proto generated script that can be used to correctly generate all proto files for the go sdk. This new tool will add any necessary options to the proto compiler and generate the proto files relative to the go sdk root to ensure that the generated files have a namespaced file path in their metadata. If you want to generate a proto file in the go sdk, simply use this script in the go:generate directive, the rest will be taken care of by the script. * Updates the go sdk to use the new genproto script to correctly generate proto bindings. Updates the README for how to generate the model proto bindings into the SDK * Regenerates all go sdk proto bindings * Updates to the python build system. gen_protos.py now supports the new namespaced structure of the Beam model. It does this by supporting arbitrary directory structures of proto files by calculating and replacing the generated imports with relative imports with the generated source. Additionally, it will generate bindings that allow for imports of the form `from apache_beam.portability.api import beam_runner_api_pb2` so that the SDK is not dependent on the potentially changing structure of the generated bindings within `api`. Imports of the form `from apache_beam.portability.api.org.apache.beam.model import beam_runner_api_pb2` are still supported. setup.py now attempts to generate the proto bindings on invocation since the package structure must exist before the wheel can be created. * Updates to various build, ignore, and linting tooling for python in order to support the new python output structure * Updates to the python SDK to use the new import structure created by gen_protos
显示
- .gitignore 1 个添加, 2 个删除.gitignore
- build.gradle.kts 1 个添加, 1 个删除build.gradle.kts
- model/fn-execution/src/main/proto/org/apache/beam/model/fn_execution/v1/beam_fn_api.proto 3 个添加, 3 个删除...o/org/apache/beam/model/fn_execution/v1/beam_fn_api.proto
- model/fn-execution/src/main/proto/org/apache/beam/model/fn_execution/v1/beam_provision_api.proto 2 个添加, 2 个删除...pache/beam/model/fn_execution/v1/beam_provision_api.proto
- model/interactive/src/main/proto/org/apache/beam/model/interactive/v1/beam_interactive_api.proto 1 个添加, 1 个删除...ache/beam/model/interactive/v1/beam_interactive_api.proto
- model/job-management/src/main/proto/org/apache/beam/model/job_management/v1/beam_artifact_api.proto 1 个添加, 1 个删除...ache/beam/model/job_management/v1/beam_artifact_api.proto
- model/job-management/src/main/proto/org/apache/beam/model/job_management/v1/beam_expansion_api.proto 1 个添加, 1 个删除...che/beam/model/job_management/v1/beam_expansion_api.proto
- model/job-management/src/main/proto/org/apache/beam/model/job_management/v1/beam_job_api.proto 3 个添加, 3 个删除...rg/apache/beam/model/job_management/v1/beam_job_api.proto
- model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/beam_runner_api.proto 1 个添加, 1 个删除...o/org/apache/beam/model/pipeline/v1/beam_runner_api.proto
- model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/endpoints.proto 0 个添加, 0 个删除...n/proto/org/apache/beam/model/pipeline/v1/endpoints.proto
- model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/external_transforms.proto 2 个添加, 2 个删除...g/apache/beam/model/pipeline/v1/external_transforms.proto
- model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/metrics.proto 1 个添加, 1 个删除...ain/proto/org/apache/beam/model/pipeline/v1/metrics.proto
- model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/schema.proto 0 个添加, 0 个删除...main/proto/org/apache/beam/model/pipeline/v1/schema.proto
- model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/standard_window_fns.proto 1 个添加, 1 个删除...g/apache/beam/model/pipeline/v1/standard_window_fns.proto
- sdks/go/pkg/beam/core/runtime/graphx/v1/gen.go 1 个添加, 1 个删除sdks/go/pkg/beam/core/runtime/graphx/v1/gen.go
- sdks/go/pkg/beam/core/runtime/graphx/v1/v1.pb.go 333 个添加, 331 个删除sdks/go/pkg/beam/core/runtime/graphx/v1/v1.pb.go
- sdks/go/pkg/beam/core/runtime/harness/gen.go 1 个添加, 1 个删除sdks/go/pkg/beam/core/runtime/harness/gen.go
- sdks/go/pkg/beam/core/runtime/harness/session.proto 1 个添加, 1 个删除sdks/go/pkg/beam/core/runtime/harness/session.proto
- sdks/go/pkg/beam/core/runtime/harness/session/session.pb.go 125 个添加, 120 个删除sdks/go/pkg/beam/core/runtime/harness/session/session.pb.go
- sdks/go/pkg/beam/model/PROTOBUF.md 3 个添加, 7 个删除sdks/go/pkg/beam/model/PROTOBUF.md
加载中
想要评论请 注册 或 登录