* docker-compose

* stubs&README

* AIO

* fix

* nit

* WF triggers

* fix

* fix

Merge pull request #23954: Emit job ids via side output in TriggerFileLoads process to keep beam.Flatten() happy for Spark and Flink runners

* Created using Colaboratory

* Created using Colaboratory

* Implement text description feedback

* add apache license

* add dataframe describe method

* add reference to ml preprocessing notebook to df notebook

* update colab example with PR feedback and one-hot encoding implementation

* add reference to notebook in data processing webpage

* fix typos

* Include TODO to install from latest release once beam 2.43.0 is out in hidden metadata

* add reference to issue #23961

Co-authored-by: Philippe Moussalli <philippe.moussalli@ml6.eu>

* Initial draft of Batched DoFn user guide

* Add Batched DoFn link to left nav

* Minor tweaks

* Address most review comments

* Address review comments

* Fix typo in batch to element example

* Apply suggestions from code review

Co-authored-by: Pablo <pabloem@users.noreply.github.com>

* Remove pyarrow section until 2.44.0 is out

Co-authored-by: Pablo <pabloem@users.noreply.github.com>

* Add dummy ingestion component.

* Remove build script

* Add preprocessing component

* Add dummy model training component.

* Add pipeline compile step.

* Add tfx preprocessing orchestration

* Update argparse help string

* Change image size after resize step

* Rename compile_pipeline to pipeline

* Fix docstring

* Restructure preprocessing function

* Change indent to two spaces

* Fix pylint

* Rename directory containing the mlorchestration examples

* Add documentation to beam site

* Add link to full code examples

* Add trailing newline

* Remove unnecessary file

* Remove whitespaces for paragraph terminatin from markdown

* Fix python formatting with yapf

* Fix pylint violations

* Remove hanging TODOs and add brief README to point to the documentation on the website

* Update sdks/python/apache_beam/examples/ml-orchestration/kfp/components/ingestion/src/ingest.py

Co-authored-by: Andy Ye <andyye333@gmail.com>

* Update website/www/site/content/en/documentation/ml/orchestration.md

Co-authored-by: Danny McCormick <dannymccormick@google.com>

* Update website/www/site/content/en/documentation/ml/orchestration.md

Co-authored-by: Andy Ye <andyye333@gmail.com>

* Update website/www/site/content/en/documentation/ml/orchestration.md

Co-authored-by: Andy Ye <andyye333@gmail.com>

* Make argparse arguments for kfp components required

* Commit documentation suggestions and clarify pipeline execution snippet

* Fix yapf formatting and trailing whitespaces

* Add apache license and fix relative link to full code sample on website

* Clarify the requirements file for pipeline compilation and execution

* Explicitly link to the requirements for kfp pipeline execution

* Remove or replace 'Dummy' to more inclusive language

* Change global variables to variables provided by argparse

* Remove global config variables an add additional argparse arguments

* Fix python formatting

* Update website documentation to new code with argparse instead of global config variables

* Fix linting and formatting

* Fix linting and formatting

* Add missing link to yaml specification file in the docs

* Change DoFn to simple function with beam.Map

* Fix python formatting

* Fix python formatting

Co-authored-by: arnevandendorpe <arne.vandendorpe@ml6.eu>
Co-authored-by: Andy Ye <andyye333@gmail.com>
Co-authored-by: Danny McCormick <dannymccormick@google.com>

* enabled test_display_data

* cleaned up commented out lines

* Add integration tests for SparkReceiverIO

* Refactoring. Resolve comments.

* Add CustomOffsetRangeTracker. Fix reading from RabbitMqReceiver

* Fix compile

* Fix rabbitmq yaml file

* Refactoring

* Add tests for SDF

Co-authored-by: Sorokin Andrey <sorokin1990@gmail.com>

Bumps [actions/setup-java](https://github.com/actions/setup-java) from 3.5.1 to 3.6.0.
- [Release notes](https://github.com/actions/setup-java/releases)
- [Commits](https://github.com/actions/setup-java/compare/v3.5.1...v3.6.0

)

---
updated-dependencies:
- dependency-name: actions/setup-java
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Fix tests for examples not running in Dataflow

* Remove unused test

* Add todos to enable test for Dataflow

* Refactor utilities functions to create and read files

* Fix lint errors

* Fix lint errors and skip tests that require gcsio and is not available

* Refactor read file function and remove gcsio dependency

* Add notebook for doing remote inference

* Add documentation for overview page

* Add missing link in notebook, edit documentation

* add PR suggestion in documentation

* Remove duplicated sentence and add space

Co-authored-by: Shubham Krishna <“shubham.krishna@ml6.eu”>

* initial draft

* reuse parquet sink for writing rows and pa.Tables

* lint

* fix imports

* cr - rename value to table

* add doc string and example test

* fix doc strings

* specify doctest group to separate tests

Co-authored-by: Evan Sadler <easadler@gmail.com>

* Add jupyter notebook for using RunInference with sklearn, pytorch and tensorflow

* Add loading of sklearn model from GCS

* edit documentation

Co-authored-by: Shubham Krishna <“shubham.krishna@ml6.eu”>

Fix javadoc link, move off of deprecated getProperties() method to getPropertiesOrThrow()

This test suite seems to be disabled: https://ci-beam.apache.org/job/beam_PostCommit_Java11_Examples_Dataflow_Portability/

I suspect what we want to test here is the same as "Run Java 11 Examples on Dataflow Runner V2" which is running fine: https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2_java11/

* Upgrade Akvelon editor (#23415)

* Fix running folded code (#23415)

* Fix sharing folded code (#23415)

* [Playground] [Backend] updated cache component to handle precompiled object requests correctly

* [Playground] [Backend] added context with timeout instead of goroutine

* [Playground] [Backend] edited cache request timeout variable

* [Playground] [Backend] refactoring after review

* [Playground] [Backend] changed default value in readme

* [Playground] [Backend] removed transaction for ci cd datastore client

Merge pull request #23891: Fix Beam Sql Calcite Util does not support CHAR, VARCHAR, BINARY, VARBINARY

Fixes #23747

* Add brief descriptions about end-to-end ML Pipelines

* Fix wording and small nitpicks

Co-authored-by: Shubham Krishna <“shubham.krishna@ml6.eu”>