The Options interface isn't supposed to be exposed to end users. This can also avoid potential circular dependencies.

[BEAM-6721] Set numShards dynamically for TextIO.write()

[BEAM-13104] ParquetIO: SplitReadFn must read the whole block

* Add environment_service.go and structures for beam sdk, network envs and application envs

* Add executor config structure to beam environment structure
Fix tests, add new

* add not yet supported

* add java configuration file

* Add executor builder

* resolve conflicts with master

* adapt server to new envs

* Bugfix for a moved file

* test bugfix

Co-authored-by: Ilya Kozyrev <ilya.kozyrev@akvelon.com>

The test names seem to have changed since they are not being excluded so swap to using test category exclusion. See https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/1378/testReport/

* [BEAM-13048]: add keyboard shortcuts to playground

* [BEAM-13048]: update shortcuts

* [BEAM-13048]: fix lint errors

[BEAM-13062][Playground] Update errors in Protobuf

* [BEAM-11758] Update basics page: Aggregation, Runner, UDF, Schema

* Address most review feedback

* Fix type in intro list

* Address remaining two feedback comments

Merge pull request #15770 from [BEAM-13095][Playground] Using working directory from environment in LifeCycle instead of using hardcoded backend directory 

* [BEAM-13095][Playground]
Change the logic of 'lc` so that it uses workingDir

* [BEAM-13095][Playground]
Update comments;
Add using of filepath.Join() to construct filePath;

* [BEAM-13095][Playground]
Fix comment

* [BEAM-13095][Playground]
update go.sum

[BEAM-12998] [Playground] Environment service

Merge pull request #15714 from [BEAM-13005] [Playground] Implement local cache for pipelines' states

* Implement local cache for pipelines' states

* [BEAM-13005][Playground]
add context;
add redis option to NewCache;

* Edit GetValue method

* [BEAM-13005][Playground]
remove constructor from cache.go;
rename constructor method from newLocalCache to New;
move localCache files to new folder (local);
add comments;

* Resolve comments

* [BEAM-13005][Playground]
minor fix

* add documentation for exported methods

* fix documentation

Co-authored-by: AydarZaynutdinov <aydar.zaynutdinov@akvelon.com>
Co-authored-by: Ilya Kozyrev <ilya.kozyrev@akvelon.com>

* Implement the Log service

* Resolve comments

update README file;
generate files for backend

Co-authored-by: AydarZaynutdinov <aydar.zaynutdinov@akvelon.com>
Co-authored-by: daria.malkova <daria.malkova@akvelon.com>

add STATUS_COMPILE_ERROR;
add STATUS_RUN_TIMEOUT;
update proto command in README file to generate files;

* Rewriting for readability

* Address comments

* Address comments

* Addressing comments. Simplifying logic

[BEAM-13015] Create a multiplexer that sends Elements based upon instruction id allowing for an inbound observer responsible for the entire instruction id. (#15747)

* [BEAM-13015] Create a multiplexer that sends Elements based upon instruction id allowing for an inbound observer responsible for the entire instruction id.

* fixup! checkstyle

* Update sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataGrpcMultiplexer2.java

Co-authored-by: Brian Hulette <hulettbh@gmail.com>

* fixup! Address PR comments.

Co-authored-by: Brian Hulette <hulettbh@gmail.com>

* [BEAM-11275] Updated CHANGES.md with new improvements

* [BEAM-11275] Removed contextlib

* [BEAM-11275] Added logic to download files from supported filesystems

* [BEAM-11275] Update logs and catch larger scope Exception

* [BEAM-11275] Formatting

* [BEAM-11275] Download GCS file with FileSystems.open file handle

* [BEAM-11275] Fixed file decompression issue

* [BEAM-11275] Formatting

* [BEAM-11275] Added tests for Stager._download_file

* [BEAM-11275] Removed unused import

* [BEAM-11275] Fix Python formatting

* lazy creation of source ssplits

* formatting

[BEAM-12830] Avoiding read-only Go module cache in Gradle config.

* [BEAM-11758] Update basics page: Pipeline, PCollection, PTransform

* Address review feedback

* Minor PCollection text tweaks

./gradlew cleanTest was getting various errors from Go due to the Go module caches inside /.gogradle directories being read-only, so clean was unable to delete them. I adjusted this with the following:

1. Add -modcacherw to GOFLAGS on any commonly used gogradle tasks.
2. Avoid using gofmt on nested Gradle modules, as it would try to format the contents of /.gogradle.

* [BEAM-13019] Add `containsInAnyOrder` with matchers to the `IterableAssert` interface

* Fix the test.

* Add a test

* Fix a test

* Fix another test.

Confirmed all tests pass with `./gradlew compileTestJava`

* Add a no-argument containsInAnyOrder to remove the ambiguity.

* Update sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java

* Update sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java

Co-authored-by: Lukasz Cwik <lcwik@google.com>

* Implement Beam Playground gradle tasks

* Fix trailing whitespace

* Remove format from precommit

* Fix order of includes and add descriptions

* [BEAM-13008] add generated files and fix analysis issues (#2)

* Reconcile errors with playgroundPrecommit output

Co-authored-by: Aydar Farrakhov <stranniknm@gmail.com>

[BEAM-11087] Add default WindowMappingFn from Main to Side Input windows, validation test, unit tests (#15743)

Merge pull request #15381: [BEAM-10990] Elasticsearch response filtering and [BEAM-5172] Tries to reduce ES UTest flakiness

* Updates cross-language transform URNs to use the new convention

* Fix the Spanner Insert URN.

Co-authored-by: Brian Hulette <hulettbh@gmail.com>

* Fix the Kafka test

* Fixes the Pub/Sub test failure

Co-authored-by: Brian Hulette <hulettbh@gmail.com>

Merge pull request #15750 from [BEAM-13077][Playground] LifeCycle.CreateFolders() after several uses returns error 

[BEAM-13077][Playground] LifeCycle.CreateFolders() after several uses returns error