- 4月 08, 2020
-
-
由 Vikas Singh 创作于
CNKAF-541: Reduce the number of tests run in RandomClusterTest Subclasses of RandomClusterTess were running 48 tests each. The tests were groups in 4 sets of 6x2 tests. This change reduces the number to 4 sets of 4x2 tests. In all total number of tests run is now 32. The reduction in time it takes for the tests to run is about 50% (6m 39s to 3m 9s) as the indexes were used to generate # of partitions/replica etc, thus getting rid of higher numbers resulted in getting rid of longer running tests. The change also categorizes the test as IntegrationTest so it will not get run if unitTest goal is used.
- 4月 07, 2020
-
-
由 David Jacot 创作于
-
由 David Jacot 创作于
-
由 Apoorv Mittal 创作于
* METRICS-1232 - Move whitelist check to individual methods * METRICS-1232 - Moving whitelist check at single place * Revert "METRICS-1232 - Moving whitelist check at single place" This reverts commit ccfd740bd840dcdc31b190fedaf27691a6c33d58. * Modified kafka metrics filtering and test cases for derived metrics filtering * Refactored kafka Metrics Collector
-
由 Brian Bushree 创作于
-
由 Stanislav Kozlovski 创作于
It was found that checking for quotas in throttled replicas accounted for 5% of a CPU profile when all replicas' throttle was enabled. The majority of the time is spent in constructing the QuotaViolationException (stacktrace + String construction). Given that the exception is purposefully ignored, it makes sense to skip its generation when recording quotas for replication traffic
-
由 Steve Rodrigues 创作于
Add a unit test case to ensure the generated CruiseControl config actually has the desired regex.
- 4月 06, 2020
-
-
由 David Jacot 创作于
-
由 David Jacot 创作于
-
由 Jason Gustafson 创作于
If the high-watermark is updated in the middle of a read with the `read_committed` isolation level, it is possible to return data above the LSO. In the worst case, this can lead to the read of an aborted transaction. The root cause is that the logic depends on reading the high-watermark twice. We fix the problem by reading it once and caching the value. Reviewers: David Arthur <mumrah@gmail.com>, Guozhang Wang <wangguoz@gmail.com>, Ismael Juma <ismael@juma.me.uk>
-
- 4月 05, 2020
-
-
由 Lucas Bradstreet 创作于
https://github.com/apache/kafka/commit/3e9d1c1411c5268de382f9dfcc95bdf66d0063a0 introduced skipKeyValueIterator(s) which were intended to be used, but in this case were created but were not used in offset validation. ce-kafka results after the benchmark was fixed to instantiate the BrokerTopicStats outside of the benchmark measurements. These show an ~85% improvement in CPU and a 50% improvement in allocations. ``` Before: Benchmark (bufferSupplierStr) (bytes) (compressionType) (maxBatchSize) (messageSize) (messageVersion) Mode Cnt Score Error Units RecordBatchIterationBenchmark.measureValidation NO_CACHING RANDOM LZ4 1 1000 2 thrpt 15 167739.973 ± 2856.103 ops/s RecordBatchIterationBenchmark.measureValidation:·gc.alloc.rate.norm NO_CACHING RANDOM LZ4 1 1000 2 thrpt 15 137336.002 ± 0.002 B/op RecordBatchIterationBenchmark.measureValidation NO_CACHING RANDOM LZ4 2 1000 2 thrpt 15 171799.380 ± 496.011 ops/s RecordBatchIterationBenchmark.measureValidation:·gc.alloc.rate.norm NO_CACHING RANDOM LZ4 2 1000 2 thrpt 15 137360.002 ± 0.002 B/op RecordBatchIterationBenchmark.measureValidation NO_CACHING RANDOM LZ4 10 1000 2 thrpt 15 172052.847 ± 365.869 ops/s RecordBatchIterationBenchmark.measureValidation:·gc.alloc.rate.norm NO_CACHING RANDOM LZ4 10 1000 2 thrpt 15 137312.001 ± 0.002 B/op After: Benchmark (bufferSupplierStr) (bytes) (compressionType) (maxBatchSize) (messageSize) (messageVersion) Mode Cnt Score Error Units RecordBatchIterationBenchmark.measureValidation NO_CACHING RANDOM LZ4 1 1000 2 thrpt 15 311084.661 ± 3645.225 ops/s RecordBatchIterationBenchmark.measureValidation:·gc.alloc.rate.norm NO_CACHING RANDOM LZ4 1 1000 2 thrpt 15 69832.001 ± 0.001 B/op RecordBatchIterationBenchmark.measureValidation NO_CACHING RANDOM LZ4 2 1000 2 thrpt 15 298843.146 ± 9059.394 ops/s RecordBatchIterationBenchmark.measureValidation:·gc.alloc.rate.norm NO_CACHING RANDOM LZ4 2 1000 2 thrpt 15 69832.001 ± 0.001 B/op RecordBatchIterationBenchmark.measureValidation NO_CACHING RANDOM LZ4 10 1000 2 thrpt 15 306849.740 ± 8257.428 ops/s RecordBatchIterationBenchmark.measureValidation:·gc.alloc.rate.norm NO_CACHING RANDOM LZ4 10 1000 2 thrpt 15 69832.001 ± 0.001 B/op ``` A subset of the benchmark results follow. Looks like a 20% improvement in validation performance and a 40% reduction in garbage allocation for 1-2 batch sizes. **# Parameters: (bufferSupplierStr = NO_CACHING, bytes = RANDOM, compressionType = LZ4, maxBatchSize = 1, messageSize = 1000, messageVersion = 2)** Before: Result "org.apache.kafka.jmh.record.RecordBatchIterationBenchmark.measureValidation": 64851.837 ±(99.9%) 944.248 ops/s [Average] (min, avg, max) = (64505.317, 64851.837, 65114.359), stdev = 245.218 CI (99.9%): [63907.589, 65796.084] (assumes normal distribution) "org.apache.kafka.jmh.record.RecordBatchIterationBenchmark.measureValidation:·gc.alloc.rate.norm": 164088.003 ±(99.9%) 0.004 B/op [Average] (min, avg, max) = (164088.001, 164088.003, 164088.004), stdev = 0.001 CI (99.9%): [164087.998, 164088.007] (assumes normal distribution) After: Result "org.apache.kafka.jmh.record.RecordBatchIterationBenchmark.measureValidation": 78910.273 ±(99.9%) 707.024 ops/s [Average] (min, avg, max) = (78785.486, 78910.273, 79234.007), stdev = 183.612 CI (99.9%): [78203.249, 79617.297] (assumes normal distribution) "org.apache.kafka.jmh.record.RecordBatchIterationBenchmark.measureValidation:·gc.alloc.rate.norm": 96440.002 ±(99.9%) 0.001 B/op [Average] (min, avg, max) = (96440.002, 96440.002, 96440.002), stdev = 0.001 CI (99.9%): [96440.002, 96440.003] (assumes normal distribution) **# Parameters: (bufferSupplierStr = NO_CACHING, bytes = RANDOM, compressionType = LZ4, maxBatchSize = 2, messageSize = 1000, messageVersion = 2)** Before: Result "org.apache.kafka.jmh.record.RecordBatchIterationBenchmark.measureValidation": 64815.364 ±(99.9%) 639.309 ops/s [Average] (min, avg, max) = (64594.545, 64815.364, 64983.305), stdev = 166.026 CI (99.9%): [64176.056, 65454.673] (assumes normal distribution) "org.apache.kafka.jmh.record.RecordBatchIterationBenchmark.measureValidation:·gc.alloc.rate.norm": 163944.003 ±(99.9%) 0.001 B/op [Average] (min, avg, max) = (163944.002, 163944.003, 163944.003), stdev = 0.001 CI (99.9%): [163944.002, 163944.004] (assumes normal distribution) After: Result "org.apache.kafka.jmh.record.RecordBatchIterationBenchmark.measureValidation": 77075.096 ±(99.9%) 201.092 ops/s [Average] (min, avg, max) = (77021.537, 77075.096, 77129.693), stdev = 52.223 CI (99.9%): [76874.003, 77276.188] (assumes normal distribution) "org.apache.kafka.jmh.record.RecordBatchIterationBenchmark.measureValidation:·gc.alloc.rate.norm": 96504.002 ±(99.9%) 0.003 B/op [Average] (min, avg, max) = (96504.001, 96504.002, 96504.003), stdev = 0.001 CI (99.9%): [96503.999, 96504.005] (assumes normal distribution) Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, Ismael Juma <ismael@juma.me.uk> ce-kafka: added interceptor and interceptor stats arguments for validateMessagesAndAssignOffsetsCompressed Conflicts: checkstyle/import-control-jmh-benchmarks.xml
- 4月 04, 2020
-
-
由 Lucas Bradstreet 创作于
Adds or fixes heartbeat metrics: - Fixes the TierTopicManager/TierTopicConsumer heartbeat. - Adds a TierTasks heartbeat. - Adds a TierDeletedPartitionsCoordinator heartbeat. I'm not sure how useful this one is. - Adds a number of metrics related to the TierTopicManager including the number of online, immigrating, and initialized partitions, the max time listening for a materialization, and the number of listeners. System tests: - Adds a check_cluster_state assertion to system tests which checks the above heartbeats, and the archiver PartitionsInError metric. - Checks all the above heartbeats in system tests. - Reduces the difference between AK and ce-kafka for the reassign system test. - Uses the new confluent.tier.segment.hotset.roll.min.bytes in the tiered storage system tests to enforce a minimum segment size.
-
由 Bob Barrett 创作于
* CPKAFKA-4753: Add metrics tracking total txn time and time spent in a pending state * Move metrics constants under existing object class * Remove per-producer and pending time metrics, add sensors for timeouts and state errors * Tests WIP * Add metrics tests * PR feedback * Make state and pending State of TransactionMetadata volatile * PR feedback * Close JmxReporter in TraansactionStateManagerTest * PR feedback
-
由 David Mao 创作于
- 4月 03, 2020
-
-
由 Rajini Sivaram 创作于
Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>
-
由 Aishwarya Gune 创作于
Add kafka StartRebalance protocol and request/response classes. Co-authored-by: Aishwarya <aishwarya.gune@Aishwarya-Gunes-MBP13.local> Co-authored-by: Stanislav Kozlovski <stanislav_kozlovski@outlook.com>
-
由 Srini Dandu 创作于
Signed-off-by: sdanduConf <sdandu@confluent.io>
-
-
由 Boyang Chen 创作于
As documented in the KIP: We shall set `transaction.timout.ms` default to 10000 ms (10 seconds) on Kafka Streams. Reviewer: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
-
由 Daniel 创作于
KAFKA-9778: Add methods to validate and assert connector configurations in integration tests with EmbeddedConnectCluster (#8359) * Add validateConnector functionality to the EmbeddedConnectCluster * PR Revision - added ConnectException conversion, validateConnectorConfig calls to ExampleConnectIntegrationTest * PR revision - Added method to EmbeddedConnectClusterAssertions Reviewers: Konstantine Karantasis <konstantine@confluent.io>, Chia-Ping Tsai <chia7712@gmail.com>
-
由 Vikas Singh 创作于
KafkaSampleStore checks that the two topics that it uses to store historical data are present at startup. If it finds any issue it aborts the Cruise Control startup. It can run into error for many reasons but the one common is that topic replication factor isn't met. This is because during bootstrap first broker will become controller, it will try to start CC and then SampleStore will complain that topic replication factor is 3 whereas there are only 1 alive brokers and then abort the CC startup. This makes it difficult to start brokers and have CC running. This change refactors the code to add the check as part of CC startup check. Tested by running broker in different setups to confirm that it works.
-
由 Lucas Bradstreet 创作于
Includes a ce-kafka related benchmark of authorizer code used by TenantAclProvider.
-
由 A. Sophie Blee-Goldman 创作于
Adds a new TaskAssignor implementation, currently hidden behind an internal feature flag, that implements the high availability algorithm of KIP-441. Reviewers: Bruno Cadonna <bruno@confluent.io>, John Roesler <vvcephei@apache.org>
-
由 Ron Dagostino 创作于
Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>