Skip to content
代码片段 群组 项目
  1. 10月 14, 2020
  2. 10月 07, 2020
  3. 9月 23, 2020
  4. 9月 22, 2020
  5. 9月 21, 2020
    • Chia-Ping Tsai's avatar
      KAFKA-10438: Lazy initialization of record header to reduce memory usage (#9223) · 4b6d8da9
      Chia-Ping Tsai 创作于
      There are no checks on the header key so instantiating key (bytes to string) is unnecessary.
      One implication is that conversion failures will be detected a bit later, but this is consistent
      with how we handle the header value.
      
      **JMH RESULT**
      
      1. ops: +12%
      1. The optimization of memory usage is very small as the cost of creating extra ```ByteBuffer``` is
      almost same to byte array copy (used to construct ```String```). Using large key results in better
      improvement but I don't think large key is common case.
      
      **BEFORE**
      ```
      Benchmark                                                                     (bufferSupplierStr)  (bytes)  (compressionType)  (headerKeySize)  (maxBatchSize)  (maxHeaderSize)  (messageSize)  (messageVersion)   Mode  Cnt        Score      Error   Units
      RecordBatchIterationBenchmark.measureValidation                                        NO_CACHING   RANDOM               NONE               10             200                5           1000                 2  thrpt   15  2035938.174 ± 1653.566   ops/s
      RecordBatchIterationBenchmark.measureValidation:·gc.alloc.rate.norm                    NO_CACHING   RANDOM               NONE               10             200                5           1000                 2  thrpt   15     2040.000 ±    0.001    B/op
      ```
      
      ```
      Benchmark                                                                     (bufferSupplierStr)  (bytes)  (compressionType)  (headerKeySize)  (maxBatchSize)  (maxHeaderSize)  (messageSize)  (messageVersion)   Mode  Cnt        Score      Error   Units
      RecordBatchIterationBenchmark.measureValidation                                        NO_CACHING   RANDOM               NONE               30             200                5           1000                 2  thrpt   15  1979193.376 ± 1239.286   ops/s
      RecordBatchIterationBenchmark.measureValidation:·gc.alloc.rate.norm                    NO_CACHING   RANDOM               NONE               30             200                5           1000                 2  thrpt   15     2120.000 ±    0.001    B/op
      ```
      
      
      **AFTER**
      
      ```
      Benchmark                                                                     (bufferSupplierStr)  (bytes)  (compressionType)  (headerKeySize)  (maxBatchSize)  (maxHeaderSize)  (messageSize)  (messageVersion)   Mode  Cnt        Score      Error   Units
      RecordBatchIterationBenchmark.measureValidation                                        NO_CACHING   RANDOM               NONE               10             200                5           1000                 2  thrpt   15  2289115.973 ± 2661.856   ops/s
      RecordBatchIterationBenchmark.measureValidation:·gc.alloc.rate.norm                    NO_CACHING   RANDOM               NONE               10             200                5           1000                 2  thrpt   15     2032.000 ±    0.001    B/op
      ```
      
      ```
      Benchmark                                                                     (bufferSupplierStr)  (bytes)  (compressionType)  (headerKeySize)  (maxBatchSize)  (maxHeaderSize)  (messageSize)  (messageVersion)   Mode  Cnt        Score     Error   Units
      RecordBatchIterationBenchmark.measureValidation                                        NO_CACHING   RANDOM               NONE               30             200                5           1000                 2  thrpt   15  2222625.706 ± 908.358   ops/s
      RecordBatchIterationBenchmark.measureValidation:·gc.alloc.rate.norm                    NO_CACHING   RANDOM               NONE               30             200                5           1000                 2  thrpt   15     2040.000 ±   0.001    B/op
      ```
      
      Reviewers: Ismael Juma <ismael@juma.me.uk>
      4b6d8da9
    • Ismael Juma's avatar
      MINOR: Replace Java 14 with Java 15 in the README (#9298) · aa0cd667
      Ismael Juma 创作于
      We have been testing with the Java 15 release candidate for a few weeks and
      it has now been declared final.
      
      Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, David Jacot <djacot@confluent.io>, Lee Dongjin <dongjin@apache.org>
      aa0cd667
  6. 9月 20, 2020
  7. 9月 19, 2020
    • Zach Zhang's avatar
      MINOR: Log warn message with details when there's kerberos login issue (#9236) · 2a27e0dd
      Zach Zhang 创作于
      Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>, Manikumar Reddy <manikumar.reddy@gmail.com>
      2a27e0dd
    • Tom Bentley's avatar
      MINOR: Generator config-specific HTML ids (#8878) · 59c1d4ec
      Tom Bentley 创作于
      Currently the docs have HTML ids for each config key. That doesn't work
      correctly for config keys like bootstrap.servers which occur across
      producer, consumer, admin configs: We generate duplicate ids. So arrange
      for each config to prefix the ids it generates with the HTML id of its
      section heading.
      
      Reviewers: Mickael Maison <mickael.maison@gmail.com>, Manikumar Reddy <manikumar.reddy@gmail.com>
      59c1d4ec
    • Luke Chen's avatar
      KAFKA-8098: fix the flaky test by disabling the auto commit to avoid member rejoining · 9a32ea9e
      Luke Chen 创作于
      In the test, we first test removing 1 member from group, and then test removing the other 2 members from group, and it failed sometimes at the 2nd member number assert. After investigation, I found it's because we enabled auto commit for the consumers(default setting), and the removed consumer offset commit will get the `UNKNOWN_MEMBER_ID` error, which will then make the member rejoin. (check ConsumerCoordinator#OffsetCommitResponseHandler) So, that's why after the 2nd members removing, the members will sometimes be not empty.
      
      I set the consumer config to disable the auto commit to fix this issue. Thanks.
      
      Author: Luke Chen <showuon@gmail.com>
      
      Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>
      
      Closes #9062 from showuon/KAFKA-8098
      9a32ea9e
  8. 9月 18, 2020
  9. 9月 17, 2020
  10. 9月 16, 2020
    • Bruno Cadonna's avatar
      MINOR: Add unit tests for StreamsRebalanceListener (#9258) · f28713f9
      Bruno Cadonna 创作于
      Reviewers: Walker Carlson <wcarlson@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
      f28713f9
    • Jason Gustafson's avatar
      KAFKA-10435; Fetch protocol changes for KIP-595 (#9275) · 634c9175
      Jason Gustafson 创作于
      This patch bumps the `Fetch` protocol as specified by KIP-595: https://cwiki.apache.org/confluence/display/KAFKA/KIP-595%3A+A+Raft+Protocol+for+the+Metadata+Quorum. The main differences are the following:
      
      - Truncation detection 
      - Leader discovery through the response
      - Flexible version support
      
      The most notable change is truncation detection. This patch adds logic in the request handling path to detect truncation, but it does not change the replica fetchers to make use of this capability. This will be done separately.
      
      Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>
      634c9175
    • Bruno Cadonna's avatar
      KAFKA-10292: Set min.insync.replicas to 1 of __consumer_offsets (#9286) · a46c07ec
      Bruno Cadonna 创作于
      The test StreamsBrokerBounceTest.test_all_brokers_bounce() fails on
      2.5 because in the last stage of the test there is only one broker
      left and the offset commit cannot succeed because the
      min.insync.replicas of __consumer_offsets is set to 2 and acks is
      set to all. This causes a time out and extends the closing of the
      Kafka Streams client to beyond the duration passed to the close
      method of the client.
      
      This affects especially the 2.5 branch since there Kafka Streams
      commits offsets for each task, i.e., close() needs to wait for the
      timeout for each task. In 2.6 and trunk the offset commit is done
      per thread, so close() does only need to wait for one time out per
      stream thread.
      
      I opened this PR on trunk, since the test could also become
      flaky on trunk and we want to avoid diverging system tests across
      branches.
      
      A more complete solution would be to improve the test by defining
      a better success criteria.
      
      Reviewers: Guozhang Wang <wangguoz@gmail.com>
      a46c07ec
  11. 9月 15, 2020
  12. 9月 14, 2020
    • Luke Chen's avatar
      KAFKA-8362: fix the old checkpoint won't be removed after alter log dir (#9178) · 77a0bba1
      Luke Chen 创作于
      In KIP-113, we support replicas movement between log directories. But while the directory change, we forgot to remove the topicPartition offset data in old directory, which will cause there are more than 1 checkpoint copy stayed in the logs for the altered topicPartition. And it'll let the LogCleaner get stuck due to it's possible to always get the old topicPartition offset data from the old checkpoint file.
      
      I added one more parameter topicPartitionToBeRemoved in updateCheckpoints() method. So, if the update parameter is None (as before), we'll do the remove action to remove the topicPartitionToBeRemoved data in dir, otherwise, update the data as before.
      
      Reviewers: Jun Rao <junrao@gmail.com>
      77a0bba1
    • Luke Chen's avatar
      update source link in interactive query page (#9261) · 77f6175d
      Luke Chen 创作于
      Currently, the source reference are all pointing to the 1.0 version codes,
      which is obviously wrong. Update to the current dotVersion.
      
      Reviewers: John Roesler <vvcephei@apache.org>
      77f6175d
    • Ismael Juma's avatar
      MINOR: Update junit to 5.7.0 (#9282) · 96f55027
      Ismael Juma 创作于
      The final release is now out:
      https://junit.org/junit5/docs/5.7.0/release-notes/index.html
      
      Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>
      96f55027
  13. 9月 12, 2020
  14. 9月 11, 2020
    • leah's avatar
      KAFKA-5636: Add Sliding Windows documentation (#9264) · 8260d7cd
      leah 创作于
      Add necessary documentation for KIP-450, adding sliding window aggregations to KStreams
      
      Reviewers: A. Sophie Blee-Goldman <sophie@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
      8260d7cd
    • Ismael Juma's avatar
      KAFKA-10447: Migrate tools module to JUnit 5 (#9231) · 7d0086e0
      Ismael Juma 创作于
      This change sets the groundwork for migrating other modules incrementally.
      
      Main changes:
      - Replace `junit` 4.13 with `junit-jupiter` and `junit-vintage` 5.7.0-RC1.
      - All modules except for `tools` depend on `junit-vintage`.
      - `tools` depends on `junit-jupiter`.
      - Convert `tools` tests to JUnit 5.
      - Update `PushHttpMetricsReporterTest` to use `mockito` instead of `powermock` and `easymock`
      (powermock doesn't seem to work well with JUnit 5 and we don't need it since mockito can mock
      static methods).
      - Update `mockito` to 3.5.7.
      - Update `TestUtils` to use JUnit 5 assertions since `tools` depends on it.
      
      Unrelated clean-ups:
      - Remove `unit` from package names in a few `core` tests.
      - Replace `try/catch/fail` with `assertThrows` in a number of places.
      - Tag `CoordinatorTest` as integration test.
      - Remove unnecessary type parameters when invoking methods and constructors.
      
      Tested with IntelliJ and gradle. Verified that the following commands work as expected:
      * ./gradlew tools:unitTest
      * ./gradlew tools:integrationTest
      * ./gradlew tools:test
      * ./gradlew core:unitTest
      * ./gradlew core:integrationTest
      * ./gradlew clients:test
      
      Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>
      7d0086e0
    • Guozhang Wang's avatar
      KAFKA-10134: Enable heartbeat during PrepareRebalance and Depend On State For Poll Timeout (#8834) · 7e7bb184
      Guozhang Wang 创作于
      1. Split the consumer coordinator's REBALANCING state into PREPARING_REBALANCE and COMPLETING_REBALANCE. The first is when the join group request is sent, and the second is after the join group response is received. During the first state we should still not send hb since it shares the same socket with the join group request and the group coordinator has disabled timeout, however when we transit to the second state we should start sending hb in case leader's assign takes long time. This is also for fixing KAFKA-10122.
      
      2. When deciding coordinator#timeToNextPoll, do not count in timeToNextHeartbeat if the state is in UNJOINED or PREPARING_REBALANCE since we would disable hb and hence its timer would not be updated.
      
      3. On the broker side, allow hb received during PREPARING_REBALANCE, return NONE error code instead of REBALANCE_IN_PROGRESS. However on client side, we still need to ignore REBALANCE_IN_PROGRESS if state is COMPLETING_REBALANCE in case it is talking to an old versioned broker.
      
      4. Piggy-backing a log4j improvement on the broker coordinator for triggering rebalance reason, as I found it a bit blurred during the investigation. Also subsumed #9038 with log4j improvements.
      
      The tricky part for allowing hb during COMPLETING_REBALANCE is in two parts: 1) before the sync-group response is received, a hb response may have reset the generation; also after the sync-group response but before the callback is triggered, a hb response can still reset the generation, we need to handle both cases by checking the generation / state. 2) with the hb thread enabled, the sync-group request may be sent by the hb thread even if the caller thread did not call poll yet.
      
      Reviewers: A. Sophie Blee-Goldman <sophie@confluent.io>, Boyang Chen <boyang@confluent.io>, John Roesler <john@confluent.io>
      7e7bb184
    • John Roesler's avatar
      MINOR: Add debug logs for StreamThread (#9267) · 5ee3ecb1
      John Roesler 创作于
      Add debug logs to see when Streams calls poll, process, commit, etc.
      
      Reviewers: Walker Carlson <wcarlson@confluent.io>, Guozhang Wang <guozhang@apache.org>
      5ee3ecb1
    • David Jacot's avatar
      MINOR: Address flaky `KafkaAdminClient` tests (#9091) · e4eab377
      David Jacot 创作于
      Fixes flakiness in `KafkaAdminClientTest` as a result of #8864. Addresses the following flaky tests:
      
      - testAlterReplicaLogDirsPartialFailure
      - testDescribeLogDirsPartialFailure
      - testMetadataRetries
      
      Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, Jason Gustafson <jason@confluent.io>
      e4eab377
    • Jason Gustafson's avatar
      MINOR: Fix JSON generation of nested structs with non-matching type/name (#9277) · e7f1cffd
      Jason Gustafson 创作于
      The schema specification allows a struct type name to differ from the field name. This works with the generated `Message` classes, but not with the generated JSON converter. The patch fixes the problem, which is that the type name is getting replaced with the field name when the struct is registered in the `StructRegistry`.
      
      Reviewers: Colin P. McCabe <cmccabe@apache.org>
      e7f1cffd
加载中