Skip to content
代码片段 群组 项目
  1. 6月 17, 2020
  2. 3月 03, 2020
  3. 3月 01, 2020
    • Confluent Jenkins Bot's avatar
    • Brian Bushree's avatar
      MINOR: add wait_for_assigned_partitions to console-consumer (#8192) · bd4feef3
      Brian Bushree 创作于
      what/why
      the throttling_test was broken by this PR (#7785) since it depends on the consumer having partitions-assigned before starting the producer
      
      this PR provides the ability to wait for partitions to be assigned in the console consumer before considering it started.
      
      caveat
      this does not support starting up the JmxTool inside the console-consumer for custom metrics while using this wait_until_partitions_assigned flag since the code assumes one JmxTool running per node.
      
      I think a proper fix for this would be to make JmxTool its own standalone single-node service
      
      alternatives
      we could use the EndToEnd test suite which uses the verifiable producer/consumer under the hood but I found that there were more changes necessary to get this working unfortunately (specifically doesn't seem like this test suite plays nicely with the ProducerPerformanceService)
      
      Reviewers: Mathew Wong <mwong@confluent.io>, Bill Bejeck <bbejeck.com>
      bd4feef3
  4. 2月 28, 2020
    • Bill Bejeck's avatar
      MINOR: Update upgrade guide for ZK (#8182) · a696296f
      Bill Bejeck 创作于
      Adding the ZK upgrade information to the Notable Changes section for 2.4.1
      
      Reviewers: Ron Dagostino <rdagostino@confluent.io>, Jim Galasyn <jim.galasyn@confluent.io>
      a696296f
    • Confluent Jenkins Bot's avatar
    • Matthew Wong's avatar
      throttle consumer timeout increase (#8188) · e60eb69a
      Matthew Wong 创作于
      The test_throttled_reassignment test fails because the consumer that is used to validate reassignment does not start on time to consume all messages. This does not seem like an issue with the throttling of the reassignment, since increasing the timeout allowed the test to pass multiple consecutive runs locally.
      
      This test seemed to rely on the default JmxTool for the console consumer that was removed in this commit: 179d0d73
      The console consumer would check to see if it had partitions assigned to it before beginning to consume. Although the test occasionally failed with the JmxTool, it began to fail much more after the removal.
      
      Error messages of failures followed the below format with varying numbers of missed messages. They are the first messages by the producer.
      
      535 acked message did not make it to the Consumer. They are: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19...plus 515 more. Total Acked: 192792, Total Consumed: 192259. We validated that the first 535 of these missing messages correctly made it into Kafka's data files. This suggests they were lost on their way to the consumer.
      In the scope of the test, this error suggests that the test is falling into the race condition described in produce_consume_validate.py, which has the timeout to prevent the consumer from missing initial messages.
      
      This can serve as a temporary fix until the logic of consumer startup is addressed further.
      
      Reviewers: Jason Gustafson <jason@confluent.io>, Bill Bejeck <bbejeck@gmail.com>
      e60eb69a
  5. 2月 27, 2020
  6. 2月 26, 2020
  7. 2月 25, 2020
  8. 2月 22, 2020
  9. 2月 21, 2020
    • Confluent Jenkins Bot's avatar
    • David Mao's avatar
      KAFKA-6266: Repeated occurrence of WARN Resetting first dirty offset … (#8136) · ce4aa9c0
      David Mao 创作于
      Previously, checkpointed offsets for a log were only updated if the log was chosen for cleaning once the cleaning job completes. This caused issues in cases where logs with invalid checkpointed offsets would repeatedly emit warnings if the log with an invalid cleaning checkpoint wasn't chosen for cleaning.
      
      Proposed fix is to update the checkpointed offset for logs with invalid checkpoints regardless of whether it gets chosen for cleaning.
      
      Reviewers: Anna Povzner <anna@confluent.io>, Jun Rao <junrao@gmail.com>
      ce4aa9c0
    • Michael Viamari's avatar
      KAFKA-9533: ValueTransform forwards `null` values (#8108) · 5e0d6e8d
      Michael Viamari 创作于
      Fixes a bug where KStream#transformValues would forward null values from the provided ValueTransform#transform operation.
      
      A test was added for verification KStreamTransformValuesTest#shouldEmitNoRecordIfTransformReturnsNull. A parallel test for non-key ValueTransformer was not added, as they share the same code path.
      
      Reviewers: Bruno Cadonna <bruno@confluent.io>, Bill Bejeck <bbejeck@gmail.com>
      5e0d6e8d
  10. 2月 19, 2020
  11. 2月 18, 2020
  12. 2月 17, 2020
    • Boyang Chen's avatar
      KAFKA-9535; Update metadata before retrying partitions when fetching offsets (#8088) · 667ec2dd
      Boyang Chen 创作于
      Today if we attempt to list offsets with a fenced leader epoch, consumer will retry without updating the metadata until the timeout is reached. This affects synchronous APIs such as `offsetsForTimes`, `beginningOffsets`, and `endOffsets`. The fix in this patch is to trigger the metadata update call whenever we see a retriable error before additional attempts.
      
      Reviewers: Jason Gustafson <jason@confluent.io>
      667ec2dd
  13. 2月 14, 2020
  14. 2月 13, 2020
  15. 2月 12, 2020
    • John Roesler's avatar
      KAFKA-9500: Fix FK Join Topology (#8015) · 62233f10
      John Roesler 创作于
      Corrects a flaw leading to an exception while building topologies that include both:
      
      * A foreign-key join with the result not explicitly materialized
      * An operation after the join that requires source materialization
      
      Also corrects a flaw in TopologyTestDriver leading to output records being enqueued in the wrong order under some (presumably rare) circumstances.
      
      Cherry-pick of 1681c78f from trunk
      Cherry-pick of 1679839c from 2.5
      
      Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
      62233f10
    • John Roesler's avatar
      KAKFA-9503: Fix TopologyTestDriver output order (#8065) · 42062a89
      John Roesler 创作于
      Migrates TopologyTestDriver processing to be closer to the same processing/ordering
      semantics as KafkaStreams. This corrects the output order for recursive topologies
      as reported in KAFKA-9503, and also works similarly in the case of task idling.
      
      Cherry-pick of 998f1520 from trunk
      Cherry-pick of 7b71cb92 from 2.5
      
      Reviewers: Matthias J. Sax <matthias@confluent.io>
      42062a89
    • John Roesler's avatar
      KAFKA-9390: Make serde pseudo-topics unique (#8054) · 4a0859ab
      John Roesler 创作于
      During the discussion for KIP-213, we decided to pass "pseudo-topics"
      to the internal serdes we use to construct the wrapper serdes for
      CombinedKey and hashing the left-hand-side value. However, during
      the implementation, this strategy wasn't fully implemented, and we wound
      up using the same topic name for a few different data types.
      
      Cherry-pick of e16859dc from trunk
      Cherry-pick of 557ad551 from 2.5
      
      Reviewers: Guozhang Wang <guozhang@confluent.io>
      4a0859ab
    • Konstantine Karantasis's avatar
      MINOR: Start using Response and replace IOException in EmbeddedConnectCluster for failures (#8055) · 92ce678a
      Konstantine Karantasis 创作于
      Changed `EmbeddedConnectCluster` to add utility methods that return `Response`, throw `ConnectException` instead of `IOException` for failures, and deprecate the old methods that returned primitive types rather than `Response`. 
      
      Also introduce common assertions for embedded clusters under `EmbeddedConnectClusterAssertions`.
      
      Author: Konstantine Karantasis <konstantine@confluent.io>
      Reviewer: Randall Hauch <rhauch@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
      92ce678a
    • A. Sophie Blee-Goldman's avatar
      KAFKA-9540: Move "Could not find the standby task while closing it" log to debug level (#8092) · 2b3a5379
      A. Sophie Blee-Goldman 创作于
      Reviewers: Guozhang Wang <wangguoz@gmail.com>
      2b3a5379
加载中