Skip to content
代码片段 群组 项目
  1. 3月 31, 2022
    • A. Sophie Blee-Goldman's avatar
      MINOR: log warning when topology override for cache size is non-zero (#11959) · 1317f3f7
      A. Sophie Blee-Goldman 创作于
      Since the topology-level cache size config only controls whether we disable the caching layer entirely for that topology, setting it to anything other than 0 has no effect. The actual cache memory is still just split evenly between the threads, and shared by all topologies.
      
      It's possible we'll want to change this in the future, but for now we should make sure to log a warning so that users who do try to set this override to some nonzero value are made aware that it doesn't work like this.
      
      Also includes some minor refactoring plus a fix for an off-by-one error in #11796
      
      Reviewers: Luke Chen <showuon@gmail.com>, Walker Carlson <wcarlson@confluent.io>, Sagar Rao <sagarmeansocean@gmail.com>
      1317f3f7
    • Jason Gustafson's avatar
      MINOR: Move `KafkaYammerMetrics` to server-common (#11970) · b2cb6caa
      Jason Gustafson 创作于
      With major server components like the new quorum controller being moved outside of the `core` module, it is useful to have shared dependencies moved into `server-common`. An example of this is Yammer metrics which server components still rely heavily upon. All server components should have access to the default registry used by the broker so that new metrics can be registered and metric naming conventions should be standardized. This is particularly important in KRaft where we are attempting to recreate identically named metrics in the controller context. This patch takes a step in this direction. It moves `KafkaYammerMetrics` into `server-common` and it implements
      standard metric naming utilities there. 
      
      Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>
      b2cb6caa
    • Konstantine Karantasis's avatar
      KAFKA-13748: Do not include file stream connectors in Connect's CLASSPATH and... · dd62ef2e
      Konstantine Karantasis 创作于
      KAFKA-13748: Do not include file stream connectors in Connect's CLASSPATH and plugin.path by default (#11908)
      
      With this change we stop including the non-production grade connectors that are meant to be used for demos and quick starts by default in the CLASSPATH and plugin.path of Connect deployments. The package of these connector will still be shipped with the Apache Kafka distribution and will be available for explicit inclusion. 
      
      The changes have been tested through the system tests and the existing unit and integration tests. 
      
      Reviewers: Mickael Maison <mickael.maison@gmail.com>, Randall Hauch <rhauch@gmail.com>
      dd62ef2e
    • Mike Lothian's avatar
      KAFKA-13660: Switch log4j12 to reload4j (#11743) · ecb0e8ee
      Mike Lothian 创作于
      
      This bumps the slf4j version to 1.7.36 and swaps out log4j 1.2.17 with
      reload4j 1.2.19
      
      Signed-off-by: default avatarMike Lothian <mike@fireburn.co.uk>
      
      Reviewers: Luke Chen <showuon@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Bruno Cadonna <cadonna@apache.org>
      ecb0e8ee
    • Edwin's avatar
      KAFKA-13775: CVE-2020-36518 - Upgrade jackson-databind to 2.12.6.1 (#11962) · 76ca62a3
      Edwin 创作于
      CVE-2020-36518 vulnerability affects jackson-databind (see GHSA-57j2-w4cx-62h2).
      
      Upgrading to jackson-databind version 2.12.6.1 addresses this CVE.
      
      Reviewers: Luke Chen <showuon@gmail.com>, Bruno Cadonna <cadonna@apache.org>
      76ca62a3
    • Edoardo Comar's avatar
      MINOR: Increase wait in ZooKeeperClientTest (#11973) · bb60eb86
      Edoardo Comar 创作于
      Increase wait in ZooKeeperClientTest.testReinitializeAfterAuthFailure
      so that the testcase of https://github.com/apache/kafka/pull/11563
      actually fails without the corresponding source code fix.
      Followup of https://issues.apache.org/jira/browse/KAFKA-13461
      
      .
      
      Co-Authored-By: default avatarGantigmaa Selenge <gantigmaa.selenge1@uk.ibm.com>
      Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>
      bb60eb86
  2. 3月 30, 2022
    • sunshujie1990's avatar
      KAFKA-13719: Fix connector restart cause duplicate tasks (#11869) · 8e205b50
      sunshujie1990 创作于
      
      
      Reviewers: Mickael Maison <mickael.maison@gmail.com>, Luke Chen <showuon@gmail.com>, Chris Egerton <fearthecellos@gmail.com>
      Co-authored-by: default avatarChris Egerton <fearthecellos@gmail.com>
      8e205b50
    • Bruno Cadonna's avatar
      Upgrade RocksDB from 6.27.3 to 6.29.4.1 (#11967) · 3904652b
      Bruno Cadonna 创作于
      RocksDB 6.27.3 does not run on arm64 M1 Macs which would prevent people on this platform to run Kafka Streams. Thus, this PR upgrades RocksDB to 6.29.4.1 which contains the following fix to allow to run RocksDB on arm64 M1 Macs:
      
      facebook/rocksdb#7720
      
      The source compatibility report between 6.27.3 and 6.29.4.1 (attached to the ticket) reports a couple of incompatibilities. However, the incompatibilities do not seem to affect Kafka Streams' backwards compatibility.
      
          The changes to class RocksDB only apply when inheriting from RocksDB. RocksDB is not exposed to users in Streams.
          The changes to class WriteBatch and class WriteBatchInterface also only apply with inheritance. Both classes are not exposed to users in Streams.
          -The change to enum SanityLevel seem also not to apply to Streams since SanityLevel is only used in ConfigOptions which is only used to load options from files and properties objects. Loading options from files or properties is not exposed to users in Streams.
      
      Reviewers: Bill Bejeck <bbejeck@apache.org>, Matthias J. Sax <mjsax@apache.org>, A. Sophie Blee-Goldman <ableegoldman@apache.org>
      3904652b
    • Ismael Juma's avatar
      KAFKA-13418: Support key updates with TLS 1.3 (#11966) · 5aed1780
      Ismael Juma 创作于
      Key updates with TLS 1.3 trigger code paths similar to renegotiation with TLS 1.2.
      Update the read/write paths not to throw an exception in this case (kept the exception
      in the `handshake` method).
      
      With the default configuration, key updates happen after 2^37 bytes are encrypted.
      There is a security property to adjust this configuration, but the change has to be
      done before it is used for the first time and it cannot be changed after that. As such,
      it is best done via a system test (filed KAFKA-13779).
      
      To validate the change, I wrote a unit test that forces key updates and manually ran
      a producer workload that produced more than 2^37 bytes. Both cases failed without
      these changes and pass with them.
      
      Note that Shylaja Kokoori attached a patch with the SslTransportLayer fix and hence
      included them as a co-author of this change.
      
      Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>
      
      Co-authored-by: Shylaja Kokoori
      5aed1780
  3. 3月 29, 2022
    • David Jacot's avatar
      MINOR: Disable SocketServerTest.closingChannelWithBufferedReceives and... · f2aa0c43
      David Jacot 创作于
      MINOR: Disable SocketServerTest.closingChannelWithBufferedReceives and SocketServerTest.remoteCloseWithoutBufferedReceives (#11960)
      
      This reverts commit d706d6ca.
      
      Reviewers: Bruno Cadonna <cadonna@apache.org>
      f2aa0c43
    • Levani Kokhreidze's avatar
      KAFKA-6718: Add documentation for KIP-708 (#11923) · 35ae4f24
      Levani Kokhreidze 创作于
      
      Adds documentation for KIP-708: Rack awareness for Kafka Streams
      
      Co-authored-by: default avatarBruno Cadonna <cadonna@apache.org>
      
      Reviewers: Luke Chen <showuon@gmail.com>, Bruno Cadonna <cadonna@apache.org>
      35ae4f24
    • bozhao12's avatar
      KAFKA-13767; Fetch from consumers should return immediately when preferred... · db2485cb
      bozhao12 创作于
      KAFKA-13767; Fetch from consumers should return immediately when preferred read replica is defined by the leader (#11942)
      
      When a replica selector is configured, the partition leader computes a preferred read replica for any fetch from the consumers. When the preferred read replica is not the leader, the leader returns the preferred read replica with `FetchDataInfo(LogOffsetMetadata.UnknownOffsetMetadata, MemoryRecords.EMPTY)` to the `ReplicaManager`. This causes the fetch to go into in the fetch purgatory because the exit conditions are not met. In turns out that the delayed fetch is not completed until the timeout is reached because the delayed fetch ignores partition with an unknown offset (-1). If the fetch contains only one partition, the fetch is unnecessarily delayed by the timeout time (500ms by default) to only inform the consumer that it has to read from a follower.
      
      This patch fixes the issue by completing the fetch request immediately when a preferred read replica is defined.
      
      Reviewers: David Jacot <djacot@confluent.io>
      db2485cb
    • Guozhang Wang's avatar
      MINOR: Fix log4j entry in RepartitionTopics (#11958) · 19a62697
      Guozhang Wang 创作于
      I noticed two issues in the log4j entry:
      
      1. It's formatted as "{}...{}" + param1, param2; effectively it is one param only, and the printed line is effectively mis-aligned: we always print Subtopology [sourceTopics set] was missing source topics {}
      2. Even fix 1) is not enough, since topologyName may be null. On the other hand I think the original goal is not to print the topology name but the sub-topology id since it's within the per-sub-topology loop.
      
      Reviewers: Guozhang Wang <wangguoz@gmail.com>
      19a62697
    • Sanjana Kaundinya's avatar
      MINOR: Improved display names for parameterized KRaft and ZK tests (#11957) · 2a270591
      Sanjana Kaundinya 创作于
      This patch adds display names for KRaft and ZK tests. Without this, it becomes hard to understand in Jenkins test reports which test failed. With this addition, it becomes more clear which method in the test suite fails.
      
      Reviewers: Colin P. McCabe <cmccabe@apache.org>
      2a270591
    • John Roesler's avatar
      MINOR: Fix stream-join metadata (#11952) · 7243facb
      John Roesler 创作于
      #11356 inadvertently changed
      the (undefined) header forwarding behavior of stream-stream joins.
      
      This change does not define the behavior, but just restores the prior
      undefined behavior for continuity's sake. Defining the header-forwarding
      behavior is future work.
      
      Reviewers: Matthias J. Sax <mjsax@apache.org>, Jorge Esteban Quilcate Otoya <quilcate.jorge@gmail.com>
      7243facb
  4. 3月 28, 2022
    • Tim Patterson's avatar
      KAFKA-13600: Kafka Streams - Fall back to most caught up client if no caught... · 110bccac
      Tim Patterson 创作于
      KAFKA-13600: Kafka Streams - Fall back to most caught up client if no caught up clients exist (#11760)
      
      The task assignor is modified to consider the Streams client with the most caught up states if no Streams client exists that is caught up, i.e., the lag of the states on that client is less than the acceptable recovery lag.  
      
      Unit test for case task assignment where no caught up nodes exist.
      Existing unit and integration tests to verify no other behaviour has been changed
      
      Co-authored-by: default avatarBruno Cadonna <cadonna@apache.org>
      
      Reviewer: Bruno Cadonna <cadonna@apache.org>
      110bccac
  5. 3月 27, 2022
  6. 3月 26, 2022
    • David Jacot's avatar
      MINOR: Renable SocketServerTest.closingChannelWithBufferedReceives and... · d706d6ca
      David Jacot 创作于
      MINOR: Renable SocketServerTest.closingChannelWithBufferedReceives and SocketServerTest.remoteCloseWithoutBufferedReceives (#11927)
      
      Reviewers: Guozhang Wang <wangguoz@gmail.com>
      d706d6ca
    • Luke Chen's avatar
      KAFKA-4801: don't verify assignment during broker up and down in... · 0461382a
      Luke Chen 创作于
      KAFKA-4801: don't verify assignment during broker up and down in testConsumptionWithBrokerFailures (#11949)
      
      In this test, we have another thread to let broker down and up, to test if consumer can still work as expected. During the broker down and up, we tried to verify the assignment is as what we expected. But the rebalance will keep triggering while broker down and up. It doesn't make sense to verify the assignment here. Remove it to make the test reliable.
      
      Reviewers: Guozhang Wang <wangguoz@gmail.com>
      0461382a
    • Luke Chen's avatar
      KAFKA-10405: Set purge interval explicitly in PurgeRepartitionTopicIntegrationTest (#11948) · 0586f544
      Luke Chen 创作于
      In KIP-811, we added a new config repartition.purge.interval.ms to set repartition purge interval. In this flaky test, we expected the purge interval is the same as commit interval, which is not correct anymore (default is 30 sec). Set the purge interval explicitly to fix this issue.
      
      Reviewers: Bruno Cadonna <cadonna@apache.org>, Guozhang Wang <wangguoz@gmail.com>
      0586f544
  7. 3月 25, 2022
    • Randall Hauch's avatar
      KAFKA-13770: Restore compatibility with KafkaBasedLog using older Kafka brokers (#11946) · ce883892
      Randall Hauch 创作于
      The `retryEndOffsets(…)` method in `TopicAdmin` recently added (KAFKA-12879, #11797) to allow the `KafkaBasedLog.start()` method to retry any failures reading the last offsets for a topic. However, this introduce a regression when talking to older brokers (0.10.x or earlier).
      
      The `KafkaBasedLog` already had logic that expected an `UnsupportedVersionException` thrown by the admin client when a Kafka API is not available on an older broker, but the new retry logic in `TopicAdmin` did not account for this and wrapped the exception, thereby breaking the `KafkaBasedLog` logic and preventing startup.
      
      The fix is to propagate this `UnsupportedVersionException` from the `TopicAdmin.retryEndOffsets(…)` method. Added a new unit test that first replicated the problem before the fix, and verified the fix corrects the problem.
      ce883892
    • John Roesler's avatar
      MINOR: Add extra notice about IQv2 compatibility (#11944) · 46df7ee9
      John Roesler 创作于
      Added an extra notice about IQv2's API compatibility, as discussed in the KIP-796 vote thread.
      
      Reviewers: Bill Bejeck <bbejeck@apache.org>, @Kvicii 
      46df7ee9
    • Rohan's avatar
      KAFKA-13692: include metadata wait time in total blocked time (#11805) · 01533e3d
      Rohan 创作于
      This patch includes metadata wait time in total blocked time. First, this patch adds a new metric for total producer time spent waiting on metadata, called metadata-wait-time-ms-total. Then, this time is included in the total blocked time computed from StreamsProducer.
      
      Reviewers: Guozhang Wang <wangguoz@gmail.com>
      01533e3d
    • RivenSun's avatar
      KAFKA-13689: optimize the log output of logUnused method (#11940) · c5bc2688
      RivenSun 创作于
      Optimize the log output of logUnused method.
      
      Reviewers: Guozhang Wang <wangguoz@gmail.com>
      c5bc2688
  8. 3月 24, 2022
    • Yang Yu's avatar
      KAFKA-13761: KafkaLog4jAppender deadlocks when idempotence is enabled (#11939) · db724f23
      Yang Yu 创作于
      When a log entry is appended to a Kafka topic using KafkaLog4jAppender, the producer.send operation
      may hit a deadlock if the producer network thread also tries to append a log at the same log level.
      This issue is triggered when idempotence is enabled for the KafkaLog4jAppender and the producer
      tries to acquire the TransactionManager lock.
      
      This is a temporary workaround to avoid deadlocks by disabling idempotence explicitly in
      KafkaLog4jAppender.
      
      Reviewers: Luke Chen <showuon@gmail.com>, Ismael Juma <ismael@juma.me.uk>
      db724f23
    • David Jacot's avatar
      MINOR: A few cleanups in BrokerToControllerChannelManager (#11937) · 12bb2315
      David Jacot 创作于
      Make the code style more consistent
      
      Reviewers: Luke Chen <showuon@gmail.com>
      12bb2315
    • Liam Clarke-Hutchinson's avatar
      KAFKA-13672: Race condition in DynamicBrokerConfig (#11920) · e8f09007
      Liam Clarke-Hutchinson 创作于
      Reviewers: David Jacot <djacot@confluent.io>, Luke Chen <showuon@gmail.com>
      e8f09007
    • John Roesler's avatar
      KAFKA-13714: Fix cache flush position (#11926) · 322a065b
      John Roesler 创作于
      The caching store layers were passing down writes into lower store layers upon eviction, but not setting the context to the evicted records' context. Instead, the context was from whatever unrelated record was being processed at the time.
      
      Reviewers: Matthias J. Sax <mjsax@apache.org>
      322a065b
    • Konstantine Karantasis's avatar
      KAFKA-13759: Disable idempotence by default in producers instantiated by Connect (#11933) · 6ce69021
      Konstantine Karantasis 创作于
      With AK 3.0, idempotence was enabled by default in Kafka producers. However, if idempotence is enabled, Connect won't be able to communicate via its producers with Kafka brokers older than version 0.11. Perhaps more importantly, for brokers older than version 2.8 the IDEMPOTENT_WRITE ACL is required to be granted to the principal of the Connect worker.
      
      Therefore this commit disables producer idempotence by default to all the producers instantiated by Connect. Users can still choose to enable producer idempotence by explicitly setting the right worker and/or connector properties.
      
      The changes were tested via existing unit, integration and system tests.
      
      Reviewers: Randall Hauch <rhauch@gmail.com>
      6ce69021
    • John Roesler's avatar
      MINOR: Clarify how to publish specific projects to the local repo (#11938) · dcd09de1
      John Roesler 创作于
      The current README instruction for local publishing boils the ocean by building and installing every jar in the project with both 2.12 and 2.13. While that is some times what people want to do, they are also often trying to just build a specific jar.
      
      Reviewers: Bill Bejeck <bbejeck@apache.org>
      dcd09de1
  9. 3月 23, 2022
  10. 3月 22, 2022
  11. 3月 21, 2022
加载中