Skip to content
代码片段 群组 项目
派生自 oss-mirrors / ICV / kafka
源项目有可见性限制。
用户头像
Raman Verma 编辑于
KSTORAGE-724 Have TierTasks writing to online tier partitions make progress when few tier partitions are offline (#2374)

When one or more tier topic partitions are offline, TierArchiver and TierDeletionManager
reading / writing to these partitions will not be able to make progress. The corresponding transition
methods will get stuck until these tier topic partitions come back online.
The way TierTasks are scheduled currently, the scheduler will not call TierTasks#transition on any
other TierTasks till the stuck tasks make progress. This commit removes the limit on the number of
tasks that are scheduled at any given time from numThreads to MaxInt
Hence the thread pool will make progress on any TierTask that is not reading / writing to the offline
tier topic partitions.

Note that this will reduce the effectiveness of the existing archive "min lag" strategy https://confluentinc.atlassian.net/wiki/spaces/KSTORAGE/pages/878117031/One+Pager+Tier+Archiver+Strategy

, which attempts to maintain 0 lag on as many partitions as possible, even if it means effectively giving up on others. Given how severe it is for some partitions to stop making progress, we have deemed this tradeoff to be worthwhile.

Co-authored-by: default avatarLucas Bradstreet <lucas@confluent.io>
95e788de
历史
名称 最后提交 最后更新