Skip to content
代码片段 群组 项目
未验证 提交 95e788de 编辑于 作者: Raman Verma's avatar Raman Verma 提交者: GitHub
浏览文件

KSTORAGE-724 Have TierTasks writing to online tier partitions make progress...

KSTORAGE-724 Have TierTasks writing to online tier partitions make progress when few tier partitions are offline (#2374)

When one or more tier topic partitions are offline, TierArchiver and TierDeletionManager
reading / writing to these partitions will not be able to make progress. The corresponding transition
methods will get stuck until these tier topic partitions come back online.
The way TierTasks are scheduled currently, the scheduler will not call TierTasks#transition on any
other TierTasks till the stuck tasks make progress. This commit removes the limit on the number of
tasks that are scheduled at any given time from numThreads to MaxInt
Hence the thread pool will make progress on any TierTask that is not reading / writing to the offline
tier topic partitions.

Note that this will reduce the effectiveness of the existing archive "min lag" strategy https://confluentinc.atlassian.net/wiki/spaces/KSTORAGE/pages/878117031/One+Pager+Tier+Archiver+Strategy

, which attempts to maintain 0 lag on as many partitions as possible, even if it means effectively giving up on others. Given how severe it is for some partitions to stop making progress, we have deemed this tradeoff to be worthwhile.

Co-authored-by: default avatarLucas Bradstreet <lucas@confluent.io>
上级 255e0713
加载中
加载中
0% 加载中 .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑!
想要评论请 注册