KSTORAGE-724 Have TierTasks writing to online tier partitions make progress...
KSTORAGE-724 Have TierTasks writing to online tier partitions make progress when few tier partitions are offline (#2374) When one or more tier topic partitions are offline, TierArchiver and TierDeletionManager reading / writing to these partitions will not be able to make progress. The corresponding transition methods will get stuck until these tier topic partitions come back online. The way TierTasks are scheduled currently, the scheduler will not call TierTasks#transition on any other TierTasks till the stuck tasks make progress. This commit removes the limit on the number of tasks that are scheduled at any given time from numThreads to MaxInt Hence the thread pool will make progress on any TierTask that is not reading / writing to the offline tier topic partitions. Note that this will reduce the effectiveness of the existing archive "min lag" strategy https://confluentinc.atlassian.net/wiki/spaces/KSTORAGE/pages/878117031/One+Pager+Tier+Archiver+Strategy , which attempts to maintain 0 lag on as many partitions as possible, even if it means effectively giving up on others. Given how severe it is for some partitions to stop making progress, we have deemed this tradeoff to be worthwhile. Co-authored-by: Lucas Bradstreet <lucas@confluent.io>
显示
- core/src/main/java/kafka/tier/topic/TierTopicConsumer.java 1 个添加, 1 个删除core/src/main/java/kafka/tier/topic/TierTopicConsumer.java
- core/src/main/scala/kafka/tier/tasks/TierTasks.scala 9 个添加, 16 个删除core/src/main/scala/kafka/tier/tasks/TierTasks.scala
- core/src/main/scala/kafka/tier/tasks/archive/ArchiveTask.scala 5 个添加, 1 个删除...src/main/scala/kafka/tier/tasks/archive/ArchiveTask.scala
- core/src/test/scala/kafka/tier/tasks/archive/ArchiveTaskTest.scala 32 个添加, 4 个删除...test/scala/kafka/tier/tasks/archive/ArchiveTaskTest.scala
- core/src/test/scala/kafka/tier/topic/TierTopicPartitionOfflineTest.scala 170 个添加, 0 个删除...cala/kafka/tier/topic/TierTopicPartitionOfflineTest.scala
加载中
想要评论请 注册 或 登录