- 9月 16, 2020
-
-
由 David Jacot 创作于
- 9月 15, 2020
-
-
由 David Jacot 创作于
This PR fixes two issues that have been introduced by #9114. - When the metric was switched from Rate to TokenBucket in the ControllerMutationQuotaManager, the metrics were mixed up. That broke the quota update path. - When a quota is updated, the ClientQuotaManager updates the MetricConfig of the KafkaMetric. That update was not reflected into the Sensor so the Sensor was still using the MetricConfig that it has been created with. Reviewers: Anna Povzner <anna@confluent.io>, Rajini Sivaram <rajinisivaram@googlemail.com>
-
由 Alok Thatikunta 创作于
KSTORAGE-972: Modify TopicIdPartition.toString() implementation to use ':' and '-' as delimiters (#2553) * Modify toString method * Address PR comments * Fix write fencing file test method * Address PR comments
-
由 anatasiavela 创作于
The jackson dependencies should not be added to the clients. This is a rework of CNKAF-1138 to use the new generated json converter class instead to avoid adding the jackson dependencies on the clients.
-
由 Colin Patrick McCabe 创作于
For the generated message code, put the JSON conversion functionality in a separate JsonConverter class. Make MessageDataGenerator simply another generator class, alongside the new JsonConverterGenerator class. Move some of the utility functions from MessageDataGenerator into FieldSpec and other places, so that they can be used by other generator classes. Use argparse4j to support a better command-line for the generator.
- 9月 13, 2020
-
-
由 Lucas Bradstreet 创作于
This PR improves the validation of the retention stress test by starting a lagging consumer at the end of the test and which reads from the beginning of the topic. Initial timestamps for the lagging consumer are validated in the same way as they are periodically validated while writing.
- 9月 12, 2020
-
-
由 Kowshik Prakasam 创作于
-
由 Agam Brahma 创作于
* KC-883: Added config for new, large soak cluster topic. Bumped up soak-clients version Bumped up soak-clients version Bumped up soak-clients version * changes to source soak cluster topology * Add producer/consumer load for destination soak cluster. * Bump up version for helm package Co-authored-by: Agam Brahma <abrahma@confluent.io> Co-authored-by: Sanjana Kaundinya <skaundinya@gmail.com>
-
由 Jeff Kim 创作于
-
由 Soumyarka Mondal 创作于
* Migrated AuditJob away from KafkaScheduler Refactored AuditManager to inherit ShutdownableThread model reviewer: @rohitshekhar29
- 9月 11, 2020
-
-
由 Stanislav Kozlovski 创作于
This patch removes all uses of the DiskFailureDetector and MetricAnomalyDetector in SBK's anomaly detection code. Since we don't provide an interface to enable these, don't have plans to enable soon and are looking at refactoring the anomaly detection (self-healing) code, it makes sense to simplify it as much as possible first
-
由 Alok Thatikunta 创作于
* Validate partitions before fencing * Fix build failure and tests * Add integration test * Fix indentation issues * Add more integration tests and address comments * PR comments * Address PR comments * PR comments
-
由 Raman Verma 创作于
KSTORAGE-724 Have TierTasks writing to online tier partitions make progress when few tier partitions are offline (#2374) When one or more tier topic partitions are offline, TierArchiver and TierDeletionManager reading / writing to these partitions will not be able to make progress. The corresponding transition methods will get stuck until these tier topic partitions come back online. The way TierTasks are scheduled currently, the scheduler will not call TierTasks#transition on any other TierTasks till the stuck tasks make progress. This commit removes the limit on the number of tasks that are scheduled at any given time from numThreads to MaxInt Hence the thread pool will make progress on any TierTask that is not reading / writing to the offline tier topic partitions. Note that this will reduce the effectiveness of the existing archive "min lag" strategy https://confluentinc.atlassian.net/wiki/spaces/KSTORAGE/pages/878117031/One+Pager+Tier+Archiver+Strategy , which attempts to maintain 0 lag on as many partitions as possible, even if it means effectively giving up on others. Given how severe it is for some partitions to stop making progress, we have deemed this tradeoff to be worthwhile. Co-authored-by: Lucas Bradstreet <lucas@confluent.io>
-
由 Gardner Vickers 创作于
KSTORAGE-725: Exclude segments where the high offset is lower than the lowerBoundOffset for tiering. (#1796) * KSTORAGE-725: Only qualify segments where the last offset is larger than the upperBoundOffset for tiering. When dealing with topics which were previously compacted and have been converted to non compacted topics, holes in the log can cause MergedLog::localLogSegments to return segments for which the largest offset in the segment is less than the lowerBoundOffset. This PR adds an additional filter to the set of candidate segments for tiering which enforces that segments with a high offset less than the lower bound offset are excluded from tiering.
-
由 An Qi Wu 创作于
-
由 Kowshik Prakasam 创作于
* KSTORAGE-1075: Add more debug info for consumption tracking * Add max.partition.fetch.bytes config * Improvements
- 9月 10, 2020
-
-
由 Lucas Bradstreet 创作于
-
由 Tim Fox 创作于
-
由 Anastasia Vela 创作于
Currently request and response samples are outputted in a format that is JSON-like, so they aren't easily parseable. Once they're emitted as JSONs via the auto-generated schema, it'll be easier to load and parse them into other tools like druid and angle grinder. Included tests that ensure the outputted string is a parseable JSON. The following are some benchmark tests before and after the change respectively (will keep adding as I run more) BEFORE: ``` Benchmark (autoCreateTopic) (partitionCount) (topicCount) Mode Cnt Score Error Units MetadataRequestBenchmark.testRequestToJson true 10 500 avgt 15 638.312 ± 5.816 ns/op MetadataRequestBenchmark.testRequestToJson true 10 1000 avgt 15 659.132 ± 10.050 ns/op MetadataRequestBenchmark.testRequestToJson true 10 5000 avgt 15 633.611 ± 3.839 ns/op MetadataRequestBenchmark.testRequestToJson true 20 500 avgt 15 657.749 ± 18.178 ns/op MetadataRequestBenchmark.testRequestToJson true 20 1000 avgt 15 658.238 ± 1.981 ns/op MetadataRequestBenchmark.testRequestToJson true 20 5000 avgt 15 645.021 ± 1.231 ns/op MetadataRequestBenchmark.testRequestToJson true 50 500 avgt 15 655.577 ± 4.638 ns/op MetadataRequestBenchmark.testRequestToJson true 50 1000 avgt 15 625.718 ± 3.041 ns/op MetadataRequestBenchmark.testRequestToJson true 50 5000 avgt 15 588.551 ± 2.465 ns/op MetadataRequestBenchmark.testRequestToJson false 10 500 avgt 15 621.131 ± 5.757 ns/op MetadataRequestBenchmark.testRequestToJson false 10 1000 avgt 15 594.636 ± 1.447 ns/op MetadataRequestBenchmark.testRequestToJson false 10 5000 avgt 15 611.533 ± 0.716 ns/op MetadataRequestBenchmark.testRequestToJson false 20 500 avgt 15 633.064 ± 2.681 ns/op MetadataRequestBenchmark.testRequestToJson false 20 1000 avgt 15 625.159 ± 2.476 ns/op MetadataRequestBenchmark.testRequestToJson false 20 5000 avgt 15 634.090 ± 0.941 ns/op MetadataRequestBenchmark.testRequestToJson false 50 500 avgt 15 648.524 ± 4.881 ns/op MetadataRequestBenchmark.testRequestToJson false 50 1000 avgt 15 630.730 ± 2.502 ns/op MetadataRequestBenchmark.testRequestToJson false 50 5000 avgt 15 631.709 ± 2.584 ns/op ``` ``` Benchmark (partitionCount) (topicCount) Mode Cnt Score Error Units FetchRequestBenchmark.testRequestToJson 3 10 avgt 15 14438.239 ± 467.117 ns/op FetchRequestBenchmark.testRequestToJson 3 500 avgt 15 700629.960 ± 49586.125 ns/op FetchRequestBenchmark.testRequestToJson 3 1000 avgt 15 1442081.215 ± 75845.206 ns/op FetchRequestBenchmark.testRequestToJson 10 10 avgt 15 46798.987 ± 995.295 ns/op FetchRequestBenchmark.testRequestToJson 10 500 avgt 15 2519359.200 ± 118128.637 ns/op FetchRequestBenchmark.testRequestToJson 10 1000 avgt 15 4840100.254 ± 182686.228 ns/op FetchRequestBenchmark.testRequestToJson 20 10 avgt 15 90212.311 ± 664.419 ns/op FetchRequestBenchmark.testRequestToJson 20 500 avgt 15 4918544.705 ± 223759.389 ns/op FetchRequestBenchmark.testRequestToJson 20 1000 avgt 15 10009446.657 ± 381793.614 ns/op ``` ``` Benchmark (partitionCount) (topicCount) Mode Cnt Score Error Units ProduceRequestBenchmark.testRequestToJson 3 10 avgt 15 54.673 ± 0.152 ns/op ProduceRequestBenchmark.testRequestToJson 3 500 avgt 15 58.601 ± 0.252 ns/op ProduceRequestBenchmark.testRequestToJson 3 1000 avgt 15 59.004 ± 0.257 ns/op ProduceRequestBenchmark.testRequestToJson 10 10 avgt 15 55.197 ± 0.383 ns/op ProduceRequestBenchmark.testRequestToJson 10 500 avgt 15 58.182 ± 0.267 ns/op ProduceRequestBenchmark.testRequestToJson 10 1000 avgt 15 62.475 ± 0.158 ns/op ProduceRequestBenchmark.testRequestToJson 20 10 avgt 15 57.036 ± 0.133 ns/op ProduceRequestBenchmark.testRequestToJson 20 500 avgt 15 60.630 ± 0.558 ns/op ProduceRequestBenchmark.testRequestToJson 20 1000 avgt 15 59.172 ± 0.380 ns/op ``` ``` Benchmark (partitionCount) (topicCount) Mode Cnt Score Error Units ProduceRequestBenchmark.testRequestToJson 3 10 avgt 15 68.553 ± 1.481 ns/op ProduceRequestBenchmark.testRequestToJson 3 500 avgt 15 57.801 ± 0.116 ns/op ProduceRequestBenchmark.testRequestToJson 3 1000 avgt 15 57.402 ± 0.761 ns/op ProduceRequestBenchmark.testRequestToJson 10 10 avgt 15 56.241 ± 0.623 ns/op ProduceRequestBenchmark.testRequestToJson 10 500 avgt 15 56.714 ± 0.552 ns/op ProduceRequestBenchmark.testRequestToJson 10 1000 avgt 15 58.318 ± 0.583 ns/op ProduceRequestBenchmark.testRequestToJson 20 10 avgt 15 54.154 ± 0.181 ns/op ProduceRequestBenchmark.testRequestToJson 20 500 avgt 15 58.505 ± 0.113 ns/op ProduceRequestBenchmark.testRequestToJson 20 1000 avgt 15 59.857 ± 0.224 ns/op ``` AFTER: ``` Benchmark (autoCreateTopic) (partitionCount) (topicCount) Mode Cnt Score Error Units MetadataRequestBenchmark.testRequestToJson true 10 500 avgt 15 759.007 ± 5.810 ns/op MetadataRequestBenchmark.testRequestToJson true 10 1000 avgt 15 760.656 ± 1.718 ns/op MetadataRequestBenchmark.testRequestToJson true 10 5000 avgt 15 767.757 ± 3.772 ns/op MetadataRequestBenchmark.testRequestToJson true 20 500 avgt 15 743.840 ± 5.150 ns/op MetadataRequestBenchmark.testRequestToJson true 20 1000 avgt 15 743.902 ± 3.262 ns/op MetadataRequestBenchmark.testRequestToJson true 20 5000 avgt 15 743.548 ± 1.922 ns/op MetadataRequestBenchmark.testRequestToJson true 50 500 avgt 15 725.522 ± 1.401 ns/op MetadataRequestBenchmark.testRequestToJson true 50 1000 avgt 15 813.193 ± 5.587 ns/op MetadataRequestBenchmark.testRequestToJson true 50 5000 avgt 15 755.759 ± 5.628 ns/op MetadataRequestBenchmark.testRequestToJson false 10 500 avgt 15 758.947 ± 1.367 ns/op MetadataRequestBenchmark.testRequestToJson false 10 1000 avgt 15 746.381 ± 5.538 ns/op MetadataRequestBenchmark.testRequestToJson false 10 5000 avgt 15 764.001 ± 2.020 ns/op MetadataRequestBenchmark.testRequestToJson false 20 500 avgt 15 762.420 ± 3.982 ns/op MetadataRequestBenchmark.testRequestToJson false 20 1000 avgt 15 763.226 ± 4.754 ns/op MetadataRequestBenchmark.testRequestToJson false 20 5000 avgt 15 755.896 ± 3.370 ns/op MetadataRequestBenchmark.testRequestToJson false 50 500 avgt 15 750.817 ± 1.920 ns/op MetadataRequestBenchmark.testRequestToJson false 50 1000 avgt 15 743.267 ± 1.704 ns/op MetadataRequestBenchmark.testRequestToJson false 50 5000 avgt 15 767.195 ± 4.538 ns/op ``` ``` Benchmark (partitionCount) (topicCount) Mode Cnt Score Error Units FetchRequestBenchmark.testRequestToJson 3 10 avgt 15 15621.968 ± 311.178 ns/op FetchRequestBenchmark.testRequestToJson 3 500 avgt 15 979861.270 ± 4145.852 ns/op FetchRequestBenchmark.testRequestToJson 3 1000 avgt 15 2014514.490 ± 19232.641 ns/op FetchRequestBenchmark.testRequestToJson 10 10 avgt 15 51342.178 ± 664.855 ns/op FetchRequestBenchmark.testRequestToJson 10 500 avgt 15 3365754.881 ± 35635.229 ns/op FetchRequestBenchmark.testRequestToJson 10 1000 avgt 15 6878298.687 ± 149076.429 ns/op FetchRequestBenchmark.testRequestToJson 20 10 avgt 15 103642.356 ± 233.205 ns/op FetchRequestBenchmark.testRequestToJson 20 500 avgt 15 6741431.033 ± 34171.026 ns/op FetchRequestBenchmark.testRequestToJson 20 1000 avgt 15 14141536.794 ± 27466.924 ns/op ``` ``` Benchmark (partitionCount) (topicCount) Mode Cnt Score Error Units ProduceRequestBenchmark.testRequestToJson 3 10 avgt 15 394.597 ± 10.652 ns/op ProduceRequestBenchmark.testRequestToJson 3 500 avgt 15 397.004 ± 8.379 ns/op ProduceRequestBenchmark.testRequestToJson 3 1000 avgt 15 391.991 ± 6.674 ns/op ProduceRequestBenchmark.testRequestToJson 10 10 avgt 15 396.227 ± 3.773 ns/op ProduceRequestBenchmark.testRequestToJson 10 500 avgt 15 396.368 ± 8.007 ns/op ProduceRequestBenchmark.testRequestToJson 10 1000 avgt 15 404.346 ± 13.188 ns/op ProduceRequestBenchmark.testRequestToJson 20 10 avgt 15 393.117 ± 10.497 ns/op ProduceRequestBenchmark.testRequestToJson 20 500 avgt 15 396.988 ± 6.357 ns/op ProduceRequestBenchmark.testRequestToJson 20 1000 avgt 15 397.248 ± 1.790 ns/op ``` ``` Benchmark (partitionCount) (topicCount) Mode Cnt Score Error Units ListOffsetRequestBenchmark.testRequestToJson 3 10 avgt 15 341.671 ± 0.704 ns/op ListOffsetRequestBenchmark.testRequestToJson 3 500 avgt 15 322.988 ± 0.972 ns/op ListOffsetRequestBenchmark.testRequestToJson 3 1000 avgt 15 328.373 ± 0.589 ns/op ListOffsetRequestBenchmark.testRequestToJson 10 10 avgt 15 313.740 ± 0.848 ns/op ListOffsetRequestBenchmark.testRequestToJson 10 500 avgt 15 349.826 ± 0.358 ns/op ListOffsetRequestBenchmark.testRequestToJson 10 1000 avgt 15 348.513 ± 6.119 ns/op ListOffsetRequestBenchmark.testRequestToJson 20 10 avgt 15 339.147 ± 1.250 ns/op ListOffsetRequestBenchmark.testRequestToJson 20 500 avgt 15 315.552 ± 0.444 ns/op ListOffsetRequestBenchmark.testRequestToJson 20 1000 avgt 15 348.114 ± 0.780 ns/op ```
-
由 David Mao 创作于
The various TenantMetrics sensor classes can keep around references to expired Sensors resulting in dropped metrics. We should be aware of sensors potentially expiring and reinitialize them if necessary.
-
由 Soumyarka Mondal 创作于
* AuditJob reports durability lapse for relevant S3 error codes * Introduced time.sleep within AuditJob for throttled tierObjStore access * TierMetadataValidator uses read method to drain the buffer Relevant unit tests have been added, in addition to DEVEL cluster testing. reviewer: @rohitshekhar29