Skip to content
代码片段 群组 项目
  1. 4月 08, 2020
  2. 4月 07, 2020
  3. 4月 06, 2020
  4. 4月 05, 2020
    • ConfluentSemaphore's avatar
    • Lucas Bradstreet's avatar
      KAFKA-9820: validateMessagesAndAssignOffsetsCompressed allocates unused iterator (#8422) (#1477) · 15b50960
      Lucas Bradstreet 创作于
      https://github.com/apache/kafka/commit/3e9d1c1411c5268de382f9dfcc95bdf66d0063a0 introduced skipKeyValueIterator(s) which were intended to be used, but in this case were created but were not used in offset validation.
      
      ce-kafka results after the benchmark was fixed to instantiate the BrokerTopicStats outside of the benchmark measurements. These show an ~85% improvement in CPU and a 50% improvement in allocations.
      
      ```
      Before:
      Benchmark                                                                     (bufferSupplierStr)  (bytes)  (compressionType)  (maxBatchSize)  (messageSize)  (messageVersion)   Mode  Cnt       Score      Error   Units
      RecordBatchIterationBenchmark.measureValidation                                        NO_CACHING   RANDOM                LZ4               1           1000                 2  thrpt   15  167739.973 ± 2856.103   ops/s
      RecordBatchIterationBenchmark.measureValidation:·gc.alloc.rate.norm                    NO_CACHING   RANDOM                LZ4               1           1000                 2  thrpt   15  137336.002 ±    0.002    B/op
      RecordBatchIterationBenchmark.measureValidation                                        NO_CACHING   RANDOM                LZ4               2           1000                 2  thrpt   15  171799.380 ±  496.011   ops/s
      RecordBatchIterationBenchmark.measureValidation:·gc.alloc.rate.norm                    NO_CACHING   RANDOM                LZ4               2           1000                 2  thrpt   15  137360.002 ±    0.002    B/op
      RecordBatchIterationBenchmark.measureValidation                                        NO_CACHING   RANDOM                LZ4              10           1000                 2  thrpt   15  172052.847 ±  365.869   ops/s
      RecordBatchIterationBenchmark.measureValidation:·gc.alloc.rate.norm                    NO_CACHING   RANDOM                LZ4              10           1000                 2  thrpt   15  137312.001 ±    0.002    B/op
      
      After:
      Benchmark                                                                     (bufferSupplierStr)  (bytes)  (compressionType)  (maxBatchSize)  (messageSize)  (messageVersion)   Mode  Cnt       Score      Error   Units
      RecordBatchIterationBenchmark.measureValidation                                        NO_CACHING   RANDOM                LZ4               1           1000                 2  thrpt   15  311084.661 ± 3645.225   ops/s
      RecordBatchIterationBenchmark.measureValidation:·gc.alloc.rate.norm                    NO_CACHING   RANDOM                LZ4               1           1000                 2  thrpt   15   69832.001 ±    0.001    B/op
      RecordBatchIterationBenchmark.measureValidation                                        NO_CACHING   RANDOM                LZ4               2           1000                 2  thrpt   15  298843.146 ± 9059.394   ops/s
      RecordBatchIterationBenchmark.measureValidation:·gc.alloc.rate.norm                    NO_CACHING   RANDOM                LZ4               2           1000                 2  thrpt   15   69832.001 ±    0.001    B/op
      RecordBatchIterationBenchmark.measureValidation                                        NO_CACHING   RANDOM                LZ4              10           1000                 2  thrpt   15  306849.740 ± 8257.428   ops/s
      RecordBatchIterationBenchmark.measureValidation:·gc.alloc.rate.norm                    NO_CACHING   RANDOM                LZ4              10           1000                 2  thrpt   15   69832.001 ±    0.001    B/op
      ```
      
      
      A subset of the benchmark results follow. Looks like a 20% improvement in validation performance and a 40% reduction in garbage allocation for 1-2 batch sizes.
      
      **# Parameters: (bufferSupplierStr = NO_CACHING, bytes = RANDOM, compressionType = LZ4, maxBatchSize = 1, messageSize = 1000, messageVersion = 2)**
      
      Before:
      Result "org.apache.kafka.jmh.record.RecordBatchIterationBenchmark.measureValidation":
        64851.837 ±(99.9%) 944.248 ops/s [Average]
        (min, avg, max) = (64505.317, 64851.837, 65114.359), stdev = 245.218
        CI (99.9%): [63907.589, 65796.084] (assumes normal distribution)
      
      "org.apache.kafka.jmh.record.RecordBatchIterationBenchmark.measureValidation:·gc.alloc.rate.norm":
        164088.003 ±(99.9%) 0.004 B/op [Average]
        (min, avg, max) = (164088.001, 164088.003, 164088.004), stdev = 0.001
        CI (99.9%): [164087.998, 164088.007] (assumes normal distribution)
      
      After:
      
      Result "org.apache.kafka.jmh.record.RecordBatchIterationBenchmark.measureValidation":
        78910.273 ±(99.9%) 707.024 ops/s [Average]
        (min, avg, max) = (78785.486, 78910.273, 79234.007), stdev = 183.612
        CI (99.9%): [78203.249, 79617.297] (assumes normal distribution)
      
      "org.apache.kafka.jmh.record.RecordBatchIterationBenchmark.measureValidation:·gc.alloc.rate.norm":
        96440.002 ±(99.9%) 0.001 B/op [Average]
        (min, avg, max) = (96440.002, 96440.002, 96440.002), stdev = 0.001
        CI (99.9%): [96440.002, 96440.003] (assumes normal distribution)
      
       **# Parameters: (bufferSupplierStr = NO_CACHING, bytes = RANDOM, compressionType = LZ4, maxBatchSize = 2, messageSize = 1000, messageVersion = 2)**
      
      Before:
      Result "org.apache.kafka.jmh.record.RecordBatchIterationBenchmark.measureValidation":
        64815.364 ±(99.9%) 639.309 ops/s [Average]
        (min, avg, max) = (64594.545, 64815.364, 64983.305), stdev = 166.026
        CI (99.9%): [64176.056, 65454.673] (assumes normal distribution)
                                                                                                                                                                                              "org.apache.kafka.jmh.record.RecordBatchIterationBenchmark.measureValidation:·gc.alloc.rate.norm":
        163944.003 ±(99.9%) 0.001 B/op [Average]
        (min, avg, max) = (163944.002, 163944.003, 163944.003), stdev = 0.001
        CI (99.9%): [163944.002, 163944.004] (assumes normal distribution)
      
      After:
      Result "org.apache.kafka.jmh.record.RecordBatchIterationBenchmark.measureValidation":
        77075.096 ±(99.9%) 201.092 ops/s [Average]
        (min, avg, max) = (77021.537, 77075.096, 77129.693), stdev = 52.223
        CI (99.9%): [76874.003, 77276.188] (assumes normal distribution)
      
      "org.apache.kafka.jmh.record.RecordBatchIterationBenchmark.measureValidation:·gc.alloc.rate.norm":
        96504.002 ±(99.9%) 0.003 B/op [Average]
        (min, avg, max) = (96504.001, 96504.002, 96504.003), stdev = 0.001
        CI (99.9%): [96503.999, 96504.005] (assumes normal distribution)
      
      Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, Ismael Juma <ismael@juma.me.uk>
      
      ce-kafka: added interceptor and interceptor stats arguments for validateMessagesAndAssignOffsetsCompressed
      
      Conflicts:
      	checkstyle/import-control-jmh-benchmarks.xml
      15b50960
  5. 4月 04, 2020
  6. 4月 03, 2020
加载中