Kafka
  1. Kafka
  2. KAFKA-741

Improve log cleaning dedupe buffer efficiency

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: 0.8.1
    • Component/s: None
    • Labels:
      None

      Description

      Two good suggestions:
      1. Use a probing scheme to increase density without increasing the collision rate
      2. Only count unique updates to the offset map (i.e. if the key is all zero, don't count it) when computing the load. Dynamically choose the end offset based on when the map is full.

      Would be good to investigate these things.

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Resolved Resolved
          34d 20h 35m 1 Jay Kreps 04/Mar/13 23:55
          Tony Stevenson made changes -
          Workflow Apache Kafka Workflow [ 13052956 ] no-reopen-closed, patch-avail [ 13055523 ]
          Tony Stevenson made changes -
          Workflow no-reopen-closed, patch-avail [ 12748049 ] Apache Kafka Workflow [ 13052956 ]
          Jay Kreps made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Duplicate [ 3 ]
          Jay Kreps made changes -
          Link This issue duplicates KAFKA-739 [ KAFKA-739 ]
          Hide
          Jay Kreps added a comment -

          This issue is fixed in the patch for KAFKA-739. It removes the duplication using a probing scheme and counts updates correctly.

          Show
          Jay Kreps added a comment - This issue is fixed in the patch for KAFKA-739 . It removes the duplication using a probing scheme and counts updates correctly.
          Jay Kreps made changes -
          Field Original Value New Value
          Issue Type Bug [ 1 ] Improvement [ 4 ]
          Assignee Jay Kreps [ jkreps ]
          Fix Version/s 0.8.1 [ 12322960 ]
          Jay Kreps created issue -

            People

            • Assignee:
              Jay Kreps
              Reporter:
              Jay Kreps
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development