Kafka
  1. Kafka
  2. KAFKA-188

Support multiple data directories

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: None
    • Labels:
      None

      Description

      Currently we allow only a single data directory. This means that a multi-disk configuration needs to be a RAID array or LVM volume or something like that to be mounted as a single directory.

      For a high-throughput low-reliability configuration this would mean RAID0 striping. Common wisdom in Hadoop land has it that a JBOD setup that just mounts each disk as a separate directory and does application-level balancing over these results in about 30% write-improvement. For example see this claim here:
      http://old.nabble.com/Re%3A-RAID-vs.-JBOD-p21466110.html

      It is not clear to me why this would be the case--it seems the RAID controller should be able to balance writes as well as the application so it may depend on the details of the setup.

      Nonetheless this would be really easy to implement, all you need to do is add multiple data directories and balance partition creation over these disks.

      One problem this might cause is if a particular topic is much larger than the others it might unbalance the load across the disks. The partition->disk assignment policy should probably attempt to evenly spread each topic to avoid this, rather than just trying keep the number of partitions balanced between disks.

      1. KAFKA-188-v8.patch
        55 kB
        Jay Kreps
      2. KAFKA-188-v7.patch
        50 kB
        Jay Kreps
      3. KAFKA-188-v6.patch
        49 kB
        Jay Kreps
      4. KAFKA-188-v5.patch
        49 kB
        Jay Kreps
      5. KAFKA-188-v4.patch
        42 kB
        Jay Kreps
      6. KAFKA-188-v3.patch
        42 kB
        Jay Kreps
      7. KAFKA-188-v2.patch
        42 kB
        Jay Kreps
      8. KAFKA-188.patch
        22 kB
        Jay Kreps

        Issue Links

          Activity

            People

            • Assignee:
              Jay Kreps
              Reporter:
              Jay Kreps
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development