Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.20.0
    • Component/s: io, regionserver
    • Labels:
      None

      Description

      Consider supporting:

      • 2GB store files
      • 1TB per node (500 store files)
      • Cell values up to ~100MB
      • Typical use case of RS running with 1GB of heap only

      Some ideas:

      • Drop MapFile and make a custom store file format with (competing) design goals:
        • heap efficiency
        • fast lookups
        • minimize I/O operations
        • optimize for typical DFS blocksizes (8MB, 64MB)
      • MRU cache for filehandles and store file indexes
      • Memory mapped store file indexes – don't hold the indexes in heap; rely on the OS blockcache for performance
      • "Zero copy" I/O from IPC to store file and vice versa, like NIO buffers

        Activity

          People

          • Assignee:
            Unassigned
            Reporter:
            Andrew Purtell
          • Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development