Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-20800

Master-orchestrated compactions

    XMLWordPrintableJSON

    Details

    • Type: Umbrella
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Compaction
    • Labels:
      None

      Description

      An umbrella issue for having compactions go via the Master so we can have a centralized arbitrator of cluster i/o. If we put Master in the way, we can do stuff like:

      • Ask the Master for current cluster compaction state; what is running, what is blocked
      • Master can manage cluster-wide compaction policy and/or throttling/or blocking of compaction i/os.
      • Master can schedule when and where compactions run so we can guard against the pathological where all RegionServers decide now is the time to major compact bringing on a compaction storm.

      Other side-benefits might include being able to farm out the compaction work to another process – e.g. the splice machine model of having spark run the compactions – or just to a separate compactor that we might i/o nice.

      • We'll need to figure how to externalize the CompactionRequest so it can be passed over RPC.
      • We'll need to have something like a CompactionManager in the Master process that keeps up current cluster state.

      MOB needs a compaction fabric it can use. Its compactions are currently Master-based only and so don't scale. It could make use of this mechanism to ask the Master to farm out its compaction requests.

      This is an umbrella issue. I thought I'd filed one already on this topic but can't find it. Will shut it down if I trip over it.

        Attachments

          Activity

            People

            • Assignee:
              mogoel Mohit Goel
              Reporter:
              stack Michael Stack
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated: