Uploaded image for project: 'Accumulo'
  1. Accumulo
  2. ACCUMULO-2889

Batch metadata table updates for new walogs

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Incomplete
    • Affects Version/s: 1.5.1, 1.6.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Currently, when we update the Metadata table with new loggers, we will update the metadata for each tablet serially. We could optimize this to instead use a batchwriter to send all metadata updates for all tablets in a batch.

      A few special cases include:

      • What if the !METADATA tablet was included in the batch?
      • What about the root tablet?

      Benefit:
      In one of our clusters, we're experiencing particularly slow HDFS operations leading to large oscillations in ingest performance. We haven't isolated the cause in HDFS but when we profile the tservers, we noticed that they were waiting for metadata table operations to complete. This would target the waiting.

      Potential downsides:
      Given the existing locking scheme, it looks like we may have to lock a tablet for slightly longer (we'll lock for the duration of the batch).

        Attachments

        1. ACCUMULO-2889.0.patch.txt
          30 kB
          Jonathan Park
        2. run_all.sh
          0.3 kB
          Jonathan Park
        3. batch_perf_test.sh
          0.5 kB
          Jonathan Park
        4. start-ingest.sh
          2 kB
          Jonathan Park
        5. ACCUMULO-2889.1.patch
          35 kB
          Jonathan Park
        6. accumulo-2889-withpatch.png
          464 kB
          Jonathan Park
        7. accumulo-2889_withoutpatch.png
          475 kB
          Jonathan Park
        8. ACCUMULO-2889.2.patch
          33 kB
          Jonathan Park

          Issue Links

            Activity

              People

              • Assignee:
                parkjsung Jonathan Park
                Reporter:
                parkjsung Jonathan Park
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: