Uploaded image for project: 'Accumulo'
  1. Accumulo
  2. ACCUMULO-3259

Randomwalk Bulk.xml create queued threads without bound.

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.5.2, 1.6.1
    • 1.6.2, 1.7.0
    • test
    • None

    Description

      While looking at ACCUMULO-3257, I saw that Bulk.xml really goes crazy creating new threads of bulk-imported related work without respect to the actual available resources.

      For example, on a single node after about 5 minutes, I can see a backlog of about 600+ tasks.

      This doesn't really hurt things server-side because the threadpool running these tasks is bounded, but it unnecessarily increases the load on the test runner. We can estimate the "throughput" of the system based on the number of tservers available, skip the queueing of new tasks when we have an excessive number already queued (relative to the number of tservers available), and then wait a bit before we continue to the next node in Bulk.xml.

      Attachments

        Activity

          People

            elserj Josh Elser
            elserj Josh Elser
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 20m
                20m