Uploaded image for project: 'Accumulo'
  1. Accumulo
  2. ACCUMULO-4021

bulk imports slow file garbage collection

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.6.3
    • Fix Version/s: 1.6.5, 1.7.1, 1.8.0
    • Component/s: gc
    • Labels:
      None
    • Environment:

      large production system

      Description

      On a large system, bulk imports slow file garbage collection to a crawl. The total number of files to be deleted was about 14 million. Initially, it would run quickly, but then slow down, to the point where only a few files would be deleted every few minutes. The jvm was only using 50% of the CPU (and therefore, probably not GC thrashing). JStacks showed the collector scanning the metadata table to remove referenced files from the delete list.

      If the bulk ingest requests were stopped, the GC completed quickly.

        Attachments

          Activity

            People

            • Assignee:
              ecn Eric C. Newton
              Reporter:
              ecn Eric C. Newton
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 0.5h
                0.5h