Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-9832

Runaway disk usage for ASAN test run

    XMLWordPrintableJSON

    Details

    • Target Version:
    • Epic Color:
      ghx-label-1

      Description

      A recent ASAN run saw hundreds of test failures due to the HDFS NameNode going into safe mode due to lack of disk space:

      2020-06-05 01:03:36,366 WARN org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space available on volume '/dev/nvme0n1p1' is 4419584, which is below the configured reserved amount 104857600
      2020-06-05 01:03:36,366 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on available disk space. Entering safe mode.

      Metrics about diskspace usage at the time show the device (which had a capacity of 300GB) getting to 99-100% disk usage. Previous runs on the same configuration usually stayed in the 35-40% range. The core job with the same commits also stayed in the 35-40% range.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              joemcdonnell Joe McDonnell
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: