Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-8273

Log aggregation does not warn if HDFS quota in target directory is exceeded

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.1.0
    • Fix Version/s: 3.2.0
    • Component/s: log-aggregation
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      It appears that if an HDFS space quota is set on a target directory for log aggregation and the quota is already exceeded when log aggregation is attempted, zero-byte log files will be written to the HDFS directory, however NodeManager logs do not reflect a failure to write the files successfully (i.e. there are no ERROR or WARN messages to this effect).

      An improvement may be worth investigating to alert users to this scenario, as otherwise logs for a YARN application may be missing both on HDFS and locally (after local log cleanup is done) and the user may not otherwise be informed.

      Steps to reproduce:

      • Set a small HDFS space quota on /tmp/logs/username/logs (e.g. 2MB)
      • Write files to HDFS such that /tmp/logs/username/logs is almost 2MB full
      • Run a Spark or MR job in the cluster
      • Observe that zero byte files are written to HDFS after job completion
      • Observe that YARN container logs are also not present on the NM hosts (or are deleted after yarn.nodemanager.delete.debug-delay-sec)
      • Observe that no ERROR or WARN messages appear to be logged in the NM role log

        Attachments

        1. YARN-8273.000.patch
          19 kB
          Gergo Repas
        2. YARN-8273.001.patch
          17 kB
          Gergo Repas
        3. YARN-8273.002.patch
          18 kB
          Gergo Repas
        4. YARN-8273.003.patch
          22 kB
          Gergo Repas
        5. YARN-8273.004.patch
          24 kB
          Gergo Repas
        6. YARN-8273.005.patch
          24 kB
          Gergo Repas
        7. YARN-8273.006.patch
          24 kB
          Gergo Repas

          Issue Links

            Activity

              People

              • Assignee:
                grepas Gergo Repas
                Reporter:
                grepas Gergo Repas
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: