Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-2724

If an unreadable file is encountered during log aggregation then aggregated file in HDFS badly formed

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.5.1
    • 2.6.0
    • log-aggregation
    • None
    • Reviewed

    Description

      Look into the log output snippet. It looks like there is an issue during aggregation when an unreadable file is encountered. Likely, this results in bad encoding.

      LogType: command-13.json
      LogLength: 13934
      Log Contents:
      Error aggregating log file. Log file : /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_000004/command-13.json/grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_000004/command-13.json (Permission denied)command-3.json13983Error aggregating log file. Log file : /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_000004/command-3.json/grid/0/yarn/log/application_1413865041660_0002/contaierrors-13.txt0660_0002_01_000004/command-3.json (Permission denied)
                    errors-3.txt0gc.log-20141021044514484052014-10-21T04:45:12.046+0000: 5.134: [GC2014-10-21T04:45:12.046+0000: 5.134: [ParNew: 163840K->15575K(184320K), 0.0488700 secs] 163840K->15575K(1028096K), 0.0492510 secs] [Times: user=0.06 sys=0.01, real=0.05 secs]
      2014-10-21T04:45:14.939+0000: 8.027: [GC2014-10-21T04:45:14.939+0000: 8.027: [ParNew: 179415K->11865K(184320K), 0.0941310 secs] 179415K->17228K(1028096K), 0.0943140 secs] [Times: user=0.13 sys=0.04, real=0.09 secs]
      2014-10-21T04:46:42.099+0000: 95.187: [GC2014-10-21T04:46:42.099+0000: 95.187: [ParNew: 175705K->12802K(184320K), 0.0466420 secs] 181068K->18164K(1028096K), 0.0468490 secs] [Times: user=0.06 sys=0.00, real=0.04 secs]
      

      Specifically, look at the text after the exception text. There should be two more entries for log files but none exist. This is likely due to the fact that command-13.json is expected to be of length 13934 but its is not as the file was never read.

      I think, it should have been

      LogType: command-13.json
      LogLength: <Length of the exception text>
      Log Contents:
      Error aggregating log file. Log file : /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_000004/command-13.json/grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_000004/command-13.json (Permission denied)command-3.json13983Error aggregating log file. Log file : /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_000004/command-3.json/grid/0/yarn/log/application_1413865041660_0002/contaierrors-13.txt0660_0002_01_000004/command-3.json (Permission denied)
      
      LogType: errors-3.txt
      LogLength:0
      Log Contents:
      
      LogType:gc.log
      LogLength:???
      Log Contents:
      ......-20141021044514484052014-10-21T04:45:12.046+0000: 5.134: [GC2014-10-21T04:45:12.046+0000: 5.134: [ParNew: 163840K- .......
      

      Attachments

        1. YARN-2724.1.patch
          6 kB
          Xuan Gong
        2. YARN-2724.2.patch
          6 kB
          Xuan Gong
        3. YARN-2724.3.patch
          6 kB
          Xuan Gong
        4. YARN-2724.4.patch
          6 kB
          Xuan Gong
        5. YARN-2724.5.patch
          6 kB
          Xuan Gong

        Issue Links

          Activity

            People

              xgong Xuan Gong
              sumitmohanty Sumit Mohanty
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: