Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-10855

yarn logs cli fails to retrieve logs if any TFile is corrupt or empty

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.2.2, 2.10.1, 3.4.0, 3.3.1
    • 3.4.0
    • yarn
    • None

    Description

      When attempting to retrieve yarn logs via the CLI command, it failed with the following stack trace (on branch-2.10):

      yarn logs -applicationId application_1591017890475_1049740 > logs
      20/06/05 19:15:50 INFO client.RMProxy: Connecting to ResourceManager 
      20/06/05 19:15:51 INFO client.AHSProxy: Connecting to Application History server 
      Exception in thread "main" java.io.EOFException: Cannot seek to negative offset
      	at org.apache.hadoop.hdfs.DFSInputStream.seek(DFSInputStream.java:1701)
      	at org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:65)
      	at org.apache.hadoop.io.file.tfile.BCFile$Reader.<init>(BCFile.java:624)
      	at org.apache.hadoop.io.file.tfile.TFile$Reader.<init>(TFile.java:804)
      	at org.apache.hadoop.yarn.logaggregation.AggregatedLogFormat$LogReader.<init>(AggregatedLogFormat.java:503)
      	at org.apache.hadoop.yarn.logaggregation.LogCLIHelpers.dumpAllContainersLogs(LogCLIHelpers.java:227)
      	at org.apache.hadoop.yarn.client.cli.LogsCLI.run(LogsCLI.java:333)
      	at org.apache.hadoop.yarn.client.cli.LogsCLI.main(LogsCLI.java:367) 
      

      The problem was that there was a zero-length TFile for one of the containers in the application aggregated log directory in hdfs. When we removed the zero length file, yarn logs was able to retrieve the logs.

      A corrupt or zero length TFile for one container should not prevent loading logs for the rest of the application.

      Attachments

        1. YARN-10855.001.patch
          5 kB
          Jim Brennan
        2. YARN-10855.002.patch
          7 kB
          Jim Brennan
        3. YARN-10855.003.patch
          8 kB
          Jim Brennan

        Activity

          People

            jbrennan Jim Brennan
            jbrennan Jim Brennan
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: