Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-1670

aggregated log writer can write more log data then it says is the log length

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 0.23.10, 2.2.0, 3.0.0-alpha1
    • 0.23.11, 2.4.0
    • None
    • None

    Description

      We have seen exceptions when using 'yarn logs' to read log files.
      at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
      at java.lang.Long.parseLong(Long.java:441)
      at java.lang.Long.parseLong(Long.java:483)
      at org.apache.hadoop.yarn.logaggregation.AggregatedLogFormat$LogReader.readAContainerLogsForALogType(AggregatedLogFormat.java:518)
      at org.apache.hadoop.yarn.logaggregation.LogDumper.dumpAContainerLogs(LogDumper.java:178)
      at org.apache.hadoop.yarn.logaggregation.LogDumper.run(LogDumper.java:130)
      at org.apache.hadoop.yarn.logaggregation.LogDumper.main(LogDumper.java:246)

      We traced it down to the reader trying to read the file type of the next file but where it reads is still log data from the previous file. What happened was the Log Length was written as a certain size but the log data was actually longer then that.

      Inside of the write() routine in LogValue it first writes what the logfile length is, but then when it goes to write the log itself it just goes to the end of the file. There is a race condition here where if someone is still writing to the file when it goes to be aggregated the length written could be to small.

      We should have the write() routine stop when it writes whatever it said was the length. It would be nice if we could somehow tell the user it might be truncated but I'm not sure of a good way to do this.

      We also noticed that a bug in readAContainerLogsForALogType where it is using an int for curRead whereas it should be using a long.

      while (len != -1 && curRead < fileLength) {

      This isn't actually a problem right now as it looks like the underlying decoder is doing the right thing and the len condition exits.

      Attachments

        1. YARN-1670.patch
          2 kB
          Mit Desai
        2. YARN-1670.patch
          2 kB
          Mit Desai
        3. YARN-1670-b23.patch
          2 kB
          Mit Desai
        4. YARN-1670-v2.patch
          7 kB
          Mit Desai
        5. YARN-1670-v2-b23.patch
          7 kB
          Mit Desai
        6. YARN-1670-v3.patch
          6 kB
          Mit Desai
        7. YARN-1670-v3-b23.patch
          6 kB
          Mit Desai
        8. YARN-1670-v4.patch
          2 kB
          Mit Desai
        9. YARN-1670-v4.patch
          7 kB
          Mit Desai
        10. YARN-1670-v4-b23.patch
          6 kB
          Mit Desai
        11. YARN-1670-v4-b23.patch
          6 kB
          Mit Desai

        Activity

          People

            mitdesai Mit Desai
            tgraves Thomas Graves
            Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: