Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-2733

Aggregated log file may be corrupted if IOException happens at copying

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: log-aggregation
    • Labels:
      None

      Description

      In LogValue.write:

                while ((len = in.read(buf)) != -1) {
                  //If buffer contents within fileLength, write
                  if (len < bytesLeft) {
                    out.write(buf, 0, len);
                    bytesLeft-=len;
                  }
                  //else only write contents within fileLength, then exit early
                  else {
                    out.write(buf, 0, (int)bytesLeft);
                    break;
                  }
                }
      

      If IOException happens at any iteration, copying is going to be stopped, and exception message will be appended to the end. It is very likely that the total copied content length is smaller than the file length which has been written into the aggregated file in advance. Later on, when the reader uses this length to parse the content, error is going to happen because of reading more bytes than actually being written.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                zjshen Zhijie Shen
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated: