Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-2733

Aggregated log file may be corrupted if IOException happens at copying

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • log-aggregation
    • None

    Description

      In LogValue.write:

                while ((len = in.read(buf)) != -1) {
                  //If buffer contents within fileLength, write
                  if (len < bytesLeft) {
                    out.write(buf, 0, len);
                    bytesLeft-=len;
                  }
                  //else only write contents within fileLength, then exit early
                  else {
                    out.write(buf, 0, (int)bytesLeft);
                    break;
                  }
                }
      

      If IOException happens at any iteration, copying is going to be stopped, and exception message will be appended to the end. It is very likely that the total copied content length is smaller than the file length which has been written into the aggregated file in advance. Later on, when the reader uses this length to parse the content, error is going to happen because of reading more bytes than actually being written.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              zjshen Zhijie Shen
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated: