[YARN-2733] Aggregated log file may be corrupted if IOException happens at copying

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: log-aggregation
• Assignee: Unassigned
• Reporter: Zhijie Shen
• Created: 23/Oct/14 05:25
• Updated: 23/Oct/14 20:21

Description

In LogValue.write:

          while ((len = in.read(buf)) != -1) {
            //If buffer contents within fileLength, write
            if (len < bytesLeft) {
              out.write(buf, 0, len);
              bytesLeft-=len;
            }
            //else only write contents within fileLength, then exit early
            else {
              out.write(buf, 0, (int)bytesLeft);
              break;
            }
          }

If IOException happens at any iteration, copying is going to be stopped, and exception message will be appended to the end. It is very likely that the total copied content length is smaller than the file length which has been written into the aggregated file in advance. Later on, when the reader uses this length to parse the content, error is going to happen because of reading more bytes than actually being written.

Issue Links

• relates to: YARN-2724 If an unreadable file is encountered during log aggregation then aggregated file in HDFS badly formed