Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3596

Improve FSEditLog pre-allocation in branch-1

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.1.0, 1.2.0
    • Component/s: None
    • Labels:
      None
    • Target Version/s:

      Description

      Implement HDFS-3510 in branch-1. This will improve FSEditLog preallocation to decrease the incidence of corrupted logs after disk full conditions. (See HDFS-3510 for a longer description.)

      1. HDFS-3596-b1.001.patch
        6 kB
        Colin Patrick McCabe
      2. HDFS-3596-branch1.1-001.patch
        7 kB
        Jing Zhao

        Issue Links

          Activity

          Hide
          Matt Foley added a comment -

          Closed upon release of Hadoop-1.1.0.

          Show
          Matt Foley added a comment - Closed upon release of Hadoop-1.1.0.
          Hide
          Jing Zhao added a comment -

          By the way, I have run local test earlier and all the unit tests have passed.

          Show
          Jing Zhao added a comment - By the way, I have run local test earlier and all the unit tests have passed.
          Hide
          Suresh Srinivas added a comment -

          I committed the patch to 1.1.0 to be picked up in case 1.1.0 RC is re spun or in subsequent 1.1.x release.

          Show
          Suresh Srinivas added a comment - I committed the patch to 1.1.0 to be picked up in case 1.1.0 RC is re spun or in subsequent 1.1.x release.
          Hide
          Colin Patrick McCabe added a comment -

          HDFS-3596-branch1.1-001.patch looks good to me. Thanks, Jing.

          Show
          Colin Patrick McCabe added a comment - HDFS-3596 -branch1.1-001.patch looks good to me. Thanks, Jing.
          Hide
          Suresh Srinivas added a comment -

          +1 for the patch.

          The 1.1 branch patch looks good to me. I think this is a good change to add to 1.1, if there is another RC going to come out. Colin, if you have time, do you want to take a quick look at the patch as well.

          Show
          Suresh Srinivas added a comment - +1 for the patch. The 1.1 branch patch looks good to me. I think this is a good change to add to 1.1, if there is another RC going to come out. Colin, if you have time, do you want to take a quick look at the patch as well.
          Hide
          Jing Zhao added a comment -

          The patch for branch-1.1.

          Show
          Jing Zhao added a comment - The patch for branch-1.1.
          Hide
          Colin Patrick McCabe added a comment -

          Yes, I think you're right. Luckily, it should be extremely rare to write more than 1 MB of edits in a single batch, so this will seldom if ever be an issue. However, we should fix this for the sake of consistency.

          Show
          Colin Patrick McCabe added a comment - Yes, I think you're right. Luckily, it should be extremely rare to write more than 1 MB of edits in a single batch, so this will seldom if ever be an issue. However, we should fix this for the sake of consistency.
          Hide
          Jing Zhao added a comment -

          In the new preallocate() function, I guess

          PREALLOCATION_BUFFER.position(0);
          while (need > 0) {
          do

          { size += fc.write(PREALLOCATION_BUFFER, size); } while (PREALLOCATION_BUFFER.remaining() > 0);
          need -= fillCapacity;
          total += fillCapacity;
          }

          should be:

          while (need > 0) {
          PREALLOCATION_BUFFER.position(0);
          do { size += fc.write(PREALLOCATION_BUFFER, size); }

          while (PREALLOCATION_BUFFER.remaining() > 0);
          need -= fillCapacity;
          total += fillCapacity;
          }

          Otherwise seems only 1MB will be allocated even if need is larger than 1MB.
          I will create another JIRA and upload the patch for it.

          Show
          Jing Zhao added a comment - In the new preallocate() function, I guess PREALLOCATION_BUFFER.position(0); while (need > 0) { do { size += fc.write(PREALLOCATION_BUFFER, size); } while (PREALLOCATION_BUFFER.remaining() > 0); need -= fillCapacity; total += fillCapacity; } should be: while (need > 0) { PREALLOCATION_BUFFER.position(0); do { size += fc.write(PREALLOCATION_BUFFER, size); } while (PREALLOCATION_BUFFER.remaining() > 0); need -= fillCapacity; total += fillCapacity; } Otherwise seems only 1MB will be allocated even if need is larger than 1MB. I will create another JIRA and upload the patch for it.
          Hide
          Matt Foley added a comment -

          +1. Committed to branch-1.

          The available patch applies to branch-1 but not to branch-1.1, so marking fixed in 1.2.0.

          Show
          Matt Foley added a comment - +1. Committed to branch-1. The available patch applies to branch-1 but not to branch-1.1, so marking fixed in 1.2.0.
          Hide
          Colin Patrick McCabe added a comment -

          Thanks for the correction, Matt.

          Show
          Colin Patrick McCabe added a comment - Thanks for the correction, Matt.
          Hide
          Matt Foley added a comment -

          "Target version" indicates where a jira is intended to be fixed. "Fixed version" indicates where it has been fixed. Please set "Fixed version" only after a reviewed patch is committed. Thanks.

          Show
          Matt Foley added a comment - "Target version" indicates where a jira is intended to be fixed. "Fixed version" indicates where it has been fixed. Please set "Fixed version" only after a reviewed patch is committed. Thanks.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12535273/HDFS-3596-b1.001.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2745//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12535273/HDFS-3596-b1.001.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2745//console This message is automatically generated.
          Hide
          Colin Patrick McCabe added a comment -

          Patch for branch-1.

          Tested with: TestCheckpoint, TestEditLog, TestNameNodeRecovery, TestEditLogLoading, TestNameNodeMXBean, TestSaveNamespace, TestSecurityTokenEditLog, TestStorageDirectoryFailure, TestEditLogToleration, TestStorageRestore

          Show
          Colin Patrick McCabe added a comment - Patch for branch-1. Tested with: TestCheckpoint, TestEditLog, TestNameNodeRecovery, TestEditLogLoading, TestNameNodeMXBean, TestSaveNamespace, TestSecurityTokenEditLog, TestStorageDirectoryFailure, TestEditLogToleration, TestStorageRestore
          Hide
          Colin Patrick McCabe added a comment -

          Let's fix the edit log test

          Show
          Colin Patrick McCabe added a comment - Let's fix the edit log test

            People

            • Assignee:
              Colin Patrick McCabe
              Reporter:
              Colin Patrick McCabe
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development