Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8480

Fix performance and timeout issues in HDFS-7929 by using hard-links to preserve old edit logs instead of copying them

    Details

    • Target Version/s:

      Description

      HDFS-7929 copies existing edit logs to the storage directory of the upgraded NameNode. This slows down the upgrade process. This JIRA aims to use hard-linking instead of per-op copying to achieve the same goal.

      1. HDFS-8480.00.patch
        2 kB
        Zhe Zhang
      2. HDFS-8480.01.patch
        6 kB
        Zhe Zhang
      3. HDFS-8480.02.patch
        7 kB
        Zhe Zhang
      4. HDFS-8480.03.patch
        6 kB
        Zhe Zhang
      5. HDFS-8480-branch-2.6.1.txt
        7 kB
        Vinod Kumar Vavilapalli

        Issue Links

          Activity

          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Zhe Zhang/Andrew Wang,

          The root-cause HDFS-7929 was in 2.7.0. Do we have any proposals that we can get in shortly for 2.7.1 for this issue. Otherwise, we should consider moving this out, assuming this is not a major regression. Please comment, tx.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Zhe Zhang / Andrew Wang , The root-cause HDFS-7929 was in 2.7.0. Do we have any proposals that we can get in shortly for 2.7.1 for this issue. Otherwise, we should consider moving this out, assuming this is not a major regression. Please comment, tx.
          Hide
          andrew.wang Andrew Wang added a comment -

          If we don't get this in, we should probably revert HDFS-7929 from branch-2.7, since it can greatly slow down JN startup. Can retarget this for 2.8.0 then.

          Show
          andrew.wang Andrew Wang added a comment - If we don't get this in, we should probably revert HDFS-7929 from branch-2.7, since it can greatly slow down JN startup. Can retarget this for 2.8.0 then.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Maybe I can take a look in a little bit...

          Show
          cmccabe Colin P. McCabe added a comment - Maybe I can take a look in a little bit...
          Hide
          zhz Zhe Zhang added a comment -

          JIRA server down. Here's a simple patch:
          for (String s : fileNameList) {

          • File prevFile = new File(tmpDir, s);
          • Preconditions.checkState(prevFile.canRead(),
          • "Edits log file " + s + " is not readable.");
          • File newFile = new File(curDir, prevFile.getName());
          • Preconditions.checkState(newFile.createNewFile(),
          • "Cannot create new edits log file in " + curDir);
          • EditLogFileInputStream in = new EditLogFileInputStream(prevFile);
          • EditLogFileOutputStream out =
          • new EditLogFileOutputStream(conf, newFile, 512*1024);
          • FSEditLogOp logOp = in.nextValidOp();
          • while (logOp != null) { - out.write(logOp); - logOp = in.nextOp(); - }
          • out.setReadyToFlush();
          • out.flushAndSync(true);
          • out.close();
          • in.close();
            + Path prevFile = new File(tmpDir, s).toPath();
            + Path newFile = new File(curDir, s).toPath();
            + Files.createLink(newFile, prevFile);
            }

          On Thu, Jun 4, 2015 at 7:10 PM Colin Patrick McCabe (JIRA) <jira@apache.org>

          Show
          zhz Zhe Zhang added a comment - JIRA server down. Here's a simple patch: for (String s : fileNameList) { File prevFile = new File(tmpDir, s); Preconditions.checkState(prevFile.canRead(), "Edits log file " + s + " is not readable."); File newFile = new File(curDir, prevFile.getName()); Preconditions.checkState(newFile.createNewFile(), "Cannot create new edits log file in " + curDir); EditLogFileInputStream in = new EditLogFileInputStream(prevFile); EditLogFileOutputStream out = new EditLogFileOutputStream(conf, newFile, 512*1024); FSEditLogOp logOp = in.nextValidOp(); while (logOp != null) { - out.write(logOp); - logOp = in.nextOp(); - } out.setReadyToFlush(); out.flushAndSync(true); out.close(); in.close(); + Path prevFile = new File(tmpDir, s).toPath(); + Path newFile = new File(curDir, s).toPath(); + Files.createLink(newFile, prevFile); } On Thu, Jun 4, 2015 at 7:10 PM Colin Patrick McCabe (JIRA) <jira@apache.org>
          Hide
          zhz Zhe Zhang added a comment -

          Initial patch. Will add a test in next rev.

          Show
          zhz Zhe Zhang added a comment - Initial patch. Will add a test in next rev.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 15m 5s Findbugs (version ) appears to be broken on trunk.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 30s There were no new javac warning messages.
          +1 javadoc 9m 34s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 50s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 35s mvn install still works.
          +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse.
          +1 findbugs 3m 15s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 13s Pre-build of native portion
          +1 hdfs tests 160m 59s Tests passed in hadoop-hdfs.
              203m 1s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12737987/HDFS-8480.00.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 7588585
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11245/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11245/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11245/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 15m 5s Findbugs (version ) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 30s There were no new javac warning messages. +1 javadoc 9m 34s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 50s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse. +1 findbugs 3m 15s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 13s Pre-build of native portion +1 hdfs tests 160m 59s Tests passed in hadoop-hdfs.     203m 1s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12737987/HDFS-8480.00.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 7588585 hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11245/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11245/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11245/console This message was automatically generated.
          Hide
          cmccabe Colin P. McCabe added a comment -

          +1, pending adding the unit test you talked about earlier. Thanks, Zhe Zhang.

          Show
          cmccabe Colin P. McCabe added a comment - +1, pending adding the unit test you talked about earlier. Thanks, Zhe Zhang .
          Hide
          zhz Zhe Zhang added a comment -

          Adding a unit test simulating an upgrade with layout version change. It fails with the following error:

          org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$LogHeaderCorruptException: Unexpected version of the file system log file: -64. Current version = -63.
          	at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.readLogVersion(EditLogFileInputStream.java:387)
          	at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.init(EditLogFileInputStream.java:145)
          	at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOpImpl(EditLogFileInputStream.java:191)
          	at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOp(EditLogFileInputStream.java:249)
          	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
          	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
          	at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
          	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
          	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:191)
          	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:144)
          	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:835)
          	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:692)
          	at org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:394)
          	at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:281)
          	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:983)
          	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:690)
          	at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:584)
          	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:644)
          	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:809)
          	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:793)
          	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1482)
          	at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1114)
          	at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:985)
          	at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:814)
          	at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:471)
          	at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:430)
          	at org.apache.hadoop.hdfs.TestDFSUpgrade.createCluster(TestDFSUpgrade.java:218)
          	at org.apache.hadoop.hdfs.TestDFSUpgrade.testPreserveEditLogs(TestDFSUpgrade.java:505)
          

          I'm trying to find a way to address this issue.

          Show
          zhz Zhe Zhang added a comment - Adding a unit test simulating an upgrade with layout version change. It fails with the following error: org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$LogHeaderCorruptException: Unexpected version of the file system log file: -64. Current version = -63. at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.readLogVersion(EditLogFileInputStream.java:387) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.init(EditLogFileInputStream.java:145) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOpImpl(EditLogFileInputStream.java:191) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOp(EditLogFileInputStream.java:249) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151) at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:191) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:144) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:835) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:692) at org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:394) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:281) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:983) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:690) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:584) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:644) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:809) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:793) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1482) at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1114) at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:985) at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:814) at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:471) at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:430) at org.apache.hadoop.hdfs.TestDFSUpgrade.createCluster(TestDFSUpgrade.java:218) at org.apache.hadoop.hdfs.TestDFSUpgrade.testPreserveEditLogs(TestDFSUpgrade.java:505) I'm trying to find a way to address this issue.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 23s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 47s There were no new javac warning messages.
          +1 javadoc 9m 55s There were no new javadoc warning messages.
          +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 2m 16s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 34s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 3m 17s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 18s Pre-build of native portion
          -1 hdfs tests 161m 15s Tests failed in hadoop-hdfs.
              208m 43s  



          Reason Tests
          Failed unit tests hadoop.hdfs.TestDFSUpgrade



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12739349/HDFS-8480.01.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / eef7b50
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11333/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11333/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11333/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 23s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 47s There were no new javac warning messages. +1 javadoc 9m 55s There were no new javadoc warning messages. +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 2m 16s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 3m 17s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 18s Pre-build of native portion -1 hdfs tests 161m 15s Tests failed in hadoop-hdfs.     208m 43s   Reason Tests Failed unit tests hadoop.hdfs.TestDFSUpgrade Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12739349/HDFS-8480.01.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / eef7b50 hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11333/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11333/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11333/console This message was automatically generated.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Appreciate the movement on this Zhe Zhang and Colin P. McCabe!

          Zhe Zhang, assuming everything else is addressed, can you look at the test issue reported by Jenkins so that we can get this in 2.7.1? Thanks again!

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Appreciate the movement on this Zhe Zhang and Colin P. McCabe ! Zhe Zhang , assuming everything else is addressed, can you look at the test issue reported by Jenkins so that we can get this in 2.7.1? Thanks again!
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Zhe Zhang oddly I cannot reproduce the Jenkins test failure with your patch applied locally.

          Show
          arpitagarwal Arpit Agarwal added a comment - Zhe Zhang oddly I cannot reproduce the Jenkins test failure with your patch applied locally.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks for the comments Vinod and Arpit.

          I tried the patch again and got the same failure. To verify that the failure is caused by edit log version I changed the following line and removed the -1:

          out.create(NameNodeLayoutVersion.CURRENT_LAYOUT_VERSION - 1); -> out.create(NameNodeLayoutVersion.CURRENT_LAYOUT_VERSION);
          

          And then the test passes. I think this expected when EditLogFileInputStream#init() is called with verifyLayoutVersion as true.

          Show
          zhz Zhe Zhang added a comment - Thanks for the comments Vinod and Arpit. I tried the patch again and got the same failure. To verify that the failure is caused by edit log version I changed the following line and removed the -1 : out.create(NameNodeLayoutVersion.CURRENT_LAYOUT_VERSION - 1); -> out.create(NameNodeLayoutVersion.CURRENT_LAYOUT_VERSION); And then the test passes. I think this expected when EditLogFileInputStream#init() is called with verifyLayoutVersion as true.
          Hide
          zhz Zhe Zhang added a comment -

          The pre-created hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/name1/current/ directory has the following files:

          VERSION
          edits_0000000000000000001-0000000000000000031
          edits_0000000000000000032-0000000000000000061
          fsimage_0000000000000000000
          fsimage_0000000000000000000.md5
          fsimage_0000000000000000031
          fsimage_0000000000000000031.md5
          seen_txid
          

          This emulates a state with last checkpoint at txID 31. During upgrade NN will try to read and interpret all edit logs after txID 31 (in this case the edit log file edits_0000000000000000032-0000000000000000061). If any new edit logs have an old layout version, an error will occur.

          Uploading new patch which only manipulates the layout version of old edit files. However, the inotify part still fails because of the same EditLogFileInputStream#init() error. Colin P. McCabe Could you take a look and see if the test was correctly setup?

          Show
          zhz Zhe Zhang added a comment - The pre-created hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/name1/current/ directory has the following files: VERSION edits_0000000000000000001-0000000000000000031 edits_0000000000000000032-0000000000000000061 fsimage_0000000000000000000 fsimage_0000000000000000000.md5 fsimage_0000000000000000031 fsimage_0000000000000000031.md5 seen_txid This emulates a state with last checkpoint at txID 31. During upgrade NN will try to read and interpret all edit logs after txID 31 (in this case the edit log file edits_0000000000000000032-0000000000000000061 ). If any new edit logs have an old layout version, an error will occur. Uploading new patch which only manipulates the layout version of old edit files. However, the inotify part still fails because of the same EditLogFileInputStream#init() error. Colin P. McCabe Could you take a look and see if the test was correctly setup?
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 11s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 20s There were no new javac warning messages.
          +1 javadoc 9m 33s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 2m 18s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 31s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 3m 13s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 15s Pre-build of native portion
          -1 hdfs tests 102m 48s Tests failed in hadoop-hdfs.
              149m 7s  



          Reason Tests
          Failed unit tests hadoop.hdfs.TestDFSUpgrade
            hadoop.hdfs.TestAppendSnapshotTruncate
          Timed out tests org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12739790/HDFS-8480.02.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 32e39d8
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11367/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11367/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11367/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 11s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 20s There were no new javac warning messages. +1 javadoc 9m 33s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 2m 18s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 31s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 3m 13s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 15s Pre-build of native portion -1 hdfs tests 102m 48s Tests failed in hadoop-hdfs.     149m 7s   Reason Tests Failed unit tests hadoop.hdfs.TestDFSUpgrade   hadoop.hdfs.TestAppendSnapshotTruncate Timed out tests org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12739790/HDFS-8480.02.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 32e39d8 hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11367/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11367/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11367/console This message was automatically generated.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Hi Zhe Zhang,

          We have to distinguish between two cases where the edit log is read:
          1. when the edit log is read NN during startup to catch up with edits that didn't make it into the fsimage yet
          2. when the NN reads the edit log to handle an inotify request

          Case #1 should be validating that the version is the newest version, since the upgrade process ensures this.

          Case #2 should not, since otherwise we will fail when reading old edit logs.

          Show
          cmccabe Colin P. McCabe added a comment - Hi Zhe Zhang , We have to distinguish between two cases where the edit log is read: 1. when the edit log is read NN during startup to catch up with edits that didn't make it into the fsimage yet 2. when the NN reads the edit log to handle an inotify request Case #1 should be validating that the version is the newest version, since the upgrade process ensures this. Case #2 should not, since otherwise we will fail when reading old edit logs.
          Hide
          zhz Zhe Zhang added a comment -

          Patch 02 actually made a very silly mistake and emulates an old layout version by CURRENT_LAYOUT_VERSION - 1 instead of incrementing it by 1.

          So the conclusion is EditLogInputStream has the logic of reading from older layout versions. And it works for both upgrade and inotify scenarios.

          Show
          zhz Zhe Zhang added a comment - Patch 02 actually made a very silly mistake and emulates an old layout version by CURRENT_LAYOUT_VERSION - 1 instead of incrementing it by 1. So the conclusion is EditLogInputStream has the logic of reading from older layout versions. And it works for both upgrade and inotify scenarios.
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 55s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 32s There were no new javac warning messages.
          +1 javadoc 9m 40s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 2m 12s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 34s mvn install still works.
          +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse.
          +1 findbugs 3m 15s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 14s Pre-build of native portion
          +1 hdfs tests 161m 44s Tests passed in hadoop-hdfs.
              208m 7s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12740571/HDFS-8480.03.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 49f5d20
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11415/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11415/testReport/
          Java 1.7.0_55
          uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11415/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 55s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 32s There were no new javac warning messages. +1 javadoc 9m 40s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 2m 12s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse. +1 findbugs 3m 15s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 14s Pre-build of native portion +1 hdfs tests 161m 44s Tests passed in hadoop-hdfs.     208m 7s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12740571/HDFS-8480.03.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 49f5d20 hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11415/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11415/testReport/ Java 1.7.0_55 uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11415/console This message was automatically generated.
          Hide
          zhz Zhe Zhang added a comment -

          The test in the patch verifies that the earliest edit log op in the pre-created /current directory (a mkdir of path /TestUpgrade) was successfully retrieved.

          Show
          zhz Zhe Zhang added a comment - The test in the patch verifies that the earliest edit log op in the pre-created /current directory (a mkdir of path /TestUpgrade ) was successfully retrieved.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Thanks, Zhe Zhang. +1.

          Show
          cmccabe Colin P. McCabe added a comment - Thanks, Zhe Zhang . +1.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8046 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8046/)
          HDFS-8480. Fix performance and timeout issues in HDFS-7929 by using hard-links to preserve old edit logs, instead of copying them. (Zhe Zhang via Colin P. McCabe) (cmccabe: rev 7b424f938c3c306795d574792b086d84e4f06425)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNUpgradeUtil.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgrade.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8046 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8046/ ) HDFS-8480 . Fix performance and timeout issues in HDFS-7929 by using hard-links to preserve old edit logs, instead of copying them. (Zhe Zhang via Colin P. McCabe) (cmccabe: rev 7b424f938c3c306795d574792b086d84e4f06425) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNUpgradeUtil.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgrade.java
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Colin for the review! And helpful comments from Vinod, Andrew, and Arpit.

          Show
          zhz Zhe Zhang added a comment - Thanks Colin for the review! And helpful comments from Vinod, Andrew, and Arpit.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #967 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/967/)
          HDFS-8480. Fix performance and timeout issues in HDFS-7929 by using hard-links to preserve old edit logs, instead of copying them. (Zhe Zhang via Colin P. McCabe) (cmccabe: rev 7b424f938c3c306795d574792b086d84e4f06425)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNUpgradeUtil.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgrade.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #967 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/967/ ) HDFS-8480 . Fix performance and timeout issues in HDFS-7929 by using hard-links to preserve old edit logs, instead of copying them. (Zhe Zhang via Colin P. McCabe) (cmccabe: rev 7b424f938c3c306795d574792b086d84e4f06425) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNUpgradeUtil.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgrade.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #237 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/237/)
          HDFS-8480. Fix performance and timeout issues in HDFS-7929 by using hard-links to preserve old edit logs, instead of copying them. (Zhe Zhang via Colin P. McCabe) (cmccabe: rev 7b424f938c3c306795d574792b086d84e4f06425)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgrade.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNUpgradeUtil.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #237 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/237/ ) HDFS-8480 . Fix performance and timeout issues in HDFS-7929 by using hard-links to preserve old edit logs, instead of copying them. (Zhe Zhang via Colin P. McCabe) (cmccabe: rev 7b424f938c3c306795d574792b086d84e4f06425) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgrade.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNUpgradeUtil.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2165 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2165/)
          HDFS-8480. Fix performance and timeout issues in HDFS-7929 by using hard-links to preserve old edit logs, instead of copying them. (Zhe Zhang via Colin P. McCabe) (cmccabe: rev 7b424f938c3c306795d574792b086d84e4f06425)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgrade.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNUpgradeUtil.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2165 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2165/ ) HDFS-8480 . Fix performance and timeout issues in HDFS-7929 by using hard-links to preserve old edit logs, instead of copying them. (Zhe Zhang via Colin P. McCabe) (cmccabe: rev 7b424f938c3c306795d574792b086d84e4f06425) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgrade.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNUpgradeUtil.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #226 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/226/)
          HDFS-8480. Fix performance and timeout issues in HDFS-7929 by using hard-links to preserve old edit logs, instead of copying them. (Zhe Zhang via Colin P. McCabe) (cmccabe: rev 7b424f938c3c306795d574792b086d84e4f06425)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgrade.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNUpgradeUtil.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #226 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/226/ ) HDFS-8480 . Fix performance and timeout issues in HDFS-7929 by using hard-links to preserve old edit logs, instead of copying them. (Zhe Zhang via Colin P. McCabe) (cmccabe: rev 7b424f938c3c306795d574792b086d84e4f06425) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgrade.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNUpgradeUtil.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #235 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/235/)
          HDFS-8480. Fix performance and timeout issues in HDFS-7929 by using hard-links to preserve old edit logs, instead of copying them. (Zhe Zhang via Colin P. McCabe) (cmccabe: rev 7b424f938c3c306795d574792b086d84e4f06425)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgrade.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNUpgradeUtil.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #235 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/235/ ) HDFS-8480 . Fix performance and timeout issues in HDFS-7929 by using hard-links to preserve old edit logs, instead of copying them. (Zhe Zhang via Colin P. McCabe) (cmccabe: rev 7b424f938c3c306795d574792b086d84e4f06425) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgrade.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNUpgradeUtil.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2183 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2183/)
          HDFS-8480. Fix performance and timeout issues in HDFS-7929 by using hard-links to preserve old edit logs, instead of copying them. (Zhe Zhang via Colin P. McCabe) (cmccabe: rev 7b424f938c3c306795d574792b086d84e4f06425)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNUpgradeUtil.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgrade.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2183 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2183/ ) HDFS-8480 . Fix performance and timeout issues in HDFS-7929 by using hard-links to preserve old edit logs, instead of copying them. (Zhe Zhang via Colin P. McCabe) (cmccabe: rev 7b424f938c3c306795d574792b086d84e4f06425) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNUpgradeUtil.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgrade.java
          Hide
          mingma Ming Ma added a comment -

          Thanks Zhe Zhang.

          The test code simulates older version edit by creating EditLogFileOutputStream with the prior version. However, the FSEditLogOp's writeFields implementation doesn't distinguish different versions and always write with the latest features.

          For example, if this patch is applied directly on the top of the 2.6 release where CURRENT_LAYOUT_VERSION is set to -60, the test will fail because even though EditLogFileOutputStream is created with version -59, OP_ADD's writeFields will add storagePolicyId to the edit. During the edit loading, it will skip the reading of storagePolicyId given it has version -59 and thus cause checksum error.

          To simulates old edit for test purpose, it seems we need to have FSEditLogOp support writing optional data depending on the version requested. So in this case, if the version is set to -59, it shouldn't write storagePolicyId. This functionality seems only useful for testing upgrade scenario.

          Thought?

          Show
          mingma Ming Ma added a comment - Thanks Zhe Zhang . The test code simulates older version edit by creating EditLogFileOutputStream with the prior version. However, the FSEditLogOp's writeFields implementation doesn't distinguish different versions and always write with the latest features. For example, if this patch is applied directly on the top of the 2.6 release where CURRENT_LAYOUT_VERSION is set to -60, the test will fail because even though EditLogFileOutputStream is created with version -59, OP_ADD's writeFields will add storagePolicyId to the edit. During the edit loading, it will skip the reading of storagePolicyId given it has version -59 and thus cause checksum error. To simulates old edit for test purpose, it seems we need to have FSEditLogOp support writing optional data depending on the version requested. So in this case, if the version is set to -59, it shouldn't write storagePolicyId. This functionality seems only useful for testing upgrade scenario. Thought?
          Hide
          zhz Zhe Zhang added a comment -

          Ming Ma Thanks for the good catch! Yes this is indeed a flaw in the patch.

          I can't think of another use case requiring us to extend FSEditLogOp as you suggested. But testing upgrade scenarios itself is a pretty strong motivation. The only alternative is to manually create some old edit log files.

          Once we agree upon a plan I'm happy to make the change.

          Show
          zhz Zhe Zhang added a comment - Ming Ma Thanks for the good catch! Yes this is indeed a flaw in the patch. I can't think of another use case requiring us to extend FSEditLogOp as you suggested. But testing upgrade scenarios itself is a pretty strong motivation. The only alternative is to manually create some old edit log files. Once we agree upon a plan I'm happy to make the change.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Good catch, Ming Ma. If we make a change later to the edit log format, this test will break because we'll be writing out an edit log with the new format, but the old version ID.

          I think we should simply check in a small binary edit log in order to do this testing. This is similar to what we do with other "ugprade scenario" tests. As Zhe Zhang mentioned, it would be a lot of effort to support writing out edit logs in older formats (we have literally dozens of versions).

          Show
          cmccabe Colin P. McCabe added a comment - Good catch, Ming Ma . If we make a change later to the edit log format, this test will break because we'll be writing out an edit log with the new format, but the old version ID. I think we should simply check in a small binary edit log in order to do this testing. This is similar to what we do with other "ugprade scenario" tests. As Zhe Zhang mentioned, it would be a lot of effort to support writing out edit logs in older formats (we have literally dozens of versions).
          Hide
          zhz Zhe Zhang added a comment -

          Thanks for the discussion Ming and Colin!

          I created HDFS-8846 to add old-version edit logs for testing.

          Show
          zhz Zhe Zhang added a comment - Thanks for the discussion Ming and Colin! I created HDFS-8846 to add old-version edit logs for testing.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Thanks, Zhe Zhang.

          Show
          cmccabe Colin P. McCabe added a comment - Thanks, Zhe Zhang .
          Hide
          mingma Ming Ma added a comment -

          Make sense. It is easier to check in an edit log with an older version. Thanks Zhe Zhang and Colin P. McCabe.

          Show
          mingma Ming Ma added a comment - Make sense. It is easier to check in an edit log with an older version. Thanks Zhe Zhang and Colin P. McCabe .
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Pulled this into 2.6.1. As Ming mentioned above, the test fails after this patch, so pulled in HDFS-8846 too.

          Had to fix merge and import conflicts, and make some minor change to reflect 2.6.1. Ran compilation before the push.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Pulled this into 2.6.1. As Ming mentioned above, the test fails after this patch, so pulled in HDFS-8846 too. Had to fix merge and import conflicts, and make some minor change to reflect 2.6.1. Ran compilation before the push.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Attaching patch that I committed to 2.6.1.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Attaching patch that I committed to 2.6.1.

            People

            • Assignee:
              zhz Zhe Zhang
              Reporter:
              zhz Zhe Zhang
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development