Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8593

Calculation of effective layout version mishandles comparison to current layout version in storage.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: namenode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      HDFS-8432 introduced the concept of a minimum compatible layout version so that downgrade is applicable in a wider set of circumstances. This includes logic for determining if the current layout version in storage is within the bounds of the minimum compatible layout version. There is an inverted comparison in this logic, which can result in an incorrect calculation.

        Issue Links

          Activity

          Hide
          cnauroth Chris Nauroth added a comment -

          I'm attaching a patch to flip the comparison. This is the correct logic when we consider that layout versions grow in the negative direction. Thank you to Jing Zhao for reviewing the HDFS-8432 code and finding this bug.

          Show
          cnauroth Chris Nauroth added a comment - I'm attaching a patch to flip the comparison. This is the correct logic when we consider that layout versions grow in the negative direction. Thank you to Jing Zhao for reviewing the HDFS-8432 code and finding this bug.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 53s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 41s There were no new javac warning messages.
          +1 javadoc 9m 57s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 15s The applied patch generated 1 new checkstyle issues (total was 262, now 262).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 35s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 3m 18s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 17s Pre-build of native portion
          +1 hdfs tests 161m 12s Tests passed in hadoop-hdfs.
              208m 6s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12739401/HDFS-8593.001.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / eef7b50
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11337/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11337/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11337/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11337/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 53s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 41s There were no new javac warning messages. +1 javadoc 9m 57s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 15s The applied patch generated 1 new checkstyle issues (total was 262, now 262). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 3m 18s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 17s Pre-build of native portion +1 hdfs tests 161m 12s Tests passed in hadoop-hdfs.     208m 6s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12739401/HDFS-8593.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / eef7b50 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11337/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11337/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11337/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11337/console This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          The checkstyle warning is not actionable.

          Show
          cnauroth Chris Nauroth added a comment - The checkstyle warning is not actionable.
          Hide
          jingzhao Jing Zhao added a comment -

          Thanks for the fix, Chris! The patch looks pretty good to me. +1.

          Show
          jingzhao Jing Zhao added a comment - Thanks for the fix, Chris! The patch looks pretty good to me. +1.
          Hide
          cnauroth Chris Nauroth added a comment -

          Thank you for the review, Jing. I have committed this to trunk and branch-2.

          Show
          cnauroth Chris Nauroth added a comment - Thank you for the review, Jing. I have committed this to trunk and branch-2.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8014 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8014/)
          HDFS-8593. Calculation of effective layout version mishandles comparison to current layout version in storage. Contributed by Chris Nauroth. (cnauroth: rev b8341f1cd89791c51b396ad531ec7fcc631be149)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8014 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8014/ ) HDFS-8593 . Calculation of effective layout version mishandles comparison to current layout version in storage. Contributed by Chris Nauroth. (cnauroth: rev b8341f1cd89791c51b396ad531ec7fcc631be149) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #958 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/958/)
          HDFS-8593. Calculation of effective layout version mishandles comparison to current layout version in storage. Contributed by Chris Nauroth. (cnauroth: rev b8341f1cd89791c51b396ad531ec7fcc631be149)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #958 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/958/ ) HDFS-8593 . Calculation of effective layout version mishandles comparison to current layout version in storage. Contributed by Chris Nauroth. (cnauroth: rev b8341f1cd89791c51b396ad531ec7fcc631be149) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #228 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/228/)
          HDFS-8593. Calculation of effective layout version mishandles comparison to current layout version in storage. Contributed by Chris Nauroth. (cnauroth: rev b8341f1cd89791c51b396ad531ec7fcc631be149)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #228 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/228/ ) HDFS-8593 . Calculation of effective layout version mishandles comparison to current layout version in storage. Contributed by Chris Nauroth. (cnauroth: rev b8341f1cd89791c51b396ad531ec7fcc631be149) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #217 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/217/)
          HDFS-8593. Calculation of effective layout version mishandles comparison to current layout version in storage. Contributed by Chris Nauroth. (cnauroth: rev b8341f1cd89791c51b396ad531ec7fcc631be149)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #217 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/217/ ) HDFS-8593 . Calculation of effective layout version mishandles comparison to current layout version in storage. Contributed by Chris Nauroth. (cnauroth: rev b8341f1cd89791c51b396ad531ec7fcc631be149) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #2156 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2156/)
          HDFS-8593. Calculation of effective layout version mishandles comparison to current layout version in storage. Contributed by Chris Nauroth. (cnauroth: rev b8341f1cd89791c51b396ad531ec7fcc631be149)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2156 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2156/ ) HDFS-8593 . Calculation of effective layout version mishandles comparison to current layout version in storage. Contributed by Chris Nauroth. (cnauroth: rev b8341f1cd89791c51b396ad531ec7fcc631be149) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #226 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/226/)
          HDFS-8593. Calculation of effective layout version mishandles comparison to current layout version in storage. Contributed by Chris Nauroth. (cnauroth: rev b8341f1cd89791c51b396ad531ec7fcc631be149)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #226 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/226/ ) HDFS-8593 . Calculation of effective layout version mishandles comparison to current layout version in storage. Contributed by Chris Nauroth. (cnauroth: rev b8341f1cd89791c51b396ad531ec7fcc631be149) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2174 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2174/)
          HDFS-8593. Calculation of effective layout version mishandles comparison to current layout version in storage. Contributed by Chris Nauroth. (cnauroth: rev b8341f1cd89791c51b396ad531ec7fcc631be149)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2174 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2174/ ) HDFS-8593 . Calculation of effective layout version mishandles comparison to current layout version in storage. Contributed by Chris Nauroth. (cnauroth: rev b8341f1cd89791c51b396ad531ec7fcc631be149) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

            People

            • Assignee:
              cnauroth Chris Nauroth
              Reporter:
              cnauroth Chris Nauroth
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development