Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10877

Make RemoteEditLogManifest.committedTxnId optional in Protocol Buffers

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: 3.0.0-alpha2
    • Component/s: qjm
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change
    • Release Note:
      A new protobuf field added to RemoteEditLogManifest was mistakenly marked as required. This changes the field to optional, preserving compatibility with 2.x releases but breaking compatibility with 3.0.0-alpha1.

      Description

      HDFS-10519 introduced a new field in the RemoteEditLogManifest message. It can be made optional to improve wire-compatibility with previous versions.

      1. HDFS-10877.001.patch
        4 kB
        Sean Mackrory
      2. HDFS-10877.002.patch
        4 kB
        Sean Mackrory

        Issue Links

          Activity

          Hide
          mackrorysd Sean Mackrory added a comment -

          Attaching a patch that modifies the .proto definition and the unit test for it. Every use of this field that I can see can already be INVALID_TXID by default anyway, so I believe this is a safe change.

          Show
          mackrorysd Sean Mackrory added a comment - Attaching a patch that modifies the .proto definition and the unit test for it. Every use of this field that I can see can already be INVALID_TXID by default anyway, so I believe this is a safe change.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 8s trunk passed
          +1 compile 0m 51s trunk passed
          +1 checkstyle 0m 26s trunk passed
          +1 mvnsite 1m 1s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 2m 0s trunk passed
          +1 javadoc 1m 4s trunk passed
          +1 mvninstall 0m 52s the patch passed
          +1 compile 0m 50s the patch passed
          +1 cc 0m 50s the patch passed
          +1 javac 0m 50s the patch passed
          -0 checkstyle 0m 26s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 56 unchanged - 0 fixed = 57 total (was 56)
          +1 mvnsite 1m 2s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 4s the patch passed
          +1 javadoc 1m 1s the patch passed
          -1 unit 64m 40s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          86m 49s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.TestNamenodeCapacityReport
            hadoop.hdfs.TestErasureCodeBenchmarkThroughput
            hadoop.hdfs.TestBlockStoragePolicy



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10877
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12829420/HDFS-10877.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 9f6b73cf3914 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e45307c
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16811/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16811/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16811/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16811/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 8s trunk passed +1 compile 0m 51s trunk passed +1 checkstyle 0m 26s trunk passed +1 mvnsite 1m 1s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 2m 0s trunk passed +1 javadoc 1m 4s trunk passed +1 mvninstall 0m 52s the patch passed +1 compile 0m 50s the patch passed +1 cc 0m 50s the patch passed +1 javac 0m 50s the patch passed -0 checkstyle 0m 26s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 56 unchanged - 0 fixed = 57 total (was 56) +1 mvnsite 1m 2s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 4s the patch passed +1 javadoc 1m 1s the patch passed -1 unit 64m 40s hadoop-hdfs in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 86m 49s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.TestNamenodeCapacityReport   hadoop.hdfs.TestErasureCodeBenchmarkThroughput   hadoop.hdfs.TestBlockStoragePolicy Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10877 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12829420/HDFS-10877.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 9f6b73cf3914 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e45307c Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16811/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16811/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16811/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16811/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          andrew.wang Andrew Wang added a comment -

          Thanks for working on this Sean! Patch LGTM, though we should fix that long line that checkstyle picked up. Test failures look unrelated.

          +1 pending

          Show
          andrew.wang Andrew Wang added a comment - Thanks for working on this Sean! Patch LGTM, though we should fix that long line that checkstyle picked up. Test failures look unrelated. +1 pending
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 26s trunk passed
          +1 compile 0m 45s trunk passed
          +1 checkstyle 0m 28s trunk passed
          +1 mvnsite 0m 53s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 47s trunk passed
          +1 javadoc 0m 57s trunk passed
          +1 mvninstall 0m 50s the patch passed
          +1 compile 0m 46s the patch passed
          +1 cc 0m 46s the patch passed
          +1 javac 0m 46s the patch passed
          +1 checkstyle 0m 25s the patch passed
          +1 mvnsite 0m 50s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 50s the patch passed
          +1 javadoc 0m 56s the patch passed
          -1 unit 62m 50s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          84m 2s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestFileChecksum
            hadoop.fs.viewfs.TestViewFileSystemHdfs
          Timed out junit tests org.apache.hadoop.hdfs.server.namenode.TestQuotaByStorageType



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10877
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12829687/HDFS-10877.002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 9d2872cbdb68 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 964e546
          Default Java 1.8.0_101
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16826/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16826/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16826/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 26s trunk passed +1 compile 0m 45s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 0m 53s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 47s trunk passed +1 javadoc 0m 57s trunk passed +1 mvninstall 0m 50s the patch passed +1 compile 0m 46s the patch passed +1 cc 0m 46s the patch passed +1 javac 0m 46s the patch passed +1 checkstyle 0m 25s the patch passed +1 mvnsite 0m 50s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 50s the patch passed +1 javadoc 0m 56s the patch passed -1 unit 62m 50s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 84m 2s Reason Tests Failed junit tests hadoop.hdfs.TestFileChecksum   hadoop.fs.viewfs.TestViewFileSystemHdfs Timed out junit tests org.apache.hadoop.hdfs.server.namenode.TestQuotaByStorageType Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10877 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12829687/HDFS-10877.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 9d2872cbdb68 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 964e546 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/16826/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16826/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16826/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          andrew.wang Andrew Wang added a comment -

          Committed to trunk, thanks for the contribution Sean!

          Show
          andrew.wang Andrew Wang added a comment - Committed to trunk, thanks for the contribution Sean!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10474 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10474/)
          HDFS-10877. Make RemoteEditLogManifest.committedTxnId optional in (wang: rev 40acacee085494ca52205d37449a46c058d5d325)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/RemoteEditLogManifest.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/proto/HdfsServer.proto
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10474 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10474/ ) HDFS-10877 . Make RemoteEditLogManifest.committedTxnId optional in (wang: rev 40acacee085494ca52205d37449a46c058d5d325) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/RemoteEditLogManifest.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/proto/HdfsServer.proto

            People

            • Assignee:
              mackrorysd Sean Mackrory
              Reporter:
              mackrorysd Sean Mackrory
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development