Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11498

Make RestCsrfPreventionHandler and WebHdfsHandler compatible with Netty 4.0

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha3
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Release Note:
      This JIRA sets the Netty 4 dependency to 4.0.23. This is an incompatible change for the 3.0 release line, as 3.0.0-alpha1 and 3.0.0-alpha2 depended on Netty 4.1.0.Beta5.

      Description

      Per discussion in HADOOP-13866, it looks like we can change 2.8.0 back to exposing Netty 4.0, but still be ABI compatible with Netty 4.1 for users like HBase that want to swap out the version.

      1. HDFS-11498.001.patch
        5 kB
        Jason Lowe
      2. HDFS-11498.branch-2.001.patch
        5 kB
        Andrew Wang

        Issue Links

          Activity

          Hide
          andrew.wang Andrew Wang added a comment -

          Patch attached. I tested by doing a clean build with the patch applied, then again with the patch from HADOOP-13866 on top that bumps Netty from 4.0 to 4.1.

          Show
          andrew.wang Andrew Wang added a comment - Patch attached. I tested by doing a clean build with the patch applied, then again with the patch from HADOOP-13866 on top that bumps Netty from 4.0 to 4.1.
          Hide
          andrew.wang Andrew Wang added a comment -

          Rename it to run against branch-2.

          I might want to split the change: commit the java changes to trunk and the branch-2s, then the pom.xml change to just the branch-2s. This way we aren't doing custom backports to trunk vs. the branch-2s.

          Show
          andrew.wang Andrew Wang added a comment - Rename it to run against branch-2. I might want to split the change: commit the java changes to trunk and the branch-2s, then the pom.xml change to just the branch-2s. This way we aren't doing custom backports to trunk vs. the branch-2s.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 14m 43s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 1m 44s Maven dependency ordering for branch
          +1 mvninstall 8m 20s branch-2 passed
          +1 compile 6m 59s branch-2 passed with JDK v1.8.0_121
          +1 compile 7m 17s branch-2 passed with JDK v1.7.0_121
          +1 checkstyle 1m 33s branch-2 passed
          +1 mvnsite 1m 10s branch-2 passed
          +1 mvneclipse 0m 29s branch-2 passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 2m 10s branch-2 passed
          +1 javadoc 1m 6s branch-2 passed with JDK v1.8.0_121
          +1 javadoc 1m 50s branch-2 passed with JDK v1.7.0_121
          0 mvndep 0m 17s Maven dependency ordering for patch
          +1 mvninstall 0m 54s the patch passed
          +1 compile 7m 0s the patch passed with JDK v1.8.0_121
          -1 javac 7m 0s root-jdk1.8.0_121 with JDK v1.8.0_121 generated 1 new + 887 unchanged - 30 fixed = 888 total (was 917)
          +1 compile 7m 24s the patch passed with JDK v1.7.0_121
          -1 javac 7m 24s root-jdk1.7.0_121 with JDK v1.7.0_121 generated 1 new + 983 unchanged - 30 fixed = 984 total (was 1013)
          +1 checkstyle 1m 30s the patch passed
          +1 mvnsite 1m 12s the patch passed
          +1 mvneclipse 0m 28s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 0s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 2m 30s the patch passed
          +1 javadoc 1m 9s the patch passed with JDK v1.8.0_121
          +1 javadoc 1m 59s the patch passed with JDK v1.7.0_121
          +1 unit 0m 10s hadoop-project in the patch passed with JDK v1.7.0_121.
          -1 unit 57m 12s hadoop-hdfs in the patch failed with JDK v1.7.0_121.
          +1 asflicense 0m 33s The patch does not generate ASF License warnings.
          189m 51s



          Reason Tests
          JDK v1.8.0_121 Failed junit tests hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
          JDK v1.8.0_121 Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration
          JDK v1.7.0_121 Failed junit tests hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain
          JDK v1.7.0_121 Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Issue HDFS-11498
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12856016/HDFS-11498.branch-2.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux edf8558336ed 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / b26870c
          Default Java 1.7.0_121
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121
          findbugs v3.0.0
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/18552/artifact/patchprocess/diff-compile-javac-root-jdk1.8.0_121.txt
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/18552/artifact/patchprocess/diff-compile-javac-root-jdk1.7.0_121.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18552/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_121.txt
          JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18552/testReport/
          modules C: hadoop-project hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18552/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 14m 43s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 1m 44s Maven dependency ordering for branch +1 mvninstall 8m 20s branch-2 passed +1 compile 6m 59s branch-2 passed with JDK v1.8.0_121 +1 compile 7m 17s branch-2 passed with JDK v1.7.0_121 +1 checkstyle 1m 33s branch-2 passed +1 mvnsite 1m 10s branch-2 passed +1 mvneclipse 0m 29s branch-2 passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 2m 10s branch-2 passed +1 javadoc 1m 6s branch-2 passed with JDK v1.8.0_121 +1 javadoc 1m 50s branch-2 passed with JDK v1.7.0_121 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 0m 54s the patch passed +1 compile 7m 0s the patch passed with JDK v1.8.0_121 -1 javac 7m 0s root-jdk1.8.0_121 with JDK v1.8.0_121 generated 1 new + 887 unchanged - 30 fixed = 888 total (was 917) +1 compile 7m 24s the patch passed with JDK v1.7.0_121 -1 javac 7m 24s root-jdk1.7.0_121 with JDK v1.7.0_121 generated 1 new + 983 unchanged - 30 fixed = 984 total (was 1013) +1 checkstyle 1m 30s the patch passed +1 mvnsite 1m 12s the patch passed +1 mvneclipse 0m 28s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 2m 30s the patch passed +1 javadoc 1m 9s the patch passed with JDK v1.8.0_121 +1 javadoc 1m 59s the patch passed with JDK v1.7.0_121 +1 unit 0m 10s hadoop-project in the patch passed with JDK v1.7.0_121. -1 unit 57m 12s hadoop-hdfs in the patch failed with JDK v1.7.0_121. +1 asflicense 0m 33s The patch does not generate ASF License warnings. 189m 51s Reason Tests JDK v1.8.0_121 Failed junit tests hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting JDK v1.8.0_121 Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration JDK v1.7.0_121 Failed junit tests hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain JDK v1.7.0_121 Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Issue HDFS-11498 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12856016/HDFS-11498.branch-2.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux edf8558336ed 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / b26870c Default Java 1.7.0_121 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-HDFS-Build/18552/artifact/patchprocess/diff-compile-javac-root-jdk1.8.0_121.txt javac https://builds.apache.org/job/PreCommit-HDFS-Build/18552/artifact/patchprocess/diff-compile-javac-root-jdk1.7.0_121.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18552/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_121.txt JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18552/testReport/ modules C: hadoop-project hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18552/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          LGTM, +1.

          Show
          ajisakaa Akira Ajisaka added a comment - LGTM, +1.
          Hide
          jlowe Jason Lowe added a comment -

          +1, branch-2 patch looks good to me. The javac warning is unrelated as are the unit tests, and the tests pass on branch-2 for me with the patch applied.

          I believe the simplest path forward is to commit this patch to trunk, branch-2, branch-2.8 and branch-2.8.0 then update HADOOP-13866 accordingly. We do not want to ship a 4.x beta even on trunk.

          Uploading essentially the same patch (except for line number offsets) for trunk, and I'll commit this pending that Jenkins run.

          Show
          jlowe Jason Lowe added a comment - +1, branch-2 patch looks good to me. The javac warning is unrelated as are the unit tests, and the tests pass on branch-2 for me with the patch applied. I believe the simplest path forward is to commit this patch to trunk, branch-2, branch-2.8 and branch-2.8.0 then update HADOOP-13866 accordingly. We do not want to ship a 4.x beta even on trunk. Uploading essentially the same patch (except for line number offsets) for trunk, and I'll commit this pending that Jenkins run.
          Hide
          jlowe Jason Lowe added a comment -

          I was under the impression this would go in relatively cleanly for branch-2.8, but that does not seem to be the case. There is a minor conflict on some imports in the patch, but even after those are resolved the build is broken after this patch. DatanodeHttpServer fails to compile after the netty-all version change. I suspect it is because HDFS-8377 was semi-reverted from branch-2 but not branch-2.8. Seems like we need to cherry-pick the fix for HDFS-11376 (really the revert patch in HDFS-8377) to branch-2.8 then apply this change. Thoughts?

          Show
          jlowe Jason Lowe added a comment - I was under the impression this would go in relatively cleanly for branch-2.8, but that does not seem to be the case. There is a minor conflict on some imports in the patch, but even after those are resolved the build is broken after this patch. DatanodeHttpServer fails to compile after the netty-all version change. I suspect it is because HDFS-8377 was semi-reverted from branch-2 but not branch-2.8. Seems like we need to cherry-pick the fix for HDFS-11376 (really the revert patch in HDFS-8377 ) to branch-2.8 then apply this change. Thoughts?
          Hide
          andrew.wang Andrew Wang added a comment -

          Yea, sorry for only testing on branch-2, I assumed it'd be a smooth backport to 2.8 as well.

          +1 on Jason's recommendation. I'll work on backporting HDFS-11376, then I'll apply this JIRA (HDFS-11498) everywhere. Given that there are two +1s on this pretty straightforward change, I'll go ahead and commit everything if I can do it with minimal cherry-pick conflicts.

          Show
          andrew.wang Andrew Wang added a comment - Yea, sorry for only testing on branch-2, I assumed it'd be a smooth backport to 2.8 as well. +1 on Jason's recommendation. I'll work on backporting HDFS-11376 , then I'll apply this JIRA ( HDFS-11498 ) everywhere. Given that there are two +1s on this pretty straightforward change, I'll go ahead and commit everything if I can do it with minimal cherry-pick conflicts.
          Hide
          andrew.wang Andrew Wang added a comment -

          Thanks again for the reviews Akira and Jason, after backporting the revert of HDFS-8377, the patch here applied cleanly. I've committed to all the branches back through branch-2.8.0, we can resume discussion on the Netty 4.1 upgrade over at HADOOP-13866.

          Show
          andrew.wang Andrew Wang added a comment - Thanks again for the reviews Akira and Jason, after backporting the revert of HDFS-8377 , the patch here applied cleanly. I've committed to all the branches back through branch-2.8.0, we can resume discussion on the Netty 4.1 upgrade over at HADOOP-13866 .
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11357 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11357/)
          HDFS-11498. Make RestCsrfPreventionHandler and WebHdfsHandler compatible (wang: rev 5e74196ede9bfc20eb6d6fe3aa6a0e5c47a40fdd)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/webhdfs/WebHdfsHandler.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/RestCsrfPreventionFilterHandler.java
          • (edit) hadoop-project/pom.xml
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11357 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11357/ ) HDFS-11498 . Make RestCsrfPreventionHandler and WebHdfsHandler compatible (wang: rev 5e74196ede9bfc20eb6d6fe3aa6a0e5c47a40fdd) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/webhdfs/WebHdfsHandler.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/RestCsrfPreventionFilterHandler.java (edit) hadoop-project/pom.xml

            People

            • Assignee:
              andrew.wang Andrew Wang
              Reporter:
              andrew.wang Andrew Wang
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development