Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 0.3.0, 0.4.0
    • None

    Description

      Based on the discussion in HDDS-525

      The host HTTP header sometimes contains the port, sometimes not (with aws cli we have the port, with mitm proxy we doesn't). Would be easier to remove it anyway to make it easier to configure the s3 gateway.

      Attachments

        1. HDDS-573.01.patch
          3 kB
          Bharat Viswanadham
        2. HDDS-573.00.patch
          3 kB
          Danilo Perez

        Activity

          I have not got what is the change we need here, as from host we take the hostname matching with domain name. So, what additional change we need for this?

           

          bharat Bharat Viswanadham added a comment - I have not got what is the change we need here, as from host we take the hostname matching with domain name. So, what additional change we need for this?  

          ping elek to know what changes are required for this.

           

          cc danilocop

          bharat Bharat Viswanadham added a comment - ping elek to know what changes are required for this.   cc danilocop
          elek Marton Elek added a comment -

          I would remove the port from actual hostname before the domain name matching.

          Sometimes I need to add the port to 'ozone.s3g.domain.name', sometimes not. (in case of direct aws cli usage the port is part of the host name but in case of mitm proxy it's removed by the proxy).

          Not a big deal but would make my life easier

          elek Marton Elek added a comment - I would remove the port from actual hostname before the domain name matching. Sometimes I need to add the port to 'ozone.s3g.domain.name', sometimes not. (in case of direct aws cli usage the port is part of the host name but in case of mitm proxy it's removed by the proxy). Not a big deal but would make my life easier

          elek

          I have not tested with mitmproxy, if it solves the testing issue, we can definitely fix it. It is a small change.

           

          danilocop

          For this change, we need to update VirtualHostStyleFilter.java. 

           

          bharat Bharat Viswanadham added a comment - elek I have not tested with mitmproxy, if it solves the testing issue, we can definitely fix it. It is a small change.   danilocop For this change, we need to update VirtualHostStyleFilter.java.   
          danilocop Danilo Perez added a comment -

          Hello bharatviswa 

           

          Ack.!!! thank you

          danilocop Danilo Perez added a comment - Hello bharatviswa     Ack.!!! thank you
          bharat Bharat Viswanadham added a comment - - edited

          Hi danilocop

          Thank You for fixing this issue.

          Patch LGTM. As discussed offline we can add a simple unit test for this new change, where we can have URL with port, and config withoutthe port. So this way we can check newly added code behavior.

           

          Edit: I think no need of adding a new test, as the existing tests will fail because

          ozone.s3g.domain.name is set to s3 gateway http address. So, we need to update it to remove the port name from it, to make tests pass. This will be enough to test the new code.

           

          bharat Bharat Viswanadham added a comment - - edited Hi danilocop Thank You for fixing this issue. Patch LGTM. As discussed offline we can add a simple unit test for this new change, where we can have URL with port, and config withoutthe port. So this way we can check newly added code behavior.   Edit: I think no need of adding a new test, as the existing tests will fail because ozone.s3g.domain.name is set to s3 gateway http address. So, we need to update it to remove the port name from it, to make tests pass. This will be enough to test the new code.  
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
                trunk Compile Tests
          +1 mvninstall 22m 16s trunk passed
          +1 compile 0m 25s trunk passed
          +1 checkstyle 0m 16s trunk passed
          +1 mvnsite 0m 27s trunk passed
          +1 shadedclient 12m 6s branch has no errors when building and testing our client artifacts.
          +1 findbugs 0m 34s trunk passed
          +1 javadoc 0m 19s trunk passed
                Patch Compile Tests
          +1 mvninstall 0m 26s the patch passed
          +1 compile 0m 21s the patch passed
          +1 javac 0m 21s the patch passed
          -0 checkstyle 0m 12s hadoop-ozone/s3gateway: The patch generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0)
          +1 mvnsite 0m 23s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 shadedclient 12m 33s patch has no errors when building and testing our client artifacts.
          +1 findbugs 0m 38s the patch passed
          +1 javadoc 0m 16s the patch passed
                Other Tests
          -1 unit 0m 28s s3gateway in the patch failed.
          +1 asflicense 0m 25s The patch does not generate ASF License warnings.
          52m 47s



          Reason Tests
          Failed junit tests hadoop.ozone.s3.TestVirtualHostStyleFilter



          Subsystem Report/Notes
          Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1
          JIRA Issue HDDS-573
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12945064/HDDS-573.00.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
          uname Linux 93ba44f33620 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/patchprocess/precommit/personality/provided.sh
          git revision trunk / f6498af
          maven version: Apache Maven 3.3.9
          Default Java 1.8.0_181
          findbugs v3.1.0-RC1
          checkstyle https://builds.apache.org/job/PreCommit-HDDS-Build/1479/artifact/out/diff-checkstyle-hadoop-ozone_s3gateway.txt
          unit https://builds.apache.org/job/PreCommit-HDDS-Build/1479/artifact/out/patch-unit-hadoop-ozone_s3gateway.txt
          Test Results https://builds.apache.org/job/PreCommit-HDDS-Build/1479/testReport/
          Max. process+thread count 308 (vs. ulimit of 10000)
          modules C: hadoop-ozone/s3gateway U: hadoop-ozone/s3gateway
          Console output https://builds.apache.org/job/PreCommit-HDDS-Build/1479/console
          Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.       trunk Compile Tests +1 mvninstall 22m 16s trunk passed +1 compile 0m 25s trunk passed +1 checkstyle 0m 16s trunk passed +1 mvnsite 0m 27s trunk passed +1 shadedclient 12m 6s branch has no errors when building and testing our client artifacts. +1 findbugs 0m 34s trunk passed +1 javadoc 0m 19s trunk passed       Patch Compile Tests +1 mvninstall 0m 26s the patch passed +1 compile 0m 21s the patch passed +1 javac 0m 21s the patch passed -0 checkstyle 0m 12s hadoop-ozone/s3gateway: The patch generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) +1 mvnsite 0m 23s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedclient 12m 33s patch has no errors when building and testing our client artifacts. +1 findbugs 0m 38s the patch passed +1 javadoc 0m 16s the patch passed       Other Tests -1 unit 0m 28s s3gateway in the patch failed. +1 asflicense 0m 25s The patch does not generate ASF License warnings. 52m 47s Reason Tests Failed junit tests hadoop.ozone.s3.TestVirtualHostStyleFilter Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 JIRA Issue HDDS-573 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12945064/HDDS-573.00.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux 93ba44f33620 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / f6498af maven version: Apache Maven 3.3.9 Default Java 1.8.0_181 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HDDS-Build/1479/artifact/out/diff-checkstyle-hadoop-ozone_s3gateway.txt unit https://builds.apache.org/job/PreCommit-HDDS-Build/1479/artifact/out/patch-unit-hadoop-ozone_s3gateway.txt Test Results https://builds.apache.org/job/PreCommit-HDDS-Build/1479/testReport/ Max. process+thread count 308 (vs. ulimit of 10000) modules C: hadoop-ozone/s3gateway U: hadoop-ozone/s3gateway Console output https://builds.apache.org/job/PreCommit-HDDS-Build/1479/console Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          danilocop Danilo Perez added a comment -

          Hello bharatviswa

           

          Thank you! As we discussed offline, I'm attaching the new patch with the new changes in place.

           

          danilocop Danilo Perez added a comment - Hello bharatviswa   Thank you! As we discussed offline, I'm attaching the new patch with the new changes in place.  
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 7s HDDS-573 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 7s HDDS-573 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue HDDS-573 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12945220/HDDS-573.00.patch Console output https://builds.apache.org/job/PreCommit-HDDS-Build/1494/console Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

          Hi danilocop

          Can you rebase the patch, it is no more applying to trunk.

           

          And no need to delete old patches, you can upload a new patch with .01.

          bharat Bharat Viswanadham added a comment - Hi danilocop Can you rebase the patch, it is no more applying to trunk.   And no need to delete old patches, you can upload a new patch with .01.
          bharat Bharat Viswanadham added a comment - - edited

          Thank You danilocop for this fix.

          I have used your patch and rebased on top of the trunk to make forward progress.

          bharat Bharat Viswanadham added a comment - - edited Thank You danilocop for this fix. I have used your patch and rebased on top of the trunk to make forward progress.
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 38s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
                trunk Compile Tests
          +1 mvninstall 28m 15s trunk passed
          +1 compile 0m 31s trunk passed
          +1 checkstyle 0m 18s trunk passed
          +1 mvnsite 0m 35s trunk passed
          +1 shadedclient 14m 6s branch has no errors when building and testing our client artifacts.
          +1 findbugs 0m 43s trunk passed
          +1 javadoc 0m 24s trunk passed
                Patch Compile Tests
          +1 mvninstall 0m 38s the patch passed
          +1 compile 0m 29s the patch passed
          +1 javac 0m 29s the patch passed
          +1 checkstyle 0m 14s the patch passed
          +1 mvnsite 0m 32s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 shadedclient 14m 37s patch has no errors when building and testing our client artifacts.
          +1 findbugs 0m 46s the patch passed
          +1 javadoc 0m 19s the patch passed
                Other Tests
          -1 unit 0m 33s s3gateway in the patch failed.
          +1 asflicense 0m 29s The patch does not generate ASF License warnings.
          64m 46s



          Reason Tests
          Failed junit tests hadoop.ozone.s3.endpoint.TestRootList



          Subsystem Report/Notes
          Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f
          JIRA Issue HDDS-573
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12945939/HDDS-573.01.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
          uname Linux 125f2a8ded71 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14 08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/patchprocess/precommit/personality/provided.sh
          git revision trunk / 199703f
          maven version: Apache Maven 3.3.9
          Default Java 1.8.0_181
          findbugs v3.1.0-RC1
          unit https://builds.apache.org/job/PreCommit-HDDS-Build/1534/artifact/out/patch-unit-hadoop-ozone_s3gateway.txt
          Test Results https://builds.apache.org/job/PreCommit-HDDS-Build/1534/testReport/
          Max. process+thread count 340 (vs. ulimit of 10000)
          modules C: hadoop-ozone/s3gateway U: hadoop-ozone/s3gateway
          Console output https://builds.apache.org/job/PreCommit-HDDS-Build/1534/console
          Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 38s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 28m 15s trunk passed +1 compile 0m 31s trunk passed +1 checkstyle 0m 18s trunk passed +1 mvnsite 0m 35s trunk passed +1 shadedclient 14m 6s branch has no errors when building and testing our client artifacts. +1 findbugs 0m 43s trunk passed +1 javadoc 0m 24s trunk passed       Patch Compile Tests +1 mvninstall 0m 38s the patch passed +1 compile 0m 29s the patch passed +1 javac 0m 29s the patch passed +1 checkstyle 0m 14s the patch passed +1 mvnsite 0m 32s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedclient 14m 37s patch has no errors when building and testing our client artifacts. +1 findbugs 0m 46s the patch passed +1 javadoc 0m 19s the patch passed       Other Tests -1 unit 0m 33s s3gateway in the patch failed. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 64m 46s Reason Tests Failed junit tests hadoop.ozone.s3.endpoint.TestRootList Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f JIRA Issue HDDS-573 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12945939/HDDS-573.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux 125f2a8ded71 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14 08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / 199703f maven version: Apache Maven 3.3.9 Default Java 1.8.0_181 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-HDDS-Build/1534/artifact/out/patch-unit-hadoop-ozone_s3gateway.txt Test Results https://builds.apache.org/job/PreCommit-HDDS-Build/1534/testReport/ Max. process+thread count 340 (vs. ulimit of 10000) modules C: hadoop-ozone/s3gateway U: hadoop-ozone/s3gateway Console output https://builds.apache.org/job/PreCommit-HDDS-Build/1534/console Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          elek Marton Elek added a comment -

          +1 tested and worked well:

           

          curl -v  -X HEAD  http://b1xxx.localhost:9878/ 
           
          Warning: Setting custom HTTP method to HEAD with -X/--request may not work the
          Warning: way you want. Consider using -I/--head instead.
          *   Trying ::1...
          * TCP_NODELAY set
          * Connected to b1xxx.localhost (::1) port 9878 (#0)
          > HEAD / HTTP/1.1
          > Host: b1xxx.localhost:9878
          > User-Agent: curl/7.61.1
          > Accept: */*
          >
          < HTTP/1.1 200 OK
          < Date: Mon, 29 Oct 2018 13:42:52 GMT
          < Cache-Control: no-cache
          < Expires: Mon, 29 Oct 2018 13:42:52 GMT
          < Date: Mon, 29 Oct 2018 13:42:52 GMT
          < Pragma: no-cache
          < Content-Type: text/xml
          < X-Content-Type-Options: nosniff
          < X-FRAME-OPTIONS: SAMEORIGIN
          < X-XSS-Protection: 1; mode=block
          < Server: Ozone
          < x-amz-id-2: Kri68tFVQM4y15
          < x-amz-request-id: 00e5fc67-9d30-4588-9947-89234d6e5a0d
          < Content-Length: 0
          <
          * Connection #0 to host b1xxx.localhost left intact
          
          curl -v -H 'Host:b1xxx.localhost'  -X HEAD  http://b1xxx.localhost:9878/
          Warning: Setting custom HTTP method to HEAD with -X/--request may not work the
          Warning: way you want. Consider using -I/--head instead.
          *   Trying ::1...
          * TCP_NODELAY set
          * Connected to b1xxx.localhost (::1) port 9878 (#0)
          > HEAD / HTTP/1.1
          > Host:b1xxx.localhost
          > User-Agent: curl/7.61.1
          > Accept: */*
          >
          < HTTP/1.1 200 OK
          < Date: Mon, 29 Oct 2018 13:42:58 GMT
          < Cache-Control: no-cache
          < Expires: Mon, 29 Oct 2018 13:42:58 GMT
          < Date: Mon, 29 Oct 2018 13:42:58 GMT
          < Pragma: no-cache
          < Content-Type: text/xml
          < X-Content-Type-Options: nosniff
          < X-FRAME-OPTIONS: SAMEORIGIN
          < X-XSS-Protection: 1; mode=block
          < Server: Ozone
          < x-amz-id-2: doPC3pm8qmS3S
          < x-amz-request-id: 97998e36-f7f2-4bcb-8e9c-266eacd3431b
          < Content-Length: 0
          <
          * Connection #0 to host b1xxx.localhost left intact
          

          Note: by default curl adds the port numbers but awscli + mitm doesn't

          Will commit it soon...

          elek Marton Elek added a comment - +1 tested and worked well:   curl -v  -X HEAD  http: //b1xxx.localhost:9878/   Warning: Setting custom HTTP method to HEAD with -X/--request may not work the Warning: way you want. Consider using -I/--head instead. *   Trying ::1... * TCP_NODELAY set * Connected to b1xxx.localhost (::1) port 9878 (#0) > HEAD / HTTP/1.1 > Host: b1xxx.localhost:9878 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 29 Oct 2018 13:42:52 GMT < Cache-Control: no-cache < Expires: Mon, 29 Oct 2018 13:42:52 GMT < Date: Mon, 29 Oct 2018 13:42:52 GMT < Pragma: no-cache < Content-Type: text/xml < X-Content-Type-Options: nosniff < X-FRAME-OPTIONS: SAMEORIGIN < X-XSS-Protection: 1; mode=block < Server: Ozone < x-amz-id-2: Kri68tFVQM4y15 < x-amz-request-id: 00e5fc67-9d30-4588-9947-89234d6e5a0d < Content-Length: 0 < * Connection #0 to host b1xxx.localhost left intact curl -v -H 'Host:b1xxx.localhost'   -X HEAD  http: //b1xxx.localhost:9878/ Warning: Setting custom HTTP method to HEAD with -X/--request may not work the Warning: way you want. Consider using -I/--head instead. *   Trying ::1... * TCP_NODELAY set * Connected to b1xxx.localhost (::1) port 9878 (#0) > HEAD / HTTP/1.1 > Host:b1xxx.localhost > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 29 Oct 2018 13:42:58 GMT < Cache-Control: no-cache < Expires: Mon, 29 Oct 2018 13:42:58 GMT < Date: Mon, 29 Oct 2018 13:42:58 GMT < Pragma: no-cache < Content-Type: text/xml < X-Content-Type-Options: nosniff < X-FRAME-OPTIONS: SAMEORIGIN < X-XSS-Protection: 1; mode=block < Server: Ozone < x-amz-id-2: doPC3pm8qmS3S < x-amz-request-id: 97998e36-f7f2-4bcb-8e9c-266eacd3431b < Content-Length: 0 < * Connection #0 to host b1xxx.localhost left intact Note: by default curl adds the port numbers but awscli + mitm doesn't Will commit it soon...
          elek Marton Elek added a comment -

          Committed to ozone-0.3/trunk

          Thank you very much danilocop for the very first contribution. Let me know if you need help for the next...

          elek Marton Elek added a comment - Committed to ozone-0.3/trunk Thank you very much danilocop for the very first contribution. Let me know if you need help for the next...
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15333 (See https://builds.apache.org/job/Hadoop-trunk-Commit/15333/)
          HDDS-573. Make VirtualHostStyleFilter port agnostic. Contributed by (elek: rev bfb720ebc84c976e264971fa655515093a695515)

          • (edit) hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/VirtualHostStyleFilter.java
          • (edit) hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/TestVirtualHostStyleFilter.java
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15333 (See https://builds.apache.org/job/Hadoop-trunk-Commit/15333/ ) HDDS-573 . Make VirtualHostStyleFilter port agnostic. Contributed by (elek: rev bfb720ebc84c976e264971fa655515093a695515) (edit) hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/VirtualHostStyleFilter.java (edit) hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/TestVirtualHostStyleFilter.java

          People

            danilocop Danilo Perez
            elek Marton Elek
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: