Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3239

WebAppProxy does not support a final tracking url which has query fragments and params

    Details

    • Hadoop Flags:
      Reviewed

      Description

      Examples of failures:

      Expected: http://uihost:8080/#/main/views/TEZ/0.5.2.2.2.2.0-947/tez?viewPath=%2F%23%2Ftez-app%2Fapplication_1424384418229_0005
      Actual: http://uihost:8080

      Tried with a minor change to remove the "#". Saw a different issue:

      Expected: http://uihost:8080/views/TEZ/0.5.2.2.2.2.0-947/tez?viewPath=%2F%23%2Ftez-app%2Fapplication_1424388018547_0001
      Actual: http://uihost:8080/views/TEZ/0.5.2.2.2.2.0-947/tez/

      yarn application -status appId returns the expected value correctly. However, invoking an http get on http://rm:8088/proxy/appId/ returns the wrong value.

        Activity

        Hide
        hitesh Hitesh Shah added a comment -

        Jason Lowe Jonathan Eagles Have you come across any cases such as this?

        Show
        hitesh Hitesh Shah added a comment - Jason Lowe Jonathan Eagles Have you come across any cases such as this?
        Hide
        jianhe Jian He added a comment -

        Uploaded a patch to fix the issue by appending user-provided path and query parameters to the registered tracking url.

        Show
        jianhe Jian He added a comment - Uploaded a patch to fix the issue by appending user-provided path and query parameters to the registered tracking url.
        Hide
        hadoopqa Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12700132/YARN-3239.1.patch
        against trunk revision fe7a302.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy.

        Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6696//testReport/
        Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6696//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12700132/YARN-3239.1.patch against trunk revision fe7a302. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6696//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6696//console This message is automatically generated.
        Hide
        jlowe Jason Lowe added a comment -

        the WebProxyServlet does support appending the client-provided path and query strings, as I can verify it by simply curling a regular MapReduce app's proxy URL. For example:

        $ curl 'http://rmhost:8088/proxy/application_1424469840513_0001/foo/bar/boo?viewPath=a&other=b'     
        <html>
          <head>
            <title>
              Moved
            </title>
          </head>
          <body>
            <h1>
              Moved
            </h1>
            <div>
              Content has moved 
              <a href="http://jhshost:19888/jobhistory/job/job_1424469840513_0001/foo/bar/boo?viewPath=a&amp;other=b">here</a>
            </div>
          </body>
        </html>
        

        However it does not properly handle URI fragments supplied by either the app or the client, and it doesn't honor query parameters set in the tracking URL by the app.

        Thanks for the patch, Jian! Patch looks good overall, and merging query strings between the client and the app-specified URL seems reasonable. However it doesn't look like we're honoring any fragments provided by the client, and that would break clients for any app UIs that generate fragment-containing links.

        Show
        jlowe Jason Lowe added a comment - the WebProxyServlet does support appending the client-provided path and query strings, as I can verify it by simply curling a regular MapReduce app's proxy URL. For example: $ curl 'http://rmhost:8088/proxy/application_1424469840513_0001/foo/bar/boo?viewPath=a&other=b' <html> <head> <title> Moved </title> </head> <body> <h1> Moved </h1> <div> Content has moved <a href="http://jhshost:19888/jobhistory/job/job_1424469840513_0001/foo/bar/boo?viewPath=a&amp;other=b">here</a> </div> </body> </html> However it does not properly handle URI fragments supplied by either the app or the client, and it doesn't honor query parameters set in the tracking URL by the app. Thanks for the patch, Jian! Patch looks good overall, and merging query strings between the client and the app-specified URL seems reasonable. However it doesn't look like we're honoring any fragments provided by the client, and that would break clients for any app UIs that generate fragment-containing links.
        Hide
        jianhe Jian He added a comment -

        thanks for reviewing, Jason !

        However it doesn't look like we're honoring any fragments provided by the client,

        I was trying to do that, but after doing some research, looks like the fragments are not supposed to be sent to the server. So the proxy won't receive the fragments specified by the user, they are automatically stripped off.

        Show
        jianhe Jian He added a comment - thanks for reviewing, Jason ! However it doesn't look like we're honoring any fragments provided by the client, I was trying to do that, but after doing some research, looks like the fragments are not supposed to be sent to the server. So the proxy won't receive the fragments specified by the user, they are automatically stripped off.
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        However it doesn't look like we're honoring any fragments provided by the client, and that would break clients for any app UIs that generate fragment-containing links.

        Not sure how we can even combine fragments - isn't only one fragment applicable against a URI?

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - However it doesn't look like we're honoring any fragments provided by the client, and that would break clients for any app UIs that generate fragment-containing links. Not sure how we can even combine fragments - isn't only one fragment applicable against a URI?
        Hide
        jlowe Jason Lowe added a comment -

        I was trying to do that, but after doing some research, looks like the fragments are not supposed to be sent to the server. So the proxy won't receive the fragments specified by the user, they are automatically stripped off.

        Ah, yes that makes sense. I'm +1 on the patch then.

        Not sure how we can even combine fragments - isn't only one fragment applicable against a URI?

        That's what I was wondering about and thinking we may want to complain if both the app and the client are trying to mess with fragments at the same time. However it sounds like we won't have the ability to tell if the client is using a fragment since it doesn't send it to the server.

        Show
        jlowe Jason Lowe added a comment - I was trying to do that, but after doing some research, looks like the fragments are not supposed to be sent to the server. So the proxy won't receive the fragments specified by the user, they are automatically stripped off. Ah, yes that makes sense. I'm +1 on the patch then. Not sure how we can even combine fragments - isn't only one fragment applicable against a URI? That's what I was wondering about and thinking we may want to complain if both the app and the client are trying to mess with fragments at the same time. However it sounds like we won't have the ability to tell if the client is using a fragment since it doesn't send it to the server.
        Hide
        jlowe Jason Lowe added a comment -

        Any other comments? Otherwise I will commit this tomorrow.

        Show
        jlowe Jason Lowe added a comment - Any other comments? Otherwise I will commit this tomorrow.
        Hide
        hitesh Hitesh Shah added a comment -

        Tested manually by applying this patch. Works fine with the kind of urls Tez is using.

        Show
        hitesh Hitesh Shah added a comment - Tested manually by applying this patch. Works fine with the kind of urls Tez is using.
        Hide
        jlowe Jason Lowe added a comment -

        Committing this.

        Show
        jlowe Jason Lowe added a comment - Committing this.
        Hide
        jlowe Jason Lowe added a comment -

        Thanks, Jian! I committed this to trunk and branch-2.

        Show
        jlowe Jason Lowe added a comment - Thanks, Jian! I committed this to trunk and branch-2.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #7198 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7198/)
        YARN-3239. WebAppProxy does not support a final tracking url which has query fragments and params. Contributed by Jian He (jlowe: rev 1a68fc43464d3948418f453bb2f80df7ce773097)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServlet.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7198 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7198/ ) YARN-3239 . WebAppProxy does not support a final tracking url which has query fragments and params. Contributed by Jian He (jlowe: rev 1a68fc43464d3948418f453bb2f80df7ce773097) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServlet.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #116 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/116/)
        YARN-3239. WebAppProxy does not support a final tracking url which has query fragments and params. Contributed by Jian He (jlowe: rev 1a68fc43464d3948418f453bb2f80df7ce773097)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServlet.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #116 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/116/ ) YARN-3239 . WebAppProxy does not support a final tracking url which has query fragments and params. Contributed by Jian He (jlowe: rev 1a68fc43464d3948418f453bb2f80df7ce773097) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServlet.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java hadoop-yarn-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk #850 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/850/)
        YARN-3239. WebAppProxy does not support a final tracking url which has query fragments and params. Contributed by Jian He (jlowe: rev 1a68fc43464d3948418f453bb2f80df7ce773097)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServlet.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #850 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/850/ ) YARN-3239 . WebAppProxy does not support a final tracking url which has query fragments and params. Contributed by Jian He (jlowe: rev 1a68fc43464d3948418f453bb2f80df7ce773097) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServlet.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2048 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2048/)
        YARN-3239. WebAppProxy does not support a final tracking url which has query fragments and params. Contributed by Jian He (jlowe: rev 1a68fc43464d3948418f453bb2f80df7ce773097)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServlet.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2048 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2048/ ) YARN-3239 . WebAppProxy does not support a final tracking url which has query fragments and params. Contributed by Jian He (jlowe: rev 1a68fc43464d3948418f453bb2f80df7ce773097) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServlet.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #107 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/107/)
        YARN-3239. WebAppProxy does not support a final tracking url which has query fragments and params. Contributed by Jian He (jlowe: rev 1a68fc43464d3948418f453bb2f80df7ce773097)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServlet.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #107 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/107/ ) YARN-3239 . WebAppProxy does not support a final tracking url which has query fragments and params. Contributed by Jian He (jlowe: rev 1a68fc43464d3948418f453bb2f80df7ce773097) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServlet.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #116 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/116/)
        YARN-3239. WebAppProxy does not support a final tracking url which has query fragments and params. Contributed by Jian He (jlowe: rev 1a68fc43464d3948418f453bb2f80df7ce773097)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServlet.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #116 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/116/ ) YARN-3239 . WebAppProxy does not support a final tracking url which has query fragments and params. Contributed by Jian He (jlowe: rev 1a68fc43464d3948418f453bb2f80df7ce773097) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServlet.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2066 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2066/)
        YARN-3239. WebAppProxy does not support a final tracking url which has query fragments and params. Contributed by Jian He (jlowe: rev 1a68fc43464d3948418f453bb2f80df7ce773097)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServlet.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2066 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2066/ ) YARN-3239 . WebAppProxy does not support a final tracking url which has query fragments and params. Contributed by Jian He (jlowe: rev 1a68fc43464d3948418f453bb2f80df7ce773097) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServlet.java hadoop-yarn-project/CHANGES.txt
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Patch did not apply cleanly to 2.6.1. Had to fix minor merge conflicts in WebAppProxyServlet.java and removed ProxyUtils usage in test-case.

        Pulled this into 2.6.1. Ran compilation and TestWebAppProxyServlet before the push.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Patch did not apply cleanly to 2.6.1. Had to fix minor merge conflicts in WebAppProxyServlet.java and removed ProxyUtils usage in test-case. Pulled this into 2.6.1. Ran compilation and TestWebAppProxyServlet before the push.

          People

          • Assignee:
            jianhe Jian He
            Reporter:
            hitesh Hitesh Shah
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development