Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-5532

Enable the webhdfs by default to support new HDFS web UI

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 2.3.0
    • Component/s: webhdfs
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Recently in HDFS-5444, new HDFS web UI is made as default.
      but this needs webhdfs to be enabled.

      WebHDFS is disabled by default. Lets enable it by default to support new really cool web UI.

      1. HDFS-5532.patch
        3 kB
        Vinayakumar B
      2. HDFS-5532.patch
        3 kB
        Vinayakumar B

        Issue Links

          Activity

          Vinayakumar B created issue -
          Hide
          Jing Zhao added a comment -

          +1.

          Show
          Jing Zhao added a comment - +1.
          Hide
          Vinayakumar B added a comment -

          Attached simple patch to enable the webhdfs
          Please review

          Show
          Vinayakumar B added a comment - Attached simple patch to enable the webhdfs Please review
          Vinayakumar B made changes -
          Field Original Value New Value
          Attachment HDFS-5532.patch [ 12615050 ]
          Vinayakumar B made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Affects Version/s 3.0.0 [ 12320356 ]
          Vinayakumar B made changes -
          Component/s webhdfs [ 12319200 ]
          Hide
          Haohui Mai added a comment -

          The patch looks good, but I think you don't need to start a minidfscluster at all. You can check the value by looking into HdfsConfiguration. For example,

          HdfsConfiguration conf = new HdfsConfiguration(true);
          AssertTrue(conf.getBoolean(DFS_WEBHDFS_ENABLED_KEY, false));
          
          Show
          Haohui Mai added a comment - The patch looks good, but I think you don't need to start a minidfscluster at all. You can check the value by looking into HdfsConfiguration. For example, HdfsConfiguration conf = new HdfsConfiguration( true ); AssertTrue(conf.getBoolean(DFS_WEBHDFS_ENABLED_KEY, false ));
          Hide
          Vinayakumar B added a comment -

          Thanks for taking a look at patch Haohui Mai,
          I too thought about it. That would look pretty stright too. But I wanted to check the working cluster too.
          If not required then I can remove and change as you mentioned.

          Show
          Vinayakumar B added a comment - Thanks for taking a look at patch Haohui Mai , I too thought about it. That would look pretty stright too. But I wanted to check the working cluster too. If not required then I can remove and change as you mentioned.
          Hide
          Haohui Mai added a comment -

          In my opinion we can do the simple test here and avoid the cost of bringing up a minidfscluster.

          We always have to option to merge this test with other integration tests for webhdfs if it is necessary.

          Show
          Haohui Mai added a comment - In my opinion we can do the simple test here and avoid the cost of bringing up a minidfscluster. We always have to option to merge this test with other integration tests for webhdfs if it is necessary.
          Hide
          Vinayakumar B added a comment -

          Ok then, I will update the test.

          Show
          Vinayakumar B added a comment - Ok then, I will update the test.
          Hide
          Vinayakumar B added a comment -

          Here is updated patch

          Show
          Vinayakumar B added a comment - Here is updated patch
          Vinayakumar B made changes -
          Attachment HDFS-5532.patch [ 12615052 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12615052/HDFS-5532.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5519//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5519//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12615052/HDFS-5532.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5519//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5519//console This message is automatically generated.
          Hide
          Jing Zhao added a comment -

          Thanks for the patch Vinay and thanks for the review Haohui! +1. I've committed this to trunk and branch-2.

          Show
          Jing Zhao added a comment - Thanks for the patch Vinay and thanks for the review Haohui! +1. I've committed this to trunk and branch-2.
          Jing Zhao made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 2.3.0 [ 12324588 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #4775 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4775/)
          HDFS-5532. Enable the webhdfs by default to support new HDFS web UI. Contributed by Vinay. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1544047)

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #4775 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4775/ ) HDFS-5532 . Enable the webhdfs by default to support new HDFS web UI. Contributed by Vinay. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1544047 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #398 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/398/)
          HDFS-5532. Enable the webhdfs by default to support new HDFS web UI. Contributed by Vinay. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1544047)

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #398 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/398/ ) HDFS-5532 . Enable the webhdfs by default to support new HDFS web UI. Contributed by Vinay. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1544047 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #1589 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1589/)
          HDFS-5532. Enable the webhdfs by default to support new HDFS web UI. Contributed by Vinay. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1544047)

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1589 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1589/ ) HDFS-5532 . Enable the webhdfs by default to support new HDFS web UI. Contributed by Vinay. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1544047 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #1615 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1615/)
          HDFS-5532. Enable the webhdfs by default to support new HDFS web UI. Contributed by Vinay. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1544047)

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1615 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1615/ ) HDFS-5532 . Enable the webhdfs by default to support new HDFS web UI. Contributed by Vinay. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1544047 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
          Arun C Murthy made changes -
          Fix Version/s 2.3.0 [ 12325255 ]
          Fix Version/s 2.4.0 [ 12324588 ]
          Hide
          Steve Loughran added a comment -

          This stopped my secure cluster from working as I hadn't defined dfs.web.authentication.kerberos.principal, enabling webhdfs requires this conf option to be set, so instead of getting a namenode, I got a stack trace

          2014-01-29 15:42:58,630 ERROR org.apache.hadoop.http.HttpServer2: WebHDFS and security are enabled, but configuration proper
          ty 'dfs.web.authentication.kerberos.principal' is not set.
          2014-01-29 15:42:58,630 INFO org.apache.hadoop.http.HttpServer2: Added filter 'SPNEGO' (class=org.apache.hadoop.hdfs.web.Aut
          hFilter)
          2014-01-29 15:42:58,631 INFO org.apache.hadoop.http.HttpServer2: addJerseyResourcePackage: packageName=org.apache.hadoop.hdf
          s.server.namenode.web.resources;org.apache.hadoop.hdfs.web.resources, pathSpec=/webhdfs/v1/*
          2014-01-29 15:42:58,658 INFO org.apache.hadoop.http.HttpServer2: Adding Kerberos (SPNEGO) filter to getDelegationToken
          2014-01-29 15:42:58,662 INFO org.apache.hadoop.http.HttpServer2: Adding Kerberos (SPNEGO) filter to renewDelegationToken
          2014-01-29 15:42:58,663 INFO org.apache.hadoop.http.HttpServer2: Adding Kerberos (SPNEGO) filter to cancelDelegationToken
          2014-01-29 15:42:58,663 INFO org.apache.hadoop.http.HttpServer2: Adding Kerberos (SPNEGO) filter to fsck
          2014-01-29 15:42:58,671 INFO org.apache.hadoop.http.HttpServer2: Adding Kerberos (SPNEGO) filter to getimage
          2014-01-29 15:42:58,748 INFO org.apache.hadoop.http.HttpServer2: Jetty bound to port 50070
          2014-01-29 15:42:58,748 INFO org.mortbay.log: jetty-6.1.26
          2014-01-29 15:42:58,941 INFO org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler: Login using key
          tab /home/stevel/conf/hdfs.keytab, for principal HTTP/ubuntu@COTHAM
          2014-01-29 15:42:58,981 INFO org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler: Initialized, pr
          incipal [HTTP/ubuntu@COTHAM] from keytab [/home/stevel/conf/hdfs.keytab]
          2014-01-29 15:42:58,981 WARN org.apache.hadoop.security.authentication.server.AuthenticationFilter: 'signature.secret' confi
          guration not set, using a random value as secret
          2014-01-29 15:42:58,982 WARN org.mortbay.log: failed SPNEGO: javax.servlet.ServletException: javax.servlet.ServletException:
           Principal not defined in configuration
          2014-01-29 15:42:58,982 WARN org.mortbay.log: Failed startup of context org.mortbay.jetty.webapp.WebAppContext@167a465{/,fil
          e:/home/stevel/hadoop/share/hadoop/hdfs/webapps/hdfs}
          javax.servlet.ServletException: javax.servlet.ServletException: Principal not defined in configuration
                  at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.init(KerberosAuthenticationHandler
          .java:203)
          
          Show
          Steve Loughran added a comment - This stopped my secure cluster from working as I hadn't defined dfs.web.authentication.kerberos.principal , enabling webhdfs requires this conf option to be set, so instead of getting a namenode, I got a stack trace 2014-01-29 15:42:58,630 ERROR org.apache.hadoop.http.HttpServer2: WebHDFS and security are enabled, but configuration proper ty 'dfs.web.authentication.kerberos.principal' is not set. 2014-01-29 15:42:58,630 INFO org.apache.hadoop.http.HttpServer2: Added filter 'SPNEGO' (class=org.apache.hadoop.hdfs.web.Aut hFilter) 2014-01-29 15:42:58,631 INFO org.apache.hadoop.http.HttpServer2: addJerseyResourcePackage: packageName=org.apache.hadoop.hdf s.server.namenode.web.resources;org.apache.hadoop.hdfs.web.resources, pathSpec=/webhdfs/v1/* 2014-01-29 15:42:58,658 INFO org.apache.hadoop.http.HttpServer2: Adding Kerberos (SPNEGO) filter to getDelegationToken 2014-01-29 15:42:58,662 INFO org.apache.hadoop.http.HttpServer2: Adding Kerberos (SPNEGO) filter to renewDelegationToken 2014-01-29 15:42:58,663 INFO org.apache.hadoop.http.HttpServer2: Adding Kerberos (SPNEGO) filter to cancelDelegationToken 2014-01-29 15:42:58,663 INFO org.apache.hadoop.http.HttpServer2: Adding Kerberos (SPNEGO) filter to fsck 2014-01-29 15:42:58,671 INFO org.apache.hadoop.http.HttpServer2: Adding Kerberos (SPNEGO) filter to getimage 2014-01-29 15:42:58,748 INFO org.apache.hadoop.http.HttpServer2: Jetty bound to port 50070 2014-01-29 15:42:58,748 INFO org.mortbay.log: jetty-6.1.26 2014-01-29 15:42:58,941 INFO org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler: Login using key tab /home/stevel/conf/hdfs.keytab, for principal HTTP/ubuntu@COTHAM 2014-01-29 15:42:58,981 INFO org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler: Initialized, pr incipal [HTTP/ubuntu@COTHAM] from keytab [/home/stevel/conf/hdfs.keytab] 2014-01-29 15:42:58,981 WARN org.apache.hadoop.security.authentication.server.AuthenticationFilter: 'signature.secret' confi guration not set, using a random value as secret 2014-01-29 15:42:58,982 WARN org.mortbay.log: failed SPNEGO: javax.servlet.ServletException: javax.servlet.ServletException: Principal not defined in configuration 2014-01-29 15:42:58,982 WARN org.mortbay.log: Failed startup of context org.mortbay.jetty.webapp.WebAppContext@167a465{/,fil e:/home/stevel/hadoop/share/hadoop/hdfs/webapps/hdfs} javax.servlet.ServletException: javax.servlet.ServletException: Principal not defined in configuration at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.init(KerberosAuthenticationHandler .java:203)
          Steve Loughran made changes -
          Link This issue relates to HDFS-5854 [ HDFS-5854 ]
          Hide
          Haohui Mai added a comment -

          Do you think defining a default value of the principal (say, $

          {namenode.principal}

          ) is a viable approach to fix this?

          Show
          Haohui Mai added a comment - Do you think defining a default value of the principal (say, $ {namenode.principal} ) is a viable approach to fix this?
          Hide
          Arpit Gupta added a comment -

          A property already exists for this

          dfs.web.authentication.kerberos.principal

          And there is another property that defines where your keytab is. I am not sure we should set defaults for these as we dont do them for any other principal and keytab properties.

          We should probably update any documentation we have regarding secure setup.

          Show
          Arpit Gupta added a comment - A property already exists for this dfs.web.authentication.kerberos.principal And there is another property that defines where your keytab is. I am not sure we should set defaults for these as we dont do them for any other principal and keytab properties. We should probably update any documentation we have regarding secure setup.
          Hide
          Steve Loughran added a comment -

          It's not enough to set the dfs.web principal, you have to set up more for the DFS Browser to work: see HDFS-5854 for the details.

          i think the docs need to be reviewed, but also changes.txt to highlight that this is potentially incompatible if you had a secure cluster not yet set up for webhdfs

          Show
          Steve Loughran added a comment - It's not enough to set the dfs.web principal, you have to set up more for the DFS Browser to work: see HDFS-5854 for the details. i think the docs need to be reviewed, but also changes.txt to highlight that this is potentially incompatible if you had a secure cluster not yet set up for webhdfs
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          22h 42m 1 Vinayakumar B 21/Nov/13 01:42
          Patch Available Patch Available Resolved Resolved
          5h 38m 1 Jing Zhao 21/Nov/13 07:21
          Resolved Resolved Closed Closed
          95d 13h 36m 1 Arun C Murthy 24/Feb/14 20:58

            People

            • Assignee:
              Vinayakumar B
              Reporter:
              Vinayakumar B
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development