Haohui Mai converted the DN side of webhdfs to use Netty for performance and stability. He may have some experience to share.

This is the HttpFS equvalent of HADOOP-13597. Please join discussions over there.

Patch 001

• Create HttpFSServerWebServer based on HttpServer2
• Share common code with HADOOP-13597
• Convert hadoop-hdfs-httpfs from war to jar
• Update httpfs-default.xml
• Add configuration keys
• Load deprecated environment variables and issue warning
• Add “hdfs httpfs” sub-command
• Change httpfs.sh to be a wrapper of “hdfs https”
• Fix MDCFilter NPE when browsing /conf, /logLevel, /jmx, or /stacks
• Obsolete httpfs.sh version that prints Tomcat version
• Obsolete HTTP admin port for Tomcat Manager

TESTING DONE

• hdfs dfs -ls webhdfs://localhost:14000/
• hdfs dfs -ls swebhdfs://localhost:14000/“ in SSL mode
• hdfs https, hdfs —daemon start|status|stop https
• httpfs.sh run|start|status|stop
• hadoop daemonlog
• HttpFS unit tests
• /jmx, /logLevel, /conf, /stack, /logs, and /static/index.html

TODO

• Update docs: ServerSetup.md.vm

TODO in new JIRAs:

• Switch unit tests to use HttpFSServerWebServer

My private branch: https://github.com/jzhuge/hadoop/tree/HDFS-10860.001

Here is a link to some tests I use: https://github.com/jzhuge/kms-httpfs-test.

-1 overall

This message was automatically generated.

Looking into the test failures caused by my AuthenticationFilter change.

Patch 002

• Update doc index.md, ServerSetup.md.vm, and HDFSCommands.md
• Set HttpServer2.Builder#authFilterConfigurationPrefix to integrate with HttpServer2’s secret provider

TESTING DONE

• hdfs dfs -ls webhdfs://localhost:14000/
• hdfs dfs -ls swebhdfs://localhost:14000/“ in SSL mode
• hdfs https, hdfs —daemon start|status|stop httpfs
• httpfs.sh run|start|status|stop
• hadoop daemonlog
• HttpFS unit tests
• dist-test nadoop-common and hadoop-hdfs: http://dist-test.cloudera.org/job?job_id=hadoop.jzhuge.1481879379.20252, 10 unrelated test failures.
• /jmx, /logLevel, /conf, /stack, /logs, and /static/index.html

-1 overall

This message was automatically generated.

Patch 003

• Xiao's comments in HADOOP-13597

TESTING DONE

• Bats regression tests https://github.com/jzhuge/hadoop-regression-tests in unsecure and ssl mode
• Verify docs

-1 overall

This message was automatically generated.

Thanks John Zhuge for the patch. Looks good in general - I assume we need a rebase after (IMO almost ready) HADOOP-13597 is committed, will have a closer look at that time.

Only comment is let's add a very basic test class for the new HttpFSServerWebServer since it's not used anywhere else - start/stop and a simple request/response should do. Can enhance it to cover more later if needed.

Thanks Xiao Chen. Yes, we will need a rebase after HADOOP-13597 commit.

Will add unit tests for HttpFSServerWebServer.

Patch 004

• Update CommandsManual.md and SecureMode.md

TESTING DONE

• Bats regression tests https://github.com/jzhuge/hadoop-bats-tests in insecure and ssl mode
• Verify docs

To add unit tests for HttpFSServerWebServer

-1 overall

This message was automatically generated.

Patch 005

• Add unit tests for HttpFSServerWebServer

-1 overall

This message was automatically generated.

All failed unit tests pass locally.

Patch 005 does not suffer from the issue of HADOOP-13961:

Testing #1 existing source tree

1. rm -fr ~/.m2/repository/
2. ( cd hadoop-maven-plugins/ && mvn install )
3. mvn clean
4. mvn install -DskipTests -Dmaven.javadoc.skip -Pnative

Testing #2 new source tree

1. rm -fr ~/.m2/repository/
2. mvn install -DskipTests -Dmaven.javadoc.skip -Pnative

Wait for HADOOP-13992

Thanks John Zhuge for the work.
Also reviewed HADOOP-13992, close to commit. Please rebase to a version that contains the above fix once it got committed.

Also 1 nit: TestHttpFSServerWebServer could use a global test timeout.
Will have a closer look once a rebase is provided.

Patch 006

• Use SSLFactory.readSSLConfiguration introduced by HADOOP-13992
• Document use of CredentialProvider API in CredentialProviderAPI.md and ServerSetup.md.vm
• Set TestHttpFSServerWebServer global timeout to 30 seconds

TESTING DONE

• TestHttpFSServerWebServer
• Run HTTPFS_HTTP_PORT=1234 hdfs httpfs to verify the server is running on port 1234
• Run HTTPFS_SSL_KEYSTORE_PASS=abcd hdfs httpfs to expect wrong password
• Run HTTPFS_SSL_KEYSTORE_FILE=/tmp/tt hdfs httpfs to expect invalid keystore path
• HttpFS Bats regression tests https://github.com/jzhuge/hadoop-regression-tests in insecure and ssl mode
• Verify CredentialProviderAPI.html and ServerSetup.html

-1 overall

This message was automatically generated.

TestDataNodeOutlierDetectionViaMetrics failure is unrelated and it passed locally. Filed HDFS-11381 for this test failure.

Thanks John Zhuge for revving! Looks pretty close.

Some nits / questions:

• HttpFSAuthenticationFilter.java: CONF_PREFIX can be package private
• MDCFilter:java: why do we need to check getPathInfo() being null here? Do we need to check other params?
• ServerSetup.md.vm:
  • NOTE: The script `httpfs.sh` is deprecated. It is now just a wrapper of `hdfs httpfs`. I think it's more accurate to say it's a wrapper of hdfs (which is the script being called)?
  • keytool -genkey -alias tomcat -keyalg RSA let's say bye-bye to tomcat...
  • It seems the paragraph under the keytool line doesn't need the change. Since the example is given 'As the `httpfs` Unix user'.
  • Let's also have a note in the doc that tomcat is deprecated, together with its old .sh configurations. We can point to the Deprecated Environment Variables section you added later, to give more context.
  • In Deprecated Environment Variables, please move HTTPFS_TEMP to the first line to be consistent with the config code, and to better group the 'Configuration File'
• TestHttpFSServerWebServer.java: I was thinking to add a basic test, to do a simple request after startup, to make sure the server is reachable.
• index.html: This line doesn't look right /webhdfs/v1/?op=LISTSTATUS to list all keys

• Sorry should have asked this in the KMS jira. The new index.html page for loglevel/jmx/conf/logs looks nice and handy. But how does the security work here? Have you tested this in a kerberized environment? I tried locally in a pseudo-authenticated setup, it seems I can read/set everything even without the user.name= param.

Thanks Xiao Chen for the review.

HttpFSAuthenticationFilter.java: CONF_PREFIX can be package private

Fixed

MDCFilter:java: why do we need to check getPathInfo() being null here? Do we need to check other params?

GOT NPE without the check for daemonlog request

ServerSetup.md.vm:

NOTE: The script `httpfs.sh` is deprecated. It is now just a wrapper of `hdfs httpfs`. I think it's more accurate to say it's a wrapper of hdfs (which is the script being called)?

`httpfs.sh` is a wrapper of `hdfs httpfs`, not `hdfs` which has many other sub-commands.

keytool -genkey -alias tomcat -keyalg RSA let's say bye-bye to tomcat...

Fixed

It seems the paragraph under the keytool line doesn't need the change. Since the example is given 'As the `httpfs` Unix user'.

Fixed

Let's also have a note in the doc that tomcat is deprecated, together with its old .sh configurations. We can point to the Deprecated Environment Variables section you added later, to give more context.

Fixed in section 'Start/Stop HttpFS'

In Deprecated Environment Variables, please move HTTPFS_TEMP to the first line to be consistent with the config code, and to better group the 'Configuration File'

Variables are sorted by name but ok with your suggestion.

TestHttpFSServerWebServer.java: I was thinking to add a basic test, to do a simple request after startup, to make sure the server is reachable.

Added to TestStartStop

index.html: This line doesn't look right /webhdfs/v1/?op=LISTSTATUS to list all keys

Fixed

Sorry should have asked this in the KMS jira. The new index.html page for loglevel/jmx/conf/logs looks nice and handy. But how does the security work here? Have you tested this in a kerberized environment? I tried locally in a pseudo-authenticated setup, it seems I can read/set everything even without the user.name= param.

Will investigate

MDCFilter:java: why do we need to check getPathInfo() being null here? Do we need to check other params?

No, I was wrong about NPE without the check for daemonlog request. Will investigate further,

Confirmed NPE without the check for daemonlog request, and for /jmx, /conf, and /logLevel.

Sorry should have asked this in the KMS jira. The new index.html page for loglevel/jmx/conf/logs looks nice and handy. But how does the security work here? Have you tested this in a kerberized environment? I tried locally in a pseudo-authenticated setup, it seems I can read/set everything even without the user.name= param.

/conf, /jmx, /logLevel, and /stacks do report "Authentication required" when user.name is not specified.

/logs do not require authentication. This does not seem right, will file another JIRA to follow up.

Patch 007

• Xiao’s comments
• Test simple request in TestHttpFSServerWebServer#testStartStop

TESTING DONE

• TestHttpFSServerWebServer
• HttpFS Bats regression tests https://github.com/jzhuge/hadoop-regression-tests in insecure and ssl mode
• Verify ServerSetup.html and static index.html

Thanks John Zhuge.

As discussed, there're really 2 issues with HttpServer2 security:

1. authentication should be needed to access everything
2. based on 1, what level of authorization should be there for each of these servlet.

Looking at HttpServer2 code, it seems there's a hasAdministratorAccess method. So possibly we just need some configurations set. Will play with it a bit.

+1 overall

This message was automatically generated.

HttpServer2#isInstrumentationAccessAllowed requires the following conditions to return false:

• hadoop.security.instrumentation.requires.admin set to true
• hadoop.security.authorization set to true
• remote user is not set or HttpServer2#setACL is called with an ACL that disallows the remote user

Looks like we need another configuration property to set administrators for HttpFS server instrumentation access, e.g., hadoop.httpfs.http.administrators, similar to dfs.cluster.administrators.

Will need to fix KMS to call HttpServer2#setACL. Also gotta make sure all users of HttpServer2 call setACL.

Patch 008

• Add config property hadoop.httpfs.http.administrators to control access to instrumentation servlets
• Enhance TestHttpFSServerWebServer to start server on any port of localhost

TESTING DONE

• TestHttpFSServerWebServer
• HttpFS Bats regression tests https://github.com/jzhuge/hadoop-regression-tests in insecure and ssl mode
• Verify ServerSetup.html and static index.html

-1 overall

This message was automatically generated.

Thanks John Zhuge, patch 8 looks good to me. Will let this soak for a couple of days in case other audience have comments, and recheck on Friday.

Can you open the 2 jiras for:

• servlet admin control for kms, as a follow-up of HADOOP-13597.
• research why HttpServer2's logs are not following the same security model. I looked back on that class's history, but no clue so far.

Patch 009

• Update ServerSetup.md.vm

TESTING DONE

• Verify ServerSetup.html

-1 overall

This message was automatically generated.

Upgrading this to a blocker for alpha3, looks like we're close in any case but this is something I definitely want in.

Filed 2 JIRAs:

• HADOOP-14047 Require admin to access KMS instrumentation servlets
• HADOOP-14060 KMS /logs servlet should have access control

Patch 010

• Update ServerSetup.md.vm

TESTING DONE

• - Verify /jmx, /logLevel, /conf, and /stack with hadoop.httpfs.http.administrators set to “$USER”
• No access for kerberos login "hdfs"
• Access for kerberos login “$USER"
• HttpFS Bats regression tests https://github.com/jzhuge/hadoop-bats-tests and https://github.com/jzhuge/hadoop-setup-scripts in insecure, ssl, and ssl+kerberos mode

-1 overall

This message was automatically generated.

Patch 011

• Fix a typo in patch 010

Thanks for revving John! +1 pending jenkins.

-1 overall

This message was automatically generated.

TestDataNodeVolumeFailureReporting, TestNameNodeMetadataConsistency, TestThrottledAsyncChecker, TestRollingUpgrade, TestDataNodeVolumeFailure failures are unrelated and they pass locally for me.

Filed HDFS-11396 TestNameNodeMetadataConsistency#testGenerationStampInFuture timed out.

Filed HDFS-11397 TestThrottledAsyncChecker#testCancellation timed out.

+1 on patch 11, committing.

Committed to trunk! Thanks John Zhuge for the great work here, and Bob Hansen for the early pointer.

And please add a release note.
FYI release note accepts markdowns too, see HADOOP-14003 as an example.

Thanks Xiao Chen for the great help in review and commit! Thanks Bob Hansen for the comment.

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11218 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11218/)
HDFS-10860. Switch HttpFS from Tomcat to Jetty. Contributed by John (xiao: rev 69b23632c48297ac844c58fd3e21aad10c093cc8)

• (edit) hadoop-assemblies/src/main/resources/assemblies/hadoop-httpfs-dist.xml
• (edit) hadoop-common-project/hadoop-common/src/site/markdown/CredentialProviderAPI.md
• (add) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSServerWebServer.java
• (add) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/libexec/shellprofile.d/hadoop-httpfs.sh
• (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/site/markdown/index.md
• (delete) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/tomcat/logging.properties
• (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md
• (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/conf/httpfs-env.sh
• (add) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/resources/webapps/static/index.html
• (add) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSServerWebServer.java
• (edit) hadoop-common-project/hadoop-common/src/site/markdown/CommandsManual.md
• (delete) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/tomcat/server.xml
• (delete) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/tomcat/ssl-server.xml.conf
• (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/resources/httpfs-default.xml
• (add) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/resources/webapps/webhdfs/WEB-INF/web.xml
• (delete) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/libexec/httpfs-config.sh
• (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSAuthenticationFilter.java
• (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml
• (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/sbin/httpfs.sh
• (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/servlet/MDCFilter.java
• (edit) hadoop-common-project/hadoop-common/src/site/markdown/SecureMode.md
• (delete) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/tomcat/ROOT/index.html
• (delete) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/tomcat/ROOT/WEB-INF/web.xml
• (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/site/markdown/ServerSetup.md.vm