Rev01: this is an initial patch that uses AuthenticatedURL to perform SPENGO negotiation with a Kerberized cluster.

I still need to work out a good way to be backward-compatible (i.e. support http) and also need to have a better exception handling.

But I have tested it locally and it successfully connects to a Kerberized cluster and set log level correctly.

Wei-Chiu Chuang, thank you for posting the patch.

Xiaoyu Yao, did you already have a patch in progress on an earlier JIRA for this, or am I mistaken? I just want to figure out if there is a duplicate issue. If not, then we can help code review this one.

Thanks Chris Nauroth for the heads up. Yes, we have discussed this issue offline last year. Somehow I forgot to open a ticket for it. It's good see Wei-Chiu Chuang reported it and proposed the fix recently. Let's fix it here.

Thanks Chris Nauroth and Xiaoyu Yao for comments.
How do you think about support both http and https? Should the tool use an option to specify the protocol, or let it attempt to connect to https and then retry http?
For authentication, KerberosAuthenticator automatically falls back to simple, so it should not require additional configuration.

On thing to consider is that https connection may need additional parameters to supply the path to trust store file.

And some test cases as well.

Rev02: this revision adds an option to select HTTP or HTTPS and test cases.

• I basically rewrote the client side to make it more complete. The client side SSL config (keystore/truststore) comes from Hadoop configuration files.
• Rewrote the test cases to validate client command options, and run tests using both HTTP and HTTPS.
• Updated docs to reflect the additional command line parameter.
• server side is unchanged.

I did not add tests for SPENGO authentication and this code does not automatically reads Kerberos keytab to login, as it's way too complex and beyond my current understanding of Hadoop. The user of this code has to run kinit before running this command.

I'm thinking I can use URLConnectionFactory to make the implementation more compact.

Hi Wei-Chiu Chuang,

Thanks for reporting and working this issue!

A few quick comments:
1. Make sure the default behavior is the same as without passing the new switches.
2. I think it's better to move the param parsing part in doGetLevel to the dedicated param parsing method
3. Make "http" "https" contant, such as PROTOCOL_HTTP, PROTOCOL_HTTPS, and use them at the currently hardcoded places
4. I expect to see the following test result, would you please test it out?

Thanks.

Thanks for Yongjun's comments. I'm working on it.

Rev03:
This patch tests all combination of (server_scheme, client_scheme, spnego). also, ensure backward-compatibility. In addition, I rebased to incorporate the latest change in HADOOP-13098.

Thanks and please review!

Found bug in the code when hadoop daemonlog has no parameters. I'll update the patch to check for that later.

Rev04: caught two bugs: print usage if the command has no arguments, or if the first argument is not either -getlevel or -setlevel

FYI: I tested the latest code on a CDH 5.7 cluster and it works as expected.

Rev05: fixed a few exception message and usage help message, refactored the code to output stacktrace of exceptions, renamed LogLevel#checkArguments to LogLevel#getScheme, also update Hadoop Command Guide.

The v05 patch did not trigger precommit build. Re-submit the patch to re-trigger it.

v06: trigger precommit build again.

-1 overall

This message was automatically generated.

v08. fixed checkstyle, javac and whitespace issues.

+1 overall

This message was automatically generated.

v09:

1. Changed the protocol command to "[-protocol (http|https)]" instead of "[-http|-https]".
2. Also, changed the argument parsing method.
3. Updated tests and documentation acoordingly.

-1 overall

This message was automatically generated.

The failed unit test is unrelated.

v10. minor code refactoring.

-1 overall

This message was automatically generated.

Attach the branch-2 patch. The default NN port has been changed in trunk, so need to update the doc.

+1 overall

This message was automatically generated.

-1 overall

This message was automatically generated.

Thanks Wei-Chiu Chuang for the work here, +1 on rev 10. Will commit soon,

SUCCESS: Integrated in Hadoop-trunk-Commit #9873 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9873/)
HADOOP-12847. hadoop daemonlog should support https and SPNEGO for (yzhang: rev 34cc21f6d1a293d92613defba38e8ae810db4c71)

• hadoop-common-project/hadoop-common/src/site/markdown/CommandsManual.md
• hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/log/TestLogLevel.java
• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/log/LogLevel.java

Committed to trunk and branch-2.

Thanks Wei-Chiu Chuang for the contribution, and other folks for reviewing.

Attach the branch-2.8 patch.

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10709 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10709/)
YARN-5172. Update yarn daemonlog documentation due to HADOOP-12847. (jlowe: rev b4a8fbcbbc5ea4ab3087ecf913839a53f32be113)

• (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnCommands.md