[HDFS-13690] Improve error message when creating encryption zone while KMS is unreachable

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 3.2.0
• Component/s: encryption, hdfs, kms
• Target Version/s: 3.2.0
• Hadoop Flags: Reviewed
• Assignee: Kitti Nanasi
• Reporter: Kitti Nanasi
• Created: 20/Jun/18 10:02
• Updated: 17/Jul/18 06:06
• Resolved: 16/Jul/18 20:20

Description

In failure testing, we stopped the KMS and then tried to run some encryption related commands.

hdfs crypto -createZone will complain with a short "RemoteException: Connection refused." This message could be improved to explain that we cannot connect to the KMSClientProvier.

For example, hadoop key list while KMS is down will error:

 -bash-4.1$ hadoop key list
 Cannot list keys for KeyProvider: KMSClientProvider[http://hdfs-cdh5-vanilla-1.vpc.cloudera.com:16000/kms/v1/]: Connection refusedjava.net.ConnectException: Connection refused
 at java.net.PlainSocketImpl.socketConnect(Native Method)
 at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
 at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
 at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
 at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
 at java.net.Socket.connect(Socket.java:579)
 at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
 at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
 at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
 at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
 at sun.net.www.http.HttpClient.New(HttpClient.java:308)
 at sun.net.www.http.HttpClient.New(HttpClient.java:326)
 at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:996)
 at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:932)
 at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:850)
 at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:186)
 at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:125)
 at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216)
 at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:312)
 at org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:397)
 at org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:392)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:415)
 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
 at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:392)
 at org.apache.hadoop.crypto.key.kms.KMSClientProvider.getKeys(KMSClientProvider.java:479)
 at org.apache.hadoop.crypto.key.KeyShell$ListCommand.execute(KeyShell.java:286)
 at org.apache.hadoop.crypto.key.KeyShell.run(KeyShell.java:79)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
 at org.apache.hadoop.crypto.key.KeyShell.main(KeyShell.java:513)

Issue Links

• is related to: HADOOP-15596 Stack trace should not be printed out when running hadoop key commands