[IMPALA-7124] Error during data load: Can't recover key from keystore file

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 2.13.0
• Fix Version/s: Impala 2.13.0, Impala 3.1.0
• Component/s: Infrastructure
• Target Version: Impala 2.13.0
• Epic Color: ghx-label-5
• Assignee: Tim Armstrong
• Reporter: Tim Armstrong
• Created: 05/Jun/18 23:32
• Updated: 07/Jun/18 17:12
• Resolved: 06/Jun/18 17:16

Description

https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/2388/console

19:47:07 18/06/05 19:47:07 WARN kms.LoadBalancingKMSClientProvider: Aborting since the Request has failed with all KMS providers(depending on hadoop.security.kms.client.failover.max.retries=1 setting and numProviders=1) in the group OR the exception is not recoverable
19:47:07 testkey1 has not been deleted. java.io.IOException: Can't recover key for testkey1 from keystore file:/home/ubuntu/Impala/testdata/cluster/cdh5/node-1/data/kms.keystore
19:47:07 java.io.IOException: Can't recover key for testkey1 from keystore file:/home/ubuntu/Impala/testdata/cluster/cdh5/node-1/data/kms.keystore
19:47:07 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
19:47:07 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
19:47:07 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
19:47:07 	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
19:47:07 	at org.apache.hadoop.util.HttpExceptionUtils.validateResponse(HttpExceptionUtils.java:157)
19:47:07 	at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:520)
19:47:07 	at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:481)
19:47:07 	at org.apache.hadoop.crypto.key.kms.KMSClientProvider.deleteKey(KMSClientProvider.java:864)
19:47:07 	at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$16.call(LoadBalancingKMSClientProvider.java:433)
19:47:07 	at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$16.call(LoadBalancingKMSClientProvider.java:430)
19:47:07 	at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:123)
19:47:07 	at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.deleteKey(LoadBalancingKMSClientProvider.java:430)
19:47:07 	at org.apache.hadoop.crypto.key.KeyShell$DeleteCommand.execute(KeyShell.java:451)
19:47:07 	at org.apache.hadoop.crypto.key.KeyShell.run(KeyShell.java:92)
19:47:07 	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
19:47:07 	at org.apache.hadoop.crypto.key.KeyShell.main(KeyShell.java:604)
19:47:07 Error in /home/ubuntu/Impala/testdata/bin/setup-hdfs-env.sh at line 53: hadoop key delete $KEY -f