[HBASE-10890] ExportSnapshot needs to add acquired token to job

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.96.1, 0.98.1
• Fix Version/s: 0.99.0, 0.98.2
• Component/s: snapshots
• Hadoop Flags: Reviewed
• Assignee: Jerry He
• Reporter: Jerry He
• Created: 01/Apr/14 23:13
• Updated: 21/Feb/15 23:32
• Resolved: 04/Apr/14 00:00

Description

ExportSnapshot failed when exporting to remote secure kerberos cluster

biadmin@hdtest249:/opt/ibm/biginsights/hbase/lib> /opt/ibm/biginsights/hbase/bin/hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot s1 -copy-to hdfs://bdvm197.svl.ibm.com:9000/user/biadmin/mysnapshots
14/04/01 14:07:20 INFO util.FSVisitor: No logs under directory:hdfs://hdtest249.svl.ibm.com:9000/hbase/.hbase-snapshot/s1/WALs
14/04/01 14:07:22 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
14/04/01 14:07:22 INFO compress.CodecPool: Got brand-new compressor [.deflate]
14/04/01 14:07:22 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 17 for biadmin on 9.30.195.103:9000
14/04/01 14:07:22 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 14 for biadmin on 9.30.122.206:9000
14/04/01 14:07:24 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 18 for biadmin on 9.30.195.103:9000
14/04/01 14:07:24 INFO security.TokenCache: Got dt for hdfs://hdtest249.svl.ibm.com:9000/user/biadmin/.staging/job_201404010039_0015/libjars/guava-12.0.1.jar;uri=9.30.195.103:9000;t.service=9.30.195.103:9000
14/04/01 14:07:24 INFO input.FileInputFormat: Total input paths to process : 1
14/04/01 14:07:25 INFO mapred.JobClient: Running job: job_201404010039_0015
14/04/01 14:07:26 INFO mapred.JobClient:  map 0% reduce 0%
14/04/01 14:07:41 INFO mapred.JobClient: Task Id : attempt_201404010039_0015_m_000000_0, Status : FAILED
java.io.IOException: Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: "hdtest250.svl.ibm.com/9.30.195.104"; destination host is: "bdvm197.svl.ibm.com":9000;
 
Caused by: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
        at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:173)
        at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:390)
        at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:494)
        at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:314)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:659)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:655)
        at java.security.AccessController.doPrivileged(AccessController.java:366)
        at javax.security.auth.Subject.doAs(Subject.java:572)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1502)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:654)
        ... 31 more

With HBASE-10608, we acquired the tokens, but we didn't add them to the MR job.