[HBASE-10379] Lower the msg "Request is a replay (34) - PROCESS_TGS" from logging level ERROR to WARN

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Incomplete
• Affects Version/s: 0.94.16
• Assignee: Unassigned
• Reporter: takeshi.miao
• Created: 20/Jan/14 02:38
• Updated: 17/Jun/22 15:39
• Resolved: 17/Jun/22 15:39

Description

Hi All,

Recently we got the error msg "Request is a replay (34) - PROCESS_TGS" while we are using the HBase client API to put data into HBase-0.94.16 with krb5-1.6.1 enabled. The related msg as follows...

[2014-01-15 09:40:38,452][hbase-tablepool-1-thread-3][ERROR][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1124)): PriviledgedActionException as:takeshi_miao@LAB cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Request is a replay (34) - PROCESS_TGS)]
[2014-01-15 09:40:38,453][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.logPriviledgedAction(UserGroupInformation.java:1143)): PriviledgedAction as:takeshi_miao@LAB from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)                                                                                          
[2014-01-15 09:40:38,453][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.ipc.SecureClient](org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$1.run(SecureClient.java:213)): Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Request is a replay (34) - PROCESS_TGS)]
[2014-01-15 09:40:38,454][hbase-tablepool-1-thread-3][INFO ][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.reloginFromTicketCache(UserGroupInformation.java:657)): Initiating logout for takeshi_miao@LAB
[2014-01-15 09:40:38,454][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.logout(UserGroupInformation.java:154)): hadoop logout
[2014-01-15 09:40:38,454][hbase-tablepool-1-thread-3][INFO ][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.reloginFromTicketCache(UserGroupInformation.java:667)): Initiating re-login for takeshi_miao@LAB
[2014-01-15 09:40:38,455][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.login(UserGroupInformation.java:146)): hadoop login
[2014-01-15 09:40:38,456][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:95)): hadoop login commit
[2014-01-15 09:40:38,456][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:100)): using existing subject:[takeshi_miao@LAB, UnixPrincipal: takeshi_miao, UnixNumericUserPrincipal: 501, UnixNumericGroupPrincipal [Primary Group]: 501, UnixNumericGroupPrincipal [Supplementary Group]: 502, takeshi_miao@LAB]

At the beginning, we were worry about the "data loss" occurring while we found the "Request is a replay (34) - PROCESS_TGS" (especially it is the ERROR level) in log, but after code study, this is basically NOT a data loss issue due to HBase client API would try 5 times internally (o.a.h.hbase.ipc.SecureClient, L#296, a one thread) and also 10 times of retry externally (o.a.h.hbase.client.HConnectionManager, L#1661, for all failed thread), The HTable API would also throw IOEcxeption to client code if any thread still fail after these retries.

Based on HBase users' viewpoint as us, we think this is better to change the logging level from 'ERROR' to 'WARN', due to the 'ERROR' level had been confused us for a while...But this code change may need to change boht HBase code and Hadoop code as well; so I am wondering how community think about this small thing but may be important to the pure HBase users.

mailing list
http://mail-archives.apache.org/mod_mbox/hbase-user/201401.mbox/%3CCADcMMgGiEyho0HGwgbfOUS78ymDpCo5Q0PStWAPUk40W%3DPfcFQ%40mail.gmail.com%3E

Issue Links

• relates to: HADOOP-10274 Lower the logging level from ERROR to WARN for UGI.doAs method
• relates to: HBASE-10411 [Book] Add a kerberos 'request is a replay (34)' issue at troubleshooting section