Description
On some platforms all unit tests that use minikdc fail consistently. Those tests include TestKMS, TestSaslDataTransfer, TestTimelineAuthenticationFilter, etc.
Typical failures on the unit tests:
java.lang.AssertionError: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Cannot get a KDC reply) at org.junit.Assert.fail(Assert.java:88) at org.apache.hadoop.crypto.key.kms.server.TestKMS$8$4.run(TestKMS.java:1154) at org.apache.hadoop.crypto.key.kms.server.TestKMS$8$4.run(TestKMS.java:1145) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1645) at org.apache.hadoop.crypto.key.kms.server.TestKMS.doAs(TestKMS.java:261) at org.apache.hadoop.crypto.key.kms.server.TestKMS.access$100(TestKMS.java:76)
The errors that cause this failure on the KDC server on the minikdc are a NullPointerException:
org.apache.mina.filter.codec.ProtocolDecoderException: java.lang.NullPointerException: message (Hexdump: ...) at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:234) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:48) at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:802) at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:120) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426) at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:604) at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:564) at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:553) at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:57) at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:892) at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NullPointerException: message at org.apache.mina.filter.codec.AbstractProtocolDecoderOutput.write(AbstractProtocolDecoderOutput.java:44) at org.apache.directory.server.kerberos.protocol.codec.MinaKerberosDecoder.decode(MinaKerberosDecoder.java:65) at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:224) ... 15 more
Attachments
Attachments
Issue Links
- is duplicated by
-
HDFS-9746 Some Kerberos related tests intermittently fail.
- Resolved
- is related to
-
DIRSERVER-2071 MinaKerberosDecoder fails with NullPointerException if the kerberos message is split in multiple packets
- Closed
-
DIRSERVER-2074 small default TCP receive/send buffer size causing TCP packet fragmentation on some platforms
- Closed
This is caused by fragmented TCP packets for the kerberos authentication request.
In the problem situation, the kerberos authentication request sent by the client gets fragmented into 2 packets although the size is tiny (e.g. 584 bytes). It gets split into one packet with 570 bytes of data and another with 14 bytes in this case. Tcpdump output:
It turns out there is a bug with apacheds (on which minikdc is based) where the kerberos message decoding fails with a NPE if the kerberos message is not contained in a single TCP packet (
DIRSERVER-2071).Furthermore, the TCP fragmentation itself has something to do with apacheds as well. Mina, the underlying I/O framework for apacheds, sets a pretty small receive/send buffer size by default (1 KB). This has an affect of reducing the TCP window size significantly as it is evidenced by the tcp dump above. This is causing the fragmentation.