Details
-
Test
-
Status: Resolved
-
Major
-
Resolution: Duplicate
-
None
-
None
-
None
-
None
Description
So far I've seen TestSecureNNWithQJM#testSecureMode and TestKMS#testACLs failing. More details coming in the 1st comment.
Attachments
Issue Links
- duplicates
-
HADOOP-12090 minikdc-related unit tests fail consistently on some platforms
- Patch Available
- is related to
-
HDFS-9745 TestSecureNNWithQJM#testSecureMode sometimes fails with timeouts
- Resolved
Activity
Xiao Chen
added a comment -
Log excerpt of the failure I saw in TestSecureNNWithQJM#testSecureMode:
Error Message
org.apache.hadoop.hdfs.server.namenode.EditLogInputException: Error replaying edit log at offset 0. Expected transaction ID was 1 at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:194) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151) at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:187) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:140) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:835) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:690) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:281) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1063) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:767) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:609) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:670) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:838) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:817) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1538) at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1862) at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1827) at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.restartNameNode(TestSecureNNWithQJM.java:197) at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.doNNWithQJMTest(TestSecureNNWithQJM.java:179) at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.testSecureMode(TestSecureNNWithQJM.java:159)
stdout
2016-02-01 06:41:47,944 INFO namenode.FSImage (FSImage.java:loadEdits(832)) - Reading org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream@12e8c783 expecting start txid #1 2016-02-01 06:41:47,944 INFO namenode.FSImage (FSEditLogLoader.java:loadFSEdits(139)) - Start loading edits file https://localhost:55326/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID, https://localhost:48398/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID 2016-02-01 06:41:47,945 INFO namenode.EditLogInputStream (RedundantEditLogInputStream.java:nextOp(176)) - Fast-forwarding stream 'https://localhost:55326/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID, https://localhost:48398/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID' to transaction ID 1 2016-02-01 06:41:47,945 INFO namenode.EditLogInputStream (RedundantEditLogInputStream.java:nextOp(176)) - Fast-forwarding stream 'https://localhost:55326/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID' to transaction ID 1 2016-02-01 06:41:48,131 ERROR protocol.KerberosProtocolHandler (KerberosProtocolHandler.java:exceptionCaught(157)) - /127.0.0.1:46417 EXCEPTION org.apache.mina.filter.codec.ProtocolDecoderException: java.lang.NullPointerException: message (Hexdump: 00 00 02 45 6C 82 02 41 30 82 02 3D A1 03 02 01 05 A2 03 02 01 0C A3 82 01 C4 30 82 01 C0 30 82 01 BC A1 03 02 01 01 A2 82 01 B3 04 82 01 AF 6E 82 01 AB 30 82 01 A7 A0 03 02 01 05 A1 03 02 01 0E A2 07 03 05 00 00 00 00 00 A3 81 F6 61 81 F3 30 81 F0 A0 03 02 01 05 A1 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A2 20 30 1E A0 03 02 01 02 A1 17 30 15 1B 06 6B 72 62 74 67 74 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 81 B7 30 81 B4 A0 03 02 01 11 A2 81 AC 04 81 A9 6D 55 CA 9E E9 6A 4D 9F C1 47 9A 2B 9E 75 07 73 F4 48 4A 36 AE BD 28 4D DA 3D 00 89 0D B1 70 C1 67 E7 44 0B C4 3E BF 59 3D 2B F2 EA E7 09 05 44 23 2D B6 D6 76 46 D8 DC 05 32 68 A2 2B D5 58 3D EC 5D DD 4D A1 6D F2 80 95 5F 61 2A 40 C5 D3 F7 BD A3 71 73 F2 81 DD CD B1 B3 1D 8E FA B0 70 9F 88 AD 97 7C 2A 91 DE 4F 69 5A 23 17 AA 21 99 7E 89 36 E6 1A 01 06 D2 DF 6F C3 76 15 47 81 9A 65 66 F7 CC 23 52 C5 DE 77 09 AD 66 38 94 DE 93 DC CA 24 6B C4 2B FA 1A BC FC 07 84 9B CE 0D 15 BA C7 00 8A 5C 0E 61 D8 BE 01 A4 81 98 30 81 95 A0 03 02 01 11 A2 81 8D 04 81 8A 60 DF 82 D5 14 DB 78 8D A8 E4 6D F8 FE 3A F4 AB 98 25 9D DB 51 ED 3B CE 53 C8 DC 48 1C CB EB B5 1B 5A 45 BA CD 68 0A 26 2F 8D 3A FE 75 AE 36 4B 25 B5 B8 5A C1 27 71 E3 B6 03 7D D6 2D 14 58 CD 6D 19 F0 25 D0 5A 9B 35 A6 7E 36 62 DA 28 56 0B E9 53 03 43 7B 71 D5 ED 8F 52 CE 6E 8A 23 0C 52 53 EB 42 0B 7A 6B 8C 54 EB 1C 70 FB 21 DD DF 23 B4 5E AD 42 67 65 42 61 FD DB 2D 28 C2 4D 7A 71 69 D0 74 9A 64 8A 82 A0 EC C8 A4 69 30 67 A0 07 03 05 00 00 00 00 00 A2 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 1C 30 1A A0 03 02 01 00 A1 13 30 11 1B 04 48 54 54 50 1B 09 6C 6F 63 61 6C 68 6F 73 74 A5 11 18 0F 31 39 37 30 30 31 30 31 30 30 30 30 30 30 5A A7 06 02 04 6B 88 55 A5 A8 14 30 12 02 01 12 02 01 11 02 01 10) at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:234) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:48) at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:802) at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:120) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426) at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:604) at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:564) at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:553) at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:57) at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:892) at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NullPointerException: message at org.apache.mina.filter.codec.AbstractProtocolDecoderOutput.write(AbstractProtocolDecoderOutput.java:44) at org.apache.directory.server.kerberos.protocol.codec.MinaKerberosDecoder.decode(MinaKerberosDecoder.java:65) at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:224) ... 15 more 2016-02-01 06:41:48,133 ERROR server.KERBEROS_LOG (KerberosProtocolHandler.java:exceptionCaught(158)) - /127.0.0.1:46417 EXCEPTION org.apache.mina.filter.codec.ProtocolDecoderException: java.lang.NullPointerException: message (Hexdump: 00 00 02 45 6C 82 02 41 30 82 02 3D A1 03 02 01 05 A2 03 02 01 0C A3 82 01 C4 30 82 01 C0 30 82 01 BC A1 03 02 01 01 A2 82 01 B3 04 82 01 AF 6E 82 01 AB 30 82 01 A7 A0 03 02 01 05 A1 03 02 01 0E A2 07 03 05 00 00 00 00 00 A3 81 F6 61 81 F3 30 81 F0 A0 03 02 01 05 A1 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A2 20 30 1E A0 03 02 01 02 A1 17 30 15 1B 06 6B 72 62 74 67 74 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 81 B7 30 81 B4 A0 03 02 01 11 A2 81 AC 04 81 A9 6D 55 CA 9E E9 6A 4D 9F C1 47 9A 2B 9E 75 07 73 F4 48 4A 36 AE BD 28 4D DA 3D 00 89 0D B1 70 C1 67 E7 44 0B C4 3E BF 59 3D 2B F2 EA E7 09 05 44 23 2D B6 D6 76 46 D8 DC 05 32 68 A2 2B D5 58 3D EC 5D DD 4D A1 6D F2 80 95 5F 61 2A 40 C5 D3 F7 BD A3 71 73 F2 81 DD CD B1 B3 1D 8E FA B0 70 9F 88 AD 97 7C 2A 91 DE 4F 69 5A 23 17 AA 21 99 7E 89 36 E6 1A 01 06 D2 DF 6F C3 76 15 47 81 9A 65 66 F7 CC 23 52 C5 DE 77 09 AD 66 38 94 DE 93 DC CA 24 6B C4 2B FA 1A BC FC 07 84 9B CE 0D 15 BA C7 00 8A 5C 0E 61 D8 BE 01 A4 81 98 30 81 95 A0 03 02 01 11 A2 81 8D 04 81 8A 60 DF 82 D5 14 DB 78 8D A8 E4 6D F8 FE 3A F4 AB 98 25 9D DB 51 ED 3B CE 53 C8 DC 48 1C CB EB B5 1B 5A 45 BA CD 68 0A 26 2F 8D 3A FE 75 AE 36 4B 25 B5 B8 5A C1 27 71 E3 B6 03 7D D6 2D 14 58 CD 6D 19 F0 25 D0 5A 9B 35 A6 7E 36 62 DA 28 56 0B E9 53 03 43 7B 71 D5 ED 8F 52 CE 6E 8A 23 0C 52 53 EB 42 0B 7A 6B 8C 54 EB 1C 70 FB 21 DD DF 23 B4 5E AD 42 67 65 42 61 FD DB 2D 28 C2 4D 7A 71 69 D0 74 9A 64 8A 82 A0 EC C8 A4 69 30 67 A0 07 03 05 00 00 00 00 00 A2 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 1C 30 1A A0 03 02 01 00 A1 13 30 11 1B 04 48 54 54 50 1B 09 6C 6F 63 61 6C 68 6F 73 74 A5 11 18 0F 31 39 37 30 30 31 30 31 30 30 30 30 30 30 5A A7 06 02 04 6B 88 55 A5 A8 14 30 12 02 01 12 02 01 11 02 01 10) at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:234) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:48) at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:802) at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:120) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426) at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:604) at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:564) at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:553) at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:57) at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:892) at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NullPointerException: message at org.apache.mina.filter.codec.AbstractProtocolDecoderOutput.write(AbstractProtocolDecoderOutput.java:44) at org.apache.directory.server.kerberos.protocol.codec.MinaKerberosDecoder.decode(MinaKerberosDecoder.java:65) at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:224) ... 15 more 2016-02-01 06:41:48,138 ERROR protocol.KerberosProtocolHandler (KerberosProtocolHandler.java:exceptionCaught(157)) - /127.0.0.1:46418 EXCEPTION org.apache.mina.filter.codec.ProtocolDecoderException: java.lang.NullPointerException: message (Hexdump: 00 00 02 45 6C 82 02 41 30 82 02 3D A1 03 02 01 05 A2 03 02 01 0C A3 82 01 C4 30 82 01 C0 30 82 01 BC A1 03 02 01 01 A2 82 01 B3 04 82 01 AF 6E 82 01 AB 30 82 01 A7 A0 03 02 01 05 A1 03 02 01 0E A2 07 03 05 00 00 00 00 00 A3 81 F6 61 81 F3 30 81 F0 A0 03 02 01 05 A1 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A2 20 30 1E A0 03 02 01 02 A1 17 30 15 1B 06 6B 72 62 74 67 74 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 81 B7 30 81 B4 A0 03 02 01 11 A2 81 AC 04 81 A9 6D 55 CA 9E E9 6A 4D 9F C1 47 9A 2B 9E 75 07 73 F4 48 4A 36 AE BD 28 4D DA 3D 00 89 0D B1 70 C1 67 E7 44 0B C4 3E BF 59 3D 2B F2 EA E7 09 05 44 23 2D B6 D6 76 46 D8 DC 05 32 68 A2 2B D5 58 3D EC 5D DD 4D A1 6D F2 80 95 5F 61 2A 40 C5 D3 F7 BD A3 71 73 F2 81 DD CD B1 B3 1D 8E FA B0 70 9F 88 AD 97 7C 2A 91 DE 4F 69 5A 23 17 AA 21 99 7E 89 36 E6 1A 01 06 D2 DF 6F C3 76 15 47 81 9A 65 66 F7 CC 23 52 C5 DE 77 09 AD 66 38 94 DE 93 DC CA 24 6B C4 2B FA 1A BC FC 07 84 9B CE 0D 15 BA C7 00 8A 5C 0E 61 D8 BE 01 A4 81 98 30 81 95 A0 03 02 01 11 A2 81 8D 04 81 8A 7E 36 73 CB EC 41 15 69 53 02 B8 E4 28 1A 8D 61 56 3E 30 4E 0F C6 28 7C 12 CF EB 25 1E A9 3B 28 99 68 3B CE F3 38 50 B0 DA D9 0F 25 3A 39 B4 12 39 44 A6 64 8D FA 1B 62 22 27 7E 64 D8 C4 FE D3 89 DC 82 E2 C2 21 7E 14 BE DE 89 C6 9D FD C5 3F 25 40 9A F7 2C 26 DD 16 E1 EF C5 66 A5 AD 1A 46 8E 44 9C 86 63 9B 80 BA 81 45 78 97 F4 C1 EF EF 05 68 77 2B 61 B2 DF 30 0B BF E1 94 69 44 2C 97 7D 54 52 F8 9C 88 14 56 A0 B7 A4 69 30 67 A0 07 03 05 00 00 00 00 00 A2 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 1C 30 1A A0 03 02 01 00 A1 13 30 11 1B 04 48 54 54 50 1B 09 6C 6F 63 61 6C 68 6F 73 74 A5 11 18 0F 31 39 37 30 30 31 30 31 30 30 30 30 30 30 5A A7 06 02 04 3F 0B C9 19 A8 14 30 12 02 01 12 02 01 11 02 01 10) at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:234) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:48) at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:802) at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:120) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426) at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:604) at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:564) at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:553) at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:57) at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:892) at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NullPointerException: message at org.apache.mina.filter.codec.AbstractProtocolDecoderOutput.write(AbstractProtocolDecoderOutput.java:44) at org.apache.directory.server.kerberos.protocol.codec.MinaKerberosDecoder.decode(MinaKerberosDecoder.java:65) at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:224) ... 15 more 2016-02-01 06:41:48,139 ERROR server.KERBEROS_LOG (KerberosProtocolHandler.java:exceptionCaught(158)) - /127.0.0.1:46418 EXCEPTION org.apache.mina.filter.codec.ProtocolDecoderException: java.lang.NullPointerException: message (Hexdump: 00 00 02 45 6C 82 02 41 30 82 02 3D A1 03 02 01 05 A2 03 02 01 0C A3 82 01 C4 30 82 01 C0 30 82 01 BC A1 03 02 01 01 A2 82 01 B3 04 82 01 AF 6E 82 01 AB 30 82 01 A7 A0 03 02 01 05 A1 03 02 01 0E A2 07 03 05 00 00 00 00 00 A3 81 F6 61 81 F3 30 81 F0 A0 03 02 01 05 A1 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A2 20 30 1E A0 03 02 01 02 A1 17 30 15 1B 06 6B 72 62 74 67 74 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 81 B7 30 81 B4 A0 03 02 01 11 A2 81 AC 04 81 A9 6D 55 CA 9E E9 6A 4D 9F C1 47 9A 2B 9E 75 07 73 F4 48 4A 36 AE BD 28 4D DA 3D 00 89 0D B1 70 C1 67 E7 44 0B C4 3E BF 59 3D 2B F2 EA E7 09 05 44 23 2D B6 D6 76 46 D8 DC 05 32 68 A2 2B D5 58 3D EC 5D DD 4D A1 6D F2 80 95 5F 61 2A 40 C5 D3 F7 BD A3 71 73 F2 81 DD CD B1 B3 1D 8E FA B0 70 9F 88 AD 97 7C 2A 91 DE 4F 69 5A 23 17 AA 21 99 7E 89 36 E6 1A 01 06 D2 DF 6F C3 76 15 47 81 9A 65 66 F7 CC 23 52 C5 DE 77 09 AD 66 38 94 DE 93 DC CA 24 6B C4 2B FA 1A BC FC 07 84 9B CE 0D 15 BA C7 00 8A 5C 0E 61 D8 BE 01 A4 81 98 30 81 95 A0 03 02 01 11 A2 81 8D 04 81 8A 7E 36 73 CB EC 41 15 69 53 02 B8 E4 28 1A 8D 61 56 3E 30 4E 0F C6 28 7C 12 CF EB 25 1E A9 3B 28 99 68 3B CE F3 38 50 B0 DA D9 0F 25 3A 39 B4 12 39 44 A6 64 8D FA 1B 62 22 27 7E 64 D8 C4 FE D3 89 DC 82 E2 C2 21 7E 14 BE DE 89 C6 9D FD C5 3F 25 40 9A F7 2C 26 DD 16 E1 EF C5 66 A5 AD 1A 46 8E 44 9C 86 63 9B 80 BA 81 45 78 97 F4 C1 EF EF 05 68 77 2B 61 B2 DF 30 0B BF E1 94 69 44 2C 97 7D 54 52 F8 9C 88 14 56 A0 B7 A4 69 30 67 A0 07 03 05 00 00 00 00 00 A2 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 1C 30 1A A0 03 02 01 00 A1 13 30 11 1B 04 48 54 54 50 1B 09 6C 6F 63 61 6C 68 6F 73 74 A5 11 18 0F 31 39 37 30 30 31 30 31 30 30 30 30 30 30 5A A7 06 02 04 3F 0B C9 19 A8 14 30 12 02 01 12 02 01 11 02 01 10) at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:234) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:48) at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:802) at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:120) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426) at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:604) at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:564) at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:553) at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:57) at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:892) at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NullPointerException: message at org.apache.mina.filter.codec.AbstractProtocolDecoderOutput.write(AbstractProtocolDecoderOutput.java:44) at org.apache.directory.server.kerberos.protocol.codec.MinaKerberosDecoder.decode(MinaKerberosDecoder.java:65) at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:224) ... 15 more 2016-02-01 06:41:48,139 WARN security.UserGroupInformation (UserGroupInformation.java:doAs(1696)) - PriviledgedActionException as:slave/localhost@EXAMPLE.COM (auth:KERBEROS) cause:java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Cannot get a KDC reply) 2016-02-01 06:41:48,140 ERROR namenode.EditLogInputStream (EditLogFileInputStream.java:nextOpImpl(194)) - caught exception initializing https://localhost:55326/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Cannot get a KDC reply) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:473) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:465) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693) at org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:445) at org.apache.hadoop.security.SecurityUtil.doAsCurrentUser(SecurityUtil.java:439) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog.getInputStream(EditLogFileInputStream.java:464) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.init(EditLogFileInputStream.java:141) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOpImpl(EditLogFileInputStream.java:192) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOp(EditLogFileInputStream.java:250) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151) at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151) at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:187) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:140) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:835) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:690) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:281) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1063) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:767) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:609) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:670) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:838) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:817) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1538) at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1862) at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1827) at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.restartNameNode(TestSecureNNWithQJM.java:197) at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.doNNWithQJMTest(TestSecureNNWithQJM.java:179) at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.testSecureMode(TestSecureNNWithQJM.java:159) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Cannot get a KDC reply) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:306) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:196) at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:215) at org.apache.hadoop.hdfs.web.URLConnectionFactory.openConnection(URLConnectionFactory.java:161) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:470) ... 45 more Caused by: GSSException: No valid credentials provided (Mechanism level: Cannot get a KDC reply) at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:715) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:285) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:261) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:261) ... 49 more Caused by: java.io.IOException: Cannot get a KDC reply at sun.security.krb5.KdcComm.send(KdcComm.java:250) at sun.security.krb5.KdcComm.send(KdcComm.java:191) at sun.security.krb5.KrbTgsReq.send(KrbTgsReq.java:186) at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:201) at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:292) at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:101) at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:456) at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:641) ... 56 more 2016-02-01 06:41:48,141 ERROR namenode.EditLogInputStream (RedundantEditLogInputStream.java:nextOp(221)) - Got error reading edit log input stream https://localhost:55326/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID; failing over to edit log https://localhost:48398/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream$PrematureEOFException: got premature end-of-file at txid 0; expected file to go up to 5 at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:194) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151) at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:187) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:140) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:835) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:690) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:281) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1063) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:767) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:609) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:670) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:838) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:817) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1538) at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1862) at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1827) at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.restartNameNode(TestSecureNNWithQJM.java:197) at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.doNNWithQJMTest(TestSecureNNWithQJM.java:179) at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.testSecureMode(TestSecureNNWithQJM.java:159) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2016-02-01 06:41:48,142 INFO namenode.EditLogInputStream (RedundantEditLogInputStream.java:nextOp(176)) - Fast-forwarding stream 'https://localhost:48398/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID' to transaction ID 1 2016-02-01 06:41:48,169 ERROR protocol.KerberosProtocolHandler (KerberosProtocolHandler.java:exceptionCaught(157)) - /127.0.0.1:46420 EXCEPTION org.apache.mina.filter.codec.ProtocolDecoderException: java.lang.NullPointerException: message (Hexdump: 00 00 02 45 6C 82 02 41 30 82 02 3D A1 03 02 01 05 A2 03 02 01 0C A3 82 01 C4 30 82 01 C0 30 82 01 BC A1 03 02 01 01 A2 82 01 B3 04 82 01 AF 6E 82 01 AB 30 82 01 A7 A0 03 02 01 05 A1 03 02 01 0E A2 07 03 05 00 00 00 00 00 A3 81 F6 61 81 F3 30 81 F0 A0 03 02 01 05 A1 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A2 20 30 1E A0 03 02 01 02 A1 17 30 15 1B 06 6B 72 62 74 67 74 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 81 B7 30 81 B4 A0 03 02 01 11 A2 81 AC 04 81 A9 6D 55 CA 9E E9 6A 4D 9F C1 47 9A 2B 9E 75 07 73 F4 48 4A 36 AE BD 28 4D DA 3D 00 89 0D B1 70 C1 67 E7 44 0B C4 3E BF 59 3D 2B F2 EA E7 09 05 44 23 2D B6 D6 76 46 D8 DC 05 32 68 A2 2B D5 58 3D EC 5D DD 4D A1 6D F2 80 95 5F 61 2A 40 C5 D3 F7 BD A3 71 73 F2 81 DD CD B1 B3 1D 8E FA B0 70 9F 88 AD 97 7C 2A 91 DE 4F 69 5A 23 17 AA 21 99 7E 89 36 E6 1A 01 06 D2 DF 6F C3 76 15 47 81 9A 65 66 F7 CC 23 52 C5 DE 77 09 AD 66 38 94 DE 93 DC CA 24 6B C4 2B FA 1A BC FC 07 84 9B CE 0D 15 BA C7 00 8A 5C 0E 61 D8 BE 01 A4 81 98 30 81 95 A0 03 02 01 11 A2 81 8D 04 81 8A BA 11 25 9F 01 D1 21 8A 31 30 5B 14 7A 6D A5 65 C1 87 95 F1 86 3A E1 A9 7E 3E 7F 4D FE 8E 8F 96 E4 0A 78 2B 5F 0D 1D 87 88 51 A4 5B 92 94 6A FC 87 97 2A 9C 69 81 B3 5D 5D 67 56 E9 97 97 5D 29 7B B9 B8 9E 99 8F 7A BD 3A E3 4D 6E BA 55 0C ED 3A 67 0F A9 45 B5 93 9B 48 FF FB 45 A7 23 5A B9 52 44 E4 54 D5 08 D5 2F FF 4A C4 52 16 FA 5F 1C 62 41 88 15 C1 B6 63 13 DA 16 33 56 63 37 09 63 2E 1D C8 42 55 FB FA 33 9B CC A4 69 30 67 A0 07 03 05 00 00 00 00 00 A2 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 1C 30 1A A0 03 02 01 00 A1 13 30 11 1B 04 48 54 54 50 1B 09 6C 6F 63 61 6C 68 6F 73 74 A5 11 18 0F 31 39 37 30 30 31 30 31 30 30 30 30 30 30 5A A7 06 02 04 01 AA D4 AF A8 14 30 12 02 01 12 02 01 11 02 01 10) at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:234) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:48) at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:802) at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:120) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426) at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:604) at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:564) at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:553) at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:57) at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:892) at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NullPointerException: message at org.apache.mina.filter.codec.AbstractProtocolDecoderOutput.write(AbstractProtocolDecoderOutput.java:44) at org.apache.directory.server.kerberos.protocol.codec.MinaKerberosDecoder.decode(MinaKerberosDecoder.java:65) at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:224) ... 15 more 2016-02-01 06:41:48,169 ERROR server.KERBEROS_LOG (KerberosProtocolHandler.java:exceptionCaught(158)) - /127.0.0.1:46420 EXCEPTION org.apache.mina.filter.codec.ProtocolDecoderException: java.lang.NullPointerException: message (Hexdump: 00 00 02 45 6C 82 02 41 30 82 02 3D A1 03 02 01 05 A2 03 02 01 0C A3 82 01 C4 30 82 01 C0 30 82 01 BC A1 03 02 01 01 A2 82 01 B3 04 82 01 AF 6E 82 01 AB 30 82 01 A7 A0 03 02 01 05 A1 03 02 01 0E A2 07 03 05 00 00 00 00 00 A3 81 F6 61 81 F3 30 81 F0 A0 03 02 01 05 A1 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A2 20 30 1E A0 03 02 01 02 A1 17 30 15 1B 06 6B 72 62 74 67 74 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 81 B7 30 81 B4 A0 03 02 01 11 A2 81 AC 04 81 A9 6D 55 CA 9E E9 6A 4D 9F C1 47 9A 2B 9E 75 07 73 F4 48 4A 36 AE BD 28 4D DA 3D 00 89 0D B1 70 C1 67 E7 44 0B C4 3E BF 59 3D 2B F2 EA E7 09 05 44 23 2D B6 D6 76 46 D8 DC 05 32 68 A2 2B D5 58 3D EC 5D DD 4D A1 6D F2 80 95 5F 61 2A 40 C5 D3 F7 BD A3 71 73 F2 81 DD CD B1 B3 1D 8E FA B0 70 9F 88 AD 97 7C 2A 91 DE 4F 69 5A 23 17 AA 21 99 7E 89 36 E6 1A 01 06 D2 DF 6F C3 76 15 47 81 9A 65 66 F7 CC 23 52 C5 DE 77 09 AD 66 38 94 DE 93 DC CA 24 6B C4 2B FA 1A BC FC 07 84 9B CE 0D 15 BA C7 00 8A 5C 0E 61 D8 BE 01 A4 81 98 30 81 95 A0 03 02 01 11 A2 81 8D 04 81 8A BA 11 25 9F 01 D1 21 8A 31 30 5B 14 7A 6D A5 65 C1 87 95 F1 86 3A E1 A9 7E 3E 7F 4D FE 8E 8F 96 E4 0A 78 2B 5F 0D 1D 87 88 51 A4 5B 92 94 6A FC 87 97 2A 9C 69 81 B3 5D 5D 67 56 E9 97 97 5D 29 7B B9 B8 9E 99 8F 7A BD 3A E3 4D 6E BA 55 0C ED 3A 67 0F A9 45 B5 93 9B 48 FF FB 45 A7 23 5A B9 52 44 E4 54 D5 08 D5 2F FF 4A C4 52 16 FA 5F 1C 62 41 88 15 C1 B6 63 13 DA 16 33 56 63 37 09 63 2E 1D C8 42 55 FB FA 33 9B CC A4 69 30 67 A0 07 03 05 00 00 00 00 00 A2 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 1C 30 1A A0 03 02 01 00 A1 13 30 11 1B 04 48 54 54 50 1B 09 6C 6F 63 61 6C 68 6F 73 74 A5 11 18 0F 31 39 37 30 30 31 30 31 30 30 30 30 30 30 5A A7 06 02 04 01 AA D4 AF A8 14 30 12 02 01 12 02 01 11 02 01 10) at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:234) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:48) at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:802) at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:120) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426) at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:604) at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:564) at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:553) at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:57) at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:892) at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NullPointerException: message at org.apache.mina.filter.codec.AbstractProtocolDecoderOutput.write(AbstractProtocolDecoderOutput.java:44) at org.apache.directory.server.kerberos.protocol.codec.MinaKerberosDecoder.decode(MinaKerberosDecoder.java:65) at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:224) ... 15 more 2016-02-01 06:41:48,170 WARN security.UserGroupInformation (UserGroupInformation.java:doAs(1696)) - PriviledgedActionException as:slave/localhost@EXAMPLE.COM (auth:KERBEROS) cause:java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Cannot get a KDC reply) 2016-02-01 06:41:48,170 ERROR namenode.EditLogInputStream (EditLogFileInputStream.java:nextOpImpl(194)) - caught exception initializing https://localhost:48398/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Cannot get a KDC reply) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:473) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:465) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693) at org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:445) at org.apache.hadoop.security.SecurityUtil.doAsCurrentUser(SecurityUtil.java:439) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog.getInputStream(EditLogFileInputStream.java:464) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.init(EditLogFileInputStream.java:141) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOpImpl(EditLogFileInputStream.java:192) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOp(EditLogFileInputStream.java:250) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151) at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151) at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:187) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:140) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:835) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:690) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:281) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1063) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:767) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:609) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:670) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:838) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:817) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1538) at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1862) at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1827) at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.restartNameNode(TestSecureNNWithQJM.java:197) at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.doNNWithQJMTest(TestSecureNNWithQJM.java:179) at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.testSecureMode(TestSecureNNWithQJM.java:159) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Cannot get a KDC reply) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:306) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:196) at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:215) at org.apache.hadoop.hdfs.web.URLConnectionFactory.openConnection(URLConnectionFactory.java:161) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:470) ... 45 more Caused by: GSSException: No valid credentials provided (Mechanism level: Cannot get a KDC reply) at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:715) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:285) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:261) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:261) ... 49 more Caused by: java.io.IOException: Cannot get a KDC reply at sun.security.krb5.KdcComm.send(KdcComm.java:250) at sun.security.krb5.KdcComm.send(KdcComm.java:191) at sun.security.krb5.KrbTgsReq.send(KrbTgsReq.java:186) at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:201) at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:292) at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:101) at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:456) at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:641) ... 56 more 2016-02-01 06:41:48,171 ERROR namenode.FSImage (FSEditLogLoader.java:loadEditRecords(196)) - Error replaying edit log at offset 0. Expected transaction ID was 1 org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream$PrematureEOFException: got premature end-of-file at txid 0; expected file to go up to 5 at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:194) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151) at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:187) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:140) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:835) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:690) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:281) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1063) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:767) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:609) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:670) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:838) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:817) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1538) at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1862) at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1827) at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.restartNameNode(TestSecureNNWithQJM.java:197) at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.doNNWithQJMTest(TestSecureNNWithQJM.java:179) at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.testSecureMode(TestSecureNNWithQJM.java:159) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2016-02-01 06:41:48,173 WARN namenode.FSNamesystem (FSNamesystem.java:loadFromDisk(769)) - Encountered exception loading fsimage org.apache.hadoop.hdfs.server.namenode.EditLogInputException: Error replaying edit log at offset 0. Expected transaction ID was 1 at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:200) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:140) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:835) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:690) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:281) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1063) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:767) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:609) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:670) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:838) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:817) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1538) at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1862) at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1827) at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.restartNameNode(TestSecureNNWithQJM.java:197) at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.doNNWithQJMTest(TestSecureNNWithQJM.java:179) at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.testSecureMode(TestSecureNNWithQJM.java:159) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream$PrematureEOFException: got premature end-of-file at txid 0; expected file to go up to 5 at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:194) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151) at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:187) ... 27 more 2016-02-01 06:41:48,187 INFO mortbay.log (Slf4jLog.java:info(67)) - Stopped SslSocketConnectorSecure@localhost:41570 2016-02-01 06:41:48,287 INFO hdfs.MiniDFSCluster (MiniDFSCluster.java:shutdown(1744)) - Shutting down the Mini HDFS Cluster
Xiao Chen
added a comment - Log excerpt of the failure I saw in TestSecureNNWithQJM#testSecureMode :
Error Message
org.apache.hadoop.hdfs.server.namenode.EditLogInputException: Error replaying edit log at offset 0. Expected transaction ID was 1
at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:194)
at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:187)
at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:140)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:835)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:690)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:281)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1063)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:767)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:609)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:670)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:838)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:817)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1538)
at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1862)
at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1827)
at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.restartNameNode(TestSecureNNWithQJM.java:197)
at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.doNNWithQJMTest(TestSecureNNWithQJM.java:179)
at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.testSecureMode(TestSecureNNWithQJM.java:159)
stdout
2016-02-01 06:41:47,944 INFO namenode.FSImage (FSImage.java:loadEdits(832)) - Reading org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream@12e8c783 expecting start txid #1
2016-02-01 06:41:47,944 INFO namenode.FSImage (FSEditLogLoader.java:loadFSEdits(139)) - Start loading edits file https://localhost:55326/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID, https://localhost:48398/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID
2016-02-01 06:41:47,945 INFO namenode.EditLogInputStream (RedundantEditLogInputStream.java:nextOp(176)) - Fast-forwarding stream 'https://localhost:55326/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID, https://localhost:48398/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID' to transaction ID 1
2016-02-01 06:41:47,945 INFO namenode.EditLogInputStream (RedundantEditLogInputStream.java:nextOp(176)) - Fast-forwarding stream 'https://localhost:55326/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID' to transaction ID 1
2016-02-01 06:41:48,131 ERROR protocol.KerberosProtocolHandler (KerberosProtocolHandler.java:exceptionCaught(157)) - /127.0.0.1:46417 EXCEPTION
org.apache.mina.filter.codec.ProtocolDecoderException: java.lang.NullPointerException: message (Hexdump: 00 00 02 45 6C 82 02 41 30 82 02 3D A1 03 02 01 05 A2 03 02 01 0C A3 82 01 C4 30 82 01 C0 30 82 01 BC A1 03 02 01 01 A2 82 01 B3 04 82 01 AF 6E 82 01 AB 30 82 01 A7 A0 03 02 01 05 A1 03 02 01 0E A2 07 03 05 00 00 00 00 00 A3 81 F6 61 81 F3 30 81 F0 A0 03 02 01 05 A1 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A2 20 30 1E A0 03 02 01 02 A1 17 30 15 1B 06 6B 72 62 74 67 74 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 81 B7 30 81 B4 A0 03 02 01 11 A2 81 AC 04 81 A9 6D 55 CA 9E E9 6A 4D 9F C1 47 9A 2B 9E 75 07 73 F4 48 4A 36 AE BD 28 4D DA 3D 00 89 0D B1 70 C1 67 E7 44 0B C4 3E BF 59 3D 2B F2 EA E7 09 05 44 23 2D B6 D6 76 46 D8 DC 05 32 68 A2 2B D5 58 3D EC 5D DD 4D A1 6D F2 80 95 5F 61 2A 40 C5 D3 F7 BD A3 71 73 F2 81 DD CD B1 B3 1D 8E FA B0 70 9F 88 AD 97 7C 2A 91 DE 4F 69 5A 23 17 AA 21 99 7E 89 36 E6 1A 01 06 D2 DF 6F C3 76 15 47 81 9A 65 66 F7 CC 23 52 C5 DE 77 09 AD 66 38 94 DE 93 DC CA 24 6B C4 2B FA 1A BC FC 07 84 9B CE 0D 15 BA C7 00 8A 5C 0E 61 D8 BE 01 A4 81 98 30 81 95 A0 03 02 01 11 A2 81 8D 04 81 8A 60 DF 82 D5 14 DB 78 8D A8 E4 6D F8 FE 3A F4 AB 98 25 9D DB 51 ED 3B CE 53 C8 DC 48 1C CB EB B5 1B 5A 45 BA CD 68 0A 26 2F 8D 3A FE 75 AE 36 4B 25 B5 B8 5A C1 27 71 E3 B6 03 7D D6 2D 14 58 CD 6D 19 F0 25 D0 5A 9B 35 A6 7E 36 62 DA 28 56 0B E9 53 03 43 7B 71 D5 ED 8F 52 CE 6E 8A 23 0C 52 53 EB 42 0B 7A 6B 8C 54 EB 1C 70 FB 21 DD DF 23 B4 5E AD 42 67 65 42 61 FD DB 2D 28 C2 4D 7A 71 69 D0 74 9A 64 8A 82 A0 EC C8 A4 69 30 67 A0 07 03 05 00 00 00 00 00 A2 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 1C 30 1A A0 03 02 01 00 A1 13 30 11 1B 04 48 54 54 50 1B 09 6C 6F 63 61 6C 68 6F 73 74 A5 11 18 0F 31 39 37 30 30 31 30 31 30 30 30 30 30 30 5A A7 06 02 04 6B 88 55 A5 A8 14 30 12 02 01 12 02 01 11 02 01 10)
at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:234)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:48)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:802)
at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:120)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:604)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:564)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:553)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:57)
at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:892)
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException: message
at org.apache.mina.filter.codec.AbstractProtocolDecoderOutput.write(AbstractProtocolDecoderOutput.java:44)
at org.apache.directory.server.kerberos.protocol.codec.MinaKerberosDecoder.decode(MinaKerberosDecoder.java:65)
at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:224)
... 15 more
2016-02-01 06:41:48,133 ERROR server.KERBEROS_LOG (KerberosProtocolHandler.java:exceptionCaught(158)) - /127.0.0.1:46417 EXCEPTION
org.apache.mina.filter.codec.ProtocolDecoderException: java.lang.NullPointerException: message (Hexdump: 00 00 02 45 6C 82 02 41 30 82 02 3D A1 03 02 01 05 A2 03 02 01 0C A3 82 01 C4 30 82 01 C0 30 82 01 BC A1 03 02 01 01 A2 82 01 B3 04 82 01 AF 6E 82 01 AB 30 82 01 A7 A0 03 02 01 05 A1 03 02 01 0E A2 07 03 05 00 00 00 00 00 A3 81 F6 61 81 F3 30 81 F0 A0 03 02 01 05 A1 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A2 20 30 1E A0 03 02 01 02 A1 17 30 15 1B 06 6B 72 62 74 67 74 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 81 B7 30 81 B4 A0 03 02 01 11 A2 81 AC 04 81 A9 6D 55 CA 9E E9 6A 4D 9F C1 47 9A 2B 9E 75 07 73 F4 48 4A 36 AE BD 28 4D DA 3D 00 89 0D B1 70 C1 67 E7 44 0B C4 3E BF 59 3D 2B F2 EA E7 09 05 44 23 2D B6 D6 76 46 D8 DC 05 32 68 A2 2B D5 58 3D EC 5D DD 4D A1 6D F2 80 95 5F 61 2A 40 C5 D3 F7 BD A3 71 73 F2 81 DD CD B1 B3 1D 8E FA B0 70 9F 88 AD 97 7C 2A 91 DE 4F 69 5A 23 17 AA 21 99 7E 89 36 E6 1A 01 06 D2 DF 6F C3 76 15 47 81 9A 65 66 F7 CC 23 52 C5 DE 77 09 AD 66 38 94 DE 93 DC CA 24 6B C4 2B FA 1A BC FC 07 84 9B CE 0D 15 BA C7 00 8A 5C 0E 61 D8 BE 01 A4 81 98 30 81 95 A0 03 02 01 11 A2 81 8D 04 81 8A 60 DF 82 D5 14 DB 78 8D A8 E4 6D F8 FE 3A F4 AB 98 25 9D DB 51 ED 3B CE 53 C8 DC 48 1C CB EB B5 1B 5A 45 BA CD 68 0A 26 2F 8D 3A FE 75 AE 36 4B 25 B5 B8 5A C1 27 71 E3 B6 03 7D D6 2D 14 58 CD 6D 19 F0 25 D0 5A 9B 35 A6 7E 36 62 DA 28 56 0B E9 53 03 43 7B 71 D5 ED 8F 52 CE 6E 8A 23 0C 52 53 EB 42 0B 7A 6B 8C 54 EB 1C 70 FB 21 DD DF 23 B4 5E AD 42 67 65 42 61 FD DB 2D 28 C2 4D 7A 71 69 D0 74 9A 64 8A 82 A0 EC C8 A4 69 30 67 A0 07 03 05 00 00 00 00 00 A2 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 1C 30 1A A0 03 02 01 00 A1 13 30 11 1B 04 48 54 54 50 1B 09 6C 6F 63 61 6C 68 6F 73 74 A5 11 18 0F 31 39 37 30 30 31 30 31 30 30 30 30 30 30 5A A7 06 02 04 6B 88 55 A5 A8 14 30 12 02 01 12 02 01 11 02 01 10)
at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:234)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:48)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:802)
at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:120)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:604)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:564)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:553)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:57)
at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:892)
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException: message
at org.apache.mina.filter.codec.AbstractProtocolDecoderOutput.write(AbstractProtocolDecoderOutput.java:44)
at org.apache.directory.server.kerberos.protocol.codec.MinaKerberosDecoder.decode(MinaKerberosDecoder.java:65)
at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:224)
... 15 more
2016-02-01 06:41:48,138 ERROR protocol.KerberosProtocolHandler (KerberosProtocolHandler.java:exceptionCaught(157)) - /127.0.0.1:46418 EXCEPTION
org.apache.mina.filter.codec.ProtocolDecoderException: java.lang.NullPointerException: message (Hexdump: 00 00 02 45 6C 82 02 41 30 82 02 3D A1 03 02 01 05 A2 03 02 01 0C A3 82 01 C4 30 82 01 C0 30 82 01 BC A1 03 02 01 01 A2 82 01 B3 04 82 01 AF 6E 82 01 AB 30 82 01 A7 A0 03 02 01 05 A1 03 02 01 0E A2 07 03 05 00 00 00 00 00 A3 81 F6 61 81 F3 30 81 F0 A0 03 02 01 05 A1 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A2 20 30 1E A0 03 02 01 02 A1 17 30 15 1B 06 6B 72 62 74 67 74 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 81 B7 30 81 B4 A0 03 02 01 11 A2 81 AC 04 81 A9 6D 55 CA 9E E9 6A 4D 9F C1 47 9A 2B 9E 75 07 73 F4 48 4A 36 AE BD 28 4D DA 3D 00 89 0D B1 70 C1 67 E7 44 0B C4 3E BF 59 3D 2B F2 EA E7 09 05 44 23 2D B6 D6 76 46 D8 DC 05 32 68 A2 2B D5 58 3D EC 5D DD 4D A1 6D F2 80 95 5F 61 2A 40 C5 D3 F7 BD A3 71 73 F2 81 DD CD B1 B3 1D 8E FA B0 70 9F 88 AD 97 7C 2A 91 DE 4F 69 5A 23 17 AA 21 99 7E 89 36 E6 1A 01 06 D2 DF 6F C3 76 15 47 81 9A 65 66 F7 CC 23 52 C5 DE 77 09 AD 66 38 94 DE 93 DC CA 24 6B C4 2B FA 1A BC FC 07 84 9B CE 0D 15 BA C7 00 8A 5C 0E 61 D8 BE 01 A4 81 98 30 81 95 A0 03 02 01 11 A2 81 8D 04 81 8A 7E 36 73 CB EC 41 15 69 53 02 B8 E4 28 1A 8D 61 56 3E 30 4E 0F C6 28 7C 12 CF EB 25 1E A9 3B 28 99 68 3B CE F3 38 50 B0 DA D9 0F 25 3A 39 B4 12 39 44 A6 64 8D FA 1B 62 22 27 7E 64 D8 C4 FE D3 89 DC 82 E2 C2 21 7E 14 BE DE 89 C6 9D FD C5 3F 25 40 9A F7 2C 26 DD 16 E1 EF C5 66 A5 AD 1A 46 8E 44 9C 86 63 9B 80 BA 81 45 78 97 F4 C1 EF EF 05 68 77 2B 61 B2 DF 30 0B BF E1 94 69 44 2C 97 7D 54 52 F8 9C 88 14 56 A0 B7 A4 69 30 67 A0 07 03 05 00 00 00 00 00 A2 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 1C 30 1A A0 03 02 01 00 A1 13 30 11 1B 04 48 54 54 50 1B 09 6C 6F 63 61 6C 68 6F 73 74 A5 11 18 0F 31 39 37 30 30 31 30 31 30 30 30 30 30 30 5A A7 06 02 04 3F 0B C9 19 A8 14 30 12 02 01 12 02 01 11 02 01 10)
at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:234)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:48)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:802)
at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:120)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:604)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:564)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:553)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:57)
at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:892)
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException: message
at org.apache.mina.filter.codec.AbstractProtocolDecoderOutput.write(AbstractProtocolDecoderOutput.java:44)
at org.apache.directory.server.kerberos.protocol.codec.MinaKerberosDecoder.decode(MinaKerberosDecoder.java:65)
at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:224)
... 15 more
2016-02-01 06:41:48,139 ERROR server.KERBEROS_LOG (KerberosProtocolHandler.java:exceptionCaught(158)) - /127.0.0.1:46418 EXCEPTION
org.apache.mina.filter.codec.ProtocolDecoderException: java.lang.NullPointerException: message (Hexdump: 00 00 02 45 6C 82 02 41 30 82 02 3D A1 03 02 01 05 A2 03 02 01 0C A3 82 01 C4 30 82 01 C0 30 82 01 BC A1 03 02 01 01 A2 82 01 B3 04 82 01 AF 6E 82 01 AB 30 82 01 A7 A0 03 02 01 05 A1 03 02 01 0E A2 07 03 05 00 00 00 00 00 A3 81 F6 61 81 F3 30 81 F0 A0 03 02 01 05 A1 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A2 20 30 1E A0 03 02 01 02 A1 17 30 15 1B 06 6B 72 62 74 67 74 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 81 B7 30 81 B4 A0 03 02 01 11 A2 81 AC 04 81 A9 6D 55 CA 9E E9 6A 4D 9F C1 47 9A 2B 9E 75 07 73 F4 48 4A 36 AE BD 28 4D DA 3D 00 89 0D B1 70 C1 67 E7 44 0B C4 3E BF 59 3D 2B F2 EA E7 09 05 44 23 2D B6 D6 76 46 D8 DC 05 32 68 A2 2B D5 58 3D EC 5D DD 4D A1 6D F2 80 95 5F 61 2A 40 C5 D3 F7 BD A3 71 73 F2 81 DD CD B1 B3 1D 8E FA B0 70 9F 88 AD 97 7C 2A 91 DE 4F 69 5A 23 17 AA 21 99 7E 89 36 E6 1A 01 06 D2 DF 6F C3 76 15 47 81 9A 65 66 F7 CC 23 52 C5 DE 77 09 AD 66 38 94 DE 93 DC CA 24 6B C4 2B FA 1A BC FC 07 84 9B CE 0D 15 BA C7 00 8A 5C 0E 61 D8 BE 01 A4 81 98 30 81 95 A0 03 02 01 11 A2 81 8D 04 81 8A 7E 36 73 CB EC 41 15 69 53 02 B8 E4 28 1A 8D 61 56 3E 30 4E 0F C6 28 7C 12 CF EB 25 1E A9 3B 28 99 68 3B CE F3 38 50 B0 DA D9 0F 25 3A 39 B4 12 39 44 A6 64 8D FA 1B 62 22 27 7E 64 D8 C4 FE D3 89 DC 82 E2 C2 21 7E 14 BE DE 89 C6 9D FD C5 3F 25 40 9A F7 2C 26 DD 16 E1 EF C5 66 A5 AD 1A 46 8E 44 9C 86 63 9B 80 BA 81 45 78 97 F4 C1 EF EF 05 68 77 2B 61 B2 DF 30 0B BF E1 94 69 44 2C 97 7D 54 52 F8 9C 88 14 56 A0 B7 A4 69 30 67 A0 07 03 05 00 00 00 00 00 A2 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 1C 30 1A A0 03 02 01 00 A1 13 30 11 1B 04 48 54 54 50 1B 09 6C 6F 63 61 6C 68 6F 73 74 A5 11 18 0F 31 39 37 30 30 31 30 31 30 30 30 30 30 30 5A A7 06 02 04 3F 0B C9 19 A8 14 30 12 02 01 12 02 01 11 02 01 10)
at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:234)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:48)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:802)
at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:120)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:604)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:564)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:553)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:57)
at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:892)
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException: message
at org.apache.mina.filter.codec.AbstractProtocolDecoderOutput.write(AbstractProtocolDecoderOutput.java:44)
at org.apache.directory.server.kerberos.protocol.codec.MinaKerberosDecoder.decode(MinaKerberosDecoder.java:65)
at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:224)
... 15 more
2016-02-01 06:41:48,139 WARN security.UserGroupInformation (UserGroupInformation.java:doAs(1696)) - PriviledgedActionException as:slave/localhost@EXAMPLE.COM (auth:KERBEROS) cause:java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Cannot get a KDC reply)
2016-02-01 06:41:48,140 ERROR namenode.EditLogInputStream (EditLogFileInputStream.java:nextOpImpl(194)) - caught exception initializing https://localhost:55326/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID
java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Cannot get a KDC reply)
at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:473)
at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:465)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
at org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:445)
at org.apache.hadoop.security.SecurityUtil.doAsCurrentUser(SecurityUtil.java:439)
at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog.getInputStream(EditLogFileInputStream.java:464)
at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.init(EditLogFileInputStream.java:141)
at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOpImpl(EditLogFileInputStream.java:192)
at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOp(EditLogFileInputStream.java:250)
at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:187)
at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:140)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:835)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:690)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:281)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1063)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:767)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:609)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:670)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:838)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:817)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1538)
at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1862)
at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1827)
at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.restartNameNode(TestSecureNNWithQJM.java:197)
at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.doNNWithQJMTest(TestSecureNNWithQJM.java:179)
at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.testSecureMode(TestSecureNNWithQJM.java:159)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Cannot get a KDC reply)
at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:306)
at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:196)
at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:215)
at org.apache.hadoop.hdfs.web.URLConnectionFactory.openConnection(URLConnectionFactory.java:161)
at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:470)
... 45 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Cannot get a KDC reply)
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:715)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:285)
at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:261)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:261)
... 49 more
Caused by: java.io.IOException: Cannot get a KDC reply
at sun.security.krb5.KdcComm.send(KdcComm.java:250)
at sun.security.krb5.KdcComm.send(KdcComm.java:191)
at sun.security.krb5.KrbTgsReq.send(KrbTgsReq.java:186)
at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:201)
at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:292)
at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:101)
at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:456)
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:641)
... 56 more
2016-02-01 06:41:48,141 ERROR namenode.EditLogInputStream (RedundantEditLogInputStream.java:nextOp(221)) - Got error reading edit log input stream https://localhost:55326/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID; failing over to edit log https://localhost:48398/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream$PrematureEOFException: got premature end-of-file at txid 0; expected file to go up to 5
at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:194)
at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:187)
at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:140)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:835)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:690)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:281)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1063)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:767)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:609)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:670)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:838)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:817)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1538)
at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1862)
at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1827)
at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.restartNameNode(TestSecureNNWithQJM.java:197)
at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.doNNWithQJMTest(TestSecureNNWithQJM.java:179)
at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.testSecureMode(TestSecureNNWithQJM.java:159)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
2016-02-01 06:41:48,142 INFO namenode.EditLogInputStream (RedundantEditLogInputStream.java:nextOp(176)) - Fast-forwarding stream 'https://localhost:48398/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID' to transaction ID 1
2016-02-01 06:41:48,169 ERROR protocol.KerberosProtocolHandler (KerberosProtocolHandler.java:exceptionCaught(157)) - /127.0.0.1:46420 EXCEPTION
org.apache.mina.filter.codec.ProtocolDecoderException: java.lang.NullPointerException: message (Hexdump: 00 00 02 45 6C 82 02 41 30 82 02 3D A1 03 02 01 05 A2 03 02 01 0C A3 82 01 C4 30 82 01 C0 30 82 01 BC A1 03 02 01 01 A2 82 01 B3 04 82 01 AF 6E 82 01 AB 30 82 01 A7 A0 03 02 01 05 A1 03 02 01 0E A2 07 03 05 00 00 00 00 00 A3 81 F6 61 81 F3 30 81 F0 A0 03 02 01 05 A1 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A2 20 30 1E A0 03 02 01 02 A1 17 30 15 1B 06 6B 72 62 74 67 74 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 81 B7 30 81 B4 A0 03 02 01 11 A2 81 AC 04 81 A9 6D 55 CA 9E E9 6A 4D 9F C1 47 9A 2B 9E 75 07 73 F4 48 4A 36 AE BD 28 4D DA 3D 00 89 0D B1 70 C1 67 E7 44 0B C4 3E BF 59 3D 2B F2 EA E7 09 05 44 23 2D B6 D6 76 46 D8 DC 05 32 68 A2 2B D5 58 3D EC 5D DD 4D A1 6D F2 80 95 5F 61 2A 40 C5 D3 F7 BD A3 71 73 F2 81 DD CD B1 B3 1D 8E FA B0 70 9F 88 AD 97 7C 2A 91 DE 4F 69 5A 23 17 AA 21 99 7E 89 36 E6 1A 01 06 D2 DF 6F C3 76 15 47 81 9A 65 66 F7 CC 23 52 C5 DE 77 09 AD 66 38 94 DE 93 DC CA 24 6B C4 2B FA 1A BC FC 07 84 9B CE 0D 15 BA C7 00 8A 5C 0E 61 D8 BE 01 A4 81 98 30 81 95 A0 03 02 01 11 A2 81 8D 04 81 8A BA 11 25 9F 01 D1 21 8A 31 30 5B 14 7A 6D A5 65 C1 87 95 F1 86 3A E1 A9 7E 3E 7F 4D FE 8E 8F 96 E4 0A 78 2B 5F 0D 1D 87 88 51 A4 5B 92 94 6A FC 87 97 2A 9C 69 81 B3 5D 5D 67 56 E9 97 97 5D 29 7B B9 B8 9E 99 8F 7A BD 3A E3 4D 6E BA 55 0C ED 3A 67 0F A9 45 B5 93 9B 48 FF FB 45 A7 23 5A B9 52 44 E4 54 D5 08 D5 2F FF 4A C4 52 16 FA 5F 1C 62 41 88 15 C1 B6 63 13 DA 16 33 56 63 37 09 63 2E 1D C8 42 55 FB FA 33 9B CC A4 69 30 67 A0 07 03 05 00 00 00 00 00 A2 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 1C 30 1A A0 03 02 01 00 A1 13 30 11 1B 04 48 54 54 50 1B 09 6C 6F 63 61 6C 68 6F 73 74 A5 11 18 0F 31 39 37 30 30 31 30 31 30 30 30 30 30 30 5A A7 06 02 04 01 AA D4 AF A8 14 30 12 02 01 12 02 01 11 02 01 10)
at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:234)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:48)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:802)
at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:120)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:604)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:564)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:553)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:57)
at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:892)
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException: message
at org.apache.mina.filter.codec.AbstractProtocolDecoderOutput.write(AbstractProtocolDecoderOutput.java:44)
at org.apache.directory.server.kerberos.protocol.codec.MinaKerberosDecoder.decode(MinaKerberosDecoder.java:65)
at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:224)
... 15 more
2016-02-01 06:41:48,169 ERROR server.KERBEROS_LOG (KerberosProtocolHandler.java:exceptionCaught(158)) - /127.0.0.1:46420 EXCEPTION
org.apache.mina.filter.codec.ProtocolDecoderException: java.lang.NullPointerException: message (Hexdump: 00 00 02 45 6C 82 02 41 30 82 02 3D A1 03 02 01 05 A2 03 02 01 0C A3 82 01 C4 30 82 01 C0 30 82 01 BC A1 03 02 01 01 A2 82 01 B3 04 82 01 AF 6E 82 01 AB 30 82 01 A7 A0 03 02 01 05 A1 03 02 01 0E A2 07 03 05 00 00 00 00 00 A3 81 F6 61 81 F3 30 81 F0 A0 03 02 01 05 A1 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A2 20 30 1E A0 03 02 01 02 A1 17 30 15 1B 06 6B 72 62 74 67 74 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 81 B7 30 81 B4 A0 03 02 01 11 A2 81 AC 04 81 A9 6D 55 CA 9E E9 6A 4D 9F C1 47 9A 2B 9E 75 07 73 F4 48 4A 36 AE BD 28 4D DA 3D 00 89 0D B1 70 C1 67 E7 44 0B C4 3E BF 59 3D 2B F2 EA E7 09 05 44 23 2D B6 D6 76 46 D8 DC 05 32 68 A2 2B D5 58 3D EC 5D DD 4D A1 6D F2 80 95 5F 61 2A 40 C5 D3 F7 BD A3 71 73 F2 81 DD CD B1 B3 1D 8E FA B0 70 9F 88 AD 97 7C 2A 91 DE 4F 69 5A 23 17 AA 21 99 7E 89 36 E6 1A 01 06 D2 DF 6F C3 76 15 47 81 9A 65 66 F7 CC 23 52 C5 DE 77 09 AD 66 38 94 DE 93 DC CA 24 6B C4 2B FA 1A BC FC 07 84 9B CE 0D 15 BA C7 00 8A 5C 0E 61 D8 BE 01 A4 81 98 30 81 95 A0 03 02 01 11 A2 81 8D 04 81 8A BA 11 25 9F 01 D1 21 8A 31 30 5B 14 7A 6D A5 65 C1 87 95 F1 86 3A E1 A9 7E 3E 7F 4D FE 8E 8F 96 E4 0A 78 2B 5F 0D 1D 87 88 51 A4 5B 92 94 6A FC 87 97 2A 9C 69 81 B3 5D 5D 67 56 E9 97 97 5D 29 7B B9 B8 9E 99 8F 7A BD 3A E3 4D 6E BA 55 0C ED 3A 67 0F A9 45 B5 93 9B 48 FF FB 45 A7 23 5A B9 52 44 E4 54 D5 08 D5 2F FF 4A C4 52 16 FA 5F 1C 62 41 88 15 C1 B6 63 13 DA 16 33 56 63 37 09 63 2E 1D C8 42 55 FB FA 33 9B CC A4 69 30 67 A0 07 03 05 00 00 00 00 00 A2 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 1C 30 1A A0 03 02 01 00 A1 13 30 11 1B 04 48 54 54 50 1B 09 6C 6F 63 61 6C 68 6F 73 74 A5 11 18 0F 31 39 37 30 30 31 30 31 30 30 30 30 30 30 5A A7 06 02 04 01 AA D4 AF A8 14 30 12 02 01 12 02 01 11 02 01 10)
at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:234)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:48)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:802)
at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:120)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:604)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:564)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:553)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:57)
at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:892)
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException: message
at org.apache.mina.filter.codec.AbstractProtocolDecoderOutput.write(AbstractProtocolDecoderOutput.java:44)
at org.apache.directory.server.kerberos.protocol.codec.MinaKerberosDecoder.decode(MinaKerberosDecoder.java:65)
at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:224)
... 15 more
2016-02-01 06:41:48,170 WARN security.UserGroupInformation (UserGroupInformation.java:doAs(1696)) - PriviledgedActionException as:slave/localhost@EXAMPLE.COM (auth:KERBEROS) cause:java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Cannot get a KDC reply)
2016-02-01 06:41:48,170 ERROR namenode.EditLogInputStream (EditLogFileInputStream.java:nextOpImpl(194)) - caught exception initializing https://localhost:48398/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID
java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Cannot get a KDC reply)
at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:473)
at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:465)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
at org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:445)
at org.apache.hadoop.security.SecurityUtil.doAsCurrentUser(SecurityUtil.java:439)
at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog.getInputStream(EditLogFileInputStream.java:464)
at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.init(EditLogFileInputStream.java:141)
at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOpImpl(EditLogFileInputStream.java:192)
at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOp(EditLogFileInputStream.java:250)
at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:187)
at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:140)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:835)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:690)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:281)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1063)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:767)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:609)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:670)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:838)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:817)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1538)
at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1862)
at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1827)
at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.restartNameNode(TestSecureNNWithQJM.java:197)
at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.doNNWithQJMTest(TestSecureNNWithQJM.java:179)
at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.testSecureMode(TestSecureNNWithQJM.java:159)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Cannot get a KDC reply)
at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:306)
at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:196)
at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:215)
at org.apache.hadoop.hdfs.web.URLConnectionFactory.openConnection(URLConnectionFactory.java:161)
at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:470)
... 45 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Cannot get a KDC reply)
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:715)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:285)
at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:261)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:261)
... 49 more
Caused by: java.io.IOException: Cannot get a KDC reply
at sun.security.krb5.KdcComm.send(KdcComm.java:250)
at sun.security.krb5.KdcComm.send(KdcComm.java:191)
at sun.security.krb5.KrbTgsReq.send(KrbTgsReq.java:186)
at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:201)
at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:292)
at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:101)
at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:456)
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:641)
... 56 more
2016-02-01 06:41:48,171 ERROR namenode.FSImage (FSEditLogLoader.java:loadEditRecords(196)) - Error replaying edit log at offset 0. Expected transaction ID was 1
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream$PrematureEOFException: got premature end-of-file at txid 0; expected file to go up to 5
at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:194)
at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:187)
at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:140)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:835)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:690)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:281)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1063)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:767)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:609)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:670)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:838)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:817)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1538)
at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1862)
at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1827)
at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.restartNameNode(TestSecureNNWithQJM.java:197)
at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.doNNWithQJMTest(TestSecureNNWithQJM.java:179)
at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.testSecureMode(TestSecureNNWithQJM.java:159)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
2016-02-01 06:41:48,173 WARN namenode.FSNamesystem (FSNamesystem.java:loadFromDisk(769)) - Encountered exception loading fsimage
org.apache.hadoop.hdfs.server.namenode.EditLogInputException: Error replaying edit log at offset 0. Expected transaction ID was 1
at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:200)
at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:140)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:835)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:690)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:281)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1063)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:767)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:609)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:670)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:838)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:817)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1538)
at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1862)
at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1827)
at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.restartNameNode(TestSecureNNWithQJM.java:197)
at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.doNNWithQJMTest(TestSecureNNWithQJM.java:179)
at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.testSecureMode(TestSecureNNWithQJM.java:159)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Caused by: org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream$PrematureEOFException: got premature end-of-file at txid 0; expected file to go up to 5
at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:194)
at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:187)
... 27 more
2016-02-01 06:41:48,187 INFO mortbay.log (Slf4jLog.java:info(67)) - Stopped SslSocketConnectorSecure@localhost:41570
2016-02-01 06:41:48,287 INFO hdfs.MiniDFSCluster (MiniDFSCluster.java:shutdown(1744)) - Shutting down the Mini HDFS Cluster
Daniel Templeton
added a comment - Looks like it might be HADOOP-12090 .
Xiao Chen
added a comment -
Thanks Daniel for the pointer, HADOOP-12090 looks to be the same issue. Resolving this as dup.
P.S. looks the ds jira mentioned in HADOOP-12090 is fixed in M22 which is unreleased, so I guess we'll just wait for that and upgrade.
Xiao Chen
added a comment - Thanks Daniel for the pointer, HADOOP-12090 looks to be the same issue. Resolving this as dup.
P.S. looks the ds jira mentioned in HADOOP-12090 is fixed in M22 which is unreleased, so I guess we'll just wait for that and upgrade.
TestKMS#testACLs failure I observed in https://builds.apache.org/job/PreCommit-HADOOP-Build/8504:
Error Message
Stacktrace
Standard Output