Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9746

Some Kerberos related tests intermittently fail.

Details

    • Test
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • None
    • None
    • None
    • None

    Description

      So far I've seen TestSecureNNWithQJM#testSecureMode and TestKMS#testACLs failing. More details coming in the 1st comment.

      Attachments

        Issue Links

          Activity

            xiaochen Xiao Chen added a comment -

            TestKMS#testACLs failure I observed in https://builds.apache.org/job/PreCommit-HADOOP-Build/8504:
            Error Message

            org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Connection reset)
            

            Stacktrace

            java.lang.AssertionError: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Connection reset)
            	at org.junit.Assert.fail(Assert.java:88)
            	at org.apache.hadoop.crypto.key.kms.server.TestKMS$9$6.run(TestKMS.java:1296)
            	at org.apache.hadoop.crypto.key.kms.server.TestKMS$9$6.run(TestKMS.java:1287)
            	at java.security.AccessController.doPrivileged(Native Method)
            	at javax.security.auth.Subject.doAs(Subject.java:415)
            	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1727)
            	at org.apache.hadoop.crypto.key.kms.server.TestKMS.doAs(TestKMS.java:251)
            	at org.apache.hadoop.crypto.key.kms.server.TestKMS.access$100(TestKMS.java:70)
            

            Standard Output

            Test KMS running at: http://localhost:50643/kms
            2016-02-01 21:09:45,207 WARN  LoadBalancingKMSClientProvider - KMS provider at [http://localhost:50643/kms/v1/] threw an IOException [User:client not allowed to do 'CREATE_KEY' on 'k']!!
            2016-02-01 21:09:45,207 WARN  LoadBalancingKMSClientProvider - Aborting since the Request has failed with all KMS providers in the group. !!
            2016-02-01 21:09:45,251 WARN  LoadBalancingKMSClientProvider - KMS provider at [http://localhost:50643/kms/v1/] threw an IOException [User:client not allowed to do 'CREATE_KEY' on 'k']!!
            2016-02-01 21:09:45,251 WARN  LoadBalancingKMSClientProvider - Aborting since the Request has failed with all KMS providers in the group. !!
            2016-02-01 21:09:45,319 WARN  LoadBalancingKMSClientProvider - KMS provider at [http://localhost:50643/kms/v1/] threw an IOException [User:client not allowed to do 'ROLL_NEW_VERSION' on 'k']!!
            2016-02-01 21:09:45,320 WARN  LoadBalancingKMSClientProvider - Aborting since the Request has failed with all KMS providers in the group. !!
            2016-02-01 21:09:45,359 WARN  LoadBalancingKMSClientProvider - KMS provider at [http://localhost:50643/kms/v1/] threw an IOException [User:client not allowed to do 'ROLL_NEW_VERSION' on 'k']!!
            2016-02-01 21:09:45,359 WARN  LoadBalancingKMSClientProvider - Aborting since the Request has failed with all KMS providers in the group. !!
            2016-02-01 21:09:45,410 WARN  LoadBalancingKMSClientProvider - KMS provider at [http://localhost:50643/kms/v1/] threw an IOException [User:client not allowed to do 'GET_KEYS']!!
            2016-02-01 21:09:45,410 WARN  LoadBalancingKMSClientProvider - Aborting since the Request has failed with all KMS providers in the group. !!
            2016-02-01 21:09:45,443 WARN  LoadBalancingKMSClientProvider - KMS provider at [http://localhost:50643/kms/v1/] threw an IOException [User:client not allowed to do 'GET_KEYS_METADATA']!!
            2016-02-01 21:09:45,443 WARN  LoadBalancingKMSClientProvider - Aborting since the Request has failed with all KMS providers in the group. !!
            2016-02-01 21:09:45,477 WARN  LoadBalancingKMSClientProvider - KMS provider at [http://localhost:50643/kms/v1/] threw an IOException [User:client not allowed to do 'GET_KEY_VERSION']!!
            2016-02-01 21:09:45,478 WARN  LoadBalancingKMSClientProvider - Aborting since the Request has failed with all KMS providers in the group. !!
            2016-02-01 21:09:45,521 WARN  LoadBalancingKMSClientProvider - KMS provider at [http://localhost:50643/kms/v1/] threw an IOException [User:client not allowed to do 'GET_CURRENT_KEY' on 'k']!!
            2016-02-01 21:09:45,521 WARN  LoadBalancingKMSClientProvider - Aborting since the Request has failed with all KMS providers in the group. !!
            2016-02-01 21:09:45,559 WARN  LoadBalancingKMSClientProvider - KMS provider at [http://localhost:50643/kms/v1/] threw an IOException [User:client not allowed to do 'GET_METADATA' on 'k']!!
            2016-02-01 21:09:45,559 WARN  LoadBalancingKMSClientProvider - Aborting since the Request has failed with all KMS providers in the group. !!
            2016-02-01 21:09:45,595 WARN  LoadBalancingKMSClientProvider - KMS provider at [http://localhost:50643/kms/v1/] threw an IOException [User:client not allowed to do 'GET_KEY_VERSIONS' on 'k']!!
            2016-02-01 21:09:45,596 WARN  LoadBalancingKMSClientProvider - Aborting since the Request has failed with all KMS providers in the group. !!
            2016-02-01 21:09:45,979 ERROR KerberosProtocolHandler - /127.0.0.1:42589 EXCEPTION
            org.apache.mina.filter.codec.ProtocolDecoderException: java.lang.NullPointerException: message (Hexdump: 00 00 02 45 6C 82 02 41 30 82 02 3D A1 03 02 01 05 A2 03 02 01 0C A3 82 01 C4 30 82 01 C0 30 82 01 BC A1 03 02 01 01 A2 82 01 B3 04 82 01 AF 6E 82 01 AB 30 82 01 A7 A0 03 02 01 05 A1 03 02 01 0E A2 07 03 05 00 00 00 00 00 A3 81 F6 61 81 F3 30 81 F0 A0 03 02 01 05 A1 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A2 20 30 1E A0 03 02 01 02 A1 17 30 15 1B 06 6B 72 62 74 67 74 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 81 B7 30 81 B4 A0 03 02 01 11 A2 81 AC 04 81 A9 E8 45 37 63 BA EF 73 10 9E 77 EC 44 59 5A 2E 8E 87 BE AC E8 01 2C 2C A8 82 EB 69 96 2D E7 04 17 FB FC AC 43 B2 C1 DF 28 25 64 9F D6 DF 69 90 E6 26 B8 A7 77 7E D2 A1 5A A1 1C 9F D8 10 7A ED 6F AD AA 02 09 2F FB 6D A8 B5 77 75 69 46 17 57 D7 15 8A 57 4D 08 2A 28 EA CE 09 70 0B B0 28 C0 57 E7 FF 33 81 79 0E 17 C7 52 7C DB 24 D2 FA 32 71 BE A1 A5 98 D9 82 8E 3F 96 1F 28 EF F3 AA 34 AA 79 0B B8 FC 3D 78 B7 8C A8 9A F8 23 F0 02 1C 47 85 8F 77 F9 5B 89 1C 1E DA D8 8C 93 84 BE 64 82 BE 39 B8 3A 7E 82 7B C9 86 A4 81 98 30 81 95 A0 03 02 01 11 A2 81 8D 04 81 8A 83 E7 4F 6C A4 F7 09 6F 56 32 8B 59 E0 70 08 F1 35 FE A0 42 84 5D 3D F5 82 43 04 EF 23 B1 CF 7D 57 59 9D 0D 43 12 54 0A 16 46 DE E9 99 88 F0 A7 FA B9 E6 E3 CC 81 92 57 AD F4 1D 5B 4F 81 C2 79 39 16 6A C0 32 09 D6 A5 3C B9 86 20 76 54 36 5B B3 2D 0E 97 36 53 3E 50 4B 60 90 3F F7 1E EA 6A 70 DE 75 61 5B 20 0F 58 51 C6 0A CF A5 01 68 46 FF 46 F3 7D 48 3F DB E6 A2 DA 50 6C 1F DA CB A6 9D 64 27 B1 B3 D4 FE 5B 91 51 A4 69 30 67 A0 07 03 05 00 00 00 00 00 A2 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 1C 30 1A A0 03 02 01 00 A1 13 30 11 1B 04 48 54 54 50 1B 09 6C 6F 63 61 6C 68 6F 73 74 A5 11 18 0F 31 39 37 30 30 31 30 31 30 30 30 30 30 30 5A A7 06 02 04 51 8E BF AE A8 14 30 12 02 01 12 02 01 11 02 01 10)
            	at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:234)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:48)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:802)
            	at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:120)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:604)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:564)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:553)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:57)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:892)
            	at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            	at java.lang.Thread.run(Thread.java:745)
            Caused by: java.lang.NullPointerException: message
            	at org.apache.mina.filter.codec.AbstractProtocolDecoderOutput.write(AbstractProtocolDecoderOutput.java:44)
            	at org.apache.directory.server.kerberos.protocol.codec.MinaKerberosDecoder.decode(MinaKerberosDecoder.java:65)
            	at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:224)
            	... 15 more
            2016-02-01 21:09:45,981 ERROR KERBEROS_LOG - /127.0.0.1:42589 EXCEPTION
            org.apache.mina.filter.codec.ProtocolDecoderException: java.lang.NullPointerException: message (Hexdump: 00 00 02 45 6C 82 02 41 30 82 02 3D A1 03 02 01 05 A2 03 02 01 0C A3 82 01 C4 30 82 01 C0 30 82 01 BC A1 03 02 01 01 A2 82 01 B3 04 82 01 AF 6E 82 01 AB 30 82 01 A7 A0 03 02 01 05 A1 03 02 01 0E A2 07 03 05 00 00 00 00 00 A3 81 F6 61 81 F3 30 81 F0 A0 03 02 01 05 A1 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A2 20 30 1E A0 03 02 01 02 A1 17 30 15 1B 06 6B 72 62 74 67 74 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 81 B7 30 81 B4 A0 03 02 01 11 A2 81 AC 04 81 A9 E8 45 37 63 BA EF 73 10 9E 77 EC 44 59 5A 2E 8E 87 BE AC E8 01 2C 2C A8 82 EB 69 96 2D E7 04 17 FB FC AC 43 B2 C1 DF 28 25 64 9F D6 DF 69 90 E6 26 B8 A7 77 7E D2 A1 5A A1 1C 9F D8 10 7A ED 6F AD AA 02 09 2F FB 6D A8 B5 77 75 69 46 17 57 D7 15 8A 57 4D 08 2A 28 EA CE 09 70 0B B0 28 C0 57 E7 FF 33 81 79 0E 17 C7 52 7C DB 24 D2 FA 32 71 BE A1 A5 98 D9 82 8E 3F 96 1F 28 EF F3 AA 34 AA 79 0B B8 FC 3D 78 B7 8C A8 9A F8 23 F0 02 1C 47 85 8F 77 F9 5B 89 1C 1E DA D8 8C 93 84 BE 64 82 BE 39 B8 3A 7E 82 7B C9 86 A4 81 98 30 81 95 A0 03 02 01 11 A2 81 8D 04 81 8A 83 E7 4F 6C A4 F7 09 6F 56 32 8B 59 E0 70 08 F1 35 FE A0 42 84 5D 3D F5 82 43 04 EF 23 B1 CF 7D 57 59 9D 0D 43 12 54 0A 16 46 DE E9 99 88 F0 A7 FA B9 E6 E3 CC 81 92 57 AD F4 1D 5B 4F 81 C2 79 39 16 6A C0 32 09 D6 A5 3C B9 86 20 76 54 36 5B B3 2D 0E 97 36 53 3E 50 4B 60 90 3F F7 1E EA 6A 70 DE 75 61 5B 20 0F 58 51 C6 0A CF A5 01 68 46 FF 46 F3 7D 48 3F DB E6 A2 DA 50 6C 1F DA CB A6 9D 64 27 B1 B3 D4 FE 5B 91 51 A4 69 30 67 A0 07 03 05 00 00 00 00 00 A2 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 1C 30 1A A0 03 02 01 00 A1 13 30 11 1B 04 48 54 54 50 1B 09 6C 6F 63 61 6C 68 6F 73 74 A5 11 18 0F 31 39 37 30 30 31 30 31 30 30 30 30 30 30 5A A7 06 02 04 51 8E BF AE A8 14 30 12 02 01 12 02 01 11 02 01 10)
            	at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:234)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:48)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:802)
            	at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:120)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:604)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:564)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:553)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:57)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:892)
            	at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            	at java.lang.Thread.run(Thread.java:745)
            Caused by: java.lang.NullPointerException: message
            	at org.apache.mina.filter.codec.AbstractProtocolDecoderOutput.write(AbstractProtocolDecoderOutput.java:44)
            	at org.apache.directory.server.kerberos.protocol.codec.MinaKerberosDecoder.decode(MinaKerberosDecoder.java:65)
            	at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:224)
            	... 15 more
            2016-02-01 21:09:45,987 ERROR KerberosProtocolHandler - /127.0.0.1:42590 EXCEPTION
            org.apache.mina.filter.codec.ProtocolDecoderException: java.lang.NullPointerException: message (Hexdump: 00 00 02 45 6C 82 02 41 30 82 02 3D A1 03 02 01 05 A2 03 02 01 0C A3 82 01 C4 30 82 01 C0 30 82 01 BC A1 03 02 01 01 A2 82 01 B3 04 82 01 AF 6E 82 01 AB 30 82 01 A7 A0 03 02 01 05 A1 03 02 01 0E A2 07 03 05 00 00 00 00 00 A3 81 F6 61 81 F3 30 81 F0 A0 03 02 01 05 A1 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A2 20 30 1E A0 03 02 01 02 A1 17 30 15 1B 06 6B 72 62 74 67 74 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 81 B7 30 81 B4 A0 03 02 01 11 A2 81 AC 04 81 A9 E8 45 37 63 BA EF 73 10 9E 77 EC 44 59 5A 2E 8E 87 BE AC E8 01 2C 2C A8 82 EB 69 96 2D E7 04 17 FB FC AC 43 B2 C1 DF 28 25 64 9F D6 DF 69 90 E6 26 B8 A7 77 7E D2 A1 5A A1 1C 9F D8 10 7A ED 6F AD AA 02 09 2F FB 6D A8 B5 77 75 69 46 17 57 D7 15 8A 57 4D 08 2A 28 EA CE 09 70 0B B0 28 C0 57 E7 FF 33 81 79 0E 17 C7 52 7C DB 24 D2 FA 32 71 BE A1 A5 98 D9 82 8E 3F 96 1F 28 EF F3 AA 34 AA 79 0B B8 FC 3D 78 B7 8C A8 9A F8 23 F0 02 1C 47 85 8F 77 F9 5B 89 1C 1E DA D8 8C 93 84 BE 64 82 BE 39 B8 3A 7E 82 7B C9 86 A4 81 98 30 81 95 A0 03 02 01 11 A2 81 8D 04 81 8A 76 03 CF 4B 54 6A 07 67 A5 3A CF CE 57 55 BE 1A 1D BA 77 F2 BA DE 27 5C 5D 0E 8D 5C 88 19 A1 AD A2 F5 E3 66 54 79 8A 25 2C DD 56 44 07 67 48 44 D4 94 68 BA 47 ED 20 0C B9 3C D9 A9 F2 AA 7F 81 35 E4 4F 5B 47 CB 97 EE 40 67 25 09 A6 15 FC 3D 10 BA 58 A1 7F 84 5E 4F 69 92 78 B8 04 7C 7A 1E 9A 88 DF B0 E1 5B 09 68 D4 10 D9 AB 4F 5F 4D 37 24 CB 5D 7C FE 59 9E CF 1F 95 E7 75 B6 DC 61 13 E8 18 34 9C 5D A4 B8 3F C2 C5 A4 69 30 67 A0 07 03 05 00 00 00 00 00 A2 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 1C 30 1A A0 03 02 01 00 A1 13 30 11 1B 04 48 54 54 50 1B 09 6C 6F 63 61 6C 68 6F 73 74 A5 11 18 0F 31 39 37 30 30 31 30 31 30 30 30 30 30 30 5A A7 06 02 04 44 F2 C4 87 A8 14 30 12 02 01 12 02 01 11 02 01 10)
            	at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:234)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:48)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:802)
            	at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:120)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:604)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:564)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:553)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:57)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:892)
            	at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            	at java.lang.Thread.run(Thread.java:745)
            Caused by: java.lang.NullPointerException: message
            	at org.apache.mina.filter.codec.AbstractProtocolDecoderOutput.write(AbstractProtocolDecoderOutput.java:44)
            	at org.apache.directory.server.kerberos.protocol.codec.MinaKerberosDecoder.decode(MinaKerberosDecoder.java:65)
            	at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:224)
            	... 15 more
            2016-02-01 21:09:45,988 ERROR KERBEROS_LOG - /127.0.0.1:42590 EXCEPTION
            org.apache.mina.filter.codec.ProtocolDecoderException: java.lang.NullPointerException: message (Hexdump: 00 00 02 45 6C 82 02 41 30 82 02 3D A1 03 02 01 05 A2 03 02 01 0C A3 82 01 C4 30 82 01 C0 30 82 01 BC A1 03 02 01 01 A2 82 01 B3 04 82 01 AF 6E 82 01 AB 30 82 01 A7 A0 03 02 01 05 A1 03 02 01 0E A2 07 03 05 00 00 00 00 00 A3 81 F6 61 81 F3 30 81 F0 A0 03 02 01 05 A1 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A2 20 30 1E A0 03 02 01 02 A1 17 30 15 1B 06 6B 72 62 74 67 74 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 81 B7 30 81 B4 A0 03 02 01 11 A2 81 AC 04 81 A9 E8 45 37 63 BA EF 73 10 9E 77 EC 44 59 5A 2E 8E 87 BE AC E8 01 2C 2C A8 82 EB 69 96 2D E7 04 17 FB FC AC 43 B2 C1 DF 28 25 64 9F D6 DF 69 90 E6 26 B8 A7 77 7E D2 A1 5A A1 1C 9F D8 10 7A ED 6F AD AA 02 09 2F FB 6D A8 B5 77 75 69 46 17 57 D7 15 8A 57 4D 08 2A 28 EA CE 09 70 0B B0 28 C0 57 E7 FF 33 81 79 0E 17 C7 52 7C DB 24 D2 FA 32 71 BE A1 A5 98 D9 82 8E 3F 96 1F 28 EF F3 AA 34 AA 79 0B B8 FC 3D 78 B7 8C A8 9A F8 23 F0 02 1C 47 85 8F 77 F9 5B 89 1C 1E DA D8 8C 93 84 BE 64 82 BE 39 B8 3A 7E 82 7B C9 86 A4 81 98 30 81 95 A0 03 02 01 11 A2 81 8D 04 81 8A 76 03 CF 4B 54 6A 07 67 A5 3A CF CE 57 55 BE 1A 1D BA 77 F2 BA DE 27 5C 5D 0E 8D 5C 88 19 A1 AD A2 F5 E3 66 54 79 8A 25 2C DD 56 44 07 67 48 44 D4 94 68 BA 47 ED 20 0C B9 3C D9 A9 F2 AA 7F 81 35 E4 4F 5B 47 CB 97 EE 40 67 25 09 A6 15 FC 3D 10 BA 58 A1 7F 84 5E 4F 69 92 78 B8 04 7C 7A 1E 9A 88 DF B0 E1 5B 09 68 D4 10 D9 AB 4F 5F 4D 37 24 CB 5D 7C FE 59 9E CF 1F 95 E7 75 B6 DC 61 13 E8 18 34 9C 5D A4 B8 3F C2 C5 A4 69 30 67 A0 07 03 05 00 00 00 00 00 A2 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 1C 30 1A A0 03 02 01 00 A1 13 30 11 1B 04 48 54 54 50 1B 09 6C 6F 63 61 6C 68 6F 73 74 A5 11 18 0F 31 39 37 30 30 31 30 31 30 30 30 30 30 30 5A A7 06 02 04 44 F2 C4 87 A8 14 30 12 02 01 12 02 01 11 02 01 10)
            	at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:234)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:48)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:802)
            	at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:120)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:604)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:564)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:553)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:57)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:892)
            	at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            	at java.lang.Thread.run(Thread.java:745)
            Caused by: java.lang.NullPointerException: message
            	at org.apache.mina.filter.codec.AbstractProtocolDecoderOutput.write(AbstractProtocolDecoderOutput.java:44)
            	at org.apache.directory.server.kerberos.protocol.codec.MinaKerberosDecoder.decode(MinaKerberosDecoder.java:65)
            	at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:224)
            	... 15 more
            2016-02-01 21:09:45,992 WARN  LoadBalancingKMSClientProvider - KMS provider at [http://localhost:50643/kms/v1/] threw an IOException [org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Connection reset)]!!
            2016-02-01 21:09:45,992 WARN  LoadBalancingKMSClientProvider - Aborting since the Request has failed with all KMS providers in the group. !!
            
            xiaochen Xiao Chen added a comment - TestKMS#testACLs failure I observed in https://builds.apache.org/job/PreCommit-HADOOP-Build/8504: Error Message org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Connection reset) Stacktrace java.lang.AssertionError: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Connection reset) at org.junit.Assert.fail(Assert.java:88) at org.apache.hadoop.crypto.key.kms.server.TestKMS$9$6.run(TestKMS.java:1296) at org.apache.hadoop.crypto.key.kms.server.TestKMS$9$6.run(TestKMS.java:1287) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1727) at org.apache.hadoop.crypto.key.kms.server.TestKMS.doAs(TestKMS.java:251) at org.apache.hadoop.crypto.key.kms.server.TestKMS.access$100(TestKMS.java:70) Standard Output Test KMS running at: http://localhost:50643/kms 2016-02-01 21:09:45,207 WARN LoadBalancingKMSClientProvider - KMS provider at [http://localhost:50643/kms/v1/] threw an IOException [User:client not allowed to do 'CREATE_KEY' on 'k']!! 2016-02-01 21:09:45,207 WARN LoadBalancingKMSClientProvider - Aborting since the Request has failed with all KMS providers in the group. !! 2016-02-01 21:09:45,251 WARN LoadBalancingKMSClientProvider - KMS provider at [http://localhost:50643/kms/v1/] threw an IOException [User:client not allowed to do 'CREATE_KEY' on 'k']!! 2016-02-01 21:09:45,251 WARN LoadBalancingKMSClientProvider - Aborting since the Request has failed with all KMS providers in the group. !! 2016-02-01 21:09:45,319 WARN LoadBalancingKMSClientProvider - KMS provider at [http://localhost:50643/kms/v1/] threw an IOException [User:client not allowed to do 'ROLL_NEW_VERSION' on 'k']!! 2016-02-01 21:09:45,320 WARN LoadBalancingKMSClientProvider - Aborting since the Request has failed with all KMS providers in the group. !! 2016-02-01 21:09:45,359 WARN LoadBalancingKMSClientProvider - KMS provider at [http://localhost:50643/kms/v1/] threw an IOException [User:client not allowed to do 'ROLL_NEW_VERSION' on 'k']!! 2016-02-01 21:09:45,359 WARN LoadBalancingKMSClientProvider - Aborting since the Request has failed with all KMS providers in the group. !! 2016-02-01 21:09:45,410 WARN LoadBalancingKMSClientProvider - KMS provider at [http://localhost:50643/kms/v1/] threw an IOException [User:client not allowed to do 'GET_KEYS']!! 2016-02-01 21:09:45,410 WARN LoadBalancingKMSClientProvider - Aborting since the Request has failed with all KMS providers in the group. !! 2016-02-01 21:09:45,443 WARN LoadBalancingKMSClientProvider - KMS provider at [http://localhost:50643/kms/v1/] threw an IOException [User:client not allowed to do 'GET_KEYS_METADATA']!! 2016-02-01 21:09:45,443 WARN LoadBalancingKMSClientProvider - Aborting since the Request has failed with all KMS providers in the group. !! 2016-02-01 21:09:45,477 WARN LoadBalancingKMSClientProvider - KMS provider at [http://localhost:50643/kms/v1/] threw an IOException [User:client not allowed to do 'GET_KEY_VERSION']!! 2016-02-01 21:09:45,478 WARN LoadBalancingKMSClientProvider - Aborting since the Request has failed with all KMS providers in the group. !! 2016-02-01 21:09:45,521 WARN LoadBalancingKMSClientProvider - KMS provider at [http://localhost:50643/kms/v1/] threw an IOException [User:client not allowed to do 'GET_CURRENT_KEY' on 'k']!! 2016-02-01 21:09:45,521 WARN LoadBalancingKMSClientProvider - Aborting since the Request has failed with all KMS providers in the group. !! 2016-02-01 21:09:45,559 WARN LoadBalancingKMSClientProvider - KMS provider at [http://localhost:50643/kms/v1/] threw an IOException [User:client not allowed to do 'GET_METADATA' on 'k']!! 2016-02-01 21:09:45,559 WARN LoadBalancingKMSClientProvider - Aborting since the Request has failed with all KMS providers in the group. !! 2016-02-01 21:09:45,595 WARN LoadBalancingKMSClientProvider - KMS provider at [http://localhost:50643/kms/v1/] threw an IOException [User:client not allowed to do 'GET_KEY_VERSIONS' on 'k']!! 2016-02-01 21:09:45,596 WARN LoadBalancingKMSClientProvider - Aborting since the Request has failed with all KMS providers in the group. !! 2016-02-01 21:09:45,979 ERROR KerberosProtocolHandler - /127.0.0.1:42589 EXCEPTION org.apache.mina.filter.codec.ProtocolDecoderException: java.lang.NullPointerException: message (Hexdump: 00 00 02 45 6C 82 02 41 30 82 02 3D A1 03 02 01 05 A2 03 02 01 0C A3 82 01 C4 30 82 01 C0 30 82 01 BC A1 03 02 01 01 A2 82 01 B3 04 82 01 AF 6E 82 01 AB 30 82 01 A7 A0 03 02 01 05 A1 03 02 01 0E A2 07 03 05 00 00 00 00 00 A3 81 F6 61 81 F3 30 81 F0 A0 03 02 01 05 A1 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A2 20 30 1E A0 03 02 01 02 A1 17 30 15 1B 06 6B 72 62 74 67 74 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 81 B7 30 81 B4 A0 03 02 01 11 A2 81 AC 04 81 A9 E8 45 37 63 BA EF 73 10 9E 77 EC 44 59 5A 2E 8E 87 BE AC E8 01 2C 2C A8 82 EB 69 96 2D E7 04 17 FB FC AC 43 B2 C1 DF 28 25 64 9F D6 DF 69 90 E6 26 B8 A7 77 7E D2 A1 5A A1 1C 9F D8 10 7A ED 6F AD AA 02 09 2F FB 6D A8 B5 77 75 69 46 17 57 D7 15 8A 57 4D 08 2A 28 EA CE 09 70 0B B0 28 C0 57 E7 FF 33 81 79 0E 17 C7 52 7C DB 24 D2 FA 32 71 BE A1 A5 98 D9 82 8E 3F 96 1F 28 EF F3 AA 34 AA 79 0B B8 FC 3D 78 B7 8C A8 9A F8 23 F0 02 1C 47 85 8F 77 F9 5B 89 1C 1E DA D8 8C 93 84 BE 64 82 BE 39 B8 3A 7E 82 7B C9 86 A4 81 98 30 81 95 A0 03 02 01 11 A2 81 8D 04 81 8A 83 E7 4F 6C A4 F7 09 6F 56 32 8B 59 E0 70 08 F1 35 FE A0 42 84 5D 3D F5 82 43 04 EF 23 B1 CF 7D 57 59 9D 0D 43 12 54 0A 16 46 DE E9 99 88 F0 A7 FA B9 E6 E3 CC 81 92 57 AD F4 1D 5B 4F 81 C2 79 39 16 6A C0 32 09 D6 A5 3C B9 86 20 76 54 36 5B B3 2D 0E 97 36 53 3E 50 4B 60 90 3F F7 1E EA 6A 70 DE 75 61 5B 20 0F 58 51 C6 0A CF A5 01 68 46 FF 46 F3 7D 48 3F DB E6 A2 DA 50 6C 1F DA CB A6 9D 64 27 B1 B3 D4 FE 5B 91 51 A4 69 30 67 A0 07 03 05 00 00 00 00 00 A2 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 1C 30 1A A0 03 02 01 00 A1 13 30 11 1B 04 48 54 54 50 1B 09 6C 6F 63 61 6C 68 6F 73 74 A5 11 18 0F 31 39 37 30 30 31 30 31 30 30 30 30 30 30 5A A7 06 02 04 51 8E BF AE A8 14 30 12 02 01 12 02 01 11 02 01 10) at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:234) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:48) at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:802) at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:120) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426) at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:604) at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:564) at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:553) at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:57) at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:892) at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NullPointerException: message at org.apache.mina.filter.codec.AbstractProtocolDecoderOutput.write(AbstractProtocolDecoderOutput.java:44) at org.apache.directory.server.kerberos.protocol.codec.MinaKerberosDecoder.decode(MinaKerberosDecoder.java:65) at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:224) ... 15 more 2016-02-01 21:09:45,981 ERROR KERBEROS_LOG - /127.0.0.1:42589 EXCEPTION org.apache.mina.filter.codec.ProtocolDecoderException: java.lang.NullPointerException: message (Hexdump: 00 00 02 45 6C 82 02 41 30 82 02 3D A1 03 02 01 05 A2 03 02 01 0C A3 82 01 C4 30 82 01 C0 30 82 01 BC A1 03 02 01 01 A2 82 01 B3 04 82 01 AF 6E 82 01 AB 30 82 01 A7 A0 03 02 01 05 A1 03 02 01 0E A2 07 03 05 00 00 00 00 00 A3 81 F6 61 81 F3 30 81 F0 A0 03 02 01 05 A1 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A2 20 30 1E A0 03 02 01 02 A1 17 30 15 1B 06 6B 72 62 74 67 74 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 81 B7 30 81 B4 A0 03 02 01 11 A2 81 AC 04 81 A9 E8 45 37 63 BA EF 73 10 9E 77 EC 44 59 5A 2E 8E 87 BE AC E8 01 2C 2C A8 82 EB 69 96 2D E7 04 17 FB FC AC 43 B2 C1 DF 28 25 64 9F D6 DF 69 90 E6 26 B8 A7 77 7E D2 A1 5A A1 1C 9F D8 10 7A ED 6F AD AA 02 09 2F FB 6D A8 B5 77 75 69 46 17 57 D7 15 8A 57 4D 08 2A 28 EA CE 09 70 0B B0 28 C0 57 E7 FF 33 81 79 0E 17 C7 52 7C DB 24 D2 FA 32 71 BE A1 A5 98 D9 82 8E 3F 96 1F 28 EF F3 AA 34 AA 79 0B B8 FC 3D 78 B7 8C A8 9A F8 23 F0 02 1C 47 85 8F 77 F9 5B 89 1C 1E DA D8 8C 93 84 BE 64 82 BE 39 B8 3A 7E 82 7B C9 86 A4 81 98 30 81 95 A0 03 02 01 11 A2 81 8D 04 81 8A 83 E7 4F 6C A4 F7 09 6F 56 32 8B 59 E0 70 08 F1 35 FE A0 42 84 5D 3D F5 82 43 04 EF 23 B1 CF 7D 57 59 9D 0D 43 12 54 0A 16 46 DE E9 99 88 F0 A7 FA B9 E6 E3 CC 81 92 57 AD F4 1D 5B 4F 81 C2 79 39 16 6A C0 32 09 D6 A5 3C B9 86 20 76 54 36 5B B3 2D 0E 97 36 53 3E 50 4B 60 90 3F F7 1E EA 6A 70 DE 75 61 5B 20 0F 58 51 C6 0A CF A5 01 68 46 FF 46 F3 7D 48 3F DB E6 A2 DA 50 6C 1F DA CB A6 9D 64 27 B1 B3 D4 FE 5B 91 51 A4 69 30 67 A0 07 03 05 00 00 00 00 00 A2 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 1C 30 1A A0 03 02 01 00 A1 13 30 11 1B 04 48 54 54 50 1B 09 6C 6F 63 61 6C 68 6F 73 74 A5 11 18 0F 31 39 37 30 30 31 30 31 30 30 30 30 30 30 5A A7 06 02 04 51 8E BF AE A8 14 30 12 02 01 12 02 01 11 02 01 10) at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:234) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:48) at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:802) at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:120) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426) at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:604) at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:564) at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:553) at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:57) at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:892) at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NullPointerException: message at org.apache.mina.filter.codec.AbstractProtocolDecoderOutput.write(AbstractProtocolDecoderOutput.java:44) at org.apache.directory.server.kerberos.protocol.codec.MinaKerberosDecoder.decode(MinaKerberosDecoder.java:65) at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:224) ... 15 more 2016-02-01 21:09:45,987 ERROR KerberosProtocolHandler - /127.0.0.1:42590 EXCEPTION org.apache.mina.filter.codec.ProtocolDecoderException: java.lang.NullPointerException: message (Hexdump: 00 00 02 45 6C 82 02 41 30 82 02 3D A1 03 02 01 05 A2 03 02 01 0C A3 82 01 C4 30 82 01 C0 30 82 01 BC A1 03 02 01 01 A2 82 01 B3 04 82 01 AF 6E 82 01 AB 30 82 01 A7 A0 03 02 01 05 A1 03 02 01 0E A2 07 03 05 00 00 00 00 00 A3 81 F6 61 81 F3 30 81 F0 A0 03 02 01 05 A1 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A2 20 30 1E A0 03 02 01 02 A1 17 30 15 1B 06 6B 72 62 74 67 74 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 81 B7 30 81 B4 A0 03 02 01 11 A2 81 AC 04 81 A9 E8 45 37 63 BA EF 73 10 9E 77 EC 44 59 5A 2E 8E 87 BE AC E8 01 2C 2C A8 82 EB 69 96 2D E7 04 17 FB FC AC 43 B2 C1 DF 28 25 64 9F D6 DF 69 90 E6 26 B8 A7 77 7E D2 A1 5A A1 1C 9F D8 10 7A ED 6F AD AA 02 09 2F FB 6D A8 B5 77 75 69 46 17 57 D7 15 8A 57 4D 08 2A 28 EA CE 09 70 0B B0 28 C0 57 E7 FF 33 81 79 0E 17 C7 52 7C DB 24 D2 FA 32 71 BE A1 A5 98 D9 82 8E 3F 96 1F 28 EF F3 AA 34 AA 79 0B B8 FC 3D 78 B7 8C A8 9A F8 23 F0 02 1C 47 85 8F 77 F9 5B 89 1C 1E DA D8 8C 93 84 BE 64 82 BE 39 B8 3A 7E 82 7B C9 86 A4 81 98 30 81 95 A0 03 02 01 11 A2 81 8D 04 81 8A 76 03 CF 4B 54 6A 07 67 A5 3A CF CE 57 55 BE 1A 1D BA 77 F2 BA DE 27 5C 5D 0E 8D 5C 88 19 A1 AD A2 F5 E3 66 54 79 8A 25 2C DD 56 44 07 67 48 44 D4 94 68 BA 47 ED 20 0C B9 3C D9 A9 F2 AA 7F 81 35 E4 4F 5B 47 CB 97 EE 40 67 25 09 A6 15 FC 3D 10 BA 58 A1 7F 84 5E 4F 69 92 78 B8 04 7C 7A 1E 9A 88 DF B0 E1 5B 09 68 D4 10 D9 AB 4F 5F 4D 37 24 CB 5D 7C FE 59 9E CF 1F 95 E7 75 B6 DC 61 13 E8 18 34 9C 5D A4 B8 3F C2 C5 A4 69 30 67 A0 07 03 05 00 00 00 00 00 A2 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 1C 30 1A A0 03 02 01 00 A1 13 30 11 1B 04 48 54 54 50 1B 09 6C 6F 63 61 6C 68 6F 73 74 A5 11 18 0F 31 39 37 30 30 31 30 31 30 30 30 30 30 30 5A A7 06 02 04 44 F2 C4 87 A8 14 30 12 02 01 12 02 01 11 02 01 10) at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:234) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:48) at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:802) at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:120) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426) at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:604) at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:564) at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:553) at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:57) at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:892) at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NullPointerException: message at org.apache.mina.filter.codec.AbstractProtocolDecoderOutput.write(AbstractProtocolDecoderOutput.java:44) at org.apache.directory.server.kerberos.protocol.codec.MinaKerberosDecoder.decode(MinaKerberosDecoder.java:65) at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:224) ... 15 more 2016-02-01 21:09:45,988 ERROR KERBEROS_LOG - /127.0.0.1:42590 EXCEPTION org.apache.mina.filter.codec.ProtocolDecoderException: java.lang.NullPointerException: message (Hexdump: 00 00 02 45 6C 82 02 41 30 82 02 3D A1 03 02 01 05 A2 03 02 01 0C A3 82 01 C4 30 82 01 C0 30 82 01 BC A1 03 02 01 01 A2 82 01 B3 04 82 01 AF 6E 82 01 AB 30 82 01 A7 A0 03 02 01 05 A1 03 02 01 0E A2 07 03 05 00 00 00 00 00 A3 81 F6 61 81 F3 30 81 F0 A0 03 02 01 05 A1 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A2 20 30 1E A0 03 02 01 02 A1 17 30 15 1B 06 6B 72 62 74 67 74 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 81 B7 30 81 B4 A0 03 02 01 11 A2 81 AC 04 81 A9 E8 45 37 63 BA EF 73 10 9E 77 EC 44 59 5A 2E 8E 87 BE AC E8 01 2C 2C A8 82 EB 69 96 2D E7 04 17 FB FC AC 43 B2 C1 DF 28 25 64 9F D6 DF 69 90 E6 26 B8 A7 77 7E D2 A1 5A A1 1C 9F D8 10 7A ED 6F AD AA 02 09 2F FB 6D A8 B5 77 75 69 46 17 57 D7 15 8A 57 4D 08 2A 28 EA CE 09 70 0B B0 28 C0 57 E7 FF 33 81 79 0E 17 C7 52 7C DB 24 D2 FA 32 71 BE A1 A5 98 D9 82 8E 3F 96 1F 28 EF F3 AA 34 AA 79 0B B8 FC 3D 78 B7 8C A8 9A F8 23 F0 02 1C 47 85 8F 77 F9 5B 89 1C 1E DA D8 8C 93 84 BE 64 82 BE 39 B8 3A 7E 82 7B C9 86 A4 81 98 30 81 95 A0 03 02 01 11 A2 81 8D 04 81 8A 76 03 CF 4B 54 6A 07 67 A5 3A CF CE 57 55 BE 1A 1D BA 77 F2 BA DE 27 5C 5D 0E 8D 5C 88 19 A1 AD A2 F5 E3 66 54 79 8A 25 2C DD 56 44 07 67 48 44 D4 94 68 BA 47 ED 20 0C B9 3C D9 A9 F2 AA 7F 81 35 E4 4F 5B 47 CB 97 EE 40 67 25 09 A6 15 FC 3D 10 BA 58 A1 7F 84 5E 4F 69 92 78 B8 04 7C 7A 1E 9A 88 DF B0 E1 5B 09 68 D4 10 D9 AB 4F 5F 4D 37 24 CB 5D 7C FE 59 9E CF 1F 95 E7 75 B6 DC 61 13 E8 18 34 9C 5D A4 B8 3F C2 C5 A4 69 30 67 A0 07 03 05 00 00 00 00 00 A2 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 1C 30 1A A0 03 02 01 00 A1 13 30 11 1B 04 48 54 54 50 1B 09 6C 6F 63 61 6C 68 6F 73 74 A5 11 18 0F 31 39 37 30 30 31 30 31 30 30 30 30 30 30 5A A7 06 02 04 44 F2 C4 87 A8 14 30 12 02 01 12 02 01 11 02 01 10) at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:234) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:48) at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:802) at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:120) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426) at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:604) at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:564) at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:553) at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:57) at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:892) at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NullPointerException: message at org.apache.mina.filter.codec.AbstractProtocolDecoderOutput.write(AbstractProtocolDecoderOutput.java:44) at org.apache.directory.server.kerberos.protocol.codec.MinaKerberosDecoder.decode(MinaKerberosDecoder.java:65) at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:224) ... 15 more 2016-02-01 21:09:45,992 WARN LoadBalancingKMSClientProvider - KMS provider at [http://localhost:50643/kms/v1/] threw an IOException [org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Connection reset)]!! 2016-02-01 21:09:45,992 WARN LoadBalancingKMSClientProvider - Aborting since the Request has failed with all KMS providers in the group. !!
            xiaochen Xiao Chen added a comment -

            Log excerpt of the failure I saw in TestSecureNNWithQJM#testSecureMode:
            Error Message

            org.apache.hadoop.hdfs.server.namenode.EditLogInputException: Error replaying edit log at offset 0.  Expected transaction ID was 1
            	at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:194)
            	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
            	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
            	at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
            	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
            	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:187)
            	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:140)
            	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:835)
            	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:690)
            	at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:281)
            	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1063)
            	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:767)
            	at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:609)
            	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:670)
            	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:838)
            	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:817)
            	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1538)
            	at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1862)
            	at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1827)
            	at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.restartNameNode(TestSecureNNWithQJM.java:197)
            	at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.doNNWithQJMTest(TestSecureNNWithQJM.java:179)
            	at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.testSecureMode(TestSecureNNWithQJM.java:159)
            

            stdout

            2016-02-01 06:41:47,944 INFO  namenode.FSImage (FSImage.java:loadEdits(832)) - Reading org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream@12e8c783 expecting start txid #1
            2016-02-01 06:41:47,944 INFO  namenode.FSImage (FSEditLogLoader.java:loadFSEdits(139)) - Start loading edits file https://localhost:55326/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID, https://localhost:48398/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID
            2016-02-01 06:41:47,945 INFO  namenode.EditLogInputStream (RedundantEditLogInputStream.java:nextOp(176)) - Fast-forwarding stream 'https://localhost:55326/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID, https://localhost:48398/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID' to transaction ID 1
            2016-02-01 06:41:47,945 INFO  namenode.EditLogInputStream (RedundantEditLogInputStream.java:nextOp(176)) - Fast-forwarding stream 'https://localhost:55326/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID' to transaction ID 1
            2016-02-01 06:41:48,131 ERROR protocol.KerberosProtocolHandler (KerberosProtocolHandler.java:exceptionCaught(157)) - /127.0.0.1:46417 EXCEPTION
            org.apache.mina.filter.codec.ProtocolDecoderException: java.lang.NullPointerException: message (Hexdump: 00 00 02 45 6C 82 02 41 30 82 02 3D A1 03 02 01 05 A2 03 02 01 0C A3 82 01 C4 30 82 01 C0 30 82 01 BC A1 03 02 01 01 A2 82 01 B3 04 82 01 AF 6E 82 01 AB 30 82 01 A7 A0 03 02 01 05 A1 03 02 01 0E A2 07 03 05 00 00 00 00 00 A3 81 F6 61 81 F3 30 81 F0 A0 03 02 01 05 A1 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A2 20 30 1E A0 03 02 01 02 A1 17 30 15 1B 06 6B 72 62 74 67 74 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 81 B7 30 81 B4 A0 03 02 01 11 A2 81 AC 04 81 A9 6D 55 CA 9E E9 6A 4D 9F C1 47 9A 2B 9E 75 07 73 F4 48 4A 36 AE BD 28 4D DA 3D 00 89 0D B1 70 C1 67 E7 44 0B C4 3E BF 59 3D 2B F2 EA E7 09 05 44 23 2D B6 D6 76 46 D8 DC 05 32 68 A2 2B D5 58 3D EC 5D DD 4D A1 6D F2 80 95 5F 61 2A 40 C5 D3 F7 BD A3 71 73 F2 81 DD CD B1 B3 1D 8E FA B0 70 9F 88 AD 97 7C 2A 91 DE 4F 69 5A 23 17 AA 21 99 7E 89 36 E6 1A 01 06 D2 DF 6F C3 76 15 47 81 9A 65 66 F7 CC 23 52 C5 DE 77 09 AD 66 38 94 DE 93 DC CA 24 6B C4 2B FA 1A BC FC 07 84 9B CE 0D 15 BA C7 00 8A 5C 0E 61 D8 BE 01 A4 81 98 30 81 95 A0 03 02 01 11 A2 81 8D 04 81 8A 60 DF 82 D5 14 DB 78 8D A8 E4 6D F8 FE 3A F4 AB 98 25 9D DB 51 ED 3B CE 53 C8 DC 48 1C CB EB B5 1B 5A 45 BA CD 68 0A 26 2F 8D 3A FE 75 AE 36 4B 25 B5 B8 5A C1 27 71 E3 B6 03 7D D6 2D 14 58 CD 6D 19 F0 25 D0 5A 9B 35 A6 7E 36 62 DA 28 56 0B E9 53 03 43 7B 71 D5 ED 8F 52 CE 6E 8A 23 0C 52 53 EB 42 0B 7A 6B 8C 54 EB 1C 70 FB 21 DD DF 23 B4 5E AD 42 67 65 42 61 FD DB 2D 28 C2 4D 7A 71 69 D0 74 9A 64 8A 82 A0 EC C8 A4 69 30 67 A0 07 03 05 00 00 00 00 00 A2 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 1C 30 1A A0 03 02 01 00 A1 13 30 11 1B 04 48 54 54 50 1B 09 6C 6F 63 61 6C 68 6F 73 74 A5 11 18 0F 31 39 37 30 30 31 30 31 30 30 30 30 30 30 5A A7 06 02 04 6B 88 55 A5 A8 14 30 12 02 01 12 02 01 11 02 01 10)
            	at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:234)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:48)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:802)
            	at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:120)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:604)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:564)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:553)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:57)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:892)
            	at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            	at java.lang.Thread.run(Thread.java:745)
            Caused by: java.lang.NullPointerException: message
            	at org.apache.mina.filter.codec.AbstractProtocolDecoderOutput.write(AbstractProtocolDecoderOutput.java:44)
            	at org.apache.directory.server.kerberos.protocol.codec.MinaKerberosDecoder.decode(MinaKerberosDecoder.java:65)
            	at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:224)
            	... 15 more
            2016-02-01 06:41:48,133 ERROR server.KERBEROS_LOG (KerberosProtocolHandler.java:exceptionCaught(158)) - /127.0.0.1:46417 EXCEPTION
            org.apache.mina.filter.codec.ProtocolDecoderException: java.lang.NullPointerException: message (Hexdump: 00 00 02 45 6C 82 02 41 30 82 02 3D A1 03 02 01 05 A2 03 02 01 0C A3 82 01 C4 30 82 01 C0 30 82 01 BC A1 03 02 01 01 A2 82 01 B3 04 82 01 AF 6E 82 01 AB 30 82 01 A7 A0 03 02 01 05 A1 03 02 01 0E A2 07 03 05 00 00 00 00 00 A3 81 F6 61 81 F3 30 81 F0 A0 03 02 01 05 A1 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A2 20 30 1E A0 03 02 01 02 A1 17 30 15 1B 06 6B 72 62 74 67 74 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 81 B7 30 81 B4 A0 03 02 01 11 A2 81 AC 04 81 A9 6D 55 CA 9E E9 6A 4D 9F C1 47 9A 2B 9E 75 07 73 F4 48 4A 36 AE BD 28 4D DA 3D 00 89 0D B1 70 C1 67 E7 44 0B C4 3E BF 59 3D 2B F2 EA E7 09 05 44 23 2D B6 D6 76 46 D8 DC 05 32 68 A2 2B D5 58 3D EC 5D DD 4D A1 6D F2 80 95 5F 61 2A 40 C5 D3 F7 BD A3 71 73 F2 81 DD CD B1 B3 1D 8E FA B0 70 9F 88 AD 97 7C 2A 91 DE 4F 69 5A 23 17 AA 21 99 7E 89 36 E6 1A 01 06 D2 DF 6F C3 76 15 47 81 9A 65 66 F7 CC 23 52 C5 DE 77 09 AD 66 38 94 DE 93 DC CA 24 6B C4 2B FA 1A BC FC 07 84 9B CE 0D 15 BA C7 00 8A 5C 0E 61 D8 BE 01 A4 81 98 30 81 95 A0 03 02 01 11 A2 81 8D 04 81 8A 60 DF 82 D5 14 DB 78 8D A8 E4 6D F8 FE 3A F4 AB 98 25 9D DB 51 ED 3B CE 53 C8 DC 48 1C CB EB B5 1B 5A 45 BA CD 68 0A 26 2F 8D 3A FE 75 AE 36 4B 25 B5 B8 5A C1 27 71 E3 B6 03 7D D6 2D 14 58 CD 6D 19 F0 25 D0 5A 9B 35 A6 7E 36 62 DA 28 56 0B E9 53 03 43 7B 71 D5 ED 8F 52 CE 6E 8A 23 0C 52 53 EB 42 0B 7A 6B 8C 54 EB 1C 70 FB 21 DD DF 23 B4 5E AD 42 67 65 42 61 FD DB 2D 28 C2 4D 7A 71 69 D0 74 9A 64 8A 82 A0 EC C8 A4 69 30 67 A0 07 03 05 00 00 00 00 00 A2 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 1C 30 1A A0 03 02 01 00 A1 13 30 11 1B 04 48 54 54 50 1B 09 6C 6F 63 61 6C 68 6F 73 74 A5 11 18 0F 31 39 37 30 30 31 30 31 30 30 30 30 30 30 5A A7 06 02 04 6B 88 55 A5 A8 14 30 12 02 01 12 02 01 11 02 01 10)
            	at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:234)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:48)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:802)
            	at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:120)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:604)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:564)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:553)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:57)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:892)
            	at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            	at java.lang.Thread.run(Thread.java:745)
            Caused by: java.lang.NullPointerException: message
            	at org.apache.mina.filter.codec.AbstractProtocolDecoderOutput.write(AbstractProtocolDecoderOutput.java:44)
            	at org.apache.directory.server.kerberos.protocol.codec.MinaKerberosDecoder.decode(MinaKerberosDecoder.java:65)
            	at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:224)
            	... 15 more
            2016-02-01 06:41:48,138 ERROR protocol.KerberosProtocolHandler (KerberosProtocolHandler.java:exceptionCaught(157)) - /127.0.0.1:46418 EXCEPTION
            org.apache.mina.filter.codec.ProtocolDecoderException: java.lang.NullPointerException: message (Hexdump: 00 00 02 45 6C 82 02 41 30 82 02 3D A1 03 02 01 05 A2 03 02 01 0C A3 82 01 C4 30 82 01 C0 30 82 01 BC A1 03 02 01 01 A2 82 01 B3 04 82 01 AF 6E 82 01 AB 30 82 01 A7 A0 03 02 01 05 A1 03 02 01 0E A2 07 03 05 00 00 00 00 00 A3 81 F6 61 81 F3 30 81 F0 A0 03 02 01 05 A1 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A2 20 30 1E A0 03 02 01 02 A1 17 30 15 1B 06 6B 72 62 74 67 74 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 81 B7 30 81 B4 A0 03 02 01 11 A2 81 AC 04 81 A9 6D 55 CA 9E E9 6A 4D 9F C1 47 9A 2B 9E 75 07 73 F4 48 4A 36 AE BD 28 4D DA 3D 00 89 0D B1 70 C1 67 E7 44 0B C4 3E BF 59 3D 2B F2 EA E7 09 05 44 23 2D B6 D6 76 46 D8 DC 05 32 68 A2 2B D5 58 3D EC 5D DD 4D A1 6D F2 80 95 5F 61 2A 40 C5 D3 F7 BD A3 71 73 F2 81 DD CD B1 B3 1D 8E FA B0 70 9F 88 AD 97 7C 2A 91 DE 4F 69 5A 23 17 AA 21 99 7E 89 36 E6 1A 01 06 D2 DF 6F C3 76 15 47 81 9A 65 66 F7 CC 23 52 C5 DE 77 09 AD 66 38 94 DE 93 DC CA 24 6B C4 2B FA 1A BC FC 07 84 9B CE 0D 15 BA C7 00 8A 5C 0E 61 D8 BE 01 A4 81 98 30 81 95 A0 03 02 01 11 A2 81 8D 04 81 8A 7E 36 73 CB EC 41 15 69 53 02 B8 E4 28 1A 8D 61 56 3E 30 4E 0F C6 28 7C 12 CF EB 25 1E A9 3B 28 99 68 3B CE F3 38 50 B0 DA D9 0F 25 3A 39 B4 12 39 44 A6 64 8D FA 1B 62 22 27 7E 64 D8 C4 FE D3 89 DC 82 E2 C2 21 7E 14 BE DE 89 C6 9D FD C5 3F 25 40 9A F7 2C 26 DD 16 E1 EF C5 66 A5 AD 1A 46 8E 44 9C 86 63 9B 80 BA 81 45 78 97 F4 C1 EF EF 05 68 77 2B 61 B2 DF 30 0B BF E1 94 69 44 2C 97 7D 54 52 F8 9C 88 14 56 A0 B7 A4 69 30 67 A0 07 03 05 00 00 00 00 00 A2 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 1C 30 1A A0 03 02 01 00 A1 13 30 11 1B 04 48 54 54 50 1B 09 6C 6F 63 61 6C 68 6F 73 74 A5 11 18 0F 31 39 37 30 30 31 30 31 30 30 30 30 30 30 5A A7 06 02 04 3F 0B C9 19 A8 14 30 12 02 01 12 02 01 11 02 01 10)
            	at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:234)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:48)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:802)
            	at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:120)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:604)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:564)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:553)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:57)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:892)
            	at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            	at java.lang.Thread.run(Thread.java:745)
            Caused by: java.lang.NullPointerException: message
            	at org.apache.mina.filter.codec.AbstractProtocolDecoderOutput.write(AbstractProtocolDecoderOutput.java:44)
            	at org.apache.directory.server.kerberos.protocol.codec.MinaKerberosDecoder.decode(MinaKerberosDecoder.java:65)
            	at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:224)
            	... 15 more
            2016-02-01 06:41:48,139 ERROR server.KERBEROS_LOG (KerberosProtocolHandler.java:exceptionCaught(158)) - /127.0.0.1:46418 EXCEPTION
            org.apache.mina.filter.codec.ProtocolDecoderException: java.lang.NullPointerException: message (Hexdump: 00 00 02 45 6C 82 02 41 30 82 02 3D A1 03 02 01 05 A2 03 02 01 0C A3 82 01 C4 30 82 01 C0 30 82 01 BC A1 03 02 01 01 A2 82 01 B3 04 82 01 AF 6E 82 01 AB 30 82 01 A7 A0 03 02 01 05 A1 03 02 01 0E A2 07 03 05 00 00 00 00 00 A3 81 F6 61 81 F3 30 81 F0 A0 03 02 01 05 A1 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A2 20 30 1E A0 03 02 01 02 A1 17 30 15 1B 06 6B 72 62 74 67 74 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 81 B7 30 81 B4 A0 03 02 01 11 A2 81 AC 04 81 A9 6D 55 CA 9E E9 6A 4D 9F C1 47 9A 2B 9E 75 07 73 F4 48 4A 36 AE BD 28 4D DA 3D 00 89 0D B1 70 C1 67 E7 44 0B C4 3E BF 59 3D 2B F2 EA E7 09 05 44 23 2D B6 D6 76 46 D8 DC 05 32 68 A2 2B D5 58 3D EC 5D DD 4D A1 6D F2 80 95 5F 61 2A 40 C5 D3 F7 BD A3 71 73 F2 81 DD CD B1 B3 1D 8E FA B0 70 9F 88 AD 97 7C 2A 91 DE 4F 69 5A 23 17 AA 21 99 7E 89 36 E6 1A 01 06 D2 DF 6F C3 76 15 47 81 9A 65 66 F7 CC 23 52 C5 DE 77 09 AD 66 38 94 DE 93 DC CA 24 6B C4 2B FA 1A BC FC 07 84 9B CE 0D 15 BA C7 00 8A 5C 0E 61 D8 BE 01 A4 81 98 30 81 95 A0 03 02 01 11 A2 81 8D 04 81 8A 7E 36 73 CB EC 41 15 69 53 02 B8 E4 28 1A 8D 61 56 3E 30 4E 0F C6 28 7C 12 CF EB 25 1E A9 3B 28 99 68 3B CE F3 38 50 B0 DA D9 0F 25 3A 39 B4 12 39 44 A6 64 8D FA 1B 62 22 27 7E 64 D8 C4 FE D3 89 DC 82 E2 C2 21 7E 14 BE DE 89 C6 9D FD C5 3F 25 40 9A F7 2C 26 DD 16 E1 EF C5 66 A5 AD 1A 46 8E 44 9C 86 63 9B 80 BA 81 45 78 97 F4 C1 EF EF 05 68 77 2B 61 B2 DF 30 0B BF E1 94 69 44 2C 97 7D 54 52 F8 9C 88 14 56 A0 B7 A4 69 30 67 A0 07 03 05 00 00 00 00 00 A2 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 1C 30 1A A0 03 02 01 00 A1 13 30 11 1B 04 48 54 54 50 1B 09 6C 6F 63 61 6C 68 6F 73 74 A5 11 18 0F 31 39 37 30 30 31 30 31 30 30 30 30 30 30 5A A7 06 02 04 3F 0B C9 19 A8 14 30 12 02 01 12 02 01 11 02 01 10)
            	at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:234)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:48)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:802)
            	at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:120)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:604)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:564)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:553)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:57)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:892)
            	at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            	at java.lang.Thread.run(Thread.java:745)
            Caused by: java.lang.NullPointerException: message
            	at org.apache.mina.filter.codec.AbstractProtocolDecoderOutput.write(AbstractProtocolDecoderOutput.java:44)
            	at org.apache.directory.server.kerberos.protocol.codec.MinaKerberosDecoder.decode(MinaKerberosDecoder.java:65)
            	at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:224)
            	... 15 more
            2016-02-01 06:41:48,139 WARN  security.UserGroupInformation (UserGroupInformation.java:doAs(1696)) - PriviledgedActionException as:slave/localhost@EXAMPLE.COM (auth:KERBEROS) cause:java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Cannot get a KDC reply)
            2016-02-01 06:41:48,140 ERROR namenode.EditLogInputStream (EditLogFileInputStream.java:nextOpImpl(194)) - caught exception initializing https://localhost:55326/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID
            java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Cannot get a KDC reply)
            	at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:473)
            	at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:465)
            	at java.security.AccessController.doPrivileged(Native Method)
            	at javax.security.auth.Subject.doAs(Subject.java:415)
            	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
            	at org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:445)
            	at org.apache.hadoop.security.SecurityUtil.doAsCurrentUser(SecurityUtil.java:439)
            	at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog.getInputStream(EditLogFileInputStream.java:464)
            	at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.init(EditLogFileInputStream.java:141)
            	at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOpImpl(EditLogFileInputStream.java:192)
            	at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOp(EditLogFileInputStream.java:250)
            	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
            	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
            	at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
            	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
            	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
            	at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
            	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
            	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:187)
            	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:140)
            	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:835)
            	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:690)
            	at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:281)
            	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1063)
            	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:767)
            	at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:609)
            	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:670)
            	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:838)
            	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:817)
            	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1538)
            	at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1862)
            	at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1827)
            	at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.restartNameNode(TestSecureNNWithQJM.java:197)
            	at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.doNNWithQJMTest(TestSecureNNWithQJM.java:179)
            	at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.testSecureMode(TestSecureNNWithQJM.java:159)
            	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
            	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            	at java.lang.reflect.Method.invoke(Method.java:606)
            	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
            	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
            	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
            	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
            	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
            	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
            	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
            Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Cannot get a KDC reply)
            	at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:306)
            	at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:196)
            	at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:215)
            	at org.apache.hadoop.hdfs.web.URLConnectionFactory.openConnection(URLConnectionFactory.java:161)
            	at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:470)
            	... 45 more
            Caused by: GSSException: No valid credentials provided (Mechanism level: Cannot get a KDC reply)
            	at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:715)
            	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
            	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
            	at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:285)
            	at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:261)
            	at java.security.AccessController.doPrivileged(Native Method)
            	at javax.security.auth.Subject.doAs(Subject.java:415)
            	at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:261)
            	... 49 more
            Caused by: java.io.IOException: Cannot get a KDC reply
            	at sun.security.krb5.KdcComm.send(KdcComm.java:250)
            	at sun.security.krb5.KdcComm.send(KdcComm.java:191)
            	at sun.security.krb5.KrbTgsReq.send(KrbTgsReq.java:186)
            	at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:201)
            	at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:292)
            	at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:101)
            	at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:456)
            	at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:641)
            	... 56 more
            2016-02-01 06:41:48,141 ERROR namenode.EditLogInputStream (RedundantEditLogInputStream.java:nextOp(221)) - Got error reading edit log input stream https://localhost:55326/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID; failing over to edit log https://localhost:48398/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID
            org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream$PrematureEOFException: got premature end-of-file at txid 0; expected file to go up to 5
            	at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:194)
            	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
            	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
            	at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
            	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
            	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:187)
            	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:140)
            	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:835)
            	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:690)
            	at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:281)
            	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1063)
            	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:767)
            	at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:609)
            	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:670)
            	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:838)
            	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:817)
            	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1538)
            	at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1862)
            	at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1827)
            	at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.restartNameNode(TestSecureNNWithQJM.java:197)
            	at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.doNNWithQJMTest(TestSecureNNWithQJM.java:179)
            	at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.testSecureMode(TestSecureNNWithQJM.java:159)
            	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
            	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            	at java.lang.reflect.Method.invoke(Method.java:606)
            	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
            	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
            	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
            	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
            	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
            	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
            	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
            2016-02-01 06:41:48,142 INFO  namenode.EditLogInputStream (RedundantEditLogInputStream.java:nextOp(176)) - Fast-forwarding stream 'https://localhost:48398/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID' to transaction ID 1
            2016-02-01 06:41:48,169 ERROR protocol.KerberosProtocolHandler (KerberosProtocolHandler.java:exceptionCaught(157)) - /127.0.0.1:46420 EXCEPTION
            org.apache.mina.filter.codec.ProtocolDecoderException: java.lang.NullPointerException: message (Hexdump: 00 00 02 45 6C 82 02 41 30 82 02 3D A1 03 02 01 05 A2 03 02 01 0C A3 82 01 C4 30 82 01 C0 30 82 01 BC A1 03 02 01 01 A2 82 01 B3 04 82 01 AF 6E 82 01 AB 30 82 01 A7 A0 03 02 01 05 A1 03 02 01 0E A2 07 03 05 00 00 00 00 00 A3 81 F6 61 81 F3 30 81 F0 A0 03 02 01 05 A1 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A2 20 30 1E A0 03 02 01 02 A1 17 30 15 1B 06 6B 72 62 74 67 74 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 81 B7 30 81 B4 A0 03 02 01 11 A2 81 AC 04 81 A9 6D 55 CA 9E E9 6A 4D 9F C1 47 9A 2B 9E 75 07 73 F4 48 4A 36 AE BD 28 4D DA 3D 00 89 0D B1 70 C1 67 E7 44 0B C4 3E BF 59 3D 2B F2 EA E7 09 05 44 23 2D B6 D6 76 46 D8 DC 05 32 68 A2 2B D5 58 3D EC 5D DD 4D A1 6D F2 80 95 5F 61 2A 40 C5 D3 F7 BD A3 71 73 F2 81 DD CD B1 B3 1D 8E FA B0 70 9F 88 AD 97 7C 2A 91 DE 4F 69 5A 23 17 AA 21 99 7E 89 36 E6 1A 01 06 D2 DF 6F C3 76 15 47 81 9A 65 66 F7 CC 23 52 C5 DE 77 09 AD 66 38 94 DE 93 DC CA 24 6B C4 2B FA 1A BC FC 07 84 9B CE 0D 15 BA C7 00 8A 5C 0E 61 D8 BE 01 A4 81 98 30 81 95 A0 03 02 01 11 A2 81 8D 04 81 8A BA 11 25 9F 01 D1 21 8A 31 30 5B 14 7A 6D A5 65 C1 87 95 F1 86 3A E1 A9 7E 3E 7F 4D FE 8E 8F 96 E4 0A 78 2B 5F 0D 1D 87 88 51 A4 5B 92 94 6A FC 87 97 2A 9C 69 81 B3 5D 5D 67 56 E9 97 97 5D 29 7B B9 B8 9E 99 8F 7A BD 3A E3 4D 6E BA 55 0C ED 3A 67 0F A9 45 B5 93 9B 48 FF FB 45 A7 23 5A B9 52 44 E4 54 D5 08 D5 2F FF 4A C4 52 16 FA 5F 1C 62 41 88 15 C1 B6 63 13 DA 16 33 56 63 37 09 63 2E 1D C8 42 55 FB FA 33 9B CC A4 69 30 67 A0 07 03 05 00 00 00 00 00 A2 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 1C 30 1A A0 03 02 01 00 A1 13 30 11 1B 04 48 54 54 50 1B 09 6C 6F 63 61 6C 68 6F 73 74 A5 11 18 0F 31 39 37 30 30 31 30 31 30 30 30 30 30 30 5A A7 06 02 04 01 AA D4 AF A8 14 30 12 02 01 12 02 01 11 02 01 10)
            	at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:234)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:48)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:802)
            	at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:120)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:604)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:564)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:553)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:57)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:892)
            	at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            	at java.lang.Thread.run(Thread.java:745)
            Caused by: java.lang.NullPointerException: message
            	at org.apache.mina.filter.codec.AbstractProtocolDecoderOutput.write(AbstractProtocolDecoderOutput.java:44)
            	at org.apache.directory.server.kerberos.protocol.codec.MinaKerberosDecoder.decode(MinaKerberosDecoder.java:65)
            	at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:224)
            	... 15 more
            2016-02-01 06:41:48,169 ERROR server.KERBEROS_LOG (KerberosProtocolHandler.java:exceptionCaught(158)) - /127.0.0.1:46420 EXCEPTION
            org.apache.mina.filter.codec.ProtocolDecoderException: java.lang.NullPointerException: message (Hexdump: 00 00 02 45 6C 82 02 41 30 82 02 3D A1 03 02 01 05 A2 03 02 01 0C A3 82 01 C4 30 82 01 C0 30 82 01 BC A1 03 02 01 01 A2 82 01 B3 04 82 01 AF 6E 82 01 AB 30 82 01 A7 A0 03 02 01 05 A1 03 02 01 0E A2 07 03 05 00 00 00 00 00 A3 81 F6 61 81 F3 30 81 F0 A0 03 02 01 05 A1 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A2 20 30 1E A0 03 02 01 02 A1 17 30 15 1B 06 6B 72 62 74 67 74 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 81 B7 30 81 B4 A0 03 02 01 11 A2 81 AC 04 81 A9 6D 55 CA 9E E9 6A 4D 9F C1 47 9A 2B 9E 75 07 73 F4 48 4A 36 AE BD 28 4D DA 3D 00 89 0D B1 70 C1 67 E7 44 0B C4 3E BF 59 3D 2B F2 EA E7 09 05 44 23 2D B6 D6 76 46 D8 DC 05 32 68 A2 2B D5 58 3D EC 5D DD 4D A1 6D F2 80 95 5F 61 2A 40 C5 D3 F7 BD A3 71 73 F2 81 DD CD B1 B3 1D 8E FA B0 70 9F 88 AD 97 7C 2A 91 DE 4F 69 5A 23 17 AA 21 99 7E 89 36 E6 1A 01 06 D2 DF 6F C3 76 15 47 81 9A 65 66 F7 CC 23 52 C5 DE 77 09 AD 66 38 94 DE 93 DC CA 24 6B C4 2B FA 1A BC FC 07 84 9B CE 0D 15 BA C7 00 8A 5C 0E 61 D8 BE 01 A4 81 98 30 81 95 A0 03 02 01 11 A2 81 8D 04 81 8A BA 11 25 9F 01 D1 21 8A 31 30 5B 14 7A 6D A5 65 C1 87 95 F1 86 3A E1 A9 7E 3E 7F 4D FE 8E 8F 96 E4 0A 78 2B 5F 0D 1D 87 88 51 A4 5B 92 94 6A FC 87 97 2A 9C 69 81 B3 5D 5D 67 56 E9 97 97 5D 29 7B B9 B8 9E 99 8F 7A BD 3A E3 4D 6E BA 55 0C ED 3A 67 0F A9 45 B5 93 9B 48 FF FB 45 A7 23 5A B9 52 44 E4 54 D5 08 D5 2F FF 4A C4 52 16 FA 5F 1C 62 41 88 15 C1 B6 63 13 DA 16 33 56 63 37 09 63 2E 1D C8 42 55 FB FA 33 9B CC A4 69 30 67 A0 07 03 05 00 00 00 00 00 A2 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 1C 30 1A A0 03 02 01 00 A1 13 30 11 1B 04 48 54 54 50 1B 09 6C 6F 63 61 6C 68 6F 73 74 A5 11 18 0F 31 39 37 30 30 31 30 31 30 30 30 30 30 30 5A A7 06 02 04 01 AA D4 AF A8 14 30 12 02 01 12 02 01 11 02 01 10)
            	at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:234)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:48)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:802)
            	at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:120)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
            	at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:604)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:564)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:553)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:57)
            	at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:892)
            	at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            	at java.lang.Thread.run(Thread.java:745)
            Caused by: java.lang.NullPointerException: message
            	at org.apache.mina.filter.codec.AbstractProtocolDecoderOutput.write(AbstractProtocolDecoderOutput.java:44)
            	at org.apache.directory.server.kerberos.protocol.codec.MinaKerberosDecoder.decode(MinaKerberosDecoder.java:65)
            	at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:224)
            	... 15 more
            2016-02-01 06:41:48,170 WARN  security.UserGroupInformation (UserGroupInformation.java:doAs(1696)) - PriviledgedActionException as:slave/localhost@EXAMPLE.COM (auth:KERBEROS) cause:java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Cannot get a KDC reply)
            2016-02-01 06:41:48,170 ERROR namenode.EditLogInputStream (EditLogFileInputStream.java:nextOpImpl(194)) - caught exception initializing https://localhost:48398/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID
            java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Cannot get a KDC reply)
            	at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:473)
            	at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:465)
            	at java.security.AccessController.doPrivileged(Native Method)
            	at javax.security.auth.Subject.doAs(Subject.java:415)
            	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
            	at org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:445)
            	at org.apache.hadoop.security.SecurityUtil.doAsCurrentUser(SecurityUtil.java:439)
            	at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog.getInputStream(EditLogFileInputStream.java:464)
            	at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.init(EditLogFileInputStream.java:141)
            	at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOpImpl(EditLogFileInputStream.java:192)
            	at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOp(EditLogFileInputStream.java:250)
            	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
            	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
            	at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
            	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
            	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
            	at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
            	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
            	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:187)
            	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:140)
            	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:835)
            	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:690)
            	at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:281)
            	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1063)
            	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:767)
            	at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:609)
            	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:670)
            	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:838)
            	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:817)
            	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1538)
            	at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1862)
            	at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1827)
            	at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.restartNameNode(TestSecureNNWithQJM.java:197)
            	at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.doNNWithQJMTest(TestSecureNNWithQJM.java:179)
            	at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.testSecureMode(TestSecureNNWithQJM.java:159)
            	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
            	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            	at java.lang.reflect.Method.invoke(Method.java:606)
            	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
            	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
            	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
            	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
            	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
            	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
            	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
            Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Cannot get a KDC reply)
            	at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:306)
            	at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:196)
            	at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:215)
            	at org.apache.hadoop.hdfs.web.URLConnectionFactory.openConnection(URLConnectionFactory.java:161)
            	at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:470)
            	... 45 more
            Caused by: GSSException: No valid credentials provided (Mechanism level: Cannot get a KDC reply)
            	at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:715)
            	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
            	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
            	at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:285)
            	at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:261)
            	at java.security.AccessController.doPrivileged(Native Method)
            	at javax.security.auth.Subject.doAs(Subject.java:415)
            	at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:261)
            	... 49 more
            Caused by: java.io.IOException: Cannot get a KDC reply
            	at sun.security.krb5.KdcComm.send(KdcComm.java:250)
            	at sun.security.krb5.KdcComm.send(KdcComm.java:191)
            	at sun.security.krb5.KrbTgsReq.send(KrbTgsReq.java:186)
            	at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:201)
            	at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:292)
            	at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:101)
            	at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:456)
            	at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:641)
            	... 56 more
            2016-02-01 06:41:48,171 ERROR namenode.FSImage (FSEditLogLoader.java:loadEditRecords(196)) - Error replaying edit log at offset 0.  Expected transaction ID was 1
            org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream$PrematureEOFException: got premature end-of-file at txid 0; expected file to go up to 5
            	at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:194)
            	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
            	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
            	at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
            	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
            	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:187)
            	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:140)
            	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:835)
            	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:690)
            	at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:281)
            	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1063)
            	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:767)
            	at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:609)
            	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:670)
            	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:838)
            	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:817)
            	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1538)
            	at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1862)
            	at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1827)
            	at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.restartNameNode(TestSecureNNWithQJM.java:197)
            	at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.doNNWithQJMTest(TestSecureNNWithQJM.java:179)
            	at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.testSecureMode(TestSecureNNWithQJM.java:159)
            	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
            	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            	at java.lang.reflect.Method.invoke(Method.java:606)
            	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
            	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
            	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
            	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
            	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
            	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
            	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
            2016-02-01 06:41:48,173 WARN  namenode.FSNamesystem (FSNamesystem.java:loadFromDisk(769)) - Encountered exception loading fsimage
            org.apache.hadoop.hdfs.server.namenode.EditLogInputException: Error replaying edit log at offset 0.  Expected transaction ID was 1
            	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:200)
            	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:140)
            	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:835)
            	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:690)
            	at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:281)
            	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1063)
            	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:767)
            	at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:609)
            	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:670)
            	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:838)
            	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:817)
            	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1538)
            	at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1862)
            	at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1827)
            	at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.restartNameNode(TestSecureNNWithQJM.java:197)
            	at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.doNNWithQJMTest(TestSecureNNWithQJM.java:179)
            	at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.testSecureMode(TestSecureNNWithQJM.java:159)
            	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
            	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            	at java.lang.reflect.Method.invoke(Method.java:606)
            	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
            	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
            	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
            	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
            	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
            	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
            	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
            Caused by: org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream$PrematureEOFException: got premature end-of-file at txid 0; expected file to go up to 5
            	at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:194)
            	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
            	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
            	at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
            	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
            	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:187)
            	... 27 more
            2016-02-01 06:41:48,187 INFO  mortbay.log (Slf4jLog.java:info(67)) - Stopped SslSocketConnectorSecure@localhost:41570
            2016-02-01 06:41:48,287 INFO  hdfs.MiniDFSCluster (MiniDFSCluster.java:shutdown(1744)) - Shutting down the Mini HDFS Cluster
            
            xiaochen Xiao Chen added a comment - Log excerpt of the failure I saw in TestSecureNNWithQJM#testSecureMode : Error Message org.apache.hadoop.hdfs.server.namenode.EditLogInputException: Error replaying edit log at offset 0. Expected transaction ID was 1 at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:194) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151) at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:187) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:140) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:835) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:690) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:281) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1063) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:767) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:609) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:670) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:838) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:817) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1538) at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1862) at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1827) at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.restartNameNode(TestSecureNNWithQJM.java:197) at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.doNNWithQJMTest(TestSecureNNWithQJM.java:179) at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.testSecureMode(TestSecureNNWithQJM.java:159) stdout 2016-02-01 06:41:47,944 INFO namenode.FSImage (FSImage.java:loadEdits(832)) - Reading org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream@12e8c783 expecting start txid #1 2016-02-01 06:41:47,944 INFO namenode.FSImage (FSEditLogLoader.java:loadFSEdits(139)) - Start loading edits file https://localhost:55326/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID, https://localhost:48398/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID 2016-02-01 06:41:47,945 INFO namenode.EditLogInputStream (RedundantEditLogInputStream.java:nextOp(176)) - Fast-forwarding stream 'https://localhost:55326/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID, https://localhost:48398/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID' to transaction ID 1 2016-02-01 06:41:47,945 INFO namenode.EditLogInputStream (RedundantEditLogInputStream.java:nextOp(176)) - Fast-forwarding stream 'https://localhost:55326/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID' to transaction ID 1 2016-02-01 06:41:48,131 ERROR protocol.KerberosProtocolHandler (KerberosProtocolHandler.java:exceptionCaught(157)) - /127.0.0.1:46417 EXCEPTION org.apache.mina.filter.codec.ProtocolDecoderException: java.lang.NullPointerException: message (Hexdump: 00 00 02 45 6C 82 02 41 30 82 02 3D A1 03 02 01 05 A2 03 02 01 0C A3 82 01 C4 30 82 01 C0 30 82 01 BC A1 03 02 01 01 A2 82 01 B3 04 82 01 AF 6E 82 01 AB 30 82 01 A7 A0 03 02 01 05 A1 03 02 01 0E A2 07 03 05 00 00 00 00 00 A3 81 F6 61 81 F3 30 81 F0 A0 03 02 01 05 A1 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A2 20 30 1E A0 03 02 01 02 A1 17 30 15 1B 06 6B 72 62 74 67 74 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 81 B7 30 81 B4 A0 03 02 01 11 A2 81 AC 04 81 A9 6D 55 CA 9E E9 6A 4D 9F C1 47 9A 2B 9E 75 07 73 F4 48 4A 36 AE BD 28 4D DA 3D 00 89 0D B1 70 C1 67 E7 44 0B C4 3E BF 59 3D 2B F2 EA E7 09 05 44 23 2D B6 D6 76 46 D8 DC 05 32 68 A2 2B D5 58 3D EC 5D DD 4D A1 6D F2 80 95 5F 61 2A 40 C5 D3 F7 BD A3 71 73 F2 81 DD CD B1 B3 1D 8E FA B0 70 9F 88 AD 97 7C 2A 91 DE 4F 69 5A 23 17 AA 21 99 7E 89 36 E6 1A 01 06 D2 DF 6F C3 76 15 47 81 9A 65 66 F7 CC 23 52 C5 DE 77 09 AD 66 38 94 DE 93 DC CA 24 6B C4 2B FA 1A BC FC 07 84 9B CE 0D 15 BA C7 00 8A 5C 0E 61 D8 BE 01 A4 81 98 30 81 95 A0 03 02 01 11 A2 81 8D 04 81 8A 60 DF 82 D5 14 DB 78 8D A8 E4 6D F8 FE 3A F4 AB 98 25 9D DB 51 ED 3B CE 53 C8 DC 48 1C CB EB B5 1B 5A 45 BA CD 68 0A 26 2F 8D 3A FE 75 AE 36 4B 25 B5 B8 5A C1 27 71 E3 B6 03 7D D6 2D 14 58 CD 6D 19 F0 25 D0 5A 9B 35 A6 7E 36 62 DA 28 56 0B E9 53 03 43 7B 71 D5 ED 8F 52 CE 6E 8A 23 0C 52 53 EB 42 0B 7A 6B 8C 54 EB 1C 70 FB 21 DD DF 23 B4 5E AD 42 67 65 42 61 FD DB 2D 28 C2 4D 7A 71 69 D0 74 9A 64 8A 82 A0 EC C8 A4 69 30 67 A0 07 03 05 00 00 00 00 00 A2 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 1C 30 1A A0 03 02 01 00 A1 13 30 11 1B 04 48 54 54 50 1B 09 6C 6F 63 61 6C 68 6F 73 74 A5 11 18 0F 31 39 37 30 30 31 30 31 30 30 30 30 30 30 5A A7 06 02 04 6B 88 55 A5 A8 14 30 12 02 01 12 02 01 11 02 01 10) at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:234) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:48) at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:802) at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:120) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426) at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:604) at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:564) at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:553) at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:57) at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:892) at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NullPointerException: message at org.apache.mina.filter.codec.AbstractProtocolDecoderOutput.write(AbstractProtocolDecoderOutput.java:44) at org.apache.directory.server.kerberos.protocol.codec.MinaKerberosDecoder.decode(MinaKerberosDecoder.java:65) at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:224) ... 15 more 2016-02-01 06:41:48,133 ERROR server.KERBEROS_LOG (KerberosProtocolHandler.java:exceptionCaught(158)) - /127.0.0.1:46417 EXCEPTION org.apache.mina.filter.codec.ProtocolDecoderException: java.lang.NullPointerException: message (Hexdump: 00 00 02 45 6C 82 02 41 30 82 02 3D A1 03 02 01 05 A2 03 02 01 0C A3 82 01 C4 30 82 01 C0 30 82 01 BC A1 03 02 01 01 A2 82 01 B3 04 82 01 AF 6E 82 01 AB 30 82 01 A7 A0 03 02 01 05 A1 03 02 01 0E A2 07 03 05 00 00 00 00 00 A3 81 F6 61 81 F3 30 81 F0 A0 03 02 01 05 A1 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A2 20 30 1E A0 03 02 01 02 A1 17 30 15 1B 06 6B 72 62 74 67 74 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 81 B7 30 81 B4 A0 03 02 01 11 A2 81 AC 04 81 A9 6D 55 CA 9E E9 6A 4D 9F C1 47 9A 2B 9E 75 07 73 F4 48 4A 36 AE BD 28 4D DA 3D 00 89 0D B1 70 C1 67 E7 44 0B C4 3E BF 59 3D 2B F2 EA E7 09 05 44 23 2D B6 D6 76 46 D8 DC 05 32 68 A2 2B D5 58 3D EC 5D DD 4D A1 6D F2 80 95 5F 61 2A 40 C5 D3 F7 BD A3 71 73 F2 81 DD CD B1 B3 1D 8E FA B0 70 9F 88 AD 97 7C 2A 91 DE 4F 69 5A 23 17 AA 21 99 7E 89 36 E6 1A 01 06 D2 DF 6F C3 76 15 47 81 9A 65 66 F7 CC 23 52 C5 DE 77 09 AD 66 38 94 DE 93 DC CA 24 6B C4 2B FA 1A BC FC 07 84 9B CE 0D 15 BA C7 00 8A 5C 0E 61 D8 BE 01 A4 81 98 30 81 95 A0 03 02 01 11 A2 81 8D 04 81 8A 60 DF 82 D5 14 DB 78 8D A8 E4 6D F8 FE 3A F4 AB 98 25 9D DB 51 ED 3B CE 53 C8 DC 48 1C CB EB B5 1B 5A 45 BA CD 68 0A 26 2F 8D 3A FE 75 AE 36 4B 25 B5 B8 5A C1 27 71 E3 B6 03 7D D6 2D 14 58 CD 6D 19 F0 25 D0 5A 9B 35 A6 7E 36 62 DA 28 56 0B E9 53 03 43 7B 71 D5 ED 8F 52 CE 6E 8A 23 0C 52 53 EB 42 0B 7A 6B 8C 54 EB 1C 70 FB 21 DD DF 23 B4 5E AD 42 67 65 42 61 FD DB 2D 28 C2 4D 7A 71 69 D0 74 9A 64 8A 82 A0 EC C8 A4 69 30 67 A0 07 03 05 00 00 00 00 00 A2 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 1C 30 1A A0 03 02 01 00 A1 13 30 11 1B 04 48 54 54 50 1B 09 6C 6F 63 61 6C 68 6F 73 74 A5 11 18 0F 31 39 37 30 30 31 30 31 30 30 30 30 30 30 5A A7 06 02 04 6B 88 55 A5 A8 14 30 12 02 01 12 02 01 11 02 01 10) at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:234) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:48) at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:802) at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:120) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426) at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:604) at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:564) at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:553) at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:57) at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:892) at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NullPointerException: message at org.apache.mina.filter.codec.AbstractProtocolDecoderOutput.write(AbstractProtocolDecoderOutput.java:44) at org.apache.directory.server.kerberos.protocol.codec.MinaKerberosDecoder.decode(MinaKerberosDecoder.java:65) at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:224) ... 15 more 2016-02-01 06:41:48,138 ERROR protocol.KerberosProtocolHandler (KerberosProtocolHandler.java:exceptionCaught(157)) - /127.0.0.1:46418 EXCEPTION org.apache.mina.filter.codec.ProtocolDecoderException: java.lang.NullPointerException: message (Hexdump: 00 00 02 45 6C 82 02 41 30 82 02 3D A1 03 02 01 05 A2 03 02 01 0C A3 82 01 C4 30 82 01 C0 30 82 01 BC A1 03 02 01 01 A2 82 01 B3 04 82 01 AF 6E 82 01 AB 30 82 01 A7 A0 03 02 01 05 A1 03 02 01 0E A2 07 03 05 00 00 00 00 00 A3 81 F6 61 81 F3 30 81 F0 A0 03 02 01 05 A1 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A2 20 30 1E A0 03 02 01 02 A1 17 30 15 1B 06 6B 72 62 74 67 74 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 81 B7 30 81 B4 A0 03 02 01 11 A2 81 AC 04 81 A9 6D 55 CA 9E E9 6A 4D 9F C1 47 9A 2B 9E 75 07 73 F4 48 4A 36 AE BD 28 4D DA 3D 00 89 0D B1 70 C1 67 E7 44 0B C4 3E BF 59 3D 2B F2 EA E7 09 05 44 23 2D B6 D6 76 46 D8 DC 05 32 68 A2 2B D5 58 3D EC 5D DD 4D A1 6D F2 80 95 5F 61 2A 40 C5 D3 F7 BD A3 71 73 F2 81 DD CD B1 B3 1D 8E FA B0 70 9F 88 AD 97 7C 2A 91 DE 4F 69 5A 23 17 AA 21 99 7E 89 36 E6 1A 01 06 D2 DF 6F C3 76 15 47 81 9A 65 66 F7 CC 23 52 C5 DE 77 09 AD 66 38 94 DE 93 DC CA 24 6B C4 2B FA 1A BC FC 07 84 9B CE 0D 15 BA C7 00 8A 5C 0E 61 D8 BE 01 A4 81 98 30 81 95 A0 03 02 01 11 A2 81 8D 04 81 8A 7E 36 73 CB EC 41 15 69 53 02 B8 E4 28 1A 8D 61 56 3E 30 4E 0F C6 28 7C 12 CF EB 25 1E A9 3B 28 99 68 3B CE F3 38 50 B0 DA D9 0F 25 3A 39 B4 12 39 44 A6 64 8D FA 1B 62 22 27 7E 64 D8 C4 FE D3 89 DC 82 E2 C2 21 7E 14 BE DE 89 C6 9D FD C5 3F 25 40 9A F7 2C 26 DD 16 E1 EF C5 66 A5 AD 1A 46 8E 44 9C 86 63 9B 80 BA 81 45 78 97 F4 C1 EF EF 05 68 77 2B 61 B2 DF 30 0B BF E1 94 69 44 2C 97 7D 54 52 F8 9C 88 14 56 A0 B7 A4 69 30 67 A0 07 03 05 00 00 00 00 00 A2 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 1C 30 1A A0 03 02 01 00 A1 13 30 11 1B 04 48 54 54 50 1B 09 6C 6F 63 61 6C 68 6F 73 74 A5 11 18 0F 31 39 37 30 30 31 30 31 30 30 30 30 30 30 5A A7 06 02 04 3F 0B C9 19 A8 14 30 12 02 01 12 02 01 11 02 01 10) at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:234) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:48) at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:802) at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:120) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426) at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:604) at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:564) at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:553) at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:57) at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:892) at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NullPointerException: message at org.apache.mina.filter.codec.AbstractProtocolDecoderOutput.write(AbstractProtocolDecoderOutput.java:44) at org.apache.directory.server.kerberos.protocol.codec.MinaKerberosDecoder.decode(MinaKerberosDecoder.java:65) at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:224) ... 15 more 2016-02-01 06:41:48,139 ERROR server.KERBEROS_LOG (KerberosProtocolHandler.java:exceptionCaught(158)) - /127.0.0.1:46418 EXCEPTION org.apache.mina.filter.codec.ProtocolDecoderException: java.lang.NullPointerException: message (Hexdump: 00 00 02 45 6C 82 02 41 30 82 02 3D A1 03 02 01 05 A2 03 02 01 0C A3 82 01 C4 30 82 01 C0 30 82 01 BC A1 03 02 01 01 A2 82 01 B3 04 82 01 AF 6E 82 01 AB 30 82 01 A7 A0 03 02 01 05 A1 03 02 01 0E A2 07 03 05 00 00 00 00 00 A3 81 F6 61 81 F3 30 81 F0 A0 03 02 01 05 A1 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A2 20 30 1E A0 03 02 01 02 A1 17 30 15 1B 06 6B 72 62 74 67 74 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 81 B7 30 81 B4 A0 03 02 01 11 A2 81 AC 04 81 A9 6D 55 CA 9E E9 6A 4D 9F C1 47 9A 2B 9E 75 07 73 F4 48 4A 36 AE BD 28 4D DA 3D 00 89 0D B1 70 C1 67 E7 44 0B C4 3E BF 59 3D 2B F2 EA E7 09 05 44 23 2D B6 D6 76 46 D8 DC 05 32 68 A2 2B D5 58 3D EC 5D DD 4D A1 6D F2 80 95 5F 61 2A 40 C5 D3 F7 BD A3 71 73 F2 81 DD CD B1 B3 1D 8E FA B0 70 9F 88 AD 97 7C 2A 91 DE 4F 69 5A 23 17 AA 21 99 7E 89 36 E6 1A 01 06 D2 DF 6F C3 76 15 47 81 9A 65 66 F7 CC 23 52 C5 DE 77 09 AD 66 38 94 DE 93 DC CA 24 6B C4 2B FA 1A BC FC 07 84 9B CE 0D 15 BA C7 00 8A 5C 0E 61 D8 BE 01 A4 81 98 30 81 95 A0 03 02 01 11 A2 81 8D 04 81 8A 7E 36 73 CB EC 41 15 69 53 02 B8 E4 28 1A 8D 61 56 3E 30 4E 0F C6 28 7C 12 CF EB 25 1E A9 3B 28 99 68 3B CE F3 38 50 B0 DA D9 0F 25 3A 39 B4 12 39 44 A6 64 8D FA 1B 62 22 27 7E 64 D8 C4 FE D3 89 DC 82 E2 C2 21 7E 14 BE DE 89 C6 9D FD C5 3F 25 40 9A F7 2C 26 DD 16 E1 EF C5 66 A5 AD 1A 46 8E 44 9C 86 63 9B 80 BA 81 45 78 97 F4 C1 EF EF 05 68 77 2B 61 B2 DF 30 0B BF E1 94 69 44 2C 97 7D 54 52 F8 9C 88 14 56 A0 B7 A4 69 30 67 A0 07 03 05 00 00 00 00 00 A2 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 1C 30 1A A0 03 02 01 00 A1 13 30 11 1B 04 48 54 54 50 1B 09 6C 6F 63 61 6C 68 6F 73 74 A5 11 18 0F 31 39 37 30 30 31 30 31 30 30 30 30 30 30 5A A7 06 02 04 3F 0B C9 19 A8 14 30 12 02 01 12 02 01 11 02 01 10) at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:234) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:48) at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:802) at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:120) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426) at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:604) at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:564) at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:553) at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:57) at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:892) at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NullPointerException: message at org.apache.mina.filter.codec.AbstractProtocolDecoderOutput.write(AbstractProtocolDecoderOutput.java:44) at org.apache.directory.server.kerberos.protocol.codec.MinaKerberosDecoder.decode(MinaKerberosDecoder.java:65) at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:224) ... 15 more 2016-02-01 06:41:48,139 WARN security.UserGroupInformation (UserGroupInformation.java:doAs(1696)) - PriviledgedActionException as:slave/localhost@EXAMPLE.COM (auth:KERBEROS) cause:java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Cannot get a KDC reply) 2016-02-01 06:41:48,140 ERROR namenode.EditLogInputStream (EditLogFileInputStream.java:nextOpImpl(194)) - caught exception initializing https://localhost:55326/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Cannot get a KDC reply) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:473) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:465) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693) at org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:445) at org.apache.hadoop.security.SecurityUtil.doAsCurrentUser(SecurityUtil.java:439) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog.getInputStream(EditLogFileInputStream.java:464) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.init(EditLogFileInputStream.java:141) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOpImpl(EditLogFileInputStream.java:192) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOp(EditLogFileInputStream.java:250) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151) at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151) at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:187) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:140) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:835) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:690) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:281) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1063) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:767) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:609) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:670) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:838) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:817) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1538) at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1862) at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1827) at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.restartNameNode(TestSecureNNWithQJM.java:197) at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.doNNWithQJMTest(TestSecureNNWithQJM.java:179) at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.testSecureMode(TestSecureNNWithQJM.java:159) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Cannot get a KDC reply) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:306) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:196) at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:215) at org.apache.hadoop.hdfs.web.URLConnectionFactory.openConnection(URLConnectionFactory.java:161) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:470) ... 45 more Caused by: GSSException: No valid credentials provided (Mechanism level: Cannot get a KDC reply) at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:715) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:285) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:261) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:261) ... 49 more Caused by: java.io.IOException: Cannot get a KDC reply at sun.security.krb5.KdcComm.send(KdcComm.java:250) at sun.security.krb5.KdcComm.send(KdcComm.java:191) at sun.security.krb5.KrbTgsReq.send(KrbTgsReq.java:186) at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:201) at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:292) at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:101) at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:456) at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:641) ... 56 more 2016-02-01 06:41:48,141 ERROR namenode.EditLogInputStream (RedundantEditLogInputStream.java:nextOp(221)) - Got error reading edit log input stream https://localhost:55326/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID; failing over to edit log https://localhost:48398/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream$PrematureEOFException: got premature end-of-file at txid 0; expected file to go up to 5 at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:194) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151) at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:187) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:140) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:835) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:690) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:281) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1063) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:767) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:609) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:670) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:838) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:817) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1538) at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1862) at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1827) at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.restartNameNode(TestSecureNNWithQJM.java:197) at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.doNNWithQJMTest(TestSecureNNWithQJM.java:179) at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.testSecureMode(TestSecureNNWithQJM.java:159) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2016-02-01 06:41:48,142 INFO namenode.EditLogInputStream (RedundantEditLogInputStream.java:nextOp(176)) - Fast-forwarding stream 'https://localhost:48398/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID' to transaction ID 1 2016-02-01 06:41:48,169 ERROR protocol.KerberosProtocolHandler (KerberosProtocolHandler.java:exceptionCaught(157)) - /127.0.0.1:46420 EXCEPTION org.apache.mina.filter.codec.ProtocolDecoderException: java.lang.NullPointerException: message (Hexdump: 00 00 02 45 6C 82 02 41 30 82 02 3D A1 03 02 01 05 A2 03 02 01 0C A3 82 01 C4 30 82 01 C0 30 82 01 BC A1 03 02 01 01 A2 82 01 B3 04 82 01 AF 6E 82 01 AB 30 82 01 A7 A0 03 02 01 05 A1 03 02 01 0E A2 07 03 05 00 00 00 00 00 A3 81 F6 61 81 F3 30 81 F0 A0 03 02 01 05 A1 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A2 20 30 1E A0 03 02 01 02 A1 17 30 15 1B 06 6B 72 62 74 67 74 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 81 B7 30 81 B4 A0 03 02 01 11 A2 81 AC 04 81 A9 6D 55 CA 9E E9 6A 4D 9F C1 47 9A 2B 9E 75 07 73 F4 48 4A 36 AE BD 28 4D DA 3D 00 89 0D B1 70 C1 67 E7 44 0B C4 3E BF 59 3D 2B F2 EA E7 09 05 44 23 2D B6 D6 76 46 D8 DC 05 32 68 A2 2B D5 58 3D EC 5D DD 4D A1 6D F2 80 95 5F 61 2A 40 C5 D3 F7 BD A3 71 73 F2 81 DD CD B1 B3 1D 8E FA B0 70 9F 88 AD 97 7C 2A 91 DE 4F 69 5A 23 17 AA 21 99 7E 89 36 E6 1A 01 06 D2 DF 6F C3 76 15 47 81 9A 65 66 F7 CC 23 52 C5 DE 77 09 AD 66 38 94 DE 93 DC CA 24 6B C4 2B FA 1A BC FC 07 84 9B CE 0D 15 BA C7 00 8A 5C 0E 61 D8 BE 01 A4 81 98 30 81 95 A0 03 02 01 11 A2 81 8D 04 81 8A BA 11 25 9F 01 D1 21 8A 31 30 5B 14 7A 6D A5 65 C1 87 95 F1 86 3A E1 A9 7E 3E 7F 4D FE 8E 8F 96 E4 0A 78 2B 5F 0D 1D 87 88 51 A4 5B 92 94 6A FC 87 97 2A 9C 69 81 B3 5D 5D 67 56 E9 97 97 5D 29 7B B9 B8 9E 99 8F 7A BD 3A E3 4D 6E BA 55 0C ED 3A 67 0F A9 45 B5 93 9B 48 FF FB 45 A7 23 5A B9 52 44 E4 54 D5 08 D5 2F FF 4A C4 52 16 FA 5F 1C 62 41 88 15 C1 B6 63 13 DA 16 33 56 63 37 09 63 2E 1D C8 42 55 FB FA 33 9B CC A4 69 30 67 A0 07 03 05 00 00 00 00 00 A2 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 1C 30 1A A0 03 02 01 00 A1 13 30 11 1B 04 48 54 54 50 1B 09 6C 6F 63 61 6C 68 6F 73 74 A5 11 18 0F 31 39 37 30 30 31 30 31 30 30 30 30 30 30 5A A7 06 02 04 01 AA D4 AF A8 14 30 12 02 01 12 02 01 11 02 01 10) at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:234) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:48) at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:802) at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:120) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426) at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:604) at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:564) at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:553) at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:57) at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:892) at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NullPointerException: message at org.apache.mina.filter.codec.AbstractProtocolDecoderOutput.write(AbstractProtocolDecoderOutput.java:44) at org.apache.directory.server.kerberos.protocol.codec.MinaKerberosDecoder.decode(MinaKerberosDecoder.java:65) at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:224) ... 15 more 2016-02-01 06:41:48,169 ERROR server.KERBEROS_LOG (KerberosProtocolHandler.java:exceptionCaught(158)) - /127.0.0.1:46420 EXCEPTION org.apache.mina.filter.codec.ProtocolDecoderException: java.lang.NullPointerException: message (Hexdump: 00 00 02 45 6C 82 02 41 30 82 02 3D A1 03 02 01 05 A2 03 02 01 0C A3 82 01 C4 30 82 01 C0 30 82 01 BC A1 03 02 01 01 A2 82 01 B3 04 82 01 AF 6E 82 01 AB 30 82 01 A7 A0 03 02 01 05 A1 03 02 01 0E A2 07 03 05 00 00 00 00 00 A3 81 F6 61 81 F3 30 81 F0 A0 03 02 01 05 A1 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A2 20 30 1E A0 03 02 01 02 A1 17 30 15 1B 06 6B 72 62 74 67 74 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 81 B7 30 81 B4 A0 03 02 01 11 A2 81 AC 04 81 A9 6D 55 CA 9E E9 6A 4D 9F C1 47 9A 2B 9E 75 07 73 F4 48 4A 36 AE BD 28 4D DA 3D 00 89 0D B1 70 C1 67 E7 44 0B C4 3E BF 59 3D 2B F2 EA E7 09 05 44 23 2D B6 D6 76 46 D8 DC 05 32 68 A2 2B D5 58 3D EC 5D DD 4D A1 6D F2 80 95 5F 61 2A 40 C5 D3 F7 BD A3 71 73 F2 81 DD CD B1 B3 1D 8E FA B0 70 9F 88 AD 97 7C 2A 91 DE 4F 69 5A 23 17 AA 21 99 7E 89 36 E6 1A 01 06 D2 DF 6F C3 76 15 47 81 9A 65 66 F7 CC 23 52 C5 DE 77 09 AD 66 38 94 DE 93 DC CA 24 6B C4 2B FA 1A BC FC 07 84 9B CE 0D 15 BA C7 00 8A 5C 0E 61 D8 BE 01 A4 81 98 30 81 95 A0 03 02 01 11 A2 81 8D 04 81 8A BA 11 25 9F 01 D1 21 8A 31 30 5B 14 7A 6D A5 65 C1 87 95 F1 86 3A E1 A9 7E 3E 7F 4D FE 8E 8F 96 E4 0A 78 2B 5F 0D 1D 87 88 51 A4 5B 92 94 6A FC 87 97 2A 9C 69 81 B3 5D 5D 67 56 E9 97 97 5D 29 7B B9 B8 9E 99 8F 7A BD 3A E3 4D 6E BA 55 0C ED 3A 67 0F A9 45 B5 93 9B 48 FF FB 45 A7 23 5A B9 52 44 E4 54 D5 08 D5 2F FF 4A C4 52 16 FA 5F 1C 62 41 88 15 C1 B6 63 13 DA 16 33 56 63 37 09 63 2E 1D C8 42 55 FB FA 33 9B CC A4 69 30 67 A0 07 03 05 00 00 00 00 00 A2 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 1C 30 1A A0 03 02 01 00 A1 13 30 11 1B 04 48 54 54 50 1B 09 6C 6F 63 61 6C 68 6F 73 74 A5 11 18 0F 31 39 37 30 30 31 30 31 30 30 30 30 30 30 5A A7 06 02 04 01 AA D4 AF A8 14 30 12 02 01 12 02 01 11 02 01 10) at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:234) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:48) at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:802) at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:120) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426) at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:604) at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:564) at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:553) at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:57) at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:892) at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NullPointerException: message at org.apache.mina.filter.codec.AbstractProtocolDecoderOutput.write(AbstractProtocolDecoderOutput.java:44) at org.apache.directory.server.kerberos.protocol.codec.MinaKerberosDecoder.decode(MinaKerberosDecoder.java:65) at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:224) ... 15 more 2016-02-01 06:41:48,170 WARN security.UserGroupInformation (UserGroupInformation.java:doAs(1696)) - PriviledgedActionException as:slave/localhost@EXAMPLE.COM (auth:KERBEROS) cause:java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Cannot get a KDC reply) 2016-02-01 06:41:48,170 ERROR namenode.EditLogInputStream (EditLogFileInputStream.java:nextOpImpl(194)) - caught exception initializing https://localhost:48398/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Cannot get a KDC reply) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:473) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:465) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693) at org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:445) at org.apache.hadoop.security.SecurityUtil.doAsCurrentUser(SecurityUtil.java:439) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog.getInputStream(EditLogFileInputStream.java:464) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.init(EditLogFileInputStream.java:141) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOpImpl(EditLogFileInputStream.java:192) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOp(EditLogFileInputStream.java:250) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151) at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151) at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:187) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:140) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:835) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:690) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:281) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1063) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:767) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:609) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:670) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:838) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:817) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1538) at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1862) at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1827) at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.restartNameNode(TestSecureNNWithQJM.java:197) at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.doNNWithQJMTest(TestSecureNNWithQJM.java:179) at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.testSecureMode(TestSecureNNWithQJM.java:159) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Cannot get a KDC reply) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:306) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:196) at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:215) at org.apache.hadoop.hdfs.web.URLConnectionFactory.openConnection(URLConnectionFactory.java:161) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:470) ... 45 more Caused by: GSSException: No valid credentials provided (Mechanism level: Cannot get a KDC reply) at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:715) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:285) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:261) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:261) ... 49 more Caused by: java.io.IOException: Cannot get a KDC reply at sun.security.krb5.KdcComm.send(KdcComm.java:250) at sun.security.krb5.KdcComm.send(KdcComm.java:191) at sun.security.krb5.KrbTgsReq.send(KrbTgsReq.java:186) at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:201) at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:292) at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:101) at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:456) at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:641) ... 56 more 2016-02-01 06:41:48,171 ERROR namenode.FSImage (FSEditLogLoader.java:loadEditRecords(196)) - Error replaying edit log at offset 0. Expected transaction ID was 1 org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream$PrematureEOFException: got premature end-of-file at txid 0; expected file to go up to 5 at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:194) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151) at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:187) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:140) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:835) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:690) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:281) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1063) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:767) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:609) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:670) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:838) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:817) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1538) at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1862) at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1827) at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.restartNameNode(TestSecureNNWithQJM.java:197) at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.doNNWithQJMTest(TestSecureNNWithQJM.java:179) at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.testSecureMode(TestSecureNNWithQJM.java:159) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2016-02-01 06:41:48,173 WARN namenode.FSNamesystem (FSNamesystem.java:loadFromDisk(769)) - Encountered exception loading fsimage org.apache.hadoop.hdfs.server.namenode.EditLogInputException: Error replaying edit log at offset 0. Expected transaction ID was 1 at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:200) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:140) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:835) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:690) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:281) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1063) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:767) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:609) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:670) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:838) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:817) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1538) at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1862) at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1827) at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.restartNameNode(TestSecureNNWithQJM.java:197) at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.doNNWithQJMTest(TestSecureNNWithQJM.java:179) at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.testSecureMode(TestSecureNNWithQJM.java:159) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream$PrematureEOFException: got premature end-of-file at txid 0; expected file to go up to 5 at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:194) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151) at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:187) ... 27 more 2016-02-01 06:41:48,187 INFO mortbay.log (Slf4jLog.java:info(67)) - Stopped SslSocketConnectorSecure@localhost:41570 2016-02-01 06:41:48,287 INFO hdfs.MiniDFSCluster (MiniDFSCluster.java:shutdown(1744)) - Shutting down the Mini HDFS Cluster

            Looks like it might be HADOOP-12090.

            templedf Daniel Templeton added a comment - Looks like it might be HADOOP-12090 .
            xiaochen Xiao Chen added a comment -

            Thanks Daniel for the pointer, HADOOP-12090 looks to be the same issue. Resolving this as dup.

            P.S. looks the ds jira mentioned in HADOOP-12090 is fixed in M22 which is unreleased, so I guess we'll just wait for that and upgrade.

            xiaochen Xiao Chen added a comment - Thanks Daniel for the pointer, HADOOP-12090 looks to be the same issue. Resolving this as dup. P.S. looks the ds jira mentioned in HADOOP-12090 is fixed in M22 which is unreleased, so I guess we'll just wait for that and upgrade.

            People

              xiaochen Xiao Chen
              xiaochen Xiao Chen
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: