Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.0.0
    • None
    • None

    Description

      Gianmarco De Francisci Morales asked on the user list:

      I am getting the exception in the subject when running my giraph program

      on a cluster with Kerberos authentication.

      This leads to the idea of having Kerberos authentication supported within GIRAPH. Hopefully it would use our fast GIRAPH-37 IPC, but could also interoperate with Hadoop security.

      Attachments

        1. GIRAPH-211.2.patch
          104 kB
          Avery Ching
        2. GIRAPH-211.3.patch
          104 kB
          Avery Ching
        3. GIRAPH-211.4.patch
          91 kB
          Avery Ching
        4. GIRAPH-211.patch
          91 kB
          Eugene Joseph Koontz
        5. GIRAPH-211.patch
          84 kB
          Eugene Joseph Koontz
        6. GIRAPH-211.patch
          87 kB
          Eugene Joseph Koontz
        7. GIRAPH-211.patch
          105 kB
          Eugene Joseph Koontz
        8. GIRAPH-211.patch
          69 kB
          Eugene Joseph Koontz
        9. GIRAPH-211.patch
          84 kB
          Eugene Joseph Koontz
        10. GIRAPH-211.patch
          81 kB
          Eugene Joseph Koontz
        11. GIRAPH-211.patch
          85 kB
          Eugene Joseph Koontz
        12. GIRAPH-211.patch
          52 kB
          Eugene Joseph Koontz
        13. GIRAPH-211.patch
          110 kB
          Eugene Joseph Koontz
        14. GIRAPH-211.patch
          105 kB
          Eugene Joseph Koontz
        15. GIRAPH-211-proposal.txt
          8 kB
          Eugene Joseph Koontz

        Issue Links

          Activity

            majakabiljo Maja Kabiljo added a comment -

            I talked about this to Avery, he suggested using just Netty SSL. How would you suggest that we assign certificates/keys to workers?

            majakabiljo Maja Kabiljo added a comment - I talked about this to Avery, he suggested using just Netty SSL. How would you suggest that we assign certificates/keys to workers?
            jghoman Jakob Homan added a comment -

            Certificates/keys don't scale. It would be better to piggyback on Hadoop's delegation token system, certainly while we're on HMR and even later, when we're on YARN.

            jghoman Jakob Homan added a comment - Certificates/keys don't scale. It would be better to piggyback on Hadoop's delegation token system, certainly while we're on HMR and even later, when we're on YARN.

            Interoperability with Hadoop security is a prerequisite for any use of Giraph for our use cases.

            apurtell Andrew Kyle Purtell added a comment - Interoperability with Hadoop security is a prerequisite for any use of Giraph for our use cases.
            aching Avery Ching added a comment -

            Okay, I suppose we can use the BspPolicyProvider, JobTorkenIdentifier, JobTokenSecretManager, etc. to build a light layer of security in Netty. Maja, you can start looking at RPCCommunications.java to see how feasible it is to use the same ideas from the RPC implementation to the Netty implementation.

            aching Avery Ching added a comment - Okay, I suppose we can use the BspPolicyProvider, JobTorkenIdentifier, JobTokenSecretManager, etc. to build a light layer of security in Netty. Maja, you can start looking at RPCCommunications.java to see how feasible it is to use the same ideas from the RPC implementation to the Netty implementation.
            majakabiljo Maja Kabiljo added a comment -

            I'm not sure what am I supposed to do here. My first understanding was that we want all the messages which go through the system to be protected. In that case I can't see how workers can find out about tokens of others. I guess RPC authentication is implemented in hadoop, since the only thing I see in RPC classes in Giraph is the part when the job checks whether it has privileges to make a connection. I need a bit more information here, or I can reassign the task if someone finds it easier to do it than to explain it

            majakabiljo Maja Kabiljo added a comment - I'm not sure what am I supposed to do here. My first understanding was that we want all the messages which go through the system to be protected. In that case I can't see how workers can find out about tokens of others. I guess RPC authentication is implemented in hadoop, since the only thing I see in RPC classes in Giraph is the part when the job checks whether it has privileges to make a connection. I need a bit more information here, or I can reassign the task if someone finds it easier to do it than to explain it

            My first understanding was that we want all the messages which go through the system to be protected.

            What Hadoop did fundamentally is wrap their RPC with SASL at the socket level, and then could use existing JRE support for SASL negotiation with Kerberos authentication (and transparent encryption, etc).

            Eugene could comment better, but what we did for ZooKeeper, which has IO also based on Netty, is instead tunnel the SASL authentication handshake as an extension to the existing protocol, and introduced a mode which requires that handshake to complete successfully before accepting other message types which require authenticated access.

            apurtell Andrew Kyle Purtell added a comment - My first understanding was that we want all the messages which go through the system to be protected. What Hadoop did fundamentally is wrap their RPC with SASL at the socket level, and then could use existing JRE support for SASL negotiation with Kerberos authentication (and transparent encryption, etc). Eugene could comment better, but what we did for ZooKeeper, which has IO also based on Netty, is instead tunnel the SASL authentication handshake as an extension to the existing protocol, and introduced a mode which requires that handshake to complete successfully before accepting other message types which require authenticated access.
            jghoman Jakob Homan added a comment -

            This JIRA is going to require a pretty good understanding of Hadoop's security. It may not be the best for a new contributor...

            jghoman Jakob Homan added a comment - This JIRA is going to require a pretty good understanding of Hadoop's security. It may not be the best for a new contributor...

            I'm happy to help Maja or take over this JIRA - either way I'll be following it closely! Maybe a hackathon or meetup would be a good idea.

            ekoontz Eugene Joseph Koontz added a comment - I'm happy to help Maja or take over this JIRA - either way I'll be following it closely! Maybe a hackathon or meetup would be a good idea.

            I should mention that Zookeeper supports both NIO and Netty on the server, but there is currently no Netty implementation on the Zookeeper client side: see https://issues.apache.org/jira/browse/ZOOKEEPER-823 (which is not resolved at this time).

            ekoontz Eugene Joseph Koontz added a comment - I should mention that Zookeeper supports both NIO and Netty on the server, but there is currently no Netty implementation on the Zookeeper client side: see https://issues.apache.org/jira/browse/ZOOKEEPER-823 (which is not resolved at this time).
            majakabiljo Maja Kabiljo added a comment -

            Eugene, I'm reassigning it to you. Really appreciate your offer to help, but if it requires a lot of background and we need it fast it's better this way.

            majakabiljo Maja Kabiljo added a comment - Eugene, I'm reassigning it to you. Really appreciate your offer to help, but if it requires a lot of background and we need it fast it's better this way.

            Sounds fine, Maja.

            ekoontz Eugene Joseph Koontz added a comment - Sounds fine, Maja.

            I am working on a patch for this after studying both the Hadoop RPC side and the Netty side of things. I'm posting a proposal, but, since it's long, I'll attach it as a .txt. But to get the discussion started, I want to describe the idea briefly: it uses the Job Token as a SASL credential so that one BSPWorker can authenticate itself as a client to another BSPWorker acting as a server. This is what is going on with Hadoop RPC, so we just need to replicate this behavior in the Netty implementation.

            ekoontz Eugene Joseph Koontz added a comment - I am working on a patch for this after studying both the Hadoop RPC side and the Netty side of things. I'm posting a proposal, but, since it's long, I'll attach it as a .txt. But to get the discussion started, I want to describe the idea briefly: it uses the Job Token as a SASL credential so that one BSPWorker can authenticate itself as a client to another BSPWorker acting as a server. This is what is going on with Hadoop RPC, so we just need to replicate this behavior in the Netty implementation.
            aching Avery Ching added a comment -

            Thanks for working on this Eugene. Can't wait to get rid of HadoopRPC so we only have to support one IPC instead of two.

            aching Avery Ching added a comment - Thanks for working on this Eugene. Can't wait to get rid of HadoopRPC so we only have to support one IPC instead of two.
            aching Avery Ching added a comment -

            Hey Eugene, your proposal look good btw. Let me know if you need any help doing this. I'd like to get it in asap as this is causing a lot of cruft to build up. We don't need to do the ZK security for now to remove HadoopRPC.

            aching Avery Ching added a comment - Hey Eugene, your proposal look good btw. Let me know if you need any help doing this. I'd like to get it in asap as this is causing a lot of cruft to build up. We don't need to do the ZK security for now to remove HadoopRPC.

            Thanks for reading it, Avery. I will have a patch tomorrow.

            ekoontz Eugene Joseph Koontz added a comment - Thanks for reading it, Avery. I will have a patch tomorrow.
            aching Avery Ching added a comment -

            Sounds great. I was thinking about basing it on the SecureShuffleUtils.java and ReduceTask.java, but you seem to have a more complex idea. Looking forward to it.

            aching Avery Ching added a comment - Sounds great. I was thinking about basing it on the SecureShuffleUtils.java and ReduceTask.java, but you seem to have a more complex idea. Looking forward to it.

            This patch allows SASL authentication using the JobToken, as described in

            https://issues.apache.org/jira/secure/attachment/12537279/GIRAPH-211-proposal.txt.

            Limitations (will follow up with new JIRAs as warranted)

            -Authorization is not done: that is, clients are authenticated but there are no restrictions on their ability to do RPC on the servers.

            -Clients should wait for authentication before trying to do RPC - once authorization (see above) is done, they might encounter a race where they try to do RPCs without yet being authenticated.

            -Not tested on other than hadoop 2.0.1-SNAPSHOT

            -Only works if we disable client-side channel-pooling (GIRAPH-289) and local short-circuiting of RPCs (GIRAPH-262) - these should be configurable but currently, I hard-wired both to be disabled.

            How to compile:

            mvn -Phadoop_2.0.1 clean test

            Works with the following test:

            $HADOOP_RUNTIME/bin/hadoop jar $GIRAPH_DIR/target/giraph-0.2-SNAPSHOT-for-hadoop-2.0.1-SNAPSHOT-jar-with-dependencies.jar org.apache.giraph.benchmark.PageRankBenchmark -Dgiraph.useNetty=true -e 2 -s 10 -v -V 2 -w 2

            ekoontz Eugene Joseph Koontz added a comment - This patch allows SASL authentication using the JobToken, as described in https://issues.apache.org/jira/secure/attachment/12537279/GIRAPH-211-proposal.txt . Limitations (will follow up with new JIRAs as warranted) -Authorization is not done: that is, clients are authenticated but there are no restrictions on their ability to do RPC on the servers. -Clients should wait for authentication before trying to do RPC - once authorization (see above) is done, they might encounter a race where they try to do RPCs without yet being authenticated. -Not tested on other than hadoop 2.0.1-SNAPSHOT -Only works if we disable client-side channel-pooling ( GIRAPH-289 ) and local short-circuiting of RPCs ( GIRAPH-262 ) - these should be configurable but currently, I hard-wired both to be disabled. How to compile: mvn -Phadoop_2.0.1 clean test Works with the following test: $HADOOP_RUNTIME/bin/hadoop jar $GIRAPH_DIR/target/giraph-0.2-SNAPSHOT-for-hadoop-2.0.1-SNAPSHOT-jar-with-dependencies.jar org.apache.giraph.benchmark.PageRankBenchmark -Dgiraph.useNetty=true -e 2 -s 10 -v -V 2 -w 2

            Here's some output showing client and server SASL communication based on the above test:

            [ec2-user@ip-10-176-255-76 giraph]$ cat /tmp/logs/application_1344445498302_0016/container_1344445498302_0016_01_000002/syslog | grep -i giraph | grep -i sasl
            2012-08-08 20:58:52,806 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.NettyClient: creating saslNettyClient now.
            2012-08-08 20:58:52,816 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.NettyClient: creating saslNettyClient now.
            2012-08-08 20:58:52,929 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslNettyClient: Creating SASL DIGEST-MD5 client to authenticate to service at job_1344445498302_0016
            2012-08-08 20:58:52,931 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslNettyClient: Creating SASL DIGEST-MD5 client to authenticate to service at job_1344445498302_0016
            2012-08-08 20:58:52,943 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.NettyClient: storing saslNettyClient at key: [id: 0x7e8905bd, /10.176.255.76:48238 => ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30000]
            2012-08-08 20:58:52,943 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.NettyClient: created: org.apache.giraph.comm.SaslNettyClient@2aa937cd
            2012-08-08 20:58:52,943 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.NettyClient: sending request: org.apache.giraph.comm.SaslTokenMessage@4b8e899c to remote server: ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30000
            2012-08-08 20:58:52,943 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.RequestEncoder: encoding object: org.apache.giraph.comm.SaslTokenMessage@4b8e899c
            2012-08-08 20:58:52,947 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: write() called on token of length: 0
            2012-08-08 20:58:52,947 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: write() finished writing token.
            2012-08-08 20:58:52,947 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.RequestEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslTokenMessage
            2012-08-08 20:58:52,954 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.NettyClient: storing saslNettyClient at key: [id: 0x04df8b14, /10.176.255.76:60743 => ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30001]
            2012-08-08 20:58:52,954 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.NettyClient: created: org.apache.giraph.comm.SaslNettyClient@23faa614
            2012-08-08 20:58:52,954 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.NettyClient: sending request: org.apache.giraph.comm.SaslTokenMessage@1cad7d80 to remote server: ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30001
            2012-08-08 20:58:52,954 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.RequestEncoder: encoding object: org.apache.giraph.comm.SaslTokenMessage@1cad7d80
            2012-08-08 20:58:52,954 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: write() called on token of length: 0
            2012-08-08 20:58:52,954 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: write() finished writing token.
            2012-08-08 20:58:52,954 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.RequestEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslTokenMessage
            2012-08-08 20:58:52,959 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestDecoder: decode: Got a request of type SASL_TOKEN from remote:/10.176.255.76:48238
            2012-08-08 20:58:52,959 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_TOKEN
            2012-08-08 20:58:52,959 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: reading fields from DataInput: org.jboss.netty.buffer.ChannelBufferInputStream@2b8ca663
            2012-08-08 20:58:52,959 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: token size is: 0
            2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: now reading token.
            2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: read SASL token of length: 0
            2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestDecoder: decoded an object of type: class org.apache.giraph.comm.SaslTokenMessage:org.apache.giraph.comm.SaslTokenMessage@1effc3eb
            2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestServerHandler: Got class org.apache.giraph.comm.SaslTokenMessage
            2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestServerHandler: calling doRequest on class: class org.apache.giraph.comm.SaslTokenMessage
            2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: looking for saslNettyServer on server:/10.176.255.76:30000 for client /10.176.255.76:48238
            2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: creating SaslNettyServer with secret manager: org.apache.hadoop.mapreduce.security.token.JobTokenSecretManager@658782a7
            2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: creating saslNettyServer: secret manager is: org.apache.hadoop.mapreduce.security.token.JobTokenSecretManager@658782a7
            2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: checked read-availability; now creating sasl server.
            2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: create...
            2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: creating callback handler..
            2012-08-08 20:58:52,961 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: creating SaslDigestCallback handler with secret manager: org.apache.hadoop.mapreduce.security.token.JobTokenSecretManager@658782a7
            2012-08-08 20:58:52,961 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: created callback handler: org.apache.giraph.comm.SaslNettyServer$SaslDigestCallbackHandler@1bb0889a
            2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: create was successful.
            2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: client's token's length is: 0
            2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: checking completeness of SASL (pre-response):false
            2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: responding to input token of length: 0
            2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: response token length: 108
            2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: checking completeness of SASL (post-response):false
            2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: server's token's length is: 108
            2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: Responding to client: /10.176.255.76:48238
            2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.ResponseEncoder: RESPONSE ENCODER IS ENCODING A MESSAGE:org.apache.giraph.comm.SaslTokenMessage@1effc3eb
            2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.ResponseEncoder: encoding object: org.apache.giraph.comm.SaslTokenMessage@1effc3eb
            2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.ResponseEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslTokenMessage
            2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: write() called on token of length: 108
            2012-08-08 20:58:52,968 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: write() finished writing token.
            2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: decode: Got a response of type SASL_TOKEN from server:ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30000
            2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_TOKEN
            2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: writableRequestClass: class org.apache.giraph.comm.SaslTokenMessage found for type:SASL_TOKEN
            2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: reading fields of server response:org.apache.giraph.comm.SaslTokenMessage@48c5186e
            2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: reading fields from DataInput: org.jboss.netty.buffer.ChannelBufferInputStream@3c70315
            2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: token size is: 108
            2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: now reading token.
            2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: read SASL token of length: 108
            2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: finished reading fields of server response:org.apache.giraph.comm.SaslTokenMessage@48c5186e
            2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseClientHandler: message received: org.apache.giraph.comm.SaslTokenMessage@48c5186e
            2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslNettyClient: SASL client callback: setting username: FmpvYl8xMzQ0NDQ1NDk4MzAyXzAwMTY=
            2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslNettyClient: SASL client callback: setting userPassword
            2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslNettyClient: SASL client callback: setting realm: default
            2012-08-08 20:58:52,969 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: creating SASL message with size: 274
            2012-08-08 20:58:52,969 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.NettyClient: sending request: org.apache.giraph.comm.SaslTokenMessage@7e9b59a2 to remote server: ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30000
            2012-08-08 20:58:52,969 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.RequestEncoder: encoding object: org.apache.giraph.comm.SaslTokenMessage@7e9b59a2
            2012-08-08 20:58:52,969 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: write() called on token of length: 274
            2012-08-08 20:58:52,969 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: write() finished writing token.
            2012-08-08 20:58:52,969 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.RequestEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslTokenMessage
            2012-08-08 20:58:52,969 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseClientHandler: due to reception of: org.apache.giraph.comm.SaslTokenMessage@48c5186e
            2012-08-08 20:58:52,970 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: Finished writing token to client.
            2012-08-08 20:58:52,970 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestServerHandler: done with doRequest() for class: class org.apache.giraph.comm.SaslTokenMessage
            2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestDecoder: decode: Got a request of type SASL_TOKEN from remote:/10.176.255.76:48238
            2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_TOKEN
            2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: reading fields from DataInput: org.jboss.netty.buffer.ChannelBufferInputStream@919db9e
            2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: token size is: 274
            2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: now reading token.
            2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: read SASL token of length: 274
            2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestDecoder: decoded an object of type: class org.apache.giraph.comm.SaslTokenMessage:org.apache.giraph.comm.SaslTokenMessage@a166bd
            2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestServerHandler: Got class org.apache.giraph.comm.SaslTokenMessage
            2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestServerHandler: calling doRequest on class: class org.apache.giraph.comm.SaslTokenMessage
            2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: looking for saslNettyServer on server:/10.176.255.76:30000 for client /10.176.255.76:48238
            2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: client's token's length is: 274
            2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: checking completeness of SASL (pre-response):false
            2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: responding to input token of length: 274
            2012-08-08 20:58:52,981 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: SASL server DIGEST-MD5 callback: setting password for client: job_1344445498302_0016 (auth:SIMPLE)
            2012-08-08 20:58:52,981 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: SASL server DIGEST-MD5 callback: setting canonicalized client ID: job_1344445498302_0016
            2012-08-08 20:58:52,982 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: response token length: 40
            2012-08-08 20:58:52,982 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: checking completeness of SASL (post-response):true
            2012-08-08 20:58:52,982 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: server's token's length is: 40
            2012-08-08 20:58:52,982 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: Responding to client: /10.176.255.76:48238
            2012-08-08 20:58:52,982 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.ResponseEncoder: RESPONSE ENCODER IS ENCODING A MESSAGE:org.apache.giraph.comm.SaslTokenMessage@a166bd
            2012-08-08 20:58:52,982 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.ResponseEncoder: encoding object: org.apache.giraph.comm.SaslTokenMessage@a166bd
            2012-08-08 20:58:52,982 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.ResponseEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslTokenMessage
            2012-08-08 20:58:52,982 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: write() called on token of length: 40
            2012-08-08 20:58:52,982 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: write() finished writing token.
            2012-08-08 20:58:52,982 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: decode: Got a response of type SASL_TOKEN from server:ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30000
            2012-08-08 20:58:52,982 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_TOKEN
            2012-08-08 20:58:52,983 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: writableRequestClass: class org.apache.giraph.comm.SaslTokenMessage found for type:SASL_TOKEN
            2012-08-08 20:58:52,983 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: reading fields of server response:org.apache.giraph.comm.SaslTokenMessage@4b6c06dd
            2012-08-08 20:58:52,983 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: reading fields from DataInput: org.jboss.netty.buffer.ChannelBufferInputStream@2136bdda
            2012-08-08 20:58:52,983 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: token size is: 40
            2012-08-08 20:58:52,983 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: now reading token.
            2012-08-08 20:58:52,983 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: read SASL token of length: 40
            2012-08-08 20:58:52,983 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: finished reading fields of server response:org.apache.giraph.comm.SaslTokenMessage@4b6c06dd
            2012-08-08 20:58:52,983 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseClientHandler: message received: org.apache.giraph.comm.SaslTokenMessage@4b6c06dd
            2012-08-08 20:58:52,987 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: Finished writing token to client.
            2012-08-08 20:58:52,987 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: writing out complete token now.
            2012-08-08 20:58:52,987 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.ResponseEncoder: RESPONSE ENCODER IS ENCODING A MESSAGE:org.apache.giraph.comm.SaslComplete@36511e0a
            2012-08-08 20:58:52,987 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.ResponseEncoder: encoding object: org.apache.giraph.comm.SaslComplete@36511e0a
            2012-08-08 20:58:52,987 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.ResponseEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslComplete
            2012-08-08 20:58:52,987 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslComplete: writing one byte (write).
            2012-08-08 20:58:52,987 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: wrote out complete token now.
            2012-08-08 20:58:52,987 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestServerHandler: done with doRequest() for class: class org.apache.giraph.comm.SaslTokenMessage
            2012-08-08 20:58:53,009 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestDecoder: decode: Got a request of type SASL_TOKEN from remote:/10.176.255.76:48239
            2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_TOKEN
            2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: reading fields from DataInput: org.jboss.netty.buffer.ChannelBufferInputStream@595780d9
            2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: token size is: 0
            2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: now reading token.
            2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: read SASL token of length: 0
            2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestDecoder: decoded an object of type: class org.apache.giraph.comm.SaslTokenMessage:org.apache.giraph.comm.SaslTokenMessage@7fcc5461
            2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestServerHandler: Got class org.apache.giraph.comm.SaslTokenMessage
            2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestServerHandler: calling doRequest on class: class org.apache.giraph.comm.SaslTokenMessage
            2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: looking for saslNettyServer on server:/10.176.255.76:30000 for client /10.176.255.76:48239
            2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: creating SaslNettyServer with secret manager: org.apache.hadoop.mapreduce.security.token.JobTokenSecretManager@658782a7
            2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: creating saslNettyServer: secret manager is: org.apache.hadoop.mapreduce.security.token.JobTokenSecretManager@658782a7
            2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: checked read-availability; now creating sasl server.
            2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: create...
            2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: creating callback handler..
            2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: creating SaslDigestCallback handler with secret manager: org.apache.hadoop.mapreduce.security.token.JobTokenSecretManager@658782a7
            2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: created callback handler: org.apache.giraph.comm.SaslNettyServer$SaslDigestCallbackHandler@2ecc5436
            2012-08-08 20:58:53,011 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: create was successful.
            2012-08-08 20:58:53,011 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: client's token's length is: 0
            2012-08-08 20:58:53,011 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: checking completeness of SASL (pre-response):false
            2012-08-08 20:58:53,011 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: responding to input token of length: 0
            2012-08-08 20:58:53,011 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: response token length: 108
            2012-08-08 20:58:53,011 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: checking completeness of SASL (post-response):false
            2012-08-08 20:58:53,011 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: server's token's length is: 108
            2012-08-08 20:58:53,011 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: Responding to client: /10.176.255.76:48239
            2012-08-08 20:58:53,011 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.ResponseEncoder: RESPONSE ENCODER IS ENCODING A MESSAGE:org.apache.giraph.comm.SaslTokenMessage@7fcc5461
            2012-08-08 20:58:53,012 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.ResponseEncoder: encoding object: org.apache.giraph.comm.SaslTokenMessage@7fcc5461
            2012-08-08 20:58:53,012 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.ResponseEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslTokenMessage
            2012-08-08 20:58:53,012 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: write() called on token of length: 108
            2012-08-08 20:58:53,012 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: write() finished writing token.
            2012-08-08 20:58:53,012 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: Finished writing token to client.
            2012-08-08 20:58:53,012 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestServerHandler: done with doRequest() for class: class org.apache.giraph.comm.SaslTokenMessage
            2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestDecoder: decode: Got a request of type SASL_TOKEN from remote:/10.176.255.76:48239
            2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_TOKEN
            2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: reading fields from DataInput: org.jboss.netty.buffer.ChannelBufferInputStream@24bb6086
            2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: token size is: 274
            2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: now reading token.
            2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: read SASL token of length: 274
            2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestDecoder: decoded an object of type: class org.apache.giraph.comm.SaslTokenMessage:org.apache.giraph.comm.SaslTokenMessage@9be1041
            2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestServerHandler: Got class org.apache.giraph.comm.SaslTokenMessage
            2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestServerHandler: calling doRequest on class: class org.apache.giraph.comm.SaslTokenMessage
            2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: looking for saslNettyServer on server:/10.176.255.76:30000 for client /10.176.255.76:48239
            2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: client's token's length is: 274
            2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: checking completeness of SASL (pre-response):false
            2012-08-08 20:58:53,019 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: responding to input token of length: 274
            2012-08-08 20:58:53,019 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: SASL server DIGEST-MD5 callback: setting password for client: job_1344445498302_0016 (auth:SIMPLE)
            2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: decode: Got a response of type SASL_TOKEN from server:ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30001
            2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_TOKEN
            2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: writableRequestClass: class org.apache.giraph.comm.SaslTokenMessage found for type:SASL_TOKEN
            2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: reading fields of server response:org.apache.giraph.comm.SaslTokenMessage@1d1d2066
            2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: reading fields from DataInput: org.jboss.netty.buffer.ChannelBufferInputStream@633a1778
            2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: token size is: 108
            2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: now reading token.
            2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: read SASL token of length: 108
            2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: finished reading fields of server response:org.apache.giraph.comm.SaslTokenMessage@1d1d2066
            2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseClientHandler: message received: org.apache.giraph.comm.SaslTokenMessage@1d1d2066
            2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslNettyClient: SASL client callback: setting username: FmpvYl8xMzQ0NDQ1NDk4MzAyXzAwMTY=
            2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslNettyClient: SASL client callback: setting userPassword
            2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslNettyClient: SASL client callback: setting realm: default
            2012-08-08 20:58:53,021 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: creating SASL message with size: 274
            2012-08-08 20:58:53,021 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.NettyClient: sending request: org.apache.giraph.comm.SaslTokenMessage@7adafa2c to remote server: ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30001
            2012-08-08 20:58:53,021 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.RequestEncoder: encoding object: org.apache.giraph.comm.SaslTokenMessage@7adafa2c
            2012-08-08 20:58:53,021 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: write() called on token of length: 274
            2012-08-08 20:58:53,021 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: write() finished writing token.
            2012-08-08 20:58:53,021 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.RequestEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslTokenMessage
            2012-08-08 20:58:53,021 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseClientHandler: due to reception of: org.apache.giraph.comm.SaslTokenMessage@1d1d2066
            2012-08-08 20:58:53,021 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: SASL server DIGEST-MD5 callback: setting canonicalized client ID: job_1344445498302_0016
            2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: response token length: 40
            2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: checking completeness of SASL (post-response):true
            2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: server's token's length is: 40
            2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: Responding to client: /10.176.255.76:48239
            2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.ResponseEncoder: RESPONSE ENCODER IS ENCODING A MESSAGE:org.apache.giraph.comm.SaslTokenMessage@9be1041
            2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.ResponseEncoder: encoding object: org.apache.giraph.comm.SaslTokenMessage@9be1041
            2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.ResponseEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslTokenMessage
            2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: write() called on token of length: 40
            2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: write() finished writing token.
            2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: Finished writing token to client.
            2012-08-08 20:58:53,023 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: writing out complete token now.
            2012-08-08 20:58:53,023 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.ResponseEncoder: RESPONSE ENCODER IS ENCODING A MESSAGE:org.apache.giraph.comm.SaslComplete@3e5e9db7
            2012-08-08 20:58:53,023 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.ResponseEncoder: encoding object: org.apache.giraph.comm.SaslComplete@3e5e9db7
            2012-08-08 20:58:53,023 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.ResponseEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslComplete
            2012-08-08 20:58:53,023 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: decode: Got a response of type SASL_COMPLETE from server:ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30000
            2012-08-08 20:58:53,023 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_COMPLETE
            2012-08-08 20:58:53,023 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: writableRequestClass: class org.apache.giraph.comm.SaslComplete found for type:SASL_COMPLETE
            2012-08-08 20:58:53,023 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: reading fields of server response:org.apache.giraph.comm.SaslComplete@75fc25e5
            2012-08-08 20:58:53,023 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslComplete: read fields (none)
            2012-08-08 20:58:53,024 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: finished reading fields of server response:org.apache.giraph.comm.SaslComplete@75fc25e5
            2012-08-08 20:58:53,024 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseClientHandler: message received: org.apache.giraph.comm.SaslComplete@75fc25e5
            2012-08-08 20:58:53,024 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseClientHandler: SASL Authentication complete with server: ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30000.
            2012-08-08 20:58:53,024 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseClientHandler: due to reception of: org.apache.giraph.comm.SaslComplete@75fc25e5
            2012-08-08 20:58:53,023 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslComplete: writing one byte (write).
            2012-08-08 20:58:53,024 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: wrote out complete token now.
            2012-08-08 20:58:53,024 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestServerHandler: done with doRequest() for class: class org.apache.giraph.comm.SaslTokenMessage
            2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: decode: Got a response of type SASL_TOKEN from server:ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30001
            2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_TOKEN
            2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: writableRequestClass: class org.apache.giraph.comm.SaslTokenMessage found for type:SASL_TOKEN
            2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: reading fields of server response:org.apache.giraph.comm.SaslTokenMessage@3fe88b35
            2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: reading fields from DataInput: org.jboss.netty.buffer.ChannelBufferInputStream@3c0b655a
            2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: token size is: 40
            2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: now reading token.
            2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: read SASL token of length: 40
            2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: finished reading fields of server response:org.apache.giraph.comm.SaslTokenMessage@3fe88b35
            2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseClientHandler: message received: org.apache.giraph.comm.SaslTokenMessage@3fe88b35
            2012-08-08 20:58:53,071 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: decode: Got a response of type SASL_COMPLETE from server:ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30001
            2012-08-08 20:58:53,071 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_COMPLETE
            2012-08-08 20:58:53,071 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: writableRequestClass: class org.apache.giraph.comm.SaslComplete found for type:SASL_COMPLETE
            2012-08-08 20:58:53,071 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: reading fields of server response:org.apache.giraph.comm.SaslComplete@156a9424
            2012-08-08 20:58:53,071 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslComplete: read fields (none)
            2012-08-08 20:58:53,071 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: finished reading fields of server response:org.apache.giraph.comm.SaslComplete@156a9424
            2012-08-08 20:58:53,071 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseClientHandler: message received: org.apache.giraph.comm.SaslComplete@156a9424
            2012-08-08 20:58:53,071 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseClientHandler: SASL Authentication complete with server: ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30001.
            2012-08-08 20:58:53,071 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseClientHandler: due to reception of: org.apache.giraph.comm.SaslComplete@156a9424
            
            
            ekoontz Eugene Joseph Koontz added a comment - Here's some output showing client and server SASL communication based on the above test: [ec2-user@ip-10-176-255-76 giraph]$ cat /tmp/logs/application_1344445498302_0016/container_1344445498302_0016_01_000002/syslog | grep -i giraph | grep -i sasl 2012-08-08 20:58:52,806 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.NettyClient: creating saslNettyClient now. 2012-08-08 20:58:52,816 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.NettyClient: creating saslNettyClient now. 2012-08-08 20:58:52,929 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslNettyClient: Creating SASL DIGEST-MD5 client to authenticate to service at job_1344445498302_0016 2012-08-08 20:58:52,931 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslNettyClient: Creating SASL DIGEST-MD5 client to authenticate to service at job_1344445498302_0016 2012-08-08 20:58:52,943 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.NettyClient: storing saslNettyClient at key: [id: 0x7e8905bd, /10.176.255.76:48238 => ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30000] 2012-08-08 20:58:52,943 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.NettyClient: created: org.apache.giraph.comm.SaslNettyClient@2aa937cd 2012-08-08 20:58:52,943 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.NettyClient: sending request: org.apache.giraph.comm.SaslTokenMessage@4b8e899c to remote server: ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30000 2012-08-08 20:58:52,943 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.RequestEncoder: encoding object: org.apache.giraph.comm.SaslTokenMessage@4b8e899c 2012-08-08 20:58:52,947 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: write() called on token of length: 0 2012-08-08 20:58:52,947 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: write() finished writing token. 2012-08-08 20:58:52,947 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.RequestEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:52,954 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.NettyClient: storing saslNettyClient at key: [id: 0x04df8b14, /10.176.255.76:60743 => ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30001] 2012-08-08 20:58:52,954 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.NettyClient: created: org.apache.giraph.comm.SaslNettyClient@23faa614 2012-08-08 20:58:52,954 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.NettyClient: sending request: org.apache.giraph.comm.SaslTokenMessage@1cad7d80 to remote server: ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30001 2012-08-08 20:58:52,954 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.RequestEncoder: encoding object: org.apache.giraph.comm.SaslTokenMessage@1cad7d80 2012-08-08 20:58:52,954 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: write() called on token of length: 0 2012-08-08 20:58:52,954 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: write() finished writing token. 2012-08-08 20:58:52,954 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.RequestEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:52,959 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestDecoder: decode: Got a request of type SASL_TOKEN from remote:/10.176.255.76:48238 2012-08-08 20:58:52,959 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_TOKEN 2012-08-08 20:58:52,959 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: reading fields from DataInput: org.jboss.netty.buffer.ChannelBufferInputStream@2b8ca663 2012-08-08 20:58:52,959 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: token size is: 0 2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: now reading token. 2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: read SASL token of length: 0 2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestDecoder: decoded an object of type: class org.apache.giraph.comm.SaslTokenMessage:org.apache.giraph.comm.SaslTokenMessage@1effc3eb 2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestServerHandler: Got class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestServerHandler: calling doRequest on class: class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: looking for saslNettyServer on server:/10.176.255.76:30000 for client /10.176.255.76:48238 2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: creating SaslNettyServer with secret manager: org.apache.hadoop.mapreduce.security.token.JobTokenSecretManager@658782a7 2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: creating saslNettyServer: secret manager is: org.apache.hadoop.mapreduce.security.token.JobTokenSecretManager@658782a7 2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: checked read-availability; now creating sasl server. 2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: create... 2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: creating callback handler.. 2012-08-08 20:58:52,961 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: creating SaslDigestCallback handler with secret manager: org.apache.hadoop.mapreduce.security.token.JobTokenSecretManager@658782a7 2012-08-08 20:58:52,961 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: created callback handler: org.apache.giraph.comm.SaslNettyServer$SaslDigestCallbackHandler@1bb0889a 2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: create was successful. 2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: client 's token' s length is: 0 2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: checking completeness of SASL (pre-response): false 2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: responding to input token of length: 0 2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: response token length: 108 2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: checking completeness of SASL (post-response): false 2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: server 's token' s length is: 108 2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: Responding to client: /10.176.255.76:48238 2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.ResponseEncoder: RESPONSE ENCODER IS ENCODING A MESSAGE:org.apache.giraph.comm.SaslTokenMessage@1effc3eb 2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.ResponseEncoder: encoding object: org.apache.giraph.comm.SaslTokenMessage@1effc3eb 2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.ResponseEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: write() called on token of length: 108 2012-08-08 20:58:52,968 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: write() finished writing token. 2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: decode: Got a response of type SASL_TOKEN from server:ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30000 2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_TOKEN 2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: writableRequestClass: class org.apache.giraph.comm.SaslTokenMessage found for type:SASL_TOKEN 2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: reading fields of server response:org.apache.giraph.comm.SaslTokenMessage@48c5186e 2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: reading fields from DataInput: org.jboss.netty.buffer.ChannelBufferInputStream@3c70315 2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: token size is: 108 2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: now reading token. 2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: read SASL token of length: 108 2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: finished reading fields of server response:org.apache.giraph.comm.SaslTokenMessage@48c5186e 2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseClientHandler: message received: org.apache.giraph.comm.SaslTokenMessage@48c5186e 2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslNettyClient: SASL client callback: setting username: FmpvYl8xMzQ0NDQ1NDk4MzAyXzAwMTY= 2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslNettyClient: SASL client callback: setting userPassword 2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslNettyClient: SASL client callback: setting realm: default 2012-08-08 20:58:52,969 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: creating SASL message with size: 274 2012-08-08 20:58:52,969 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.NettyClient: sending request: org.apache.giraph.comm.SaslTokenMessage@7e9b59a2 to remote server: ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30000 2012-08-08 20:58:52,969 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.RequestEncoder: encoding object: org.apache.giraph.comm.SaslTokenMessage@7e9b59a2 2012-08-08 20:58:52,969 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: write() called on token of length: 274 2012-08-08 20:58:52,969 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: write() finished writing token. 2012-08-08 20:58:52,969 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.RequestEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:52,969 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseClientHandler: due to reception of: org.apache.giraph.comm.SaslTokenMessage@48c5186e 2012-08-08 20:58:52,970 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: Finished writing token to client. 2012-08-08 20:58:52,970 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestServerHandler: done with doRequest() for class: class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestDecoder: decode: Got a request of type SASL_TOKEN from remote:/10.176.255.76:48238 2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_TOKEN 2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: reading fields from DataInput: org.jboss.netty.buffer.ChannelBufferInputStream@919db9e 2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: token size is: 274 2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: now reading token. 2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: read SASL token of length: 274 2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestDecoder: decoded an object of type: class org.apache.giraph.comm.SaslTokenMessage:org.apache.giraph.comm.SaslTokenMessage@a166bd 2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestServerHandler: Got class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestServerHandler: calling doRequest on class: class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: looking for saslNettyServer on server:/10.176.255.76:30000 for client /10.176.255.76:48238 2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: client 's token' s length is: 274 2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: checking completeness of SASL (pre-response): false 2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: responding to input token of length: 274 2012-08-08 20:58:52,981 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: SASL server DIGEST-MD5 callback: setting password for client: job_1344445498302_0016 (auth:SIMPLE) 2012-08-08 20:58:52,981 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: SASL server DIGEST-MD5 callback: setting canonicalized client ID: job_1344445498302_0016 2012-08-08 20:58:52,982 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: response token length: 40 2012-08-08 20:58:52,982 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: checking completeness of SASL (post-response): true 2012-08-08 20:58:52,982 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: server 's token' s length is: 40 2012-08-08 20:58:52,982 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: Responding to client: /10.176.255.76:48238 2012-08-08 20:58:52,982 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.ResponseEncoder: RESPONSE ENCODER IS ENCODING A MESSAGE:org.apache.giraph.comm.SaslTokenMessage@a166bd 2012-08-08 20:58:52,982 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.ResponseEncoder: encoding object: org.apache.giraph.comm.SaslTokenMessage@a166bd 2012-08-08 20:58:52,982 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.ResponseEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:52,982 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: write() called on token of length: 40 2012-08-08 20:58:52,982 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: write() finished writing token. 2012-08-08 20:58:52,982 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: decode: Got a response of type SASL_TOKEN from server:ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30000 2012-08-08 20:58:52,982 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_TOKEN 2012-08-08 20:58:52,983 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: writableRequestClass: class org.apache.giraph.comm.SaslTokenMessage found for type:SASL_TOKEN 2012-08-08 20:58:52,983 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: reading fields of server response:org.apache.giraph.comm.SaslTokenMessage@4b6c06dd 2012-08-08 20:58:52,983 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: reading fields from DataInput: org.jboss.netty.buffer.ChannelBufferInputStream@2136bdda 2012-08-08 20:58:52,983 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: token size is: 40 2012-08-08 20:58:52,983 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: now reading token. 2012-08-08 20:58:52,983 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: read SASL token of length: 40 2012-08-08 20:58:52,983 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: finished reading fields of server response:org.apache.giraph.comm.SaslTokenMessage@4b6c06dd 2012-08-08 20:58:52,983 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseClientHandler: message received: org.apache.giraph.comm.SaslTokenMessage@4b6c06dd 2012-08-08 20:58:52,987 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: Finished writing token to client. 2012-08-08 20:58:52,987 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: writing out complete token now. 2012-08-08 20:58:52,987 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.ResponseEncoder: RESPONSE ENCODER IS ENCODING A MESSAGE:org.apache.giraph.comm.SaslComplete@36511e0a 2012-08-08 20:58:52,987 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.ResponseEncoder: encoding object: org.apache.giraph.comm.SaslComplete@36511e0a 2012-08-08 20:58:52,987 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.ResponseEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslComplete 2012-08-08 20:58:52,987 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslComplete: writing one byte (write). 2012-08-08 20:58:52,987 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: wrote out complete token now. 2012-08-08 20:58:52,987 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestServerHandler: done with doRequest() for class: class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:53,009 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestDecoder: decode: Got a request of type SASL_TOKEN from remote:/10.176.255.76:48239 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_TOKEN 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: reading fields from DataInput: org.jboss.netty.buffer.ChannelBufferInputStream@595780d9 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: token size is: 0 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: now reading token. 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: read SASL token of length: 0 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestDecoder: decoded an object of type: class org.apache.giraph.comm.SaslTokenMessage:org.apache.giraph.comm.SaslTokenMessage@7fcc5461 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestServerHandler: Got class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestServerHandler: calling doRequest on class: class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: looking for saslNettyServer on server:/10.176.255.76:30000 for client /10.176.255.76:48239 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: creating SaslNettyServer with secret manager: org.apache.hadoop.mapreduce.security.token.JobTokenSecretManager@658782a7 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: creating saslNettyServer: secret manager is: org.apache.hadoop.mapreduce.security.token.JobTokenSecretManager@658782a7 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: checked read-availability; now creating sasl server. 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: create... 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: creating callback handler.. 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: creating SaslDigestCallback handler with secret manager: org.apache.hadoop.mapreduce.security.token.JobTokenSecretManager@658782a7 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: created callback handler: org.apache.giraph.comm.SaslNettyServer$SaslDigestCallbackHandler@2ecc5436 2012-08-08 20:58:53,011 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: create was successful. 2012-08-08 20:58:53,011 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: client 's token' s length is: 0 2012-08-08 20:58:53,011 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: checking completeness of SASL (pre-response): false 2012-08-08 20:58:53,011 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: responding to input token of length: 0 2012-08-08 20:58:53,011 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: response token length: 108 2012-08-08 20:58:53,011 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: checking completeness of SASL (post-response): false 2012-08-08 20:58:53,011 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: server 's token' s length is: 108 2012-08-08 20:58:53,011 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: Responding to client: /10.176.255.76:48239 2012-08-08 20:58:53,011 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.ResponseEncoder: RESPONSE ENCODER IS ENCODING A MESSAGE:org.apache.giraph.comm.SaslTokenMessage@7fcc5461 2012-08-08 20:58:53,012 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.ResponseEncoder: encoding object: org.apache.giraph.comm.SaslTokenMessage@7fcc5461 2012-08-08 20:58:53,012 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.ResponseEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:53,012 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: write() called on token of length: 108 2012-08-08 20:58:53,012 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: write() finished writing token. 2012-08-08 20:58:53,012 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: Finished writing token to client. 2012-08-08 20:58:53,012 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestServerHandler: done with doRequest() for class: class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestDecoder: decode: Got a request of type SASL_TOKEN from remote:/10.176.255.76:48239 2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_TOKEN 2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: reading fields from DataInput: org.jboss.netty.buffer.ChannelBufferInputStream@24bb6086 2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: token size is: 274 2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: now reading token. 2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: read SASL token of length: 274 2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestDecoder: decoded an object of type: class org.apache.giraph.comm.SaslTokenMessage:org.apache.giraph.comm.SaslTokenMessage@9be1041 2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestServerHandler: Got class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestServerHandler: calling doRequest on class: class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: looking for saslNettyServer on server:/10.176.255.76:30000 for client /10.176.255.76:48239 2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: client 's token' s length is: 274 2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: checking completeness of SASL (pre-response): false 2012-08-08 20:58:53,019 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: responding to input token of length: 274 2012-08-08 20:58:53,019 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: SASL server DIGEST-MD5 callback: setting password for client: job_1344445498302_0016 (auth:SIMPLE) 2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: decode: Got a response of type SASL_TOKEN from server:ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30001 2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_TOKEN 2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: writableRequestClass: class org.apache.giraph.comm.SaslTokenMessage found for type:SASL_TOKEN 2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: reading fields of server response:org.apache.giraph.comm.SaslTokenMessage@1d1d2066 2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: reading fields from DataInput: org.jboss.netty.buffer.ChannelBufferInputStream@633a1778 2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: token size is: 108 2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: now reading token. 2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: read SASL token of length: 108 2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: finished reading fields of server response:org.apache.giraph.comm.SaslTokenMessage@1d1d2066 2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseClientHandler: message received: org.apache.giraph.comm.SaslTokenMessage@1d1d2066 2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslNettyClient: SASL client callback: setting username: FmpvYl8xMzQ0NDQ1NDk4MzAyXzAwMTY= 2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslNettyClient: SASL client callback: setting userPassword 2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslNettyClient: SASL client callback: setting realm: default 2012-08-08 20:58:53,021 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: creating SASL message with size: 274 2012-08-08 20:58:53,021 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.NettyClient: sending request: org.apache.giraph.comm.SaslTokenMessage@7adafa2c to remote server: ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30001 2012-08-08 20:58:53,021 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.RequestEncoder: encoding object: org.apache.giraph.comm.SaslTokenMessage@7adafa2c 2012-08-08 20:58:53,021 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: write() called on token of length: 274 2012-08-08 20:58:53,021 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: write() finished writing token. 2012-08-08 20:58:53,021 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.RequestEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:53,021 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseClientHandler: due to reception of: org.apache.giraph.comm.SaslTokenMessage@1d1d2066 2012-08-08 20:58:53,021 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: SASL server DIGEST-MD5 callback: setting canonicalized client ID: job_1344445498302_0016 2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: response token length: 40 2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: checking completeness of SASL (post-response): true 2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: server 's token' s length is: 40 2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: Responding to client: /10.176.255.76:48239 2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.ResponseEncoder: RESPONSE ENCODER IS ENCODING A MESSAGE:org.apache.giraph.comm.SaslTokenMessage@9be1041 2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.ResponseEncoder: encoding object: org.apache.giraph.comm.SaslTokenMessage@9be1041 2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.ResponseEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: write() called on token of length: 40 2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: write() finished writing token. 2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: Finished writing token to client. 2012-08-08 20:58:53,023 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: writing out complete token now. 2012-08-08 20:58:53,023 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.ResponseEncoder: RESPONSE ENCODER IS ENCODING A MESSAGE:org.apache.giraph.comm.SaslComplete@3e5e9db7 2012-08-08 20:58:53,023 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.ResponseEncoder: encoding object: org.apache.giraph.comm.SaslComplete@3e5e9db7 2012-08-08 20:58:53,023 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.ResponseEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslComplete 2012-08-08 20:58:53,023 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: decode: Got a response of type SASL_COMPLETE from server:ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30000 2012-08-08 20:58:53,023 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_COMPLETE 2012-08-08 20:58:53,023 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: writableRequestClass: class org.apache.giraph.comm.SaslComplete found for type:SASL_COMPLETE 2012-08-08 20:58:53,023 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: reading fields of server response:org.apache.giraph.comm.SaslComplete@75fc25e5 2012-08-08 20:58:53,023 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslComplete: read fields (none) 2012-08-08 20:58:53,024 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: finished reading fields of server response:org.apache.giraph.comm.SaslComplete@75fc25e5 2012-08-08 20:58:53,024 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseClientHandler: message received: org.apache.giraph.comm.SaslComplete@75fc25e5 2012-08-08 20:58:53,024 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseClientHandler: SASL Authentication complete with server: ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30000. 2012-08-08 20:58:53,024 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseClientHandler: due to reception of: org.apache.giraph.comm.SaslComplete@75fc25e5 2012-08-08 20:58:53,023 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslComplete: writing one byte (write). 2012-08-08 20:58:53,024 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: wrote out complete token now. 2012-08-08 20:58:53,024 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestServerHandler: done with doRequest() for class: class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: decode: Got a response of type SASL_TOKEN from server:ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30001 2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_TOKEN 2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: writableRequestClass: class org.apache.giraph.comm.SaslTokenMessage found for type:SASL_TOKEN 2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: reading fields of server response:org.apache.giraph.comm.SaslTokenMessage@3fe88b35 2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: reading fields from DataInput: org.jboss.netty.buffer.ChannelBufferInputStream@3c0b655a 2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: token size is: 40 2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: now reading token. 2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: read SASL token of length: 40 2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: finished reading fields of server response:org.apache.giraph.comm.SaslTokenMessage@3fe88b35 2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseClientHandler: message received: org.apache.giraph.comm.SaslTokenMessage@3fe88b35 2012-08-08 20:58:53,071 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: decode: Got a response of type SASL_COMPLETE from server:ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30001 2012-08-08 20:58:53,071 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_COMPLETE 2012-08-08 20:58:53,071 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: writableRequestClass: class org.apache.giraph.comm.SaslComplete found for type:SASL_COMPLETE 2012-08-08 20:58:53,071 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: reading fields of server response:org.apache.giraph.comm.SaslComplete@156a9424 2012-08-08 20:58:53,071 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslComplete: read fields (none) 2012-08-08 20:58:53,071 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: finished reading fields of server response:org.apache.giraph.comm.SaslComplete@156a9424 2012-08-08 20:58:53,071 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseClientHandler: message received: org.apache.giraph.comm.SaslComplete@156a9424 2012-08-08 20:58:53,071 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseClientHandler: SASL Authentication complete with server: ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30001. 2012-08-08 20:58:53,071 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseClientHandler: due to reception of: org.apache.giraph.comm.SaslComplete@156a9424

            Tried to create a new review but hit 500 error; submitted this: https://issues.apache.org/jira/browse/INFRA-5128

            ekoontz Eugene Joseph Koontz added a comment - Tried to create a new review but hit 500 error; submitted this: https://issues.apache.org/jira/browse/INFRA-5128
            aching Avery Ching added a comment -

            Eugene, this is a nice start! A few comments/questions about the limitations:

            >-Authorization is not done: that is, clients are authenticated but there are no restrictions on their ability to do RPC on the servers.

            Can't we block until the authentication is done?

            >-Clients should wait for authentication before trying to do RPC - once authorization (see above) is done, they might encounter a race where they try to do RPCs without yet being authenticated.

            Same question as above?

            >-Not tested on other than hadoop 2.0.1-SNAPSHOT

            Would probably be nice to try on a hadoop 1.0.x if you have a chance.

            >-Only works if we disable client-side channel-pooling (GIRAPH-289) and local short-circuiting of RPCs (GIRAPH-262) - these should be configurable but currently, I hard-wired both to be disabled.

            I think it's reasonable to allow short-circuiting since there isn't a security issue here (this is the same process). As for channel pooling, can we simply authenticate once per channel?

            aching Avery Ching added a comment - Eugene, this is a nice start! A few comments/questions about the limitations: >-Authorization is not done: that is, clients are authenticated but there are no restrictions on their ability to do RPC on the servers. Can't we block until the authentication is done? >-Clients should wait for authentication before trying to do RPC - once authorization (see above) is done, they might encounter a race where they try to do RPCs without yet being authenticated. Same question as above? >-Not tested on other than hadoop 2.0.1-SNAPSHOT Would probably be nice to try on a hadoop 1.0.x if you have a chance. >-Only works if we disable client-side channel-pooling ( GIRAPH-289 ) and local short-circuiting of RPCs ( GIRAPH-262 ) - these should be configurable but currently, I hard-wired both to be disabled. I think it's reasonable to allow short-circuiting since there isn't a security issue here (this is the same process). As for channel pooling, can we simply authenticate once per channel?

            Thanks for your comments Avery! Replying to each below:

            > Can't we block until the authentication is done?

            Absolutely - we just need to choose the object to synchronize on and be careful that it's respected where it needs to be. Will include this in next patch, which will also avoid the race.

            > Would probably be nice to try on a hadoop 1.0.x if you have a chance.

            Will do.

            > I think it's reasonable to allow short-circuiting since there isn't a security issue here (this is the same process).

            Yes, I think it will be ok to trust the client in this case (specifically in GIRAPH-262 we are testing for):

            (service.getWorkerInfo().getHostnamePort().equals(remoteServerAddress)) {
            

            I just need to fix whatever bug I am masking by disabling short-curcuiting.

            > As for channel pooling, can we simply authenticate once per channel?

            Yes, I just need to figure out how to associate the sasl client with the channel when we are using channel-pooling. I use the Address => Channel map in my patch, but now with GIRAPH-262, we have Address => ChannelRotator. So this will require some changes on my patch.

            ekoontz Eugene Joseph Koontz added a comment - Thanks for your comments Avery! Replying to each below: > Can't we block until the authentication is done? Absolutely - we just need to choose the object to synchronize on and be careful that it's respected where it needs to be. Will include this in next patch, which will also avoid the race. > Would probably be nice to try on a hadoop 1.0.x if you have a chance. Will do. > I think it's reasonable to allow short-circuiting since there isn't a security issue here (this is the same process). Yes, I think it will be ok to trust the client in this case (specifically in GIRAPH-262 we are testing for): (service.getWorkerInfo().getHostnamePort().equals(remoteServerAddress)) { I just need to fix whatever bug I am masking by disabling short-curcuiting. > As for channel pooling, can we simply authenticate once per channel? Yes, I just need to figure out how to associate the sasl client with the channel when we are using channel-pooling. I use the Address => Channel map in my patch, but now with GIRAPH-262 , we have Address => ChannelRotator. So this will require some changes on my patch.

            Added review: https://reviews.apache.org/r/6609/

            Thanks to pctony for enabling git-based reviews: https://issues.apache.org/jira/browse/INFRA-5128

            ekoontz Eugene Joseph Koontz added a comment - Added review: https://reviews.apache.org/r/6609/ Thanks to pctony for enabling git-based reviews: https://issues.apache.org/jira/browse/INFRA-5128
            aching Avery Ching added a comment -

            Cool! I'll wait for your updated patch to review.

            aching Avery Ching added a comment - Cool! I'll wait for your updated patch to review.

            Any update on this? I don't know the schedule for cutting the 0.2 release, but I'd say this is a prerequisite: not only we get rid of a lot of cruft, but also the API currently includes methods (putMessages/getMessages) that are used only by the Hadoop RPC implementation.

            apresta Alessandro Presta added a comment - Any update on this? I don't know the schedule for cutting the 0.2 release, but I'd say this is a prerequisite: not only we get rid of a lot of cruft, but also the API currently includes methods (putMessages/getMessages) that are used only by the Hadoop RPC implementation.

            Hi Alessandro, I am working actively on it and hope to have a patch ready in the next day or so.
            -Eugene

            ekoontz Eugene Joseph Koontz added a comment - Hi Alessandro, I am working actively on it and hope to have a patch ready in the next day or so. -Eugene

            Awesome, looking forward to it!

            apresta Alessandro Presta added a comment - Awesome, looking forward to it!

            Improves on recent patch:

            -Does authorization: server checks for authentication being completed before allowing non-SASL client requests to be performed.

            -Client waits to send (non-SASL) requests until SASL authentication has completed.

            -Works with channel-pooling and local short-circuiting

            Limitations:

            -Not tested on other than hadoop 2.0.1-SNAPSHOT
            -Restores RequestRegistry class, which was removed in GIRAPH-313 - hope to merge better with this.
            -Needs more testing in general, including new unit tests.

            ekoontz Eugene Joseph Koontz added a comment - Improves on recent patch: -Does authorization: server checks for authentication being completed before allowing non-SASL client requests to be performed. -Client waits to send (non-SASL) requests until SASL authentication has completed. -Works with channel-pooling and local short-circuiting Limitations: -Not tested on other than hadoop 2.0.1-SNAPSHOT -Restores RequestRegistry class, which was removed in GIRAPH-313 - hope to merge better with this. -Needs more testing in general, including new unit tests.

            I manually fixed some conflicts recently, so I'm gradually making this patch more closely fit to trunk.

            Improvements:
            -Removes usage of RequestRegistry (that was removed in GIRAPH-313)

            Limitations:
            Uses preGIRAPH-313 workerThreadPool rather than requestServerHandlerFactory
            -Needs tests

            ekoontz Eugene Joseph Koontz added a comment - I manually fixed some conflicts recently, so I'm gradually making this patch more closely fit to trunk. Improvements: -Removes usage of RequestRegistry (that was removed in GIRAPH-313 ) Limitations: Uses pre GIRAPH-313 workerThreadPool rather than requestServerHandlerFactory -Needs tests

            Sorry, the last patch was missing the new source files - no wonder it was so small.

            ekoontz Eugene Joseph Koontz added a comment - Sorry, the last patch was missing the new source files - no wonder it was so small.

            https://reviews.apache.org/r/6609/diff/ updated to most recent patch.

            ekoontz Eugene Joseph Koontz added a comment - https://reviews.apache.org/r/6609/diff/ updated to most recent patch.

            Improvements:

            Removes remaining preGIRAPH-313-isms in NettyServer and NettyWorkerClient

            ekoontz Eugene Joseph Koontz added a comment - Improvements: Removes remaining pre GIRAPH-313 -isms in NettyServer and NettyWorkerClient

            https://reviews.apache.org/r/6609/diff/ updated to most recent patch (will refrain from making this comment from now on and simply keep them synced).

            ekoontz Eugene Joseph Koontz added a comment - https://reviews.apache.org/r/6609/diff/ updated to most recent patch (will refrain from making this comment from now on and simply keep them synced).
            ekoontz Eugene Joseph Koontz added a comment - merge with trunk: 55bfcb78 https://svn.apache.org/repos/asf/giraph/trunk@1378761 ( GIRAPH-317 )
            aching Avery Ching added a comment -

            Hi Eugene, any luck getting this implemented as a ChannelHandler?

            aching Avery Ching added a comment - Hi Eugene, any luck getting this implemented as a ChannelHandler?

            Hi Avery,
            Yes, I should have a new patch up today.
            -Eugene

            ekoontz Eugene Joseph Koontz added a comment - Hi Avery, Yes, I should have a new patch up today. -Eugene

            Interesting and related: "Handshaking tutorial with Netty" by Bruno de Carvalho here:

            http://biasedbit.com/handshaking-tutorial-with-netty/

            ekoontz Eugene Joseph Koontz added a comment - Interesting and related: "Handshaking tutorial with Netty" by Bruno de Carvalho here: http://biasedbit.com/handshaking-tutorial-with-netty/

            My apologies for the delay. I am in the final stages : getting it to compile on all supported Hadoop versions and fixing checkstyle errors. I have the SASL code implemented as a ChannelHandler as we discussed. Here's what the pipeline looks like on the client side, showing both authenticated and non-authenticated cases:

            NettyClient.java
               // Set up the pipeline factory.
                bootstrap.setPipelineFactory(new ChannelPipelineFactory() {
                  @Override
                  public ChannelPipeline getPipeline() throws Exception {
                    if (conf.getBoolean(GiraphJob.AUTHENTICATE,
                        GiraphJob.DEFAULT_AUTHENTICATE)) {
                      LOG.info("Using Netty with authentication.");
            
                      // Our pipeline starts with just byteCounter, and then we use
                      // addAfter() to incrementally add pipeline elements, so that we can
                      // name them for identification for removal or replacement after
                      // client is authenticated by server.
                      ChannelPipeline pipeline = Channels.pipeline(
                          byteCounter);
            
                      // After authentication finishes, the following is replaced with
                      // FixedLengthFrameDecoder (as in non-auth pipeline below):
                      pipeline.addLast("length-field-based-frame-decoder",
                          new LengthFieldBasedFrameDecoder(1024, 0, 4, 0, 4));
            
                      pipeline.addLast("request-encoder", new RequestEncoder());
            
                      // After authentication finishes, the following is removed:
                      pipeline.addLast("sasl-client-handler",
                          new SaslClientHandler(conf));
            
                      pipeline.addLast("response-handler",
                          new ResponseClientHandler(clientRequestIdRequestInfoMap, conf));
                      return pipeline;
                    } else {
                      LOG.info("Using Netty without authentication.");
                      return Channels.pipeline(
                          byteCounter,
                          new FixedLengthFrameDecoder(RequestServerHandler.RESPONSE_BYTES),
                          new RequestEncoder(),
                          new ResponseClientHandler(clientRequestIdRequestInfoMap, conf));
                    }
                  }
                });
            
            ekoontz Eugene Joseph Koontz added a comment - My apologies for the delay. I am in the final stages : getting it to compile on all supported Hadoop versions and fixing checkstyle errors. I have the SASL code implemented as a ChannelHandler as we discussed. Here's what the pipeline looks like on the client side, showing both authenticated and non-authenticated cases: NettyClient.java // Set up the pipeline factory. bootstrap.setPipelineFactory( new ChannelPipelineFactory() { @Override public ChannelPipeline getPipeline() throws Exception { if (conf.getBoolean(GiraphJob.AUTHENTICATE, GiraphJob.DEFAULT_AUTHENTICATE)) { LOG.info( "Using Netty with authentication." ); // Our pipeline starts with just byteCounter, and then we use // addAfter() to incrementally add pipeline elements, so that we can // name them for identification for removal or replacement after // client is authenticated by server. ChannelPipeline pipeline = Channels.pipeline( byteCounter); // After authentication finishes, the following is replaced with // FixedLengthFrameDecoder (as in non-auth pipeline below): pipeline.addLast( "length-field-based-frame-decoder" , new LengthFieldBasedFrameDecoder(1024, 0, 4, 0, 4)); pipeline.addLast( "request-encoder" , new RequestEncoder()); // After authentication finishes, the following is removed: pipeline.addLast( "sasl-client-handler" , new SaslClientHandler(conf)); pipeline.addLast( "response-handler" , new ResponseClientHandler(clientRequestIdRequestInfoMap, conf)); return pipeline; } else { LOG.info( "Using Netty without authentication." ); return Channels.pipeline( byteCounter, new FixedLengthFrameDecoder(RequestServerHandler.RESPONSE_BYTES), new RequestEncoder(), new ResponseClientHandler(clientRequestIdRequestInfoMap, conf)); } } });

            Passes mvn -Phadoop_trunk verify - other Hadoops coming..

            ekoontz Eugene Joseph Koontz added a comment - Passes mvn -Phadoop_trunk verify - other Hadoops coming..
            aching Avery Ching added a comment - - edited

            Eugene, this looks nice! I still see some checkstyle stuff but you're working through it I guess. Overall, the design is good. If we think about improvements we can make them later on.

            Questions/Comments:

            For consistency, can you please convert multi-line comments like

            /** Whether to use SASL with DIGEST and Hadoop Job Tokens to authenticate	
             * and authorize Netty BSP Clients to Servers. */
            

            to

            /** 
             * Whether to use SASL with DIGEST and Hadoop Job Tokens to authenticate	
             * and authorize Netty BSP Clients to Servers. 
             */
            

            Can we get rename Authorize to AuthorizeServerHandler or something else more descriptive?

            NettyClient.java

            • 372: Please wrap the LOG.info() with if (LOG.isInfoEnabled()).
            • 545-557: Can't we just go through the regular netty request part of the code? We don't need to have -2 here and can just submit the destWorkerId?

            SASL_COMPLETE -> SASL_COMPLETE_REQUEST?

            SaslTokenMessage.java can we call is SaslTokenMessageRequest?

            SaslComplete.java can we call it SaslCompleteRequest to match the other names?

            SaslComplete.java

            • 29-34: Why not get rid of these?

            SaslTokenMessage.java:

            • 86: Extra line

            Thanks again, this was a lot of work!

            aching Avery Ching added a comment - - edited Eugene, this looks nice! I still see some checkstyle stuff but you're working through it I guess. Overall, the design is good. If we think about improvements we can make them later on. Questions/Comments: For consistency, can you please convert multi-line comments like /** Whether to use SASL with DIGEST and Hadoop Job Tokens to authenticate * and authorize Netty BSP Clients to Servers. */ to /** * Whether to use SASL with DIGEST and Hadoop Job Tokens to authenticate * and authorize Netty BSP Clients to Servers. */ Can we get rename Authorize to AuthorizeServerHandler or something else more descriptive? NettyClient.java 372: Please wrap the LOG.info() with if (LOG.isInfoEnabled()). 545-557: Can't we just go through the regular netty request part of the code? We don't need to have -2 here and can just submit the destWorkerId? SASL_COMPLETE -> SASL_COMPLETE_REQUEST? SaslTokenMessage.java can we call is SaslTokenMessageRequest? SaslComplete.java can we call it SaslCompleteRequest to match the other names? SaslComplete.java 29-34: Why not get rid of these? SaslTokenMessage.java: 86: Extra line Thanks again, this was a lot of work!

            Hi Avery,
            Thanks a lot for your comments - are you sure about checkstyle? I thought I got it down to zero errors. Will look into it and reply to your other comments.

            -Eugene

            ekoontz Eugene Joseph Koontz added a comment - Hi Avery, Thanks a lot for your comments - are you sure about checkstyle? I thought I got it down to zero errors. Will look into it and reply to your other comments. -Eugene

            Updated rb; please feel free to review here:

            https://reviews.apache.org/r/6609/diff/6/

            ekoontz Eugene Joseph Koontz added a comment - Updated rb; please feel free to review here: https://reviews.apache.org/r/6609/diff/6/
            giraphqa Giraph QA added a comment -

            -1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12545806/GIRAPH-211.patch
            against trunk revision 1387840.

            +1 @author. The patch does not contain any @author tags.

            -1 tests included. The patch doesn't appear to include any new or modified tests.
            Please justify why no new tests are needed for this patch.
            Also please list what manual steps were performed to verify this patch.

            -1 javac. The patch appears to cause the build to fail.

            Console output: https://builds.apache.org/job/PreCommit-GIRAPH-Build/41//console

            This message is automatically generated.

            giraphqa Giraph QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12545806/GIRAPH-211.patch against trunk revision 1387840. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javac. The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-GIRAPH-Build/41//console This message is automatically generated.
            aching Avery Ching added a comment - - edited

            Eugene, version 6 of the diff seems to be the same as the one I reviewed. Can you please address my comments above in https://issues.apache.org/jira/browse/GIRAPH-211?focusedCommentId=13459279&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13459279 ?

            aching Avery Ching added a comment - - edited Eugene, version 6 of the diff seems to be the same as the one I reviewed. Can you please address my comments above in https://issues.apache.org/jira/browse/GIRAPH-211?focusedCommentId=13459279&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13459279 ?
            aching Avery Ching added a comment -

            Let me know if you need any help on this one. I'm looking forward into putting this in!

            aching Avery Ching added a comment - Let me know if you need any help on this one. I'm looking forward into putting this in!

            Thanks Avery. I appreciate your help. I am hoping to 1) address your comments and 2) get mvn verify to work with all profiles in a new patch in the next 2 days.
            -Eugene

            ekoontz Eugene Joseph Koontz added a comment - Thanks Avery. I appreciate your help. I am hoping to 1) address your comments and 2) get mvn verify to work with all profiles in a new patch in the next 2 days. -Eugene
            aching Avery Ching added a comment -

            Any luck with this Eugene Joseph Koontz?

            aching Avery Ching added a comment - Any luck with this Eugene Joseph Koontz ?

            Getting closer.

            -Passes:

            mvn -Phadoop_non_secure clean verify &&
            mvn -Phadoop_1.0 clean verify && 
            mvn -Phadoop_2.0.0 clean verify && 
            mvn -Phadoop_2.0.1 clean verify && 
            mvn -Phadoop_2.0.2 clean verify
            

            (I added profiles for hadoop 2.0.1 and 2.0.2 since there are
            releases available for these newer Hadoop versions).

            -Addressed most of Avery's comments on reviewboard - thank you Avery. Still,
            SaslTokenMessageRequests do not properly pass the
            destWorkerId (it still uses -2 as the destWorkerId). Working on addressing this.

            -Added new test, SaslConnectionTest, adapted from ConnectionTest. Passes,
            but needs to be expanded. I need to better understand how
            Mockito can be used to mock Job Tokens, so would appreciate help
            with how to use Mockito for this.

            -Incorporates GIRAPH-212's patch: this fixes munge flags associated with security.

            ekoontz Eugene Joseph Koontz added a comment - Getting closer. -Passes: mvn -Phadoop_non_secure clean verify && mvn -Phadoop_1.0 clean verify && mvn -Phadoop_2.0.0 clean verify && mvn -Phadoop_2.0.1 clean verify && mvn -Phadoop_2.0.2 clean verify (I added profiles for hadoop 2.0.1 and 2.0.2 since there are releases available for these newer Hadoop versions). -Addressed most of Avery's comments on reviewboard - thank you Avery. Still, SaslTokenMessageRequests do not properly pass the destWorkerId (it still uses -2 as the destWorkerId). Working on addressing this. -Added new test, SaslConnectionTest, adapted from ConnectionTest. Passes, but needs to be expanded. I need to better understand how Mockito can be used to mock Job Tokens, so would appreciate help with how to use Mockito for this. -Incorporates GIRAPH-212 's patch: this fixes munge flags associated with security.
            aching Avery Ching added a comment -

            We just committed GIRAPH-212, so only thing is to fix the -2 thing and add a test? I'd really like to get this in the next day or two. If you're busy Eugene Joseph Koontz, let me know and I can try and massage it a bit.

            aching Avery Ching added a comment - We just committed GIRAPH-212 , so only thing is to fix the -2 thing and add a test? I'd really like to get this in the next day or two. If you're busy Eugene Joseph Koontz , let me know and I can try and massage it a bit.

            Hi Avery, That's correct - we just need to fix the -2 thing and improve the test. I could use some help with either or both
            Thanks a lot.
            -Eugene

            ekoontz Eugene Joseph Koontz added a comment - Hi Avery, That's correct - we just need to fix the -2 thing and improve the test. I could use some help with either or both Thanks a lot. -Eugene

            Merged with trunk: no longer needs GIRAPH-212 inclusion thanks to Avery's commit of this.

            Small munge flag name change: changed HADOOP_1_AUTHORIZATION to HADOOP_1_SECURITY.

            ekoontz Eugene Joseph Koontz added a comment - Merged with trunk: no longer needs GIRAPH-212 inclusion thanks to Avery's commit of this. Small munge flag name change: changed HADOOP_1_AUTHORIZATION to HADOOP_1_SECURITY.
            aching Avery Ching added a comment -

            Here is an attached patch with the following changes from GIRAPH-211.patch:

            Addressed the -2 worker id by adding the task id to ChannelRotater.
            SaslComplete -> SaslCompleteRequest
            SASL_TOKEN_MESSAGE -> SASL_TOKEN_MESSAGE_REQUEST
            Fixed all minor things (javadoc, functions, added authenticate()).
            mvn -Phadoop=facebook compile fixed

            Tested with

            mvn -Phadoop_non_secure clean verify

            • passed
              mvn -Phadoop_1.0 clean verify
              = passed
              mvn -Phadoop_2.0.0 clean verify
            • passed
              mvn -Phadoop_2.0.1 clean verify
            • passed
              mvn -Phadoop=facebook compile
            • passed

            Note that hadoop-2.0.2 wasn't tested (I don't have the artifacts), but should work.

            I know that we should add more testing, but I think this is good to go for now. It seems to work and it's already very huge. Great work Eugene Joseph Koontz! Please let me know if the changes I made are acceptable, I'm fine with the current version. We should commit soon.

            aching Avery Ching added a comment - Here is an attached patch with the following changes from GIRAPH-211 .patch: Addressed the -2 worker id by adding the task id to ChannelRotater. SaslComplete -> SaslCompleteRequest SASL_TOKEN_MESSAGE -> SASL_TOKEN_MESSAGE_REQUEST Fixed all minor things (javadoc, functions, added authenticate()). mvn -Phadoop=facebook compile fixed Tested with mvn -Phadoop_non_secure clean verify passed mvn -Phadoop_1.0 clean verify = passed mvn -Phadoop_2.0.0 clean verify passed mvn -Phadoop_2.0.1 clean verify passed mvn -Phadoop=facebook compile passed Note that hadoop-2.0.2 wasn't tested (I don't have the artifacts), but should work. I know that we should add more testing, but I think this is good to go for now. It seems to work and it's already very huge. Great work Eugene Joseph Koontz ! Please let me know if the changes I made are acceptable, I'm fine with the current version. We should commit soon.
            aching Avery Ching added a comment -

            Rebased with trunk.

            aching Avery Ching added a comment - Rebased with trunk.

            Wow, thanks so much Avery. I'm going to try out your patch today. Will also try with hadoop-2.0.2.

            -Eugene

            ekoontz Eugene Joseph Koontz added a comment - Wow, thanks so much Avery. I'm going to try out your patch today. Will also try with hadoop-2.0.2. -Eugene
            aching Avery Ching added a comment -

            Thanks, let me know!

            aching Avery Ching added a comment - Thanks, let me know!
            aching Avery Ching added a comment -

            Rebased on GIRAPH-360.patch.

            aching Avery Ching added a comment - Rebased on GIRAPH-360 .patch.

            Avery, modified your patch to fix one problem: a NPE in NettyClient.sendWritableRequest() when the supplied destWorkerId param is null. This null param happens when requests are made to the master when sendWritableRequest() is called by NettyClient.authenticateOnChannel().

            With Avery's fix for GIRAPH-360, we associate each request with the taskId of the worker that is making the request, and use a pair <taskid,requestid> as the key for tracking requests in the clientRequestIdRequestInfoMap. However, the Giraph master is not associated with a taskId, and so its taskid is null.

            It wouldn't be a problem that the master has a null task id for normal Giraph requests, but for authentication, workers need to authenticate with the master just as they must do with the other workers.

            This patch works around the problem by simply not registering authentication requests (requests with type=SASL_TOKEN_MESSAGE_REQUEST) with the clientRequestIdRequestInfoMap. In other words, it's another workaround like the "-2" presence in my previous patches. I know this is not ideal, but I wanted to keep the momentum going for this JIRA.

            Rather than this workaround, I thought a possible solution would be a separate client-side pipeline component that handles request tracking. We'd move the clientRequestIdRequestInfoMap into this component, which would handles the tracking functionality that currently is spread across NettyClient.sendWritableRequest(), .waitAllRequests(), and ResponseClientHandler.messageReceived().

            Or, maybe I'm missing something obvious and easy

            Tested this patch successfully with:

            mvn -Phadoop_non_secure clean verify && mvn -Phadoop_0.20.203 clean verify && mvn -Phadoop_1.0 clean verify && mvn -Phadoop0.23 clean verify && mvn -Phadoop_2.0.0 clean verify && mvn -Phadoop_2.0.1 clean verify && mvn -Phadoop_2.0.2 clean verify
            
            ekoontz Eugene Joseph Koontz added a comment - Avery, modified your patch to fix one problem: a NPE in NettyClient.sendWritableRequest() when the supplied destWorkerId param is null. This null param happens when requests are made to the master when sendWritableRequest() is called by NettyClient.authenticateOnChannel(). With Avery's fix for GIRAPH-360 , we associate each request with the taskId of the worker that is making the request, and use a pair <taskid,requestid> as the key for tracking requests in the clientRequestIdRequestInfoMap. However, the Giraph master is not associated with a taskId, and so its taskid is null. It wouldn't be a problem that the master has a null task id for normal Giraph requests, but for authentication, workers need to authenticate with the master just as they must do with the other workers. This patch works around the problem by simply not registering authentication requests (requests with type=SASL_TOKEN_MESSAGE_REQUEST) with the clientRequestIdRequestInfoMap. In other words, it's another workaround like the "-2" presence in my previous patches. I know this is not ideal, but I wanted to keep the momentum going for this JIRA. Rather than this workaround, I thought a possible solution would be a separate client-side pipeline component that handles request tracking. We'd move the clientRequestIdRequestInfoMap into this component, which would handles the tracking functionality that currently is spread across NettyClient.sendWritableRequest(), .waitAllRequests(), and ResponseClientHandler.messageReceived(). Or, maybe I'm missing something obvious and easy Tested this patch successfully with: mvn -Phadoop_non_secure clean verify && mvn -Phadoop_0.20.203 clean verify && mvn -Phadoop_1.0 clean verify && mvn -Phadoop0.23 clean verify && mvn -Phadoop_2.0.0 clean verify && mvn -Phadoop_2.0.1 clean verify && mvn -Phadoop_2.0.2 clean verify
            aching Avery Ching added a comment -

            +1, will also create a follow up task to fix up this weirdness with the master. Thanks for all the hard work Eugene!

            aching Avery Ching added a comment - +1, will also create a follow up task to fix up this weirdness with the master. Thanks for all the hard work Eugene!

            Same as last patch except adds what I mistakenly left out of last patch:

            src/main/java/org/apache/giraph/comm/netty/handler/AuthorizeServerHandler.java
            src/main/java/org/apache/giraph/comm/netty/handler/ResponseEncoder.java

            ekoontz Eugene Joseph Koontz added a comment - Same as last patch except adds what I mistakenly left out of last patch: src/main/java/org/apache/giraph/comm/netty/handler/AuthorizeServerHandler.java src/main/java/org/apache/giraph/comm/netty/handler/ResponseEncoder.java
            hudson Hudson added a comment -

            Integrated in Giraph-trunk-Commit #233 (See https://builds.apache.org/job/Giraph-trunk-Commit/233/)
            GIRAPH-211: Add secure authentication to Netty IPC (Revision 1396722)

            Result = SUCCESS
            ekoontz : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1396722
            Files :

            • /giraph/trunk/pom.xml
            • /giraph/trunk/src/main/java/org/apache/giraph/GiraphConfiguration.java
            • /giraph/trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java
            • /giraph/trunk/src/main/java/org/apache/giraph/comm/CommunicationsInterface.java
            • /giraph/trunk/src/main/java/org/apache/giraph/comm/SecureRPCCommunications.java
            • /giraph/trunk/src/main/java/org/apache/giraph/comm/WorkerClient.java
            • /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/NettyClient.java
            • /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/NettyServer.java
            • /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClient.java
            • /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClientServer.java
            • /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/SaslNettyClient.java
            • /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/SaslNettyServer.java
            • /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/handler/AuthorizeServerHandler.java
            • /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/handler/RequestEncoder.java
            • /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/handler/ResponseEncoder.java
            • /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/handler/SaslClientHandler.java
            • /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/handler/SaslServerHandler.java
            • /giraph/trunk/src/main/java/org/apache/giraph/comm/requests/RequestType.java
            • /giraph/trunk/src/main/java/org/apache/giraph/comm/requests/SaslCompleteRequest.java
            • /giraph/trunk/src/main/java/org/apache/giraph/comm/requests/SaslTokenMessageRequest.java
            • /giraph/trunk/src/main/java/org/apache/giraph/graph/BspServiceWorker.java
            • /giraph/trunk/src/test/java/org/apache/giraph/comm/SaslConnectionTest.java
            hudson Hudson added a comment - Integrated in Giraph-trunk-Commit #233 (See https://builds.apache.org/job/Giraph-trunk-Commit/233/ ) GIRAPH-211 : Add secure authentication to Netty IPC (Revision 1396722) Result = SUCCESS ekoontz : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1396722 Files : /giraph/trunk/pom.xml /giraph/trunk/src/main/java/org/apache/giraph/GiraphConfiguration.java /giraph/trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java /giraph/trunk/src/main/java/org/apache/giraph/comm/CommunicationsInterface.java /giraph/trunk/src/main/java/org/apache/giraph/comm/SecureRPCCommunications.java /giraph/trunk/src/main/java/org/apache/giraph/comm/WorkerClient.java /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/NettyClient.java /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/NettyServer.java /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClient.java /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClientServer.java /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/SaslNettyClient.java /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/SaslNettyServer.java /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/handler/AuthorizeServerHandler.java /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/handler/RequestEncoder.java /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/handler/ResponseEncoder.java /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/handler/SaslClientHandler.java /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/handler/SaslServerHandler.java /giraph/trunk/src/main/java/org/apache/giraph/comm/requests/RequestType.java /giraph/trunk/src/main/java/org/apache/giraph/comm/requests/SaslCompleteRequest.java /giraph/trunk/src/main/java/org/apache/giraph/comm/requests/SaslTokenMessageRequest.java /giraph/trunk/src/main/java/org/apache/giraph/graph/BspServiceWorker.java /giraph/trunk/src/test/java/org/apache/giraph/comm/SaslConnectionTest.java
            hudson Hudson added a comment -

            Integrated in Giraph-trunk-Commit #234 (See https://builds.apache.org/job/Giraph-trunk-Commit/234/)
            GIRAPH-363: Fix hadoop_0.23 profile broken by GIRAPH-211 (Revision 1396858)

            Result = SUCCESS
            ekoontz : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1396858
            Files :

            • /giraph/trunk/CHANGELOG
            • /giraph/trunk/pom.xml
            • /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/SaslNettyServer.java
            hudson Hudson added a comment - Integrated in Giraph-trunk-Commit #234 (See https://builds.apache.org/job/Giraph-trunk-Commit/234/ ) GIRAPH-363 : Fix hadoop_0.23 profile broken by GIRAPH-211 (Revision 1396858) Result = SUCCESS ekoontz : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1396858 Files : /giraph/trunk/CHANGELOG /giraph/trunk/pom.xml /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/SaslNettyServer.java

            HADOOP-8078 starts an ApacheDS KDC server for use in testing Kerberos integration - hopefully this could also be used to create unit tests for GIRAPH-211.

            ekoontz Eugene Joseph Koontz added a comment - HADOOP-8078 starts an ApacheDS KDC server for use in testing Kerberos integration - hopefully this could also be used to create unit tests for GIRAPH-211 .

            People

              ekoontz Eugene Joseph Koontz
              ekoontz Eugene Joseph Koontz
              Votes:
              3 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Slack