Uploaded image for project: 'Giraph'
  1. Giraph
  2. GIRAPH-211

Add secure authentication to Netty IPC

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.0.0
    • Component/s: None
    • Labels:
      None

      Description

      Gianmarco De Francisci Morales asked on the user list:

      I am getting the exception in the subject when running my giraph program

      on a cluster with Kerberos authentication.

      This leads to the idea of having Kerberos authentication supported within GIRAPH. Hopefully it would use our fast GIRAPH-37 IPC, but could also interoperate with Hadoop security.

      1. GIRAPH-211-proposal.txt
        8 kB
        Eugene Koontz
      2. GIRAPH-211.patch
        105 kB
        Eugene Koontz
      3. GIRAPH-211.patch
        110 kB
        Eugene Koontz
      4. GIRAPH-211.patch
        52 kB
        Eugene Koontz
      5. GIRAPH-211.patch
        85 kB
        Eugene Koontz
      6. GIRAPH-211.patch
        81 kB
        Eugene Koontz
      7. GIRAPH-211.patch
        84 kB
        Eugene Koontz
      8. GIRAPH-211.patch
        69 kB
        Eugene Koontz
      9. GIRAPH-211.patch
        105 kB
        Eugene Koontz
      10. GIRAPH-211.patch
        87 kB
        Eugene Koontz
      11. GIRAPH-211.patch
        84 kB
        Eugene Koontz
      12. GIRAPH-211.patch
        91 kB
        Eugene Koontz
      13. GIRAPH-211.4.patch
        91 kB
        Avery Ching
      14. GIRAPH-211.3.patch
        104 kB
        Avery Ching
      15. GIRAPH-211.2.patch
        104 kB
        Avery Ching

        Issue Links

          Activity

          Hide
          majakabiljo Maja Kabiljo added a comment -

          I talked about this to Avery, he suggested using just Netty SSL. How would you suggest that we assign certificates/keys to workers?

          Show
          majakabiljo Maja Kabiljo added a comment - I talked about this to Avery, he suggested using just Netty SSL. How would you suggest that we assign certificates/keys to workers?
          Hide
          jghoman Jakob Homan added a comment -

          Certificates/keys don't scale. It would be better to piggyback on Hadoop's delegation token system, certainly while we're on HMR and even later, when we're on YARN.

          Show
          jghoman Jakob Homan added a comment - Certificates/keys don't scale. It would be better to piggyback on Hadoop's delegation token system, certainly while we're on HMR and even later, when we're on YARN.
          Hide
          apurtell Andrew Purtell added a comment -

          Interoperability with Hadoop security is a prerequisite for any use of Giraph for our use cases.

          Show
          apurtell Andrew Purtell added a comment - Interoperability with Hadoop security is a prerequisite for any use of Giraph for our use cases.
          Hide
          aching Avery Ching added a comment -

          Okay, I suppose we can use the BspPolicyProvider, JobTorkenIdentifier, JobTokenSecretManager, etc. to build a light layer of security in Netty. Maja, you can start looking at RPCCommunications.java to see how feasible it is to use the same ideas from the RPC implementation to the Netty implementation.

          Show
          aching Avery Ching added a comment - Okay, I suppose we can use the BspPolicyProvider, JobTorkenIdentifier, JobTokenSecretManager, etc. to build a light layer of security in Netty. Maja, you can start looking at RPCCommunications.java to see how feasible it is to use the same ideas from the RPC implementation to the Netty implementation.
          Hide
          majakabiljo Maja Kabiljo added a comment -

          I'm not sure what am I supposed to do here. My first understanding was that we want all the messages which go through the system to be protected. In that case I can't see how workers can find out about tokens of others. I guess RPC authentication is implemented in hadoop, since the only thing I see in RPC classes in Giraph is the part when the job checks whether it has privileges to make a connection. I need a bit more information here, or I can reassign the task if someone finds it easier to do it than to explain it

          Show
          majakabiljo Maja Kabiljo added a comment - I'm not sure what am I supposed to do here. My first understanding was that we want all the messages which go through the system to be protected. In that case I can't see how workers can find out about tokens of others. I guess RPC authentication is implemented in hadoop, since the only thing I see in RPC classes in Giraph is the part when the job checks whether it has privileges to make a connection. I need a bit more information here, or I can reassign the task if someone finds it easier to do it than to explain it
          Hide
          apurtell Andrew Purtell added a comment -

          My first understanding was that we want all the messages which go through the system to be protected.

          What Hadoop did fundamentally is wrap their RPC with SASL at the socket level, and then could use existing JRE support for SASL negotiation with Kerberos authentication (and transparent encryption, etc).

          Eugene could comment better, but what we did for ZooKeeper, which has IO also based on Netty, is instead tunnel the SASL authentication handshake as an extension to the existing protocol, and introduced a mode which requires that handshake to complete successfully before accepting other message types which require authenticated access.

          Show
          apurtell Andrew Purtell added a comment - My first understanding was that we want all the messages which go through the system to be protected. What Hadoop did fundamentally is wrap their RPC with SASL at the socket level, and then could use existing JRE support for SASL negotiation with Kerberos authentication (and transparent encryption, etc). Eugene could comment better, but what we did for ZooKeeper, which has IO also based on Netty, is instead tunnel the SASL authentication handshake as an extension to the existing protocol, and introduced a mode which requires that handshake to complete successfully before accepting other message types which require authenticated access.
          Hide
          jghoman Jakob Homan added a comment -

          This JIRA is going to require a pretty good understanding of Hadoop's security. It may not be the best for a new contributor...

          Show
          jghoman Jakob Homan added a comment - This JIRA is going to require a pretty good understanding of Hadoop's security. It may not be the best for a new contributor...
          Hide
          ekoontz Eugene Koontz added a comment -

          I'm happy to help Maja or take over this JIRA - either way I'll be following it closely! Maybe a hackathon or meetup would be a good idea.

          Show
          ekoontz Eugene Koontz added a comment - I'm happy to help Maja or take over this JIRA - either way I'll be following it closely! Maybe a hackathon or meetup would be a good idea.
          Hide
          ekoontz Eugene Koontz added a comment -

          I should mention that Zookeeper supports both NIO and Netty on the server, but there is currently no Netty implementation on the Zookeeper client side: see https://issues.apache.org/jira/browse/ZOOKEEPER-823 (which is not resolved at this time).

          Show
          ekoontz Eugene Koontz added a comment - I should mention that Zookeeper supports both NIO and Netty on the server, but there is currently no Netty implementation on the Zookeeper client side: see https://issues.apache.org/jira/browse/ZOOKEEPER-823 (which is not resolved at this time).
          Hide
          majakabiljo Maja Kabiljo added a comment -

          Eugene, I'm reassigning it to you. Really appreciate your offer to help, but if it requires a lot of background and we need it fast it's better this way.

          Show
          majakabiljo Maja Kabiljo added a comment - Eugene, I'm reassigning it to you. Really appreciate your offer to help, but if it requires a lot of background and we need it fast it's better this way.
          Hide
          ekoontz Eugene Koontz added a comment -

          Sounds fine, Maja.

          Show
          ekoontz Eugene Koontz added a comment - Sounds fine, Maja.
          Hide
          ekoontz Eugene Koontz added a comment -

          I am working on a patch for this after studying both the Hadoop RPC side and the Netty side of things. I'm posting a proposal, but, since it's long, I'll attach it as a .txt. But to get the discussion started, I want to describe the idea briefly: it uses the Job Token as a SASL credential so that one BSPWorker can authenticate itself as a client to another BSPWorker acting as a server. This is what is going on with Hadoop RPC, so we just need to replicate this behavior in the Netty implementation.

          Show
          ekoontz Eugene Koontz added a comment - I am working on a patch for this after studying both the Hadoop RPC side and the Netty side of things. I'm posting a proposal, but, since it's long, I'll attach it as a .txt. But to get the discussion started, I want to describe the idea briefly: it uses the Job Token as a SASL credential so that one BSPWorker can authenticate itself as a client to another BSPWorker acting as a server. This is what is going on with Hadoop RPC, so we just need to replicate this behavior in the Netty implementation.
          Hide
          aching Avery Ching added a comment -

          Thanks for working on this Eugene. Can't wait to get rid of HadoopRPC so we only have to support one IPC instead of two.

          Show
          aching Avery Ching added a comment - Thanks for working on this Eugene. Can't wait to get rid of HadoopRPC so we only have to support one IPC instead of two.
          Hide
          aching Avery Ching added a comment -

          Hey Eugene, your proposal look good btw. Let me know if you need any help doing this. I'd like to get it in asap as this is causing a lot of cruft to build up. We don't need to do the ZK security for now to remove HadoopRPC.

          Show
          aching Avery Ching added a comment - Hey Eugene, your proposal look good btw. Let me know if you need any help doing this. I'd like to get it in asap as this is causing a lot of cruft to build up. We don't need to do the ZK security for now to remove HadoopRPC.
          Hide
          ekoontz Eugene Koontz added a comment -

          Thanks for reading it, Avery. I will have a patch tomorrow.

          Show
          ekoontz Eugene Koontz added a comment - Thanks for reading it, Avery. I will have a patch tomorrow.
          Hide
          aching Avery Ching added a comment -

          Sounds great. I was thinking about basing it on the SecureShuffleUtils.java and ReduceTask.java, but you seem to have a more complex idea. Looking forward to it.

          Show
          aching Avery Ching added a comment - Sounds great. I was thinking about basing it on the SecureShuffleUtils.java and ReduceTask.java, but you seem to have a more complex idea. Looking forward to it.
          Hide
          ekoontz Eugene Koontz added a comment -

          This patch allows SASL authentication using the JobToken, as described in

          https://issues.apache.org/jira/secure/attachment/12537279/GIRAPH-211-proposal.txt.

          Limitations (will follow up with new JIRAs as warranted)

          -Authorization is not done: that is, clients are authenticated but there are no restrictions on their ability to do RPC on the servers.

          -Clients should wait for authentication before trying to do RPC - once authorization (see above) is done, they might encounter a race where they try to do RPCs without yet being authenticated.

          -Not tested on other than hadoop 2.0.1-SNAPSHOT

          -Only works if we disable client-side channel-pooling (GIRAPH-289) and local short-circuiting of RPCs (GIRAPH-262) - these should be configurable but currently, I hard-wired both to be disabled.

          How to compile:

          mvn -Phadoop_2.0.1 clean test

          Works with the following test:

          $HADOOP_RUNTIME/bin/hadoop jar $GIRAPH_DIR/target/giraph-0.2-SNAPSHOT-for-hadoop-2.0.1-SNAPSHOT-jar-with-dependencies.jar org.apache.giraph.benchmark.PageRankBenchmark -Dgiraph.useNetty=true -e 2 -s 10 -v -V 2 -w 2

          Show
          ekoontz Eugene Koontz added a comment - This patch allows SASL authentication using the JobToken, as described in https://issues.apache.org/jira/secure/attachment/12537279/GIRAPH-211-proposal.txt . Limitations (will follow up with new JIRAs as warranted) -Authorization is not done: that is, clients are authenticated but there are no restrictions on their ability to do RPC on the servers. -Clients should wait for authentication before trying to do RPC - once authorization (see above) is done, they might encounter a race where they try to do RPCs without yet being authenticated. -Not tested on other than hadoop 2.0.1-SNAPSHOT -Only works if we disable client-side channel-pooling ( GIRAPH-289 ) and local short-circuiting of RPCs ( GIRAPH-262 ) - these should be configurable but currently, I hard-wired both to be disabled. How to compile: mvn -Phadoop_2.0.1 clean test Works with the following test: $HADOOP_RUNTIME/bin/hadoop jar $GIRAPH_DIR/target/giraph-0.2-SNAPSHOT-for-hadoop-2.0.1-SNAPSHOT-jar-with-dependencies.jar org.apache.giraph.benchmark.PageRankBenchmark -Dgiraph.useNetty=true -e 2 -s 10 -v -V 2 -w 2
          Hide
          ekoontz Eugene Koontz added a comment -

          Here's some output showing client and server SASL communication based on the above test:

          [ec2-user@ip-10-176-255-76 giraph]$ cat /tmp/logs/application_1344445498302_0016/container_1344445498302_0016_01_000002/syslog | grep -i giraph | grep -i sasl
          2012-08-08 20:58:52,806 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.NettyClient: creating saslNettyClient now.
          2012-08-08 20:58:52,816 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.NettyClient: creating saslNettyClient now.
          2012-08-08 20:58:52,929 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslNettyClient: Creating SASL DIGEST-MD5 client to authenticate to service at job_1344445498302_0016
          2012-08-08 20:58:52,931 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslNettyClient: Creating SASL DIGEST-MD5 client to authenticate to service at job_1344445498302_0016
          2012-08-08 20:58:52,943 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.NettyClient: storing saslNettyClient at key: [id: 0x7e8905bd, /10.176.255.76:48238 => ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30000]
          2012-08-08 20:58:52,943 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.NettyClient: created: org.apache.giraph.comm.SaslNettyClient@2aa937cd
          2012-08-08 20:58:52,943 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.NettyClient: sending request: org.apache.giraph.comm.SaslTokenMessage@4b8e899c to remote server: ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30000
          2012-08-08 20:58:52,943 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.RequestEncoder: encoding object: org.apache.giraph.comm.SaslTokenMessage@4b8e899c
          2012-08-08 20:58:52,947 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: write() called on token of length: 0
          2012-08-08 20:58:52,947 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: write() finished writing token.
          2012-08-08 20:58:52,947 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.RequestEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslTokenMessage
          2012-08-08 20:58:52,954 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.NettyClient: storing saslNettyClient at key: [id: 0x04df8b14, /10.176.255.76:60743 => ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30001]
          2012-08-08 20:58:52,954 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.NettyClient: created: org.apache.giraph.comm.SaslNettyClient@23faa614
          2012-08-08 20:58:52,954 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.NettyClient: sending request: org.apache.giraph.comm.SaslTokenMessage@1cad7d80 to remote server: ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30001
          2012-08-08 20:58:52,954 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.RequestEncoder: encoding object: org.apache.giraph.comm.SaslTokenMessage@1cad7d80
          2012-08-08 20:58:52,954 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: write() called on token of length: 0
          2012-08-08 20:58:52,954 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: write() finished writing token.
          2012-08-08 20:58:52,954 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.RequestEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslTokenMessage
          2012-08-08 20:58:52,959 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestDecoder: decode: Got a request of type SASL_TOKEN from remote:/10.176.255.76:48238
          2012-08-08 20:58:52,959 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_TOKEN
          2012-08-08 20:58:52,959 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: reading fields from DataInput: org.jboss.netty.buffer.ChannelBufferInputStream@2b8ca663
          2012-08-08 20:58:52,959 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: token size is: 0
          2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: now reading token.
          2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: read SASL token of length: 0
          2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestDecoder: decoded an object of type: class org.apache.giraph.comm.SaslTokenMessage:org.apache.giraph.comm.SaslTokenMessage@1effc3eb
          2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestServerHandler: Got class org.apache.giraph.comm.SaslTokenMessage
          2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestServerHandler: calling doRequest on class: class org.apache.giraph.comm.SaslTokenMessage
          2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: looking for saslNettyServer on server:/10.176.255.76:30000 for client /10.176.255.76:48238
          2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: creating SaslNettyServer with secret manager: org.apache.hadoop.mapreduce.security.token.JobTokenSecretManager@658782a7
          2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: creating saslNettyServer: secret manager is: org.apache.hadoop.mapreduce.security.token.JobTokenSecretManager@658782a7
          2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: checked read-availability; now creating sasl server.
          2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: create...
          2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: creating callback handler..
          2012-08-08 20:58:52,961 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: creating SaslDigestCallback handler with secret manager: org.apache.hadoop.mapreduce.security.token.JobTokenSecretManager@658782a7
          2012-08-08 20:58:52,961 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: created callback handler: org.apache.giraph.comm.SaslNettyServer$SaslDigestCallbackHandler@1bb0889a
          2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: create was successful.
          2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: client's token's length is: 0
          2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: checking completeness of SASL (pre-response):false
          2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: responding to input token of length: 0
          2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: response token length: 108
          2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: checking completeness of SASL (post-response):false
          2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: server's token's length is: 108
          2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: Responding to client: /10.176.255.76:48238
          2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.ResponseEncoder: RESPONSE ENCODER IS ENCODING A MESSAGE:org.apache.giraph.comm.SaslTokenMessage@1effc3eb
          2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.ResponseEncoder: encoding object: org.apache.giraph.comm.SaslTokenMessage@1effc3eb
          2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.ResponseEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslTokenMessage
          2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: write() called on token of length: 108
          2012-08-08 20:58:52,968 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: write() finished writing token.
          2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: decode: Got a response of type SASL_TOKEN from server:ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30000
          2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_TOKEN
          2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: writableRequestClass: class org.apache.giraph.comm.SaslTokenMessage found for type:SASL_TOKEN
          2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: reading fields of server response:org.apache.giraph.comm.SaslTokenMessage@48c5186e
          2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: reading fields from DataInput: org.jboss.netty.buffer.ChannelBufferInputStream@3c70315
          2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: token size is: 108
          2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: now reading token.
          2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: read SASL token of length: 108
          2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: finished reading fields of server response:org.apache.giraph.comm.SaslTokenMessage@48c5186e
          2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseClientHandler: message received: org.apache.giraph.comm.SaslTokenMessage@48c5186e
          2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslNettyClient: SASL client callback: setting username: FmpvYl8xMzQ0NDQ1NDk4MzAyXzAwMTY=
          2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslNettyClient: SASL client callback: setting userPassword
          2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslNettyClient: SASL client callback: setting realm: default
          2012-08-08 20:58:52,969 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: creating SASL message with size: 274
          2012-08-08 20:58:52,969 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.NettyClient: sending request: org.apache.giraph.comm.SaslTokenMessage@7e9b59a2 to remote server: ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30000
          2012-08-08 20:58:52,969 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.RequestEncoder: encoding object: org.apache.giraph.comm.SaslTokenMessage@7e9b59a2
          2012-08-08 20:58:52,969 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: write() called on token of length: 274
          2012-08-08 20:58:52,969 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: write() finished writing token.
          2012-08-08 20:58:52,969 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.RequestEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslTokenMessage
          2012-08-08 20:58:52,969 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseClientHandler: due to reception of: org.apache.giraph.comm.SaslTokenMessage@48c5186e
          2012-08-08 20:58:52,970 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: Finished writing token to client.
          2012-08-08 20:58:52,970 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestServerHandler: done with doRequest() for class: class org.apache.giraph.comm.SaslTokenMessage
          2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestDecoder: decode: Got a request of type SASL_TOKEN from remote:/10.176.255.76:48238
          2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_TOKEN
          2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: reading fields from DataInput: org.jboss.netty.buffer.ChannelBufferInputStream@919db9e
          2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: token size is: 274
          2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: now reading token.
          2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: read SASL token of length: 274
          2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestDecoder: decoded an object of type: class org.apache.giraph.comm.SaslTokenMessage:org.apache.giraph.comm.SaslTokenMessage@a166bd
          2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestServerHandler: Got class org.apache.giraph.comm.SaslTokenMessage
          2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestServerHandler: calling doRequest on class: class org.apache.giraph.comm.SaslTokenMessage
          2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: looking for saslNettyServer on server:/10.176.255.76:30000 for client /10.176.255.76:48238
          2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: client's token's length is: 274
          2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: checking completeness of SASL (pre-response):false
          2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: responding to input token of length: 274
          2012-08-08 20:58:52,981 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: SASL server DIGEST-MD5 callback: setting password for client: job_1344445498302_0016 (auth:SIMPLE)
          2012-08-08 20:58:52,981 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: SASL server DIGEST-MD5 callback: setting canonicalized client ID: job_1344445498302_0016
          2012-08-08 20:58:52,982 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: response token length: 40
          2012-08-08 20:58:52,982 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: checking completeness of SASL (post-response):true
          2012-08-08 20:58:52,982 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: server's token's length is: 40
          2012-08-08 20:58:52,982 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: Responding to client: /10.176.255.76:48238
          2012-08-08 20:58:52,982 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.ResponseEncoder: RESPONSE ENCODER IS ENCODING A MESSAGE:org.apache.giraph.comm.SaslTokenMessage@a166bd
          2012-08-08 20:58:52,982 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.ResponseEncoder: encoding object: org.apache.giraph.comm.SaslTokenMessage@a166bd
          2012-08-08 20:58:52,982 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.ResponseEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslTokenMessage
          2012-08-08 20:58:52,982 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: write() called on token of length: 40
          2012-08-08 20:58:52,982 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: write() finished writing token.
          2012-08-08 20:58:52,982 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: decode: Got a response of type SASL_TOKEN from server:ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30000
          2012-08-08 20:58:52,982 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_TOKEN
          2012-08-08 20:58:52,983 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: writableRequestClass: class org.apache.giraph.comm.SaslTokenMessage found for type:SASL_TOKEN
          2012-08-08 20:58:52,983 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: reading fields of server response:org.apache.giraph.comm.SaslTokenMessage@4b6c06dd
          2012-08-08 20:58:52,983 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: reading fields from DataInput: org.jboss.netty.buffer.ChannelBufferInputStream@2136bdda
          2012-08-08 20:58:52,983 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: token size is: 40
          2012-08-08 20:58:52,983 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: now reading token.
          2012-08-08 20:58:52,983 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: read SASL token of length: 40
          2012-08-08 20:58:52,983 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: finished reading fields of server response:org.apache.giraph.comm.SaslTokenMessage@4b6c06dd
          2012-08-08 20:58:52,983 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseClientHandler: message received: org.apache.giraph.comm.SaslTokenMessage@4b6c06dd
          2012-08-08 20:58:52,987 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: Finished writing token to client.
          2012-08-08 20:58:52,987 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: writing out complete token now.
          2012-08-08 20:58:52,987 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.ResponseEncoder: RESPONSE ENCODER IS ENCODING A MESSAGE:org.apache.giraph.comm.SaslComplete@36511e0a
          2012-08-08 20:58:52,987 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.ResponseEncoder: encoding object: org.apache.giraph.comm.SaslComplete@36511e0a
          2012-08-08 20:58:52,987 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.ResponseEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslComplete
          2012-08-08 20:58:52,987 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslComplete: writing one byte (write).
          2012-08-08 20:58:52,987 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: wrote out complete token now.
          2012-08-08 20:58:52,987 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestServerHandler: done with doRequest() for class: class org.apache.giraph.comm.SaslTokenMessage
          2012-08-08 20:58:53,009 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestDecoder: decode: Got a request of type SASL_TOKEN from remote:/10.176.255.76:48239
          2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_TOKEN
          2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: reading fields from DataInput: org.jboss.netty.buffer.ChannelBufferInputStream@595780d9
          2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: token size is: 0
          2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: now reading token.
          2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: read SASL token of length: 0
          2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestDecoder: decoded an object of type: class org.apache.giraph.comm.SaslTokenMessage:org.apache.giraph.comm.SaslTokenMessage@7fcc5461
          2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestServerHandler: Got class org.apache.giraph.comm.SaslTokenMessage
          2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestServerHandler: calling doRequest on class: class org.apache.giraph.comm.SaslTokenMessage
          2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: looking for saslNettyServer on server:/10.176.255.76:30000 for client /10.176.255.76:48239
          2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: creating SaslNettyServer with secret manager: org.apache.hadoop.mapreduce.security.token.JobTokenSecretManager@658782a7
          2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: creating saslNettyServer: secret manager is: org.apache.hadoop.mapreduce.security.token.JobTokenSecretManager@658782a7
          2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: checked read-availability; now creating sasl server.
          2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: create...
          2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: creating callback handler..
          2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: creating SaslDigestCallback handler with secret manager: org.apache.hadoop.mapreduce.security.token.JobTokenSecretManager@658782a7
          2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: created callback handler: org.apache.giraph.comm.SaslNettyServer$SaslDigestCallbackHandler@2ecc5436
          2012-08-08 20:58:53,011 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: create was successful.
          2012-08-08 20:58:53,011 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: client's token's length is: 0
          2012-08-08 20:58:53,011 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: checking completeness of SASL (pre-response):false
          2012-08-08 20:58:53,011 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: responding to input token of length: 0
          2012-08-08 20:58:53,011 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: response token length: 108
          2012-08-08 20:58:53,011 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: checking completeness of SASL (post-response):false
          2012-08-08 20:58:53,011 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: server's token's length is: 108
          2012-08-08 20:58:53,011 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: Responding to client: /10.176.255.76:48239
          2012-08-08 20:58:53,011 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.ResponseEncoder: RESPONSE ENCODER IS ENCODING A MESSAGE:org.apache.giraph.comm.SaslTokenMessage@7fcc5461
          2012-08-08 20:58:53,012 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.ResponseEncoder: encoding object: org.apache.giraph.comm.SaslTokenMessage@7fcc5461
          2012-08-08 20:58:53,012 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.ResponseEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslTokenMessage
          2012-08-08 20:58:53,012 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: write() called on token of length: 108
          2012-08-08 20:58:53,012 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: write() finished writing token.
          2012-08-08 20:58:53,012 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: Finished writing token to client.
          2012-08-08 20:58:53,012 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestServerHandler: done with doRequest() for class: class org.apache.giraph.comm.SaslTokenMessage
          2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestDecoder: decode: Got a request of type SASL_TOKEN from remote:/10.176.255.76:48239
          2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_TOKEN
          2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: reading fields from DataInput: org.jboss.netty.buffer.ChannelBufferInputStream@24bb6086
          2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: token size is: 274
          2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: now reading token.
          2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: read SASL token of length: 274
          2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestDecoder: decoded an object of type: class org.apache.giraph.comm.SaslTokenMessage:org.apache.giraph.comm.SaslTokenMessage@9be1041
          2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestServerHandler: Got class org.apache.giraph.comm.SaslTokenMessage
          2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestServerHandler: calling doRequest on class: class org.apache.giraph.comm.SaslTokenMessage
          2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: looking for saslNettyServer on server:/10.176.255.76:30000 for client /10.176.255.76:48239
          2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: client's token's length is: 274
          2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: checking completeness of SASL (pre-response):false
          2012-08-08 20:58:53,019 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: responding to input token of length: 274
          2012-08-08 20:58:53,019 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: SASL server DIGEST-MD5 callback: setting password for client: job_1344445498302_0016 (auth:SIMPLE)
          2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: decode: Got a response of type SASL_TOKEN from server:ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30001
          2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_TOKEN
          2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: writableRequestClass: class org.apache.giraph.comm.SaslTokenMessage found for type:SASL_TOKEN
          2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: reading fields of server response:org.apache.giraph.comm.SaslTokenMessage@1d1d2066
          2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: reading fields from DataInput: org.jboss.netty.buffer.ChannelBufferInputStream@633a1778
          2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: token size is: 108
          2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: now reading token.
          2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: read SASL token of length: 108
          2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: finished reading fields of server response:org.apache.giraph.comm.SaslTokenMessage@1d1d2066
          2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseClientHandler: message received: org.apache.giraph.comm.SaslTokenMessage@1d1d2066
          2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslNettyClient: SASL client callback: setting username: FmpvYl8xMzQ0NDQ1NDk4MzAyXzAwMTY=
          2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslNettyClient: SASL client callback: setting userPassword
          2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslNettyClient: SASL client callback: setting realm: default
          2012-08-08 20:58:53,021 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: creating SASL message with size: 274
          2012-08-08 20:58:53,021 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.NettyClient: sending request: org.apache.giraph.comm.SaslTokenMessage@7adafa2c to remote server: ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30001
          2012-08-08 20:58:53,021 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.RequestEncoder: encoding object: org.apache.giraph.comm.SaslTokenMessage@7adafa2c
          2012-08-08 20:58:53,021 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: write() called on token of length: 274
          2012-08-08 20:58:53,021 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: write() finished writing token.
          2012-08-08 20:58:53,021 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.RequestEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslTokenMessage
          2012-08-08 20:58:53,021 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseClientHandler: due to reception of: org.apache.giraph.comm.SaslTokenMessage@1d1d2066
          2012-08-08 20:58:53,021 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: SASL server DIGEST-MD5 callback: setting canonicalized client ID: job_1344445498302_0016
          2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: response token length: 40
          2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: checking completeness of SASL (post-response):true
          2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: server's token's length is: 40
          2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: Responding to client: /10.176.255.76:48239
          2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.ResponseEncoder: RESPONSE ENCODER IS ENCODING A MESSAGE:org.apache.giraph.comm.SaslTokenMessage@9be1041
          2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.ResponseEncoder: encoding object: org.apache.giraph.comm.SaslTokenMessage@9be1041
          2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.ResponseEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslTokenMessage
          2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: write() called on token of length: 40
          2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: write() finished writing token.
          2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: Finished writing token to client.
          2012-08-08 20:58:53,023 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: writing out complete token now.
          2012-08-08 20:58:53,023 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.ResponseEncoder: RESPONSE ENCODER IS ENCODING A MESSAGE:org.apache.giraph.comm.SaslComplete@3e5e9db7
          2012-08-08 20:58:53,023 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.ResponseEncoder: encoding object: org.apache.giraph.comm.SaslComplete@3e5e9db7
          2012-08-08 20:58:53,023 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.ResponseEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslComplete
          2012-08-08 20:58:53,023 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: decode: Got a response of type SASL_COMPLETE from server:ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30000
          2012-08-08 20:58:53,023 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_COMPLETE
          2012-08-08 20:58:53,023 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: writableRequestClass: class org.apache.giraph.comm.SaslComplete found for type:SASL_COMPLETE
          2012-08-08 20:58:53,023 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: reading fields of server response:org.apache.giraph.comm.SaslComplete@75fc25e5
          2012-08-08 20:58:53,023 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslComplete: read fields (none)
          2012-08-08 20:58:53,024 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: finished reading fields of server response:org.apache.giraph.comm.SaslComplete@75fc25e5
          2012-08-08 20:58:53,024 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseClientHandler: message received: org.apache.giraph.comm.SaslComplete@75fc25e5
          2012-08-08 20:58:53,024 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseClientHandler: SASL Authentication complete with server: ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30000.
          2012-08-08 20:58:53,024 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseClientHandler: due to reception of: org.apache.giraph.comm.SaslComplete@75fc25e5
          2012-08-08 20:58:53,023 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslComplete: writing one byte (write).
          2012-08-08 20:58:53,024 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: wrote out complete token now.
          2012-08-08 20:58:53,024 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestServerHandler: done with doRequest() for class: class org.apache.giraph.comm.SaslTokenMessage
          2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: decode: Got a response of type SASL_TOKEN from server:ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30001
          2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_TOKEN
          2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: writableRequestClass: class org.apache.giraph.comm.SaslTokenMessage found for type:SASL_TOKEN
          2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: reading fields of server response:org.apache.giraph.comm.SaslTokenMessage@3fe88b35
          2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: reading fields from DataInput: org.jboss.netty.buffer.ChannelBufferInputStream@3c0b655a
          2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: token size is: 40
          2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: now reading token.
          2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: read SASL token of length: 40
          2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: finished reading fields of server response:org.apache.giraph.comm.SaslTokenMessage@3fe88b35
          2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseClientHandler: message received: org.apache.giraph.comm.SaslTokenMessage@3fe88b35
          2012-08-08 20:58:53,071 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: decode: Got a response of type SASL_COMPLETE from server:ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30001
          2012-08-08 20:58:53,071 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_COMPLETE
          2012-08-08 20:58:53,071 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: writableRequestClass: class org.apache.giraph.comm.SaslComplete found for type:SASL_COMPLETE
          2012-08-08 20:58:53,071 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: reading fields of server response:org.apache.giraph.comm.SaslComplete@156a9424
          2012-08-08 20:58:53,071 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslComplete: read fields (none)
          2012-08-08 20:58:53,071 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: finished reading fields of server response:org.apache.giraph.comm.SaslComplete@156a9424
          2012-08-08 20:58:53,071 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseClientHandler: message received: org.apache.giraph.comm.SaslComplete@156a9424
          2012-08-08 20:58:53,071 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseClientHandler: SASL Authentication complete with server: ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30001.
          2012-08-08 20:58:53,071 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseClientHandler: due to reception of: org.apache.giraph.comm.SaslComplete@156a9424
          
          
          Show
          ekoontz Eugene Koontz added a comment - Here's some output showing client and server SASL communication based on the above test: [ec2-user@ip-10-176-255-76 giraph]$ cat /tmp/logs/application_1344445498302_0016/container_1344445498302_0016_01_000002/syslog | grep -i giraph | grep -i sasl 2012-08-08 20:58:52,806 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.NettyClient: creating saslNettyClient now. 2012-08-08 20:58:52,816 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.NettyClient: creating saslNettyClient now. 2012-08-08 20:58:52,929 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslNettyClient: Creating SASL DIGEST-MD5 client to authenticate to service at job_1344445498302_0016 2012-08-08 20:58:52,931 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslNettyClient: Creating SASL DIGEST-MD5 client to authenticate to service at job_1344445498302_0016 2012-08-08 20:58:52,943 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.NettyClient: storing saslNettyClient at key: [id: 0x7e8905bd, /10.176.255.76:48238 => ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30000] 2012-08-08 20:58:52,943 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.NettyClient: created: org.apache.giraph.comm.SaslNettyClient@2aa937cd 2012-08-08 20:58:52,943 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.NettyClient: sending request: org.apache.giraph.comm.SaslTokenMessage@4b8e899c to remote server: ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30000 2012-08-08 20:58:52,943 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.RequestEncoder: encoding object: org.apache.giraph.comm.SaslTokenMessage@4b8e899c 2012-08-08 20:58:52,947 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: write() called on token of length: 0 2012-08-08 20:58:52,947 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: write() finished writing token. 2012-08-08 20:58:52,947 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.RequestEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:52,954 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.NettyClient: storing saslNettyClient at key: [id: 0x04df8b14, /10.176.255.76:60743 => ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30001] 2012-08-08 20:58:52,954 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.NettyClient: created: org.apache.giraph.comm.SaslNettyClient@23faa614 2012-08-08 20:58:52,954 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.NettyClient: sending request: org.apache.giraph.comm.SaslTokenMessage@1cad7d80 to remote server: ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30001 2012-08-08 20:58:52,954 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.RequestEncoder: encoding object: org.apache.giraph.comm.SaslTokenMessage@1cad7d80 2012-08-08 20:58:52,954 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: write() called on token of length: 0 2012-08-08 20:58:52,954 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: write() finished writing token. 2012-08-08 20:58:52,954 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.RequestEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:52,959 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestDecoder: decode: Got a request of type SASL_TOKEN from remote:/10.176.255.76:48238 2012-08-08 20:58:52,959 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_TOKEN 2012-08-08 20:58:52,959 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: reading fields from DataInput: org.jboss.netty.buffer.ChannelBufferInputStream@2b8ca663 2012-08-08 20:58:52,959 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: token size is: 0 2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: now reading token. 2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: read SASL token of length: 0 2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestDecoder: decoded an object of type: class org.apache.giraph.comm.SaslTokenMessage:org.apache.giraph.comm.SaslTokenMessage@1effc3eb 2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestServerHandler: Got class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestServerHandler: calling doRequest on class: class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: looking for saslNettyServer on server:/10.176.255.76:30000 for client /10.176.255.76:48238 2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: creating SaslNettyServer with secret manager: org.apache.hadoop.mapreduce.security.token.JobTokenSecretManager@658782a7 2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: creating saslNettyServer: secret manager is: org.apache.hadoop.mapreduce.security.token.JobTokenSecretManager@658782a7 2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: checked read-availability; now creating sasl server. 2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: create... 2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: creating callback handler.. 2012-08-08 20:58:52,961 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: creating SaslDigestCallback handler with secret manager: org.apache.hadoop.mapreduce.security.token.JobTokenSecretManager@658782a7 2012-08-08 20:58:52,961 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: created callback handler: org.apache.giraph.comm.SaslNettyServer$SaslDigestCallbackHandler@1bb0889a 2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: create was successful. 2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: client's token's length is: 0 2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: checking completeness of SASL (pre-response): false 2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: responding to input token of length: 0 2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: response token length: 108 2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: checking completeness of SASL (post-response): false 2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: server's token's length is: 108 2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: Responding to client: /10.176.255.76:48238 2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.ResponseEncoder: RESPONSE ENCODER IS ENCODING A MESSAGE:org.apache.giraph.comm.SaslTokenMessage@1effc3eb 2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.ResponseEncoder: encoding object: org.apache.giraph.comm.SaslTokenMessage@1effc3eb 2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.ResponseEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: write() called on token of length: 108 2012-08-08 20:58:52,968 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: write() finished writing token. 2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: decode: Got a response of type SASL_TOKEN from server:ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30000 2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_TOKEN 2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: writableRequestClass: class org.apache.giraph.comm.SaslTokenMessage found for type:SASL_TOKEN 2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: reading fields of server response:org.apache.giraph.comm.SaslTokenMessage@48c5186e 2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: reading fields from DataInput: org.jboss.netty.buffer.ChannelBufferInputStream@3c70315 2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: token size is: 108 2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: now reading token. 2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: read SASL token of length: 108 2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: finished reading fields of server response:org.apache.giraph.comm.SaslTokenMessage@48c5186e 2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseClientHandler: message received: org.apache.giraph.comm.SaslTokenMessage@48c5186e 2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslNettyClient: SASL client callback: setting username: FmpvYl8xMzQ0NDQ1NDk4MzAyXzAwMTY= 2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslNettyClient: SASL client callback: setting userPassword 2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslNettyClient: SASL client callback: setting realm: default 2012-08-08 20:58:52,969 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: creating SASL message with size: 274 2012-08-08 20:58:52,969 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.NettyClient: sending request: org.apache.giraph.comm.SaslTokenMessage@7e9b59a2 to remote server: ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30000 2012-08-08 20:58:52,969 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.RequestEncoder: encoding object: org.apache.giraph.comm.SaslTokenMessage@7e9b59a2 2012-08-08 20:58:52,969 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: write() called on token of length: 274 2012-08-08 20:58:52,969 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: write() finished writing token. 2012-08-08 20:58:52,969 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.RequestEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:52,969 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseClientHandler: due to reception of: org.apache.giraph.comm.SaslTokenMessage@48c5186e 2012-08-08 20:58:52,970 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: Finished writing token to client. 2012-08-08 20:58:52,970 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestServerHandler: done with doRequest() for class: class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestDecoder: decode: Got a request of type SASL_TOKEN from remote:/10.176.255.76:48238 2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_TOKEN 2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: reading fields from DataInput: org.jboss.netty.buffer.ChannelBufferInputStream@919db9e 2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: token size is: 274 2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: now reading token. 2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: read SASL token of length: 274 2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestDecoder: decoded an object of type: class org.apache.giraph.comm.SaslTokenMessage:org.apache.giraph.comm.SaslTokenMessage@a166bd 2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestServerHandler: Got class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestServerHandler: calling doRequest on class: class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: looking for saslNettyServer on server:/10.176.255.76:30000 for client /10.176.255.76:48238 2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: client's token's length is: 274 2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: checking completeness of SASL (pre-response): false 2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: responding to input token of length: 274 2012-08-08 20:58:52,981 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: SASL server DIGEST-MD5 callback: setting password for client: job_1344445498302_0016 (auth:SIMPLE) 2012-08-08 20:58:52,981 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: SASL server DIGEST-MD5 callback: setting canonicalized client ID: job_1344445498302_0016 2012-08-08 20:58:52,982 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: response token length: 40 2012-08-08 20:58:52,982 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: checking completeness of SASL (post-response): true 2012-08-08 20:58:52,982 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: server's token's length is: 40 2012-08-08 20:58:52,982 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: Responding to client: /10.176.255.76:48238 2012-08-08 20:58:52,982 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.ResponseEncoder: RESPONSE ENCODER IS ENCODING A MESSAGE:org.apache.giraph.comm.SaslTokenMessage@a166bd 2012-08-08 20:58:52,982 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.ResponseEncoder: encoding object: org.apache.giraph.comm.SaslTokenMessage@a166bd 2012-08-08 20:58:52,982 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.ResponseEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:52,982 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: write() called on token of length: 40 2012-08-08 20:58:52,982 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: write() finished writing token. 2012-08-08 20:58:52,982 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: decode: Got a response of type SASL_TOKEN from server:ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30000 2012-08-08 20:58:52,982 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_TOKEN 2012-08-08 20:58:52,983 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: writableRequestClass: class org.apache.giraph.comm.SaslTokenMessage found for type:SASL_TOKEN 2012-08-08 20:58:52,983 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: reading fields of server response:org.apache.giraph.comm.SaslTokenMessage@4b6c06dd 2012-08-08 20:58:52,983 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: reading fields from DataInput: org.jboss.netty.buffer.ChannelBufferInputStream@2136bdda 2012-08-08 20:58:52,983 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: token size is: 40 2012-08-08 20:58:52,983 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: now reading token. 2012-08-08 20:58:52,983 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: read SASL token of length: 40 2012-08-08 20:58:52,983 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: finished reading fields of server response:org.apache.giraph.comm.SaslTokenMessage@4b6c06dd 2012-08-08 20:58:52,983 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseClientHandler: message received: org.apache.giraph.comm.SaslTokenMessage@4b6c06dd 2012-08-08 20:58:52,987 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: Finished writing token to client. 2012-08-08 20:58:52,987 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: writing out complete token now. 2012-08-08 20:58:52,987 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.ResponseEncoder: RESPONSE ENCODER IS ENCODING A MESSAGE:org.apache.giraph.comm.SaslComplete@36511e0a 2012-08-08 20:58:52,987 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.ResponseEncoder: encoding object: org.apache.giraph.comm.SaslComplete@36511e0a 2012-08-08 20:58:52,987 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.ResponseEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslComplete 2012-08-08 20:58:52,987 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslComplete: writing one byte (write). 2012-08-08 20:58:52,987 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: wrote out complete token now. 2012-08-08 20:58:52,987 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestServerHandler: done with doRequest() for class: class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:53,009 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestDecoder: decode: Got a request of type SASL_TOKEN from remote:/10.176.255.76:48239 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_TOKEN 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: reading fields from DataInput: org.jboss.netty.buffer.ChannelBufferInputStream@595780d9 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: token size is: 0 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: now reading token. 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: read SASL token of length: 0 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestDecoder: decoded an object of type: class org.apache.giraph.comm.SaslTokenMessage:org.apache.giraph.comm.SaslTokenMessage@7fcc5461 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestServerHandler: Got class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestServerHandler: calling doRequest on class: class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: looking for saslNettyServer on server:/10.176.255.76:30000 for client /10.176.255.76:48239 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: creating SaslNettyServer with secret manager: org.apache.hadoop.mapreduce.security.token.JobTokenSecretManager@658782a7 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: creating saslNettyServer: secret manager is: org.apache.hadoop.mapreduce.security.token.JobTokenSecretManager@658782a7 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: checked read-availability; now creating sasl server. 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: create... 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: creating callback handler.. 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: creating SaslDigestCallback handler with secret manager: org.apache.hadoop.mapreduce.security.token.JobTokenSecretManager@658782a7 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: created callback handler: org.apache.giraph.comm.SaslNettyServer$SaslDigestCallbackHandler@2ecc5436 2012-08-08 20:58:53,011 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: create was successful. 2012-08-08 20:58:53,011 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: client's token's length is: 0 2012-08-08 20:58:53,011 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: checking completeness of SASL (pre-response): false 2012-08-08 20:58:53,011 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: responding to input token of length: 0 2012-08-08 20:58:53,011 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: response token length: 108 2012-08-08 20:58:53,011 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: checking completeness of SASL (post-response): false 2012-08-08 20:58:53,011 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: server's token's length is: 108 2012-08-08 20:58:53,011 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: Responding to client: /10.176.255.76:48239 2012-08-08 20:58:53,011 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.ResponseEncoder: RESPONSE ENCODER IS ENCODING A MESSAGE:org.apache.giraph.comm.SaslTokenMessage@7fcc5461 2012-08-08 20:58:53,012 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.ResponseEncoder: encoding object: org.apache.giraph.comm.SaslTokenMessage@7fcc5461 2012-08-08 20:58:53,012 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.ResponseEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:53,012 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: write() called on token of length: 108 2012-08-08 20:58:53,012 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: write() finished writing token. 2012-08-08 20:58:53,012 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: Finished writing token to client. 2012-08-08 20:58:53,012 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestServerHandler: done with doRequest() for class: class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestDecoder: decode: Got a request of type SASL_TOKEN from remote:/10.176.255.76:48239 2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_TOKEN 2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: reading fields from DataInput: org.jboss.netty.buffer.ChannelBufferInputStream@24bb6086 2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: token size is: 274 2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: now reading token. 2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: read SASL token of length: 274 2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestDecoder: decoded an object of type: class org.apache.giraph.comm.SaslTokenMessage:org.apache.giraph.comm.SaslTokenMessage@9be1041 2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestServerHandler: Got class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestServerHandler: calling doRequest on class: class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: looking for saslNettyServer on server:/10.176.255.76:30000 for client /10.176.255.76:48239 2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: client's token's length is: 274 2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: checking completeness of SASL (pre-response): false 2012-08-08 20:58:53,019 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: responding to input token of length: 274 2012-08-08 20:58:53,019 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: SASL server DIGEST-MD5 callback: setting password for client: job_1344445498302_0016 (auth:SIMPLE) 2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: decode: Got a response of type SASL_TOKEN from server:ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30001 2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_TOKEN 2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: writableRequestClass: class org.apache.giraph.comm.SaslTokenMessage found for type:SASL_TOKEN 2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: reading fields of server response:org.apache.giraph.comm.SaslTokenMessage@1d1d2066 2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: reading fields from DataInput: org.jboss.netty.buffer.ChannelBufferInputStream@633a1778 2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: token size is: 108 2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: now reading token. 2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: read SASL token of length: 108 2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: finished reading fields of server response:org.apache.giraph.comm.SaslTokenMessage@1d1d2066 2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseClientHandler: message received: org.apache.giraph.comm.SaslTokenMessage@1d1d2066 2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslNettyClient: SASL client callback: setting username: FmpvYl8xMzQ0NDQ1NDk4MzAyXzAwMTY= 2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslNettyClient: SASL client callback: setting userPassword 2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslNettyClient: SASL client callback: setting realm: default 2012-08-08 20:58:53,021 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: creating SASL message with size: 274 2012-08-08 20:58:53,021 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.NettyClient: sending request: org.apache.giraph.comm.SaslTokenMessage@7adafa2c to remote server: ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30001 2012-08-08 20:58:53,021 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.RequestEncoder: encoding object: org.apache.giraph.comm.SaslTokenMessage@7adafa2c 2012-08-08 20:58:53,021 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: write() called on token of length: 274 2012-08-08 20:58:53,021 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: write() finished writing token. 2012-08-08 20:58:53,021 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.RequestEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:53,021 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseClientHandler: due to reception of: org.apache.giraph.comm.SaslTokenMessage@1d1d2066 2012-08-08 20:58:53,021 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: SASL server DIGEST-MD5 callback: setting canonicalized client ID: job_1344445498302_0016 2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: response token length: 40 2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: checking completeness of SASL (post-response): true 2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: server's token's length is: 40 2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: Responding to client: /10.176.255.76:48239 2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.ResponseEncoder: RESPONSE ENCODER IS ENCODING A MESSAGE:org.apache.giraph.comm.SaslTokenMessage@9be1041 2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.ResponseEncoder: encoding object: org.apache.giraph.comm.SaslTokenMessage@9be1041 2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.ResponseEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: write() called on token of length: 40 2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: write() finished writing token. 2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: Finished writing token to client. 2012-08-08 20:58:53,023 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: writing out complete token now. 2012-08-08 20:58:53,023 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.ResponseEncoder: RESPONSE ENCODER IS ENCODING A MESSAGE:org.apache.giraph.comm.SaslComplete@3e5e9db7 2012-08-08 20:58:53,023 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.ResponseEncoder: encoding object: org.apache.giraph.comm.SaslComplete@3e5e9db7 2012-08-08 20:58:53,023 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.ResponseEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslComplete 2012-08-08 20:58:53,023 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: decode: Got a response of type SASL_COMPLETE from server:ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30000 2012-08-08 20:58:53,023 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_COMPLETE 2012-08-08 20:58:53,023 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: writableRequestClass: class org.apache.giraph.comm.SaslComplete found for type:SASL_COMPLETE 2012-08-08 20:58:53,023 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: reading fields of server response:org.apache.giraph.comm.SaslComplete@75fc25e5 2012-08-08 20:58:53,023 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslComplete: read fields (none) 2012-08-08 20:58:53,024 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: finished reading fields of server response:org.apache.giraph.comm.SaslComplete@75fc25e5 2012-08-08 20:58:53,024 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseClientHandler: message received: org.apache.giraph.comm.SaslComplete@75fc25e5 2012-08-08 20:58:53,024 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseClientHandler: SASL Authentication complete with server: ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30000. 2012-08-08 20:58:53,024 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseClientHandler: due to reception of: org.apache.giraph.comm.SaslComplete@75fc25e5 2012-08-08 20:58:53,023 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslComplete: writing one byte (write). 2012-08-08 20:58:53,024 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: wrote out complete token now. 2012-08-08 20:58:53,024 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestServerHandler: done with doRequest() for class: class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: decode: Got a response of type SASL_TOKEN from server:ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30001 2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_TOKEN 2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: writableRequestClass: class org.apache.giraph.comm.SaslTokenMessage found for type:SASL_TOKEN 2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: reading fields of server response:org.apache.giraph.comm.SaslTokenMessage@3fe88b35 2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: reading fields from DataInput: org.jboss.netty.buffer.ChannelBufferInputStream@3c0b655a 2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: token size is: 40 2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: now reading token. 2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: read SASL token of length: 40 2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: finished reading fields of server response:org.apache.giraph.comm.SaslTokenMessage@3fe88b35 2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseClientHandler: message received: org.apache.giraph.comm.SaslTokenMessage@3fe88b35 2012-08-08 20:58:53,071 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: decode: Got a response of type SASL_COMPLETE from server:ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30001 2012-08-08 20:58:53,071 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_COMPLETE 2012-08-08 20:58:53,071 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: writableRequestClass: class org.apache.giraph.comm.SaslComplete found for type:SASL_COMPLETE 2012-08-08 20:58:53,071 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: reading fields of server response:org.apache.giraph.comm.SaslComplete@156a9424 2012-08-08 20:58:53,071 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslComplete: read fields (none) 2012-08-08 20:58:53,071 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: finished reading fields of server response:org.apache.giraph.comm.SaslComplete@156a9424 2012-08-08 20:58:53,071 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseClientHandler: message received: org.apache.giraph.comm.SaslComplete@156a9424 2012-08-08 20:58:53,071 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseClientHandler: SASL Authentication complete with server: ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30001. 2012-08-08 20:58:53,071 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseClientHandler: due to reception of: org.apache.giraph.comm.SaslComplete@156a9424
          Hide
          ekoontz Eugene Koontz added a comment -

          Tried to create a new review but hit 500 error; submitted this: https://issues.apache.org/jira/browse/INFRA-5128

          Show
          ekoontz Eugene Koontz added a comment - Tried to create a new review but hit 500 error; submitted this: https://issues.apache.org/jira/browse/INFRA-5128
          Hide
          aching Avery Ching added a comment -

          Eugene, this is a nice start! A few comments/questions about the limitations:

          >-Authorization is not done: that is, clients are authenticated but there are no restrictions on their ability to do RPC on the servers.

          Can't we block until the authentication is done?

          >-Clients should wait for authentication before trying to do RPC - once authorization (see above) is done, they might encounter a race where they try to do RPCs without yet being authenticated.

          Same question as above?

          >-Not tested on other than hadoop 2.0.1-SNAPSHOT

          Would probably be nice to try on a hadoop 1.0.x if you have a chance.

          >-Only works if we disable client-side channel-pooling (GIRAPH-289) and local short-circuiting of RPCs (GIRAPH-262) - these should be configurable but currently, I hard-wired both to be disabled.

          I think it's reasonable to allow short-circuiting since there isn't a security issue here (this is the same process). As for channel pooling, can we simply authenticate once per channel?

          Show
          aching Avery Ching added a comment - Eugene, this is a nice start! A few comments/questions about the limitations: >-Authorization is not done: that is, clients are authenticated but there are no restrictions on their ability to do RPC on the servers. Can't we block until the authentication is done? >-Clients should wait for authentication before trying to do RPC - once authorization (see above) is done, they might encounter a race where they try to do RPCs without yet being authenticated. Same question as above? >-Not tested on other than hadoop 2.0.1-SNAPSHOT Would probably be nice to try on a hadoop 1.0.x if you have a chance. >-Only works if we disable client-side channel-pooling ( GIRAPH-289 ) and local short-circuiting of RPCs ( GIRAPH-262 ) - these should be configurable but currently, I hard-wired both to be disabled. I think it's reasonable to allow short-circuiting since there isn't a security issue here (this is the same process). As for channel pooling, can we simply authenticate once per channel?
          Hide
          ekoontz Eugene Koontz added a comment -

          Thanks for your comments Avery! Replying to each below:

          > Can't we block until the authentication is done?

          Absolutely - we just need to choose the object to synchronize on and be careful that it's respected where it needs to be. Will include this in next patch, which will also avoid the race.

          > Would probably be nice to try on a hadoop 1.0.x if you have a chance.

          Will do.

          > I think it's reasonable to allow short-circuiting since there isn't a security issue here (this is the same process).

          Yes, I think it will be ok to trust the client in this case (specifically in GIRAPH-262 we are testing for):

          (service.getWorkerInfo().getHostnamePort().equals(remoteServerAddress)) {
          

          I just need to fix whatever bug I am masking by disabling short-curcuiting.

          > As for channel pooling, can we simply authenticate once per channel?

          Yes, I just need to figure out how to associate the sasl client with the channel when we are using channel-pooling. I use the Address => Channel map in my patch, but now with GIRAPH-262, we have Address => ChannelRotator. So this will require some changes on my patch.

          Show
          ekoontz Eugene Koontz added a comment - Thanks for your comments Avery! Replying to each below: > Can't we block until the authentication is done? Absolutely - we just need to choose the object to synchronize on and be careful that it's respected where it needs to be. Will include this in next patch, which will also avoid the race. > Would probably be nice to try on a hadoop 1.0.x if you have a chance. Will do. > I think it's reasonable to allow short-circuiting since there isn't a security issue here (this is the same process). Yes, I think it will be ok to trust the client in this case (specifically in GIRAPH-262 we are testing for): (service.getWorkerInfo().getHostnamePort().equals(remoteServerAddress)) { I just need to fix whatever bug I am masking by disabling short-curcuiting. > As for channel pooling, can we simply authenticate once per channel? Yes, I just need to figure out how to associate the sasl client with the channel when we are using channel-pooling. I use the Address => Channel map in my patch, but now with GIRAPH-262 , we have Address => ChannelRotator. So this will require some changes on my patch.
          Hide
          ekoontz Eugene Koontz added a comment -

          Added review: https://reviews.apache.org/r/6609/

          Thanks to pctony for enabling git-based reviews: https://issues.apache.org/jira/browse/INFRA-5128

          Show
          ekoontz Eugene Koontz added a comment - Added review: https://reviews.apache.org/r/6609/ Thanks to pctony for enabling git-based reviews: https://issues.apache.org/jira/browse/INFRA-5128
          Hide
          aching Avery Ching added a comment -

          Cool! I'll wait for your updated patch to review.

          Show
          aching Avery Ching added a comment - Cool! I'll wait for your updated patch to review.
          Hide
          apresta Alessandro Presta added a comment -

          Any update on this? I don't know the schedule for cutting the 0.2 release, but I'd say this is a prerequisite: not only we get rid of a lot of cruft, but also the API currently includes methods (putMessages/getMessages) that are used only by the Hadoop RPC implementation.

          Show
          apresta Alessandro Presta added a comment - Any update on this? I don't know the schedule for cutting the 0.2 release, but I'd say this is a prerequisite: not only we get rid of a lot of cruft, but also the API currently includes methods (putMessages/getMessages) that are used only by the Hadoop RPC implementation.
          Hide
          ekoontz Eugene Koontz added a comment -

          Hi Alessandro, I am working actively on it and hope to have a patch ready in the next day or so.
          -Eugene

          Show
          ekoontz Eugene Koontz added a comment - Hi Alessandro, I am working actively on it and hope to have a patch ready in the next day or so. -Eugene
          Hide
          apresta Alessandro Presta added a comment -

          Awesome, looking forward to it!

          Show
          apresta Alessandro Presta added a comment - Awesome, looking forward to it!
          Hide
          ekoontz Eugene Koontz added a comment -

          Improves on recent patch:

          -Does authorization: server checks for authentication being completed before allowing non-SASL client requests to be performed.

          -Client waits to send (non-SASL) requests until SASL authentication has completed.

          -Works with channel-pooling and local short-circuiting

          Limitations:

          -Not tested on other than hadoop 2.0.1-SNAPSHOT
          -Restores RequestRegistry class, which was removed in GIRAPH-313 - hope to merge better with this.
          -Needs more testing in general, including new unit tests.

          Show
          ekoontz Eugene Koontz added a comment - Improves on recent patch: -Does authorization: server checks for authentication being completed before allowing non-SASL client requests to be performed. -Client waits to send (non-SASL) requests until SASL authentication has completed. -Works with channel-pooling and local short-circuiting Limitations: -Not tested on other than hadoop 2.0.1-SNAPSHOT -Restores RequestRegistry class, which was removed in GIRAPH-313 - hope to merge better with this. -Needs more testing in general, including new unit tests.
          Hide
          ekoontz Eugene Koontz added a comment -

          I manually fixed some conflicts recently, so I'm gradually making this patch more closely fit to trunk.

          Improvements:
          -Removes usage of RequestRegistry (that was removed in GIRAPH-313)

          Limitations:
          Uses preGIRAPH-313 workerThreadPool rather than requestServerHandlerFactory
          -Needs tests

          Show
          ekoontz Eugene Koontz added a comment - I manually fixed some conflicts recently, so I'm gradually making this patch more closely fit to trunk. Improvements: -Removes usage of RequestRegistry (that was removed in GIRAPH-313 ) Limitations: Uses pre GIRAPH-313 workerThreadPool rather than requestServerHandlerFactory -Needs tests
          Hide
          ekoontz Eugene Koontz added a comment -

          Sorry, the last patch was missing the new source files - no wonder it was so small.

          Show
          ekoontz Eugene Koontz added a comment - Sorry, the last patch was missing the new source files - no wonder it was so small.
          Hide
          ekoontz Eugene Koontz added a comment -

          https://reviews.apache.org/r/6609/diff/ updated to most recent patch.

          Show
          ekoontz Eugene Koontz added a comment - https://reviews.apache.org/r/6609/diff/ updated to most recent patch.
          Hide
          ekoontz Eugene Koontz added a comment -

          Improvements:

          Removes remaining preGIRAPH-313-isms in NettyServer and NettyWorkerClient

          Show
          ekoontz Eugene Koontz added a comment - Improvements: Removes remaining pre GIRAPH-313 -isms in NettyServer and NettyWorkerClient
          Hide
          ekoontz Eugene Koontz added a comment -

          https://reviews.apache.org/r/6609/diff/ updated to most recent patch (will refrain from making this comment from now on and simply keep them synced).

          Show
          ekoontz Eugene Koontz added a comment - https://reviews.apache.org/r/6609/diff/ updated to most recent patch (will refrain from making this comment from now on and simply keep them synced).
          Hide
          ekoontz Eugene Koontz added a comment -
          Show
          ekoontz Eugene Koontz added a comment - merge with trunk: 55bfcb78 https://svn.apache.org/repos/asf/giraph/trunk@1378761 ( GIRAPH-317 )
          Hide
          aching Avery Ching added a comment -

          Hi Eugene, any luck getting this implemented as a ChannelHandler?

          Show
          aching Avery Ching added a comment - Hi Eugene, any luck getting this implemented as a ChannelHandler?
          Hide
          ekoontz Eugene Koontz added a comment -

          Hi Avery,
          Yes, I should have a new patch up today.
          -Eugene

          Show
          ekoontz Eugene Koontz added a comment - Hi Avery, Yes, I should have a new patch up today. -Eugene
          Hide
          ekoontz Eugene Koontz added a comment -

          Interesting and related: "Handshaking tutorial with Netty" by Bruno de Carvalho here:

          http://biasedbit.com/handshaking-tutorial-with-netty/

          Show
          ekoontz Eugene Koontz added a comment - Interesting and related: "Handshaking tutorial with Netty" by Bruno de Carvalho here: http://biasedbit.com/handshaking-tutorial-with-netty/
          Hide
          ekoontz Eugene Koontz added a comment -

          My apologies for the delay. I am in the final stages : getting it to compile on all supported Hadoop versions and fixing checkstyle errors. I have the SASL code implemented as a ChannelHandler as we discussed. Here's what the pipeline looks like on the client side, showing both authenticated and non-authenticated cases:

          NettyClient.java
             // Set up the pipeline factory.
              bootstrap.setPipelineFactory(new ChannelPipelineFactory() {
                @Override
                public ChannelPipeline getPipeline() throws Exception {
                  if (conf.getBoolean(GiraphJob.AUTHENTICATE,
                      GiraphJob.DEFAULT_AUTHENTICATE)) {
                    LOG.info("Using Netty with authentication.");
          
                    // Our pipeline starts with just byteCounter, and then we use
                    // addAfter() to incrementally add pipeline elements, so that we can
                    // name them for identification for removal or replacement after
                    // client is authenticated by server.
                    ChannelPipeline pipeline = Channels.pipeline(
                        byteCounter);
          
                    // After authentication finishes, the following is replaced with
                    // FixedLengthFrameDecoder (as in non-auth pipeline below):
                    pipeline.addLast("length-field-based-frame-decoder",
                        new LengthFieldBasedFrameDecoder(1024, 0, 4, 0, 4));
          
                    pipeline.addLast("request-encoder", new RequestEncoder());
          
                    // After authentication finishes, the following is removed:
                    pipeline.addLast("sasl-client-handler",
                        new SaslClientHandler(conf));
          
                    pipeline.addLast("response-handler",
                        new ResponseClientHandler(clientRequestIdRequestInfoMap, conf));
                    return pipeline;
                  } else {
                    LOG.info("Using Netty without authentication.");
                    return Channels.pipeline(
                        byteCounter,
                        new FixedLengthFrameDecoder(RequestServerHandler.RESPONSE_BYTES),
                        new RequestEncoder(),
                        new ResponseClientHandler(clientRequestIdRequestInfoMap, conf));
                  }
                }
              });
          
          Show
          ekoontz Eugene Koontz added a comment - My apologies for the delay. I am in the final stages : getting it to compile on all supported Hadoop versions and fixing checkstyle errors. I have the SASL code implemented as a ChannelHandler as we discussed. Here's what the pipeline looks like on the client side, showing both authenticated and non-authenticated cases: NettyClient.java // Set up the pipeline factory. bootstrap.setPipelineFactory( new ChannelPipelineFactory() { @Override public ChannelPipeline getPipeline() throws Exception { if (conf.getBoolean(GiraphJob.AUTHENTICATE, GiraphJob.DEFAULT_AUTHENTICATE)) { LOG.info( "Using Netty with authentication." ); // Our pipeline starts with just byteCounter, and then we use // addAfter() to incrementally add pipeline elements, so that we can // name them for identification for removal or replacement after // client is authenticated by server. ChannelPipeline pipeline = Channels.pipeline( byteCounter); // After authentication finishes, the following is replaced with // FixedLengthFrameDecoder (as in non-auth pipeline below): pipeline.addLast( "length-field-based-frame-decoder" , new LengthFieldBasedFrameDecoder(1024, 0, 4, 0, 4)); pipeline.addLast( "request-encoder" , new RequestEncoder()); // After authentication finishes, the following is removed: pipeline.addLast( "sasl-client-handler" , new SaslClientHandler(conf)); pipeline.addLast( "response-handler" , new ResponseClientHandler(clientRequestIdRequestInfoMap, conf)); return pipeline; } else { LOG.info( "Using Netty without authentication." ); return Channels.pipeline( byteCounter, new FixedLengthFrameDecoder(RequestServerHandler.RESPONSE_BYTES), new RequestEncoder(), new ResponseClientHandler(clientRequestIdRequestInfoMap, conf)); } } });
          Hide
          ekoontz Eugene Koontz added a comment -

          Passes mvn -Phadoop_trunk verify - other Hadoops coming..

          Show
          ekoontz Eugene Koontz added a comment - Passes mvn -Phadoop_trunk verify - other Hadoops coming..
          Hide
          aching Avery Ching added a comment - - edited

          Eugene, this looks nice! I still see some checkstyle stuff but you're working through it I guess. Overall, the design is good. If we think about improvements we can make them later on.

          Questions/Comments:

          For consistency, can you please convert multi-line comments like

          /** Whether to use SASL with DIGEST and Hadoop Job Tokens to authenticate	
           * and authorize Netty BSP Clients to Servers. */
          

          to

          /** 
           * Whether to use SASL with DIGEST and Hadoop Job Tokens to authenticate	
           * and authorize Netty BSP Clients to Servers. 
           */
          

          Can we get rename Authorize to AuthorizeServerHandler or something else more descriptive?

          NettyClient.java

          • 372: Please wrap the LOG.info() with if (LOG.isInfoEnabled()).
          • 545-557: Can't we just go through the regular netty request part of the code? We don't need to have -2 here and can just submit the destWorkerId?

          SASL_COMPLETE -> SASL_COMPLETE_REQUEST?

          SaslTokenMessage.java can we call is SaslTokenMessageRequest?

          SaslComplete.java can we call it SaslCompleteRequest to match the other names?

          SaslComplete.java

          • 29-34: Why not get rid of these?

          SaslTokenMessage.java:

          • 86: Extra line

          Thanks again, this was a lot of work!

          Show
          aching Avery Ching added a comment - - edited Eugene, this looks nice! I still see some checkstyle stuff but you're working through it I guess. Overall, the design is good. If we think about improvements we can make them later on. Questions/Comments: For consistency, can you please convert multi-line comments like /** Whether to use SASL with DIGEST and Hadoop Job Tokens to authenticate * and authorize Netty BSP Clients to Servers. */ to /** * Whether to use SASL with DIGEST and Hadoop Job Tokens to authenticate * and authorize Netty BSP Clients to Servers. */ Can we get rename Authorize to AuthorizeServerHandler or something else more descriptive? NettyClient.java 372: Please wrap the LOG.info() with if (LOG.isInfoEnabled()). 545-557: Can't we just go through the regular netty request part of the code? We don't need to have -2 here and can just submit the destWorkerId? SASL_COMPLETE -> SASL_COMPLETE_REQUEST? SaslTokenMessage.java can we call is SaslTokenMessageRequest? SaslComplete.java can we call it SaslCompleteRequest to match the other names? SaslComplete.java 29-34: Why not get rid of these? SaslTokenMessage.java: 86: Extra line Thanks again, this was a lot of work!
          Hide
          ekoontz Eugene Koontz added a comment -

          Hi Avery,
          Thanks a lot for your comments - are you sure about checkstyle? I thought I got it down to zero errors. Will look into it and reply to your other comments.

          -Eugene

          Show
          ekoontz Eugene Koontz added a comment - Hi Avery, Thanks a lot for your comments - are you sure about checkstyle? I thought I got it down to zero errors. Will look into it and reply to your other comments. -Eugene
          Hide
          ekoontz Eugene Koontz added a comment -

          Updated rb; please feel free to review here:

          https://reviews.apache.org/r/6609/diff/6/

          Show
          ekoontz Eugene Koontz added a comment - Updated rb; please feel free to review here: https://reviews.apache.org/r/6609/diff/6/
          Hide
          giraphqa Giraph QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12545806/GIRAPH-211.patch
          against trunk revision 1387840.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 javac. The patch appears to cause the build to fail.

          Console output: https://builds.apache.org/job/PreCommit-GIRAPH-Build/41//console

          This message is automatically generated.

          Show
          giraphqa Giraph QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12545806/GIRAPH-211.patch against trunk revision 1387840. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javac. The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-GIRAPH-Build/41//console This message is automatically generated.
          Hide
          aching Avery Ching added a comment - - edited

          Eugene, version 6 of the diff seems to be the same as the one I reviewed. Can you please address my comments above in https://issues.apache.org/jira/browse/GIRAPH-211?focusedCommentId=13459279&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13459279 ?

          Show
          aching Avery Ching added a comment - - edited Eugene, version 6 of the diff seems to be the same as the one I reviewed. Can you please address my comments above in https://issues.apache.org/jira/browse/GIRAPH-211?focusedCommentId=13459279&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13459279 ?
          Hide
          aching Avery Ching added a comment -

          Let me know if you need any help on this one. I'm looking forward into putting this in!

          Show
          aching Avery Ching added a comment - Let me know if you need any help on this one. I'm looking forward into putting this in!
          Hide
          ekoontz Eugene Koontz added a comment -

          Thanks Avery. I appreciate your help. I am hoping to 1) address your comments and 2) get mvn verify to work with all profiles in a new patch in the next 2 days.
          -Eugene

          Show
          ekoontz Eugene Koontz added a comment - Thanks Avery. I appreciate your help. I am hoping to 1) address your comments and 2) get mvn verify to work with all profiles in a new patch in the next 2 days. -Eugene
          Hide
          aching Avery Ching added a comment -

          Any luck with this Eugene Koontz?

          Show
          aching Avery Ching added a comment - Any luck with this Eugene Koontz ?
          Hide
          ekoontz Eugene Koontz added a comment -

          Getting closer.

          -Passes:

          mvn -Phadoop_non_secure clean verify &&
          mvn -Phadoop_1.0 clean verify && 
          mvn -Phadoop_2.0.0 clean verify && 
          mvn -Phadoop_2.0.1 clean verify && 
          mvn -Phadoop_2.0.2 clean verify
          

          (I added profiles for hadoop 2.0.1 and 2.0.2 since there are
          releases available for these newer Hadoop versions).

          -Addressed most of Avery's comments on reviewboard - thank you Avery. Still,
          SaslTokenMessageRequests do not properly pass the
          destWorkerId (it still uses -2 as the destWorkerId). Working on addressing this.

          -Added new test, SaslConnectionTest, adapted from ConnectionTest. Passes,
          but needs to be expanded. I need to better understand how
          Mockito can be used to mock Job Tokens, so would appreciate help
          with how to use Mockito for this.

          -Incorporates GIRAPH-212's patch: this fixes munge flags associated with security.

          Show
          ekoontz Eugene Koontz added a comment - Getting closer. -Passes: mvn -Phadoop_non_secure clean verify && mvn -Phadoop_1.0 clean verify && mvn -Phadoop_2.0.0 clean verify && mvn -Phadoop_2.0.1 clean verify && mvn -Phadoop_2.0.2 clean verify (I added profiles for hadoop 2.0.1 and 2.0.2 since there are releases available for these newer Hadoop versions). -Addressed most of Avery's comments on reviewboard - thank you Avery. Still, SaslTokenMessageRequests do not properly pass the destWorkerId (it still uses -2 as the destWorkerId). Working on addressing this. -Added new test, SaslConnectionTest, adapted from ConnectionTest. Passes, but needs to be expanded. I need to better understand how Mockito can be used to mock Job Tokens, so would appreciate help with how to use Mockito for this. -Incorporates GIRAPH-212 's patch: this fixes munge flags associated with security.
          Hide
          aching Avery Ching added a comment -

          We just committed GIRAPH-212, so only thing is to fix the -2 thing and add a test? I'd really like to get this in the next day or two. If you're busy Eugene Koontz, let me know and I can try and massage it a bit.

          Show
          aching Avery Ching added a comment - We just committed GIRAPH-212 , so only thing is to fix the -2 thing and add a test? I'd really like to get this in the next day or two. If you're busy Eugene Koontz , let me know and I can try and massage it a bit.
          Hide
          ekoontz Eugene Koontz added a comment -

          Hi Avery, That's correct - we just need to fix the -2 thing and improve the test. I could use some help with either or both
          Thanks a lot.
          -Eugene

          Show
          ekoontz Eugene Koontz added a comment - Hi Avery, That's correct - we just need to fix the -2 thing and improve the test. I could use some help with either or both Thanks a lot. -Eugene
          Hide
          ekoontz Eugene Koontz added a comment -

          Merged with trunk: no longer needs GIRAPH-212 inclusion thanks to Avery's commit of this.

          Small munge flag name change: changed HADOOP_1_AUTHORIZATION to HADOOP_1_SECURITY.

          Show
          ekoontz Eugene Koontz added a comment - Merged with trunk: no longer needs GIRAPH-212 inclusion thanks to Avery's commit of this. Small munge flag name change: changed HADOOP_1_AUTHORIZATION to HADOOP_1_SECURITY.
          Hide
          aching Avery Ching added a comment -

          Here is an attached patch with the following changes from GIRAPH-211.patch:

          Addressed the -2 worker id by adding the task id to ChannelRotater.
          SaslComplete -> SaslCompleteRequest
          SASL_TOKEN_MESSAGE -> SASL_TOKEN_MESSAGE_REQUEST
          Fixed all minor things (javadoc, functions, added authenticate()).
          mvn -Phadoop=facebook compile fixed

          Tested with

          mvn -Phadoop_non_secure clean verify

          • passed
            mvn -Phadoop_1.0 clean verify
            = passed
            mvn -Phadoop_2.0.0 clean verify
          • passed
            mvn -Phadoop_2.0.1 clean verify
          • passed
            mvn -Phadoop=facebook compile
          • passed

          Note that hadoop-2.0.2 wasn't tested (I don't have the artifacts), but should work.

          I know that we should add more testing, but I think this is good to go for now. It seems to work and it's already very huge. Great work Eugene Koontz! Please let me know if the changes I made are acceptable, I'm fine with the current version. We should commit soon.

          Show
          aching Avery Ching added a comment - Here is an attached patch with the following changes from GIRAPH-211 .patch: Addressed the -2 worker id by adding the task id to ChannelRotater. SaslComplete -> SaslCompleteRequest SASL_TOKEN_MESSAGE -> SASL_TOKEN_MESSAGE_REQUEST Fixed all minor things (javadoc, functions, added authenticate()). mvn -Phadoop=facebook compile fixed Tested with mvn -Phadoop_non_secure clean verify passed mvn -Phadoop_1.0 clean verify = passed mvn -Phadoop_2.0.0 clean verify passed mvn -Phadoop_2.0.1 clean verify passed mvn -Phadoop=facebook compile passed Note that hadoop-2.0.2 wasn't tested (I don't have the artifacts), but should work. I know that we should add more testing, but I think this is good to go for now. It seems to work and it's already very huge. Great work Eugene Koontz ! Please let me know if the changes I made are acceptable, I'm fine with the current version. We should commit soon.
          Hide
          aching Avery Ching added a comment -

          Rebased with trunk.

          Show
          aching Avery Ching added a comment - Rebased with trunk.
          Hide
          ekoontz Eugene Koontz added a comment -

          Wow, thanks so much Avery. I'm going to try out your patch today. Will also try with hadoop-2.0.2.

          -Eugene

          Show
          ekoontz Eugene Koontz added a comment - Wow, thanks so much Avery. I'm going to try out your patch today. Will also try with hadoop-2.0.2. -Eugene
          Hide
          aching Avery Ching added a comment -

          Thanks, let me know!

          Show
          aching Avery Ching added a comment - Thanks, let me know!
          Hide
          aching Avery Ching added a comment -

          Rebased on GIRAPH-360.patch.

          Show
          aching Avery Ching added a comment - Rebased on GIRAPH-360 .patch.
          Hide
          ekoontz Eugene Koontz added a comment -

          Avery, modified your patch to fix one problem: a NPE in NettyClient.sendWritableRequest() when the supplied destWorkerId param is null. This null param happens when requests are made to the master when sendWritableRequest() is called by NettyClient.authenticateOnChannel().

          With Avery's fix for GIRAPH-360, we associate each request with the taskId of the worker that is making the request, and use a pair <taskid,requestid> as the key for tracking requests in the clientRequestIdRequestInfoMap. However, the Giraph master is not associated with a taskId, and so its taskid is null.

          It wouldn't be a problem that the master has a null task id for normal Giraph requests, but for authentication, workers need to authenticate with the master just as they must do with the other workers.

          This patch works around the problem by simply not registering authentication requests (requests with type=SASL_TOKEN_MESSAGE_REQUEST) with the clientRequestIdRequestInfoMap. In other words, it's another workaround like the "-2" presence in my previous patches. I know this is not ideal, but I wanted to keep the momentum going for this JIRA.

          Rather than this workaround, I thought a possible solution would be a separate client-side pipeline component that handles request tracking. We'd move the clientRequestIdRequestInfoMap into this component, which would handles the tracking functionality that currently is spread across NettyClient.sendWritableRequest(), .waitAllRequests(), and ResponseClientHandler.messageReceived().

          Or, maybe I'm missing something obvious and easy

          Tested this patch successfully with:

          mvn -Phadoop_non_secure clean verify && mvn -Phadoop_0.20.203 clean verify && mvn -Phadoop_1.0 clean verify && mvn -Phadoop0.23 clean verify && mvn -Phadoop_2.0.0 clean verify && mvn -Phadoop_2.0.1 clean verify && mvn -Phadoop_2.0.2 clean verify
          
          Show
          ekoontz Eugene Koontz added a comment - Avery, modified your patch to fix one problem: a NPE in NettyClient.sendWritableRequest() when the supplied destWorkerId param is null. This null param happens when requests are made to the master when sendWritableRequest() is called by NettyClient.authenticateOnChannel(). With Avery's fix for GIRAPH-360 , we associate each request with the taskId of the worker that is making the request, and use a pair <taskid,requestid> as the key for tracking requests in the clientRequestIdRequestInfoMap. However, the Giraph master is not associated with a taskId, and so its taskid is null. It wouldn't be a problem that the master has a null task id for normal Giraph requests, but for authentication, workers need to authenticate with the master just as they must do with the other workers. This patch works around the problem by simply not registering authentication requests (requests with type=SASL_TOKEN_MESSAGE_REQUEST) with the clientRequestIdRequestInfoMap. In other words, it's another workaround like the "-2" presence in my previous patches. I know this is not ideal, but I wanted to keep the momentum going for this JIRA. Rather than this workaround, I thought a possible solution would be a separate client-side pipeline component that handles request tracking. We'd move the clientRequestIdRequestInfoMap into this component, which would handles the tracking functionality that currently is spread across NettyClient.sendWritableRequest(), .waitAllRequests(), and ResponseClientHandler.messageReceived(). Or, maybe I'm missing something obvious and easy Tested this patch successfully with: mvn -Phadoop_non_secure clean verify && mvn -Phadoop_0.20.203 clean verify && mvn -Phadoop_1.0 clean verify && mvn -Phadoop0.23 clean verify && mvn -Phadoop_2.0.0 clean verify && mvn -Phadoop_2.0.1 clean verify && mvn -Phadoop_2.0.2 clean verify
          Hide
          aching Avery Ching added a comment -

          +1, will also create a follow up task to fix up this weirdness with the master. Thanks for all the hard work Eugene!

          Show
          aching Avery Ching added a comment - +1, will also create a follow up task to fix up this weirdness with the master. Thanks for all the hard work Eugene!
          Hide
          ekoontz Eugene Koontz added a comment -

          Same as last patch except adds what I mistakenly left out of last patch:

          src/main/java/org/apache/giraph/comm/netty/handler/AuthorizeServerHandler.java
          src/main/java/org/apache/giraph/comm/netty/handler/ResponseEncoder.java

          Show
          ekoontz Eugene Koontz added a comment - Same as last patch except adds what I mistakenly left out of last patch: src/main/java/org/apache/giraph/comm/netty/handler/AuthorizeServerHandler.java src/main/java/org/apache/giraph/comm/netty/handler/ResponseEncoder.java
          Hide
          hudson Hudson added a comment -

          Integrated in Giraph-trunk-Commit #233 (See https://builds.apache.org/job/Giraph-trunk-Commit/233/)
          GIRAPH-211: Add secure authentication to Netty IPC (Revision 1396722)

          Result = SUCCESS
          ekoontz : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1396722
          Files :

          • /giraph/trunk/pom.xml
          • /giraph/trunk/src/main/java/org/apache/giraph/GiraphConfiguration.java
          • /giraph/trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java
          • /giraph/trunk/src/main/java/org/apache/giraph/comm/CommunicationsInterface.java
          • /giraph/trunk/src/main/java/org/apache/giraph/comm/SecureRPCCommunications.java
          • /giraph/trunk/src/main/java/org/apache/giraph/comm/WorkerClient.java
          • /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/NettyClient.java
          • /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/NettyServer.java
          • /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClient.java
          • /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClientServer.java
          • /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/SaslNettyClient.java
          • /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/SaslNettyServer.java
          • /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/handler/AuthorizeServerHandler.java
          • /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/handler/RequestEncoder.java
          • /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/handler/ResponseEncoder.java
          • /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/handler/SaslClientHandler.java
          • /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/handler/SaslServerHandler.java
          • /giraph/trunk/src/main/java/org/apache/giraph/comm/requests/RequestType.java
          • /giraph/trunk/src/main/java/org/apache/giraph/comm/requests/SaslCompleteRequest.java
          • /giraph/trunk/src/main/java/org/apache/giraph/comm/requests/SaslTokenMessageRequest.java
          • /giraph/trunk/src/main/java/org/apache/giraph/graph/BspServiceWorker.java
          • /giraph/trunk/src/test/java/org/apache/giraph/comm/SaslConnectionTest.java
          Show
          hudson Hudson added a comment - Integrated in Giraph-trunk-Commit #233 (See https://builds.apache.org/job/Giraph-trunk-Commit/233/ ) GIRAPH-211 : Add secure authentication to Netty IPC (Revision 1396722) Result = SUCCESS ekoontz : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1396722 Files : /giraph/trunk/pom.xml /giraph/trunk/src/main/java/org/apache/giraph/GiraphConfiguration.java /giraph/trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java /giraph/trunk/src/main/java/org/apache/giraph/comm/CommunicationsInterface.java /giraph/trunk/src/main/java/org/apache/giraph/comm/SecureRPCCommunications.java /giraph/trunk/src/main/java/org/apache/giraph/comm/WorkerClient.java /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/NettyClient.java /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/NettyServer.java /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClient.java /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClientServer.java /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/SaslNettyClient.java /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/SaslNettyServer.java /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/handler/AuthorizeServerHandler.java /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/handler/RequestEncoder.java /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/handler/ResponseEncoder.java /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/handler/SaslClientHandler.java /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/handler/SaslServerHandler.java /giraph/trunk/src/main/java/org/apache/giraph/comm/requests/RequestType.java /giraph/trunk/src/main/java/org/apache/giraph/comm/requests/SaslCompleteRequest.java /giraph/trunk/src/main/java/org/apache/giraph/comm/requests/SaslTokenMessageRequest.java /giraph/trunk/src/main/java/org/apache/giraph/graph/BspServiceWorker.java /giraph/trunk/src/test/java/org/apache/giraph/comm/SaslConnectionTest.java
          Hide
          hudson Hudson added a comment -

          Integrated in Giraph-trunk-Commit #234 (See https://builds.apache.org/job/Giraph-trunk-Commit/234/)
          GIRAPH-363: Fix hadoop_0.23 profile broken by GIRAPH-211 (Revision 1396858)

          Result = SUCCESS
          ekoontz : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1396858
          Files :

          • /giraph/trunk/CHANGELOG
          • /giraph/trunk/pom.xml
          • /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/SaslNettyServer.java
          Show
          hudson Hudson added a comment - Integrated in Giraph-trunk-Commit #234 (See https://builds.apache.org/job/Giraph-trunk-Commit/234/ ) GIRAPH-363 : Fix hadoop_0.23 profile broken by GIRAPH-211 (Revision 1396858) Result = SUCCESS ekoontz : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1396858 Files : /giraph/trunk/CHANGELOG /giraph/trunk/pom.xml /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/SaslNettyServer.java
          Hide
          ekoontz Eugene Koontz added a comment -

          HADOOP-8078 starts an ApacheDS KDC server for use in testing Kerberos integration - hopefully this could also be used to create unit tests for GIRAPH-211.

          Show
          ekoontz Eugene Koontz added a comment - HADOOP-8078 starts an ApacheDS KDC server for use in testing Kerberos integration - hopefully this could also be used to create unit tests for GIRAPH-211 .

            People

            • Assignee:
              ekoontz Eugene Koontz
              Reporter:
              ekoontz Eugene Koontz
            • Votes:
              3 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development