Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Description
Gianmarco De Francisci Morales asked on the user list:
I am getting the exception in the subject when running my giraph program
on a cluster with Kerberos authentication.
This leads to the idea of having Kerberos authentication supported within GIRAPH. Hopefully it would use our fast GIRAPH-37 IPC, but could also interoperate with Hadoop security.
Attachments
Attachments
- GIRAPH-211.2.patch
- 104 kB
- Avery Ching
- GIRAPH-211.3.patch
- 104 kB
- Avery Ching
- GIRAPH-211.4.patch
- 91 kB
- Avery Ching
- GIRAPH-211.patch
- 91 kB
- Eugene Joseph Koontz
- GIRAPH-211.patch
- 84 kB
- Eugene Joseph Koontz
- GIRAPH-211.patch
- 87 kB
- Eugene Joseph Koontz
- GIRAPH-211.patch
- 105 kB
- Eugene Joseph Koontz
- GIRAPH-211.patch
- 69 kB
- Eugene Joseph Koontz
- GIRAPH-211.patch
- 84 kB
- Eugene Joseph Koontz
- GIRAPH-211.patch
- 81 kB
- Eugene Joseph Koontz
- GIRAPH-211.patch
- 85 kB
- Eugene Joseph Koontz
- GIRAPH-211.patch
- 52 kB
- Eugene Joseph Koontz
- GIRAPH-211.patch
- 110 kB
- Eugene Joseph Koontz
- GIRAPH-211.patch
- 105 kB
- Eugene Joseph Koontz
- GIRAPH-211-proposal.txt
- 8 kB
- Eugene Joseph Koontz
Issue Links
- blocks
-
GIRAPH-200 remove hadoop RPC and keep just netty
- Resolved
-
GIRAPH-282 Make netty the default RPC transport
- Resolved
- is depended upon by
-
GIRAPH-852 Add SASL unit tests
- Open
- is related to
-
GIRAPH-363 Fix hadoop_0.23 profile broken by GIRAPH-211
- Resolved
-
GIRAPH-583 Problem with authentication on Hadoop 0.23
- Resolved
-
GIRAPH-265 Enable Zookeeper security support within Giraph
- Open
-
GIRAPH-200 remove hadoop RPC and keep just netty
- Resolved
-
GIRAPH-262 Netty optimization to handle requests locally whenever possible
- Resolved
- relates to
-
GIRAPH-212 Security is busted since GIRAPH-168
- Resolved
-
GIRAPH-37 Implement Netty-backed IPC
- Resolved
-
AVRO-405 Netty-based Java RPC server
- Closed
-
AVRO-641 add SASL to socket transport
- Closed
-
HADOOP-4487 Security features for Hadoop
- Closed
-
HADOOP-6419 Change RPC layer to support SASL based mutual authentication
- Closed
-
ZOOKEEPER-823 update ZooKeeper java client to optionally use Netty for connections
- Closed
-
GIRAPH-362 Address master task id for communication for master (known issue from GIRAPH-211)
- Open
-
GIRAPH-262 Netty optimization to handle requests locally whenever possible
- Resolved
-
GIRAPH-289 Add thread and channel pooling to NettyClient and NettyServer
- Resolved
-
HADOOP-8078 Add capability to turn on security in unit tests.
- Resolved
Activity
Certificates/keys don't scale. It would be better to piggyback on Hadoop's delegation token system, certainly while we're on HMR and even later, when we're on YARN.
Interoperability with Hadoop security is a prerequisite for any use of Giraph for our use cases.
Okay, I suppose we can use the BspPolicyProvider, JobTorkenIdentifier, JobTokenSecretManager, etc. to build a light layer of security in Netty. Maja, you can start looking at RPCCommunications.java to see how feasible it is to use the same ideas from the RPC implementation to the Netty implementation.
I'm not sure what am I supposed to do here. My first understanding was that we want all the messages which go through the system to be protected. In that case I can't see how workers can find out about tokens of others. I guess RPC authentication is implemented in hadoop, since the only thing I see in RPC classes in Giraph is the part when the job checks whether it has privileges to make a connection. I need a bit more information here, or I can reassign the task if someone finds it easier to do it than to explain it
My first understanding was that we want all the messages which go through the system to be protected.
What Hadoop did fundamentally is wrap their RPC with SASL at the socket level, and then could use existing JRE support for SASL negotiation with Kerberos authentication (and transparent encryption, etc).
Eugene could comment better, but what we did for ZooKeeper, which has IO also based on Netty, is instead tunnel the SASL authentication handshake as an extension to the existing protocol, and introduced a mode which requires that handshake to complete successfully before accepting other message types which require authenticated access.
This JIRA is going to require a pretty good understanding of Hadoop's security. It may not be the best for a new contributor...
I'm happy to help Maja or take over this JIRA - either way I'll be following it closely! Maybe a hackathon or meetup would be a good idea.
I should mention that Zookeeper supports both NIO and Netty on the server, but there is currently no Netty implementation on the Zookeeper client side: see https://issues.apache.org/jira/browse/ZOOKEEPER-823 (which is not resolved at this time).
Eugene, I'm reassigning it to you. Really appreciate your offer to help, but if it requires a lot of background and we need it fast it's better this way.
I am working on a patch for this after studying both the Hadoop RPC side and the Netty side of things. I'm posting a proposal, but, since it's long, I'll attach it as a .txt. But to get the discussion started, I want to describe the idea briefly: it uses the Job Token as a SASL credential so that one BSPWorker can authenticate itself as a client to another BSPWorker acting as a server. This is what is going on with Hadoop RPC, so we just need to replicate this behavior in the Netty implementation.
Thanks for working on this Eugene. Can't wait to get rid of HadoopRPC so we only have to support one IPC instead of two.
Hey Eugene, your proposal look good btw. Let me know if you need any help doing this. I'd like to get it in asap as this is causing a lot of cruft to build up. We don't need to do the ZK security for now to remove HadoopRPC.
Sounds great. I was thinking about basing it on the SecureShuffleUtils.java and ReduceTask.java, but you seem to have a more complex idea. Looking forward to it.
This patch allows SASL authentication using the JobToken, as described in
https://issues.apache.org/jira/secure/attachment/12537279/GIRAPH-211-proposal.txt.
Limitations (will follow up with new JIRAs as warranted)
-Authorization is not done: that is, clients are authenticated but there are no restrictions on their ability to do RPC on the servers.
-Clients should wait for authentication before trying to do RPC - once authorization (see above) is done, they might encounter a race where they try to do RPCs without yet being authenticated.
-Not tested on other than hadoop 2.0.1-SNAPSHOT
-Only works if we disable client-side channel-pooling (GIRAPH-289) and local short-circuiting of RPCs (GIRAPH-262) - these should be configurable but currently, I hard-wired both to be disabled.
How to compile:
mvn -Phadoop_2.0.1 clean test
Works with the following test:
$HADOOP_RUNTIME/bin/hadoop jar $GIRAPH_DIR/target/giraph-0.2-SNAPSHOT-for-hadoop-2.0.1-SNAPSHOT-jar-with-dependencies.jar org.apache.giraph.benchmark.PageRankBenchmark -Dgiraph.useNetty=true -e 2 -s 10 -v -V 2 -w 2
Here's some output showing client and server SASL communication based on the above test:
[ec2-user@ip-10-176-255-76 giraph]$ cat /tmp/logs/application_1344445498302_0016/container_1344445498302_0016_01_000002/syslog | grep -i giraph | grep -i sasl 2012-08-08 20:58:52,806 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.NettyClient: creating saslNettyClient now. 2012-08-08 20:58:52,816 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.NettyClient: creating saslNettyClient now. 2012-08-08 20:58:52,929 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslNettyClient: Creating SASL DIGEST-MD5 client to authenticate to service at job_1344445498302_0016 2012-08-08 20:58:52,931 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslNettyClient: Creating SASL DIGEST-MD5 client to authenticate to service at job_1344445498302_0016 2012-08-08 20:58:52,943 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.NettyClient: storing saslNettyClient at key: [id: 0x7e8905bd, /10.176.255.76:48238 => ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30000] 2012-08-08 20:58:52,943 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.NettyClient: created: org.apache.giraph.comm.SaslNettyClient@2aa937cd 2012-08-08 20:58:52,943 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.NettyClient: sending request: org.apache.giraph.comm.SaslTokenMessage@4b8e899c to remote server: ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30000 2012-08-08 20:58:52,943 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.RequestEncoder: encoding object: org.apache.giraph.comm.SaslTokenMessage@4b8e899c 2012-08-08 20:58:52,947 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: write() called on token of length: 0 2012-08-08 20:58:52,947 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: write() finished writing token. 2012-08-08 20:58:52,947 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.RequestEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:52,954 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.NettyClient: storing saslNettyClient at key: [id: 0x04df8b14, /10.176.255.76:60743 => ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30001] 2012-08-08 20:58:52,954 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.NettyClient: created: org.apache.giraph.comm.SaslNettyClient@23faa614 2012-08-08 20:58:52,954 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.NettyClient: sending request: org.apache.giraph.comm.SaslTokenMessage@1cad7d80 to remote server: ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30001 2012-08-08 20:58:52,954 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.RequestEncoder: encoding object: org.apache.giraph.comm.SaslTokenMessage@1cad7d80 2012-08-08 20:58:52,954 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: write() called on token of length: 0 2012-08-08 20:58:52,954 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: write() finished writing token. 2012-08-08 20:58:52,954 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.RequestEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:52,959 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestDecoder: decode: Got a request of type SASL_TOKEN from remote:/10.176.255.76:48238 2012-08-08 20:58:52,959 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_TOKEN 2012-08-08 20:58:52,959 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: reading fields from DataInput: org.jboss.netty.buffer.ChannelBufferInputStream@2b8ca663 2012-08-08 20:58:52,959 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: token size is: 0 2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: now reading token. 2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: read SASL token of length: 0 2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestDecoder: decoded an object of type: class org.apache.giraph.comm.SaslTokenMessage:org.apache.giraph.comm.SaslTokenMessage@1effc3eb 2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestServerHandler: Got class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestServerHandler: calling doRequest on class: class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: looking for saslNettyServer on server:/10.176.255.76:30000 for client /10.176.255.76:48238 2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: creating SaslNettyServer with secret manager: org.apache.hadoop.mapreduce.security.token.JobTokenSecretManager@658782a7 2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: creating saslNettyServer: secret manager is: org.apache.hadoop.mapreduce.security.token.JobTokenSecretManager@658782a7 2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: checked read-availability; now creating sasl server. 2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: create... 2012-08-08 20:58:52,960 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: creating callback handler.. 2012-08-08 20:58:52,961 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: creating SaslDigestCallback handler with secret manager: org.apache.hadoop.mapreduce.security.token.JobTokenSecretManager@658782a7 2012-08-08 20:58:52,961 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: created callback handler: org.apache.giraph.comm.SaslNettyServer$SaslDigestCallbackHandler@1bb0889a 2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: create was successful. 2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: client's token's length is: 0 2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: checking completeness of SASL (pre-response):false 2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: responding to input token of length: 0 2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: response token length: 108 2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: checking completeness of SASL (post-response):false 2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: server's token's length is: 108 2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: Responding to client: /10.176.255.76:48238 2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.ResponseEncoder: RESPONSE ENCODER IS ENCODING A MESSAGE:org.apache.giraph.comm.SaslTokenMessage@1effc3eb 2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.ResponseEncoder: encoding object: org.apache.giraph.comm.SaslTokenMessage@1effc3eb 2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.ResponseEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:52,967 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: write() called on token of length: 108 2012-08-08 20:58:52,968 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: write() finished writing token. 2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: decode: Got a response of type SASL_TOKEN from server:ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30000 2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_TOKEN 2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: writableRequestClass: class org.apache.giraph.comm.SaslTokenMessage found for type:SASL_TOKEN 2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: reading fields of server response:org.apache.giraph.comm.SaslTokenMessage@48c5186e 2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: reading fields from DataInput: org.jboss.netty.buffer.ChannelBufferInputStream@3c70315 2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: token size is: 108 2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: now reading token. 2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: read SASL token of length: 108 2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: finished reading fields of server response:org.apache.giraph.comm.SaslTokenMessage@48c5186e 2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseClientHandler: message received: org.apache.giraph.comm.SaslTokenMessage@48c5186e 2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslNettyClient: SASL client callback: setting username: FmpvYl8xMzQ0NDQ1NDk4MzAyXzAwMTY= 2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslNettyClient: SASL client callback: setting userPassword 2012-08-08 20:58:52,968 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslNettyClient: SASL client callback: setting realm: default 2012-08-08 20:58:52,969 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: creating SASL message with size: 274 2012-08-08 20:58:52,969 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.NettyClient: sending request: org.apache.giraph.comm.SaslTokenMessage@7e9b59a2 to remote server: ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30000 2012-08-08 20:58:52,969 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.RequestEncoder: encoding object: org.apache.giraph.comm.SaslTokenMessage@7e9b59a2 2012-08-08 20:58:52,969 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: write() called on token of length: 274 2012-08-08 20:58:52,969 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: write() finished writing token. 2012-08-08 20:58:52,969 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.RequestEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:52,969 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseClientHandler: due to reception of: org.apache.giraph.comm.SaslTokenMessage@48c5186e 2012-08-08 20:58:52,970 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: Finished writing token to client. 2012-08-08 20:58:52,970 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestServerHandler: done with doRequest() for class: class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestDecoder: decode: Got a request of type SASL_TOKEN from remote:/10.176.255.76:48238 2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_TOKEN 2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: reading fields from DataInput: org.jboss.netty.buffer.ChannelBufferInputStream@919db9e 2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: token size is: 274 2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: now reading token. 2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: read SASL token of length: 274 2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestDecoder: decoded an object of type: class org.apache.giraph.comm.SaslTokenMessage:org.apache.giraph.comm.SaslTokenMessage@a166bd 2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestServerHandler: Got class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestServerHandler: calling doRequest on class: class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: looking for saslNettyServer on server:/10.176.255.76:30000 for client /10.176.255.76:48238 2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: client's token's length is: 274 2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: checking completeness of SASL (pre-response):false 2012-08-08 20:58:52,971 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: responding to input token of length: 274 2012-08-08 20:58:52,981 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: SASL server DIGEST-MD5 callback: setting password for client: job_1344445498302_0016 (auth:SIMPLE) 2012-08-08 20:58:52,981 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: SASL server DIGEST-MD5 callback: setting canonicalized client ID: job_1344445498302_0016 2012-08-08 20:58:52,982 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslNettyServer: response token length: 40 2012-08-08 20:58:52,982 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: checking completeness of SASL (post-response):true 2012-08-08 20:58:52,982 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: server's token's length is: 40 2012-08-08 20:58:52,982 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: Responding to client: /10.176.255.76:48238 2012-08-08 20:58:52,982 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.ResponseEncoder: RESPONSE ENCODER IS ENCODING A MESSAGE:org.apache.giraph.comm.SaslTokenMessage@a166bd 2012-08-08 20:58:52,982 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.ResponseEncoder: encoding object: org.apache.giraph.comm.SaslTokenMessage@a166bd 2012-08-08 20:58:52,982 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.ResponseEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:52,982 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: write() called on token of length: 40 2012-08-08 20:58:52,982 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: write() finished writing token. 2012-08-08 20:58:52,982 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: decode: Got a response of type SASL_TOKEN from server:ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30000 2012-08-08 20:58:52,982 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_TOKEN 2012-08-08 20:58:52,983 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: writableRequestClass: class org.apache.giraph.comm.SaslTokenMessage found for type:SASL_TOKEN 2012-08-08 20:58:52,983 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: reading fields of server response:org.apache.giraph.comm.SaslTokenMessage@4b6c06dd 2012-08-08 20:58:52,983 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: reading fields from DataInput: org.jboss.netty.buffer.ChannelBufferInputStream@2136bdda 2012-08-08 20:58:52,983 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: token size is: 40 2012-08-08 20:58:52,983 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: now reading token. 2012-08-08 20:58:52,983 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslTokenMessage: read SASL token of length: 40 2012-08-08 20:58:52,983 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: finished reading fields of server response:org.apache.giraph.comm.SaslTokenMessage@4b6c06dd 2012-08-08 20:58:52,983 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseClientHandler: message received: org.apache.giraph.comm.SaslTokenMessage@4b6c06dd 2012-08-08 20:58:52,987 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: Finished writing token to client. 2012-08-08 20:58:52,987 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: writing out complete token now. 2012-08-08 20:58:52,987 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.ResponseEncoder: RESPONSE ENCODER IS ENCODING A MESSAGE:org.apache.giraph.comm.SaslComplete@36511e0a 2012-08-08 20:58:52,987 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.ResponseEncoder: encoding object: org.apache.giraph.comm.SaslComplete@36511e0a 2012-08-08 20:58:52,987 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.ResponseEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslComplete 2012-08-08 20:58:52,987 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslComplete: writing one byte (write). 2012-08-08 20:58:52,987 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.SaslTokenMessage: wrote out complete token now. 2012-08-08 20:58:52,987 DEBUG [New I/O server worker #1-1] org.apache.giraph.comm.RequestServerHandler: done with doRequest() for class: class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:53,009 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestDecoder: decode: Got a request of type SASL_TOKEN from remote:/10.176.255.76:48239 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_TOKEN 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: reading fields from DataInput: org.jboss.netty.buffer.ChannelBufferInputStream@595780d9 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: token size is: 0 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: now reading token. 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: read SASL token of length: 0 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestDecoder: decoded an object of type: class org.apache.giraph.comm.SaslTokenMessage:org.apache.giraph.comm.SaslTokenMessage@7fcc5461 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestServerHandler: Got class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestServerHandler: calling doRequest on class: class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: looking for saslNettyServer on server:/10.176.255.76:30000 for client /10.176.255.76:48239 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: creating SaslNettyServer with secret manager: org.apache.hadoop.mapreduce.security.token.JobTokenSecretManager@658782a7 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: creating saslNettyServer: secret manager is: org.apache.hadoop.mapreduce.security.token.JobTokenSecretManager@658782a7 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: checked read-availability; now creating sasl server. 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: create... 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: creating callback handler.. 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: creating SaslDigestCallback handler with secret manager: org.apache.hadoop.mapreduce.security.token.JobTokenSecretManager@658782a7 2012-08-08 20:58:53,010 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: created callback handler: org.apache.giraph.comm.SaslNettyServer$SaslDigestCallbackHandler@2ecc5436 2012-08-08 20:58:53,011 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: create was successful. 2012-08-08 20:58:53,011 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: client's token's length is: 0 2012-08-08 20:58:53,011 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: checking completeness of SASL (pre-response):false 2012-08-08 20:58:53,011 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: responding to input token of length: 0 2012-08-08 20:58:53,011 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: response token length: 108 2012-08-08 20:58:53,011 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: checking completeness of SASL (post-response):false 2012-08-08 20:58:53,011 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: server's token's length is: 108 2012-08-08 20:58:53,011 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: Responding to client: /10.176.255.76:48239 2012-08-08 20:58:53,011 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.ResponseEncoder: RESPONSE ENCODER IS ENCODING A MESSAGE:org.apache.giraph.comm.SaslTokenMessage@7fcc5461 2012-08-08 20:58:53,012 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.ResponseEncoder: encoding object: org.apache.giraph.comm.SaslTokenMessage@7fcc5461 2012-08-08 20:58:53,012 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.ResponseEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:53,012 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: write() called on token of length: 108 2012-08-08 20:58:53,012 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: write() finished writing token. 2012-08-08 20:58:53,012 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: Finished writing token to client. 2012-08-08 20:58:53,012 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestServerHandler: done with doRequest() for class: class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestDecoder: decode: Got a request of type SASL_TOKEN from remote:/10.176.255.76:48239 2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_TOKEN 2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: reading fields from DataInput: org.jboss.netty.buffer.ChannelBufferInputStream@24bb6086 2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: token size is: 274 2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: now reading token. 2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: read SASL token of length: 274 2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestDecoder: decoded an object of type: class org.apache.giraph.comm.SaslTokenMessage:org.apache.giraph.comm.SaslTokenMessage@9be1041 2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestServerHandler: Got class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestServerHandler: calling doRequest on class: class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: looking for saslNettyServer on server:/10.176.255.76:30000 for client /10.176.255.76:48239 2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: client's token's length is: 274 2012-08-08 20:58:53,018 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: checking completeness of SASL (pre-response):false 2012-08-08 20:58:53,019 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: responding to input token of length: 274 2012-08-08 20:58:53,019 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: SASL server DIGEST-MD5 callback: setting password for client: job_1344445498302_0016 (auth:SIMPLE) 2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: decode: Got a response of type SASL_TOKEN from server:ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30001 2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_TOKEN 2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: writableRequestClass: class org.apache.giraph.comm.SaslTokenMessage found for type:SASL_TOKEN 2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: reading fields of server response:org.apache.giraph.comm.SaslTokenMessage@1d1d2066 2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: reading fields from DataInput: org.jboss.netty.buffer.ChannelBufferInputStream@633a1778 2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: token size is: 108 2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: now reading token. 2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: read SASL token of length: 108 2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: finished reading fields of server response:org.apache.giraph.comm.SaslTokenMessage@1d1d2066 2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseClientHandler: message received: org.apache.giraph.comm.SaslTokenMessage@1d1d2066 2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslNettyClient: SASL client callback: setting username: FmpvYl8xMzQ0NDQ1NDk4MzAyXzAwMTY= 2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslNettyClient: SASL client callback: setting userPassword 2012-08-08 20:58:53,020 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslNettyClient: SASL client callback: setting realm: default 2012-08-08 20:58:53,021 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: creating SASL message with size: 274 2012-08-08 20:58:53,021 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.NettyClient: sending request: org.apache.giraph.comm.SaslTokenMessage@7adafa2c to remote server: ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30001 2012-08-08 20:58:53,021 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.RequestEncoder: encoding object: org.apache.giraph.comm.SaslTokenMessage@7adafa2c 2012-08-08 20:58:53,021 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: write() called on token of length: 274 2012-08-08 20:58:53,021 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: write() finished writing token. 2012-08-08 20:58:53,021 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.RequestEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:53,021 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseClientHandler: due to reception of: org.apache.giraph.comm.SaslTokenMessage@1d1d2066 2012-08-08 20:58:53,021 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: SASL server DIGEST-MD5 callback: setting canonicalized client ID: job_1344445498302_0016 2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslNettyServer: response token length: 40 2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: checking completeness of SASL (post-response):true 2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: server's token's length is: 40 2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: Responding to client: /10.176.255.76:48239 2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.ResponseEncoder: RESPONSE ENCODER IS ENCODING A MESSAGE:org.apache.giraph.comm.SaslTokenMessage@9be1041 2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.ResponseEncoder: encoding object: org.apache.giraph.comm.SaslTokenMessage@9be1041 2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.ResponseEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: write() called on token of length: 40 2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: write() finished writing token. 2012-08-08 20:58:53,022 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: Finished writing token to client. 2012-08-08 20:58:53,023 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: writing out complete token now. 2012-08-08 20:58:53,023 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.ResponseEncoder: RESPONSE ENCODER IS ENCODING A MESSAGE:org.apache.giraph.comm.SaslComplete@3e5e9db7 2012-08-08 20:58:53,023 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.ResponseEncoder: encoding object: org.apache.giraph.comm.SaslComplete@3e5e9db7 2012-08-08 20:58:53,023 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.ResponseEncoder: encode: Encoding a message of type class org.apache.giraph.comm.SaslComplete 2012-08-08 20:58:53,023 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: decode: Got a response of type SASL_COMPLETE from server:ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30000 2012-08-08 20:58:53,023 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_COMPLETE 2012-08-08 20:58:53,023 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: writableRequestClass: class org.apache.giraph.comm.SaslComplete found for type:SASL_COMPLETE 2012-08-08 20:58:53,023 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: reading fields of server response:org.apache.giraph.comm.SaslComplete@75fc25e5 2012-08-08 20:58:53,023 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.SaslComplete: read fields (none) 2012-08-08 20:58:53,024 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseDecoder: finished reading fields of server response:org.apache.giraph.comm.SaslComplete@75fc25e5 2012-08-08 20:58:53,024 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseClientHandler: message received: org.apache.giraph.comm.SaslComplete@75fc25e5 2012-08-08 20:58:53,024 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseClientHandler: SASL Authentication complete with server: ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30000. 2012-08-08 20:58:53,024 DEBUG [New I/O client worker #1-1] org.apache.giraph.comm.ResponseClientHandler: due to reception of: org.apache.giraph.comm.SaslComplete@75fc25e5 2012-08-08 20:58:53,023 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslComplete: writing one byte (write). 2012-08-08 20:58:53,024 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.SaslTokenMessage: wrote out complete token now. 2012-08-08 20:58:53,024 DEBUG [New I/O server worker #1-2] org.apache.giraph.comm.RequestServerHandler: done with doRequest() for class: class org.apache.giraph.comm.SaslTokenMessage 2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: decode: Got a response of type SASL_TOKEN from server:ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30001 2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_TOKEN 2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: writableRequestClass: class org.apache.giraph.comm.SaslTokenMessage found for type:SASL_TOKEN 2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: reading fields of server response:org.apache.giraph.comm.SaslTokenMessage@3fe88b35 2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: reading fields from DataInput: org.jboss.netty.buffer.ChannelBufferInputStream@3c0b655a 2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: token size is: 40 2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: now reading token. 2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslTokenMessage: read SASL token of length: 40 2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: finished reading fields of server response:org.apache.giraph.comm.SaslTokenMessage@3fe88b35 2012-08-08 20:58:53,031 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseClientHandler: message received: org.apache.giraph.comm.SaslTokenMessage@3fe88b35 2012-08-08 20:58:53,071 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: decode: Got a response of type SASL_COMPLETE from server:ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30001 2012-08-08 20:58:53,071 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.RequestRegistry: getClass() looking for type: SASL_COMPLETE 2012-08-08 20:58:53,071 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: writableRequestClass: class org.apache.giraph.comm.SaslComplete found for type:SASL_COMPLETE 2012-08-08 20:58:53,071 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: reading fields of server response:org.apache.giraph.comm.SaslComplete@156a9424 2012-08-08 20:58:53,071 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.SaslComplete: read fields (none) 2012-08-08 20:58:53,071 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseDecoder: finished reading fields of server response:org.apache.giraph.comm.SaslComplete@156a9424 2012-08-08 20:58:53,071 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseClientHandler: message received: org.apache.giraph.comm.SaslComplete@156a9424 2012-08-08 20:58:53,071 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseClientHandler: SASL Authentication complete with server: ip-10-176-255-76.us-west-1.compute.internal/10.176.255.76:30001. 2012-08-08 20:58:53,071 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.ResponseClientHandler: due to reception of: org.apache.giraph.comm.SaslComplete@156a9424
Tried to create a new review but hit 500 error; submitted this: https://issues.apache.org/jira/browse/INFRA-5128
Eugene, this is a nice start! A few comments/questions about the limitations:
>-Authorization is not done: that is, clients are authenticated but there are no restrictions on their ability to do RPC on the servers.
Can't we block until the authentication is done?
>-Clients should wait for authentication before trying to do RPC - once authorization (see above) is done, they might encounter a race where they try to do RPCs without yet being authenticated.
Same question as above?
>-Not tested on other than hadoop 2.0.1-SNAPSHOT
Would probably be nice to try on a hadoop 1.0.x if you have a chance.
>-Only works if we disable client-side channel-pooling (GIRAPH-289) and local short-circuiting of RPCs (GIRAPH-262) - these should be configurable but currently, I hard-wired both to be disabled.
I think it's reasonable to allow short-circuiting since there isn't a security issue here (this is the same process). As for channel pooling, can we simply authenticate once per channel?
Thanks for your comments Avery! Replying to each below:
> Can't we block until the authentication is done?
Absolutely - we just need to choose the object to synchronize on and be careful that it's respected where it needs to be. Will include this in next patch, which will also avoid the race.
> Would probably be nice to try on a hadoop 1.0.x if you have a chance.
Will do.
> I think it's reasonable to allow short-circuiting since there isn't a security issue here (this is the same process).
Yes, I think it will be ok to trust the client in this case (specifically in GIRAPH-262 we are testing for):
(service.getWorkerInfo().getHostnamePort().equals(remoteServerAddress)) {
I just need to fix whatever bug I am masking by disabling short-curcuiting.
> As for channel pooling, can we simply authenticate once per channel?
Yes, I just need to figure out how to associate the sasl client with the channel when we are using channel-pooling. I use the Address => Channel map in my patch, but now with GIRAPH-262, we have Address => ChannelRotator. So this will require some changes on my patch.
Added review: https://reviews.apache.org/r/6609/
Thanks to pctony for enabling git-based reviews: https://issues.apache.org/jira/browse/INFRA-5128
Any update on this? I don't know the schedule for cutting the 0.2 release, but I'd say this is a prerequisite: not only we get rid of a lot of cruft, but also the API currently includes methods (putMessages/getMessages) that are used only by the Hadoop RPC implementation.
Hi Alessandro, I am working actively on it and hope to have a patch ready in the next day or so.
-Eugene
Improves on recent patch:
-Does authorization: server checks for authentication being completed before allowing non-SASL client requests to be performed.
-Client waits to send (non-SASL) requests until SASL authentication has completed.
-Works with channel-pooling and local short-circuiting
Limitations:
-Not tested on other than hadoop 2.0.1-SNAPSHOT
-Restores RequestRegistry class, which was removed in GIRAPH-313 - hope to merge better with this.
-Needs more testing in general, including new unit tests.
I manually fixed some conflicts recently, so I'm gradually making this patch more closely fit to trunk.
Improvements:
-Removes usage of RequestRegistry (that was removed in GIRAPH-313)
Limitations:
Uses preGIRAPH-313 workerThreadPool rather than requestServerHandlerFactory
-Needs tests
Sorry, the last patch was missing the new source files - no wonder it was so small.
https://reviews.apache.org/r/6609/diff/ updated to most recent patch.
Improvements:
Removes remaining preGIRAPH-313-isms in NettyServer and NettyWorkerClient
https://reviews.apache.org/r/6609/diff/ updated to most recent patch (will refrain from making this comment from now on and simply keep them synced).
merge with trunk: 55bfcb78 https://svn.apache.org/repos/asf/giraph/trunk@1378761 (GIRAPH-317)
Interesting and related: "Handshaking tutorial with Netty" by Bruno de Carvalho here:
My apologies for the delay. I am in the final stages : getting it to compile on all supported Hadoop versions and fixing checkstyle errors. I have the SASL code implemented as a ChannelHandler as we discussed. Here's what the pipeline looks like on the client side, showing both authenticated and non-authenticated cases:
// Set up the pipeline factory. bootstrap.setPipelineFactory(new ChannelPipelineFactory() { @Override public ChannelPipeline getPipeline() throws Exception { if (conf.getBoolean(GiraphJob.AUTHENTICATE, GiraphJob.DEFAULT_AUTHENTICATE)) { LOG.info("Using Netty with authentication."); // Our pipeline starts with just byteCounter, and then we use // addAfter() to incrementally add pipeline elements, so that we can // name them for identification for removal or replacement after // client is authenticated by server. ChannelPipeline pipeline = Channels.pipeline( byteCounter); // After authentication finishes, the following is replaced with // FixedLengthFrameDecoder (as in non-auth pipeline below): pipeline.addLast("length-field-based-frame-decoder", new LengthFieldBasedFrameDecoder(1024, 0, 4, 0, 4)); pipeline.addLast("request-encoder", new RequestEncoder()); // After authentication finishes, the following is removed: pipeline.addLast("sasl-client-handler", new SaslClientHandler(conf)); pipeline.addLast("response-handler", new ResponseClientHandler(clientRequestIdRequestInfoMap, conf)); return pipeline; } else { LOG.info("Using Netty without authentication."); return Channels.pipeline( byteCounter, new FixedLengthFrameDecoder(RequestServerHandler.RESPONSE_BYTES), new RequestEncoder(), new ResponseClientHandler(clientRequestIdRequestInfoMap, conf)); } } });
Eugene, this looks nice! I still see some checkstyle stuff but you're working through it I guess. Overall, the design is good. If we think about improvements we can make them later on.
Questions/Comments:
For consistency, can you please convert multi-line comments like
/** Whether to use SASL with DIGEST and Hadoop Job Tokens to authenticate * and authorize Netty BSP Clients to Servers. */
to
/** * Whether to use SASL with DIGEST and Hadoop Job Tokens to authenticate * and authorize Netty BSP Clients to Servers. */
Can we get rename Authorize to AuthorizeServerHandler or something else more descriptive?
NettyClient.java
- 372: Please wrap the LOG.info() with if (LOG.isInfoEnabled()).
- 545-557: Can't we just go through the regular netty request part of the code? We don't need to have -2 here and can just submit the destWorkerId?
SASL_COMPLETE -> SASL_COMPLETE_REQUEST?
SaslTokenMessage.java can we call is SaslTokenMessageRequest?
SaslComplete.java can we call it SaslCompleteRequest to match the other names?
SaslComplete.java
- 29-34: Why not get rid of these?
SaslTokenMessage.java:
- 86: Extra line
Thanks again, this was a lot of work!
Hi Avery,
Thanks a lot for your comments - are you sure about checkstyle? I thought I got it down to zero errors. Will look into it and reply to your other comments.
-Eugene
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12545806/GIRAPH-211.patch
against trunk revision 1387840.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 javac. The patch appears to cause the build to fail.
Console output: https://builds.apache.org/job/PreCommit-GIRAPH-Build/41//console
This message is automatically generated.
Eugene, version 6 of the diff seems to be the same as the one I reviewed. Can you please address my comments above in https://issues.apache.org/jira/browse/GIRAPH-211?focusedCommentId=13459279&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13459279 ?
Let me know if you need any help on this one. I'm looking forward into putting this in!
Thanks Avery. I appreciate your help. I am hoping to 1) address your comments and 2) get mvn verify to work with all profiles in a new patch in the next 2 days.
-Eugene
Getting closer.
-Passes:
mvn -Phadoop_non_secure clean verify && mvn -Phadoop_1.0 clean verify && mvn -Phadoop_2.0.0 clean verify && mvn -Phadoop_2.0.1 clean verify && mvn -Phadoop_2.0.2 clean verify
(I added profiles for hadoop 2.0.1 and 2.0.2 since there are
releases available for these newer Hadoop versions).
-Addressed most of Avery's comments on reviewboard - thank you Avery. Still,
SaslTokenMessageRequests do not properly pass the
destWorkerId (it still uses -2 as the destWorkerId). Working on addressing this.
-Added new test, SaslConnectionTest, adapted from ConnectionTest. Passes,
but needs to be expanded. I need to better understand how
Mockito can be used to mock Job Tokens, so would appreciate help
with how to use Mockito for this.
-Incorporates GIRAPH-212's patch: this fixes munge flags associated with security.
We just committed GIRAPH-212, so only thing is to fix the -2 thing and add a test? I'd really like to get this in the next day or two. If you're busy Eugene Joseph Koontz, let me know and I can try and massage it a bit.
Hi Avery, That's correct - we just need to fix the -2 thing and improve the test. I could use some help with either or both
Thanks a lot.
-Eugene
Merged with trunk: no longer needs GIRAPH-212 inclusion thanks to Avery's commit of this.
Small munge flag name change: changed HADOOP_1_AUTHORIZATION to HADOOP_1_SECURITY.
Here is an attached patch with the following changes from GIRAPH-211.patch:
Addressed the -2 worker id by adding the task id to ChannelRotater.
SaslComplete -> SaslCompleteRequest
SASL_TOKEN_MESSAGE -> SASL_TOKEN_MESSAGE_REQUEST
Fixed all minor things (javadoc, functions, added authenticate()).
mvn -Phadoop=facebook compile fixed
Tested with
mvn -Phadoop_non_secure clean verify
- passed
mvn -Phadoop_1.0 clean verify
= passed
mvn -Phadoop_2.0.0 clean verify - passed
mvn -Phadoop_2.0.1 clean verify - passed
mvn -Phadoop=facebook compile - passed
Note that hadoop-2.0.2 wasn't tested (I don't have the artifacts), but should work.
I know that we should add more testing, but I think this is good to go for now. It seems to work and it's already very huge. Great work Eugene Joseph Koontz! Please let me know if the changes I made are acceptable, I'm fine with the current version. We should commit soon.
Wow, thanks so much Avery. I'm going to try out your patch today. Will also try with hadoop-2.0.2.
-Eugene
Avery, modified your patch to fix one problem: a NPE in NettyClient.sendWritableRequest() when the supplied destWorkerId param is null. This null param happens when requests are made to the master when sendWritableRequest() is called by NettyClient.authenticateOnChannel().
With Avery's fix for GIRAPH-360, we associate each request with the taskId of the worker that is making the request, and use a pair <taskid,requestid> as the key for tracking requests in the clientRequestIdRequestInfoMap. However, the Giraph master is not associated with a taskId, and so its taskid is null.
It wouldn't be a problem that the master has a null task id for normal Giraph requests, but for authentication, workers need to authenticate with the master just as they must do with the other workers.
This patch works around the problem by simply not registering authentication requests (requests with type=SASL_TOKEN_MESSAGE_REQUEST) with the clientRequestIdRequestInfoMap. In other words, it's another workaround like the "-2" presence in my previous patches. I know this is not ideal, but I wanted to keep the momentum going for this JIRA.
Rather than this workaround, I thought a possible solution would be a separate client-side pipeline component that handles request tracking. We'd move the clientRequestIdRequestInfoMap into this component, which would handles the tracking functionality that currently is spread across NettyClient.sendWritableRequest(), .waitAllRequests(), and ResponseClientHandler.messageReceived().
Or, maybe I'm missing something obvious and easy
Tested this patch successfully with:
mvn -Phadoop_non_secure clean verify && mvn -Phadoop_0.20.203 clean verify && mvn -Phadoop_1.0 clean verify && mvn -Phadoop0.23 clean verify && mvn -Phadoop_2.0.0 clean verify && mvn -Phadoop_2.0.1 clean verify && mvn -Phadoop_2.0.2 clean verify
+1, will also create a follow up task to fix up this weirdness with the master. Thanks for all the hard work Eugene!
Same as last patch except adds what I mistakenly left out of last patch:
src/main/java/org/apache/giraph/comm/netty/handler/AuthorizeServerHandler.java
src/main/java/org/apache/giraph/comm/netty/handler/ResponseEncoder.java
Integrated in Giraph-trunk-Commit #233 (See https://builds.apache.org/job/Giraph-trunk-Commit/233/)
GIRAPH-211: Add secure authentication to Netty IPC (Revision 1396722)
Result = SUCCESS
ekoontz : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1396722
Files :
- /giraph/trunk/pom.xml
- /giraph/trunk/src/main/java/org/apache/giraph/GiraphConfiguration.java
- /giraph/trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java
- /giraph/trunk/src/main/java/org/apache/giraph/comm/CommunicationsInterface.java
- /giraph/trunk/src/main/java/org/apache/giraph/comm/SecureRPCCommunications.java
- /giraph/trunk/src/main/java/org/apache/giraph/comm/WorkerClient.java
- /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/NettyClient.java
- /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/NettyServer.java
- /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClient.java
- /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClientServer.java
- /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/SaslNettyClient.java
- /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/SaslNettyServer.java
- /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/handler/AuthorizeServerHandler.java
- /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/handler/RequestEncoder.java
- /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/handler/ResponseEncoder.java
- /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/handler/SaslClientHandler.java
- /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/handler/SaslServerHandler.java
- /giraph/trunk/src/main/java/org/apache/giraph/comm/requests/RequestType.java
- /giraph/trunk/src/main/java/org/apache/giraph/comm/requests/SaslCompleteRequest.java
- /giraph/trunk/src/main/java/org/apache/giraph/comm/requests/SaslTokenMessageRequest.java
- /giraph/trunk/src/main/java/org/apache/giraph/graph/BspServiceWorker.java
- /giraph/trunk/src/test/java/org/apache/giraph/comm/SaslConnectionTest.java
Integrated in Giraph-trunk-Commit #234 (See https://builds.apache.org/job/Giraph-trunk-Commit/234/)
GIRAPH-363: Fix hadoop_0.23 profile broken by GIRAPH-211 (Revision 1396858)
Result = SUCCESS
ekoontz : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1396858
Files :
- /giraph/trunk/CHANGELOG
- /giraph/trunk/pom.xml
- /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/SaslNettyServer.java
HADOOP-8078 starts an ApacheDS KDC server for use in testing Kerberos integration - hopefully this could also be used to create unit tests for GIRAPH-211.
I talked about this to Avery, he suggested using just Netty SSL. How would you suggest that we assign certificates/keys to workers?