Hadoop Common
  1. Hadoop Common
  2. HADOOP-6572

RPC responses may be out-of-order with respect to SASL

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: ipc, security
    • Labels:
      None

      Description

      SASL enforces its own message ordering. When RPC server sends its responses back, response A may be wrapped by SASL before response B but is put on response queue after response B. This results in RPC client receiving wrapped response B ahead of A. When the received messages are unwrapped by SASL, SASL complaints the messages are out of order.

      1. c6572-02.patch
        2 kB
        Kan Zhang
      2. 6572-bp20.patch
        2 kB
        Devaraj Das

        Issue Links

          Activity

          Hide
          Kan Zhang added a comment -

          An error log we observed.

          Error: javax.security.sasl.SaslException: DIGEST-MD5: Out of order sequencing of messages from server. Got: 9 Expected: 8 at com.sun.security.sasl.digest.DigestMD5Base$DigestIntegrity.unwrap(DigestMD5Base.java:1015) at com.sun.security.sasl.digest.DigestMD5Base.unwrap(DigestMD5Base.java:201) at org.apache.hadoop.security.SaslInputStream.readMoreData(SaslInputStream.java:97) at org.apache.hadoop.security.SaslInputStream.read(SaslInputStream.java:231) at java.io.FilterInputStream.read(FilterInputStream.java:116) at org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:329) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read(BufferedInputStream.java:237) at java.io.DataInputStream.readInt(DataInputStream.java:370) at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:620) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:565)

          Show
          Kan Zhang added a comment - An error log we observed. Error: javax.security.sasl.SaslException: DIGEST-MD5: Out of order sequencing of messages from server. Got: 9 Expected: 8 at com.sun.security.sasl.digest.DigestMD5Base$DigestIntegrity.unwrap(DigestMD5Base.java:1015) at com.sun.security.sasl.digest.DigestMD5Base.unwrap(DigestMD5Base.java:201) at org.apache.hadoop.security.SaslInputStream.readMoreData(SaslInputStream.java:97) at org.apache.hadoop.security.SaslInputStream.read(SaslInputStream.java:231) at java.io.FilterInputStream.read(FilterInputStream.java:116) at org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:329) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read(BufferedInputStream.java:237) at java.io.DataInputStream.readInt(DataInputStream.java:370) at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:620) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:565)
          Hide
          Kan Zhang added a comment -

          adding a patch that ensures call responses are added to the response queue in the same order as they are wrapped by SASL.

          Show
          Kan Zhang added a comment - adding a patch that ensures call responses are added to the response queue in the same order as they are wrapped by SASL.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12436116/c6572-02.patch
          against trunk revision 910741.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/10/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/10/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/10/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/10/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12436116/c6572-02.patch against trunk revision 910741. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/10/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/10/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/10/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/10/console This message is automatically generated.
          Hide
          Kan Zhang added a comment -

          It's not easy to write a unit test since the bug depends on a race condition among RPC handler threads. But we have run the same manual tests with the patch and the errors disappeared.

          Show
          Kan Zhang added a comment - It's not easy to write a unit test since the bug depends on a race condition among RPC handler threads. But we have run the same manual tests with the patch and the errors disappeared.
          Hide
          Devaraj Das added a comment -

          +1

          Show
          Devaraj Das added a comment - +1
          Hide
          Devaraj Das added a comment -

          I just committed this. Thanks, Kan!

          Show
          Devaraj Das added a comment - I just committed this. Thanks, Kan!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #177 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/177/)
          . Makes sure that SASL encryption and push to responder queue for the RPC response happens atomically. Contributed by Kan Zhang.

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #177 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/177/ ) . Makes sure that SASL encryption and push to responder queue for the RPC response happens atomically. Contributed by Kan Zhang.
          Hide
          Devaraj Das added a comment -

          The backported patch for Y20. Not for commit here.

          Show
          Devaraj Das added a comment - The backported patch for Y20. Not for commit here.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #255 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/255/)

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #255 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/255/ )

            People

            • Assignee:
              Kan Zhang
              Reporter:
              Kan Zhang
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development