Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7915

The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error

    Details

    • Target Version/s:

      Description

      The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error. In DataXceiver#requestShortCircuitFds, the DataNode can succeed at the first part (mark the slot as used) and fail at the second part (tell the DFSClient what it did). The "try" block for unregistering the slot only covers a failure in the first part, not the second part. In this way, a divergence can form between the views of which slots are allocated on DFSClient and on server.

      1. HDFS-7915.001.patch
        13 kB
        Colin P. McCabe
      2. HDFS-7915.002.patch
        13 kB
        Colin P. McCabe
      3. HDFS-7915.004.patch
        21 kB
        Colin P. McCabe
      4. HDFS-7915.005.patch
        21 kB
        Colin P. McCabe
      5. HDFS-7915.006.patch
        21 kB
        Colin P. McCabe
      6. HDFS-7915.branch-2.6.patch
        21 kB
        Akira Ajisaka

        Issue Links

          Activity

          Hide
          cmccabe Colin P. McCabe added a comment -

          I think putting the "unregister slot" code into a "finally" block that covers the whole function should fix this. Since the I/O we're dealing with is local (never touches the network because this is for short circuit) we can be pretty confident that if the write succeeds, the DFSClient knows about what we sent.

          Show
          cmccabe Colin P. McCabe added a comment - I think putting the "unregister slot" code into a "finally" block that covers the whole function should fix this. Since the I/O we're dealing with is local (never touches the network because this is for short circuit) we can be pretty confident that if the write succeeds, the DFSClient knows about what we sent.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Fix up the error handling in DataXceiver. A nested try block seemed like the cleanest way to go.

          The hardest part was writing the unit test. I verified that the new unit test fails without the fix.

          Show
          cmccabe Colin P. McCabe added a comment - Fix up the error handling in DataXceiver. A nested try block seemed like the cleanest way to go. The hardest part was writing the unit test. I verified that the new unit test fails without the fix.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12703855/HDFS-7915.001.patch
          against trunk revision 30c428a.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9832//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9832//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12703855/HDFS-7915.001.patch against trunk revision 30c428a. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9832//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9832//console This message is automatically generated.
          Hide
          yzhangal Yongjun Zhang added a comment -

          Hi Colin P. McCabe,

          Thanks for reporting the issue and the solution. The patch looks good in general. I have couple of comments:

          1. Can we add a log message when doing unregisterSlot below to state that "slot x is unregistered due to ..."? I think this will help future debugging of similar issue.

               if ((!success) && (registeredSlotId != null)) {
                  datanode.shortCircuitRegistry.unregisterSlot(registeredSlotId);
                }
          

          2. I applied your patch, and reverted DataXceiver, ran the test, expecting it to fail, but it did not. I wonder if I missed anything.

          Thanks.

          Show
          yzhangal Yongjun Zhang added a comment - Hi Colin P. McCabe , Thanks for reporting the issue and the solution. The patch looks good in general. I have couple of comments: 1. Can we add a log message when doing unregisterSlot below to state that "slot x is unregistered due to ..."? I think this will help future debugging of similar issue. if ((!success) && (registeredSlotId != null )) { datanode.shortCircuitRegistry.unregisterSlot(registeredSlotId); } 2. I applied your patch, and reverted DataXceiver, ran the test, expecting it to fail, but it did not. I wonder if I missed anything. Thanks.
          Hide
          cmccabe Colin P. McCabe added a comment -

          I added the log message.

          The test should now fail when DataXceiver is not modified. Previously I was injecting the failure in a slightly wrong place.

          Show
          cmccabe Colin P. McCabe added a comment - I added the log message. The test should now fail when DataXceiver is not modified. Previously I was injecting the failure in a slightly wrong place.
          Hide
          cnauroth Chris Nauroth added a comment -

          Thanks for the patch, Colin. The change looks good. In the test, is the Visitor indirection necessary, or would it be easier to add 2 VisibleForTesting getters that return the segments and slots directly to the test code?

          Show
          cnauroth Chris Nauroth added a comment - Thanks for the patch, Colin. The change looks good. In the test, is the Visitor indirection necessary, or would it be easier to add 2 VisibleForTesting getters that return the segments and slots directly to the test code?
          Hide
          yzhangal Yongjun Zhang added a comment -

          Hi Colin P. McCabe,

          Thanks for the updated patch, possible to even include what exception caused the failure and thus need to unregister the slot?
          I will try to look more about the test shortly.

          Show
          yzhangal Yongjun Zhang added a comment - Hi Colin P. McCabe , Thanks for the updated patch, possible to even include what exception caused the failure and thus need to unregister the slot? I will try to look more about the test shortly.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12704011/HDFS-7915.002.patch
          against trunk revision 344d7cb.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitCache
          org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
          org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9840//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9840//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12704011/HDFS-7915.002.patch against trunk revision 344d7cb. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitCache org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9840//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9840//console This message is automatically generated.
          Hide
          yzhangal Yongjun Zhang added a comment -

          Hi Colin,

          Thanks for the new rev. I found I made a mistake when doing earlier test, I need to include -Pnative as compile switch to enable to test. After I do that, I can see the test fail even with rev 001 after reverting DataXceiver.java. Did you speculate the problem when making rev 2?

          Some additional comments:

                fis = datanode.requestShortCircuitFdsForRead(blk, token, maxVersion);
                bld.setStatus(SUCCESS);
                bld.setShortCircuitAccessVersion(DataNode.CURRENT_BLOCK_FORMAT_VERSION);
          

          Here bld is set to SUCCESS status, without checking whether fis is null or not. However, down in the code below:

           if (fis != null) {
                  FileDescriptor fds[] = new FileDescriptor[fis.length];
                  ......
                  success = true;
           }
          

          success is set to true only when fis is not null. I saw a bit inconsistency here. Is it success when fis is null? If not, then the first section has an issue. If yes, then we can probably change success to isFisObtained.

          It seems when we do the logging below

             if ((!success) && (registeredSlotId != null)) {
                  LOG.info("Unregistering " + registeredSlotId + " because the " +
                      "requestShortCircuitFdsForRead operation failed.");
                  datanode.shortCircuitRegistry.unregisterSlot(registeredSlotId);
                }
          

          The reason that we have to unregister a slot could be an exception recorded in bld, or because of an exception not currently caught in this method.

          I think we can add code to capture the currently uncaught exception, remember it, then re-throw it. Such that when we do the logging above in the final block, we can report this exception as the reason why we are un-registering the slot in this log.

          What do you think?

          Thanks.

          Show
          yzhangal Yongjun Zhang added a comment - Hi Colin, Thanks for the new rev. I found I made a mistake when doing earlier test, I need to include -Pnative as compile switch to enable to test. After I do that, I can see the test fail even with rev 001 after reverting DataXceiver.java. Did you speculate the problem when making rev 2? Some additional comments: fis = datanode.requestShortCircuitFdsForRead(blk, token, maxVersion); bld.setStatus(SUCCESS); bld.setShortCircuitAccessVersion(DataNode.CURRENT_BLOCK_FORMAT_VERSION); Here bld is set to SUCCESS status, without checking whether fis is null or not. However, down in the code below: if (fis != null ) { FileDescriptor fds[] = new FileDescriptor[fis.length]; ...... success = true ; } success is set to true only when fis is not null. I saw a bit inconsistency here. Is it success when fis is null? If not, then the first section has an issue. If yes, then we can probably change success to isFisObtained . It seems when we do the logging below if ((!success) && (registeredSlotId != null )) { LOG.info( "Unregistering " + registeredSlotId + " because the " + "requestShortCircuitFdsForRead operation failed." ); datanode.shortCircuitRegistry.unregisterSlot(registeredSlotId); } The reason that we have to unregister a slot could be an exception recorded in bld , or because of an exception not currently caught in this method. I think we can add code to capture the currently uncaught exception, remember it, then re-throw it. Such that when we do the logging above in the final block, we can report this exception as the reason why we are un-registering the slot in this log. What do you think? Thanks.
          Hide
          yzhangal Yongjun Zhang added a comment -

          Hi Colin,

          I realize that the fis returned by fis = datanode.requestShortCircuitFdsForRead(blk, token, maxVersion); won't be null if it doesn't throw exception, So please ignore my comment about "inconsistency", but I think it'd be helpful to address the other one (include failure reason in the log). Thanks.

          Show
          yzhangal Yongjun Zhang added a comment - Hi Colin, I realize that the fis returned by fis = datanode.requestShortCircuitFdsForRead(blk, token, maxVersion); won't be null if it doesn't throw exception, So please ignore my comment about "inconsistency", but I think it'd be helpful to address the other one (include failure reason in the log). Thanks.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Here bld is set to SUCCESS status, without checking whether fis is null or not. However, down in the code below:.... success is set to true only when fis is not null. I saw a bit inconsistency here. Is it success when fis is null? If not, then the first section has an issue. If yes, then we can probably change success to isFisObtained.

          There is no inconsistency. DataNode#requestShortCircuitFdsForRead cannot return null. It can only throw an exception or return some fds. There is a difference between attempting to send a SUCCESS response to the DFSClient, and the whole function being successful. Just because we attempted to send a SUCCESS response doesn't mean we actually did it. We must actually send the fds and the response to succeed.

          I will add a Precondition check to make it clearer that fis cannot be null when a SUCCESS response is being sent.

          The reason that we have to unregister a slot could be an exception recorded in bld, or because of an exception not currently caught in this method. I think we can add code to capture the currently uncaught exception, remember it, then re-throw it. Such that when we do the logging above in the final block, we can report this exception as the reason why we are un-registering the slot in this log.

          I think this would add too much complexity. If we catch Throwable, we can't re-throw Throwable. So we'd have to have separate catch blocks for RuntimeException, IOException, and probably another block to catch other things.

          Show
          cmccabe Colin P. McCabe added a comment - Here bld is set to SUCCESS status, without checking whether fis is null or not. However, down in the code below:.... success is set to true only when fis is not null. I saw a bit inconsistency here. Is it success when fis is null? If not, then the first section has an issue. If yes, then we can probably change success to isFisObtained. There is no inconsistency. DataNode#requestShortCircuitFdsForRead cannot return null. It can only throw an exception or return some fds. There is a difference between attempting to send a SUCCESS response to the DFSClient, and the whole function being successful. Just because we attempted to send a SUCCESS response doesn't mean we actually did it. We must actually send the fds and the response to succeed. I will add a Precondition check to make it clearer that fis cannot be null when a SUCCESS response is being sent. The reason that we have to unregister a slot could be an exception recorded in bld, or because of an exception not currently caught in this method. I think we can add code to capture the currently uncaught exception, remember it, then re-throw it. Such that when we do the logging above in the final block, we can report this exception as the reason why we are un-registering the slot in this log. I think this would add too much complexity. If we catch Throwable, we can't re-throw Throwable. So we'd have to have separate catch blocks for RuntimeException, IOException, and probably another block to catch other things.
          Hide
          cmccabe Colin P. McCabe added a comment -

          I found another problem here. To explain it, I need to explain how the communication happens now.

          1. In the BlockReaderFactory, the DFSClient initiates the file descriptor request by sending:

                  [2-byte] 28 [DATA_TRANSFER_VERSION]
                  [1-byte] 87 [REQUEST_SHORT_CIRCUIT_FDS]
                  [var] OpRequestShortCircuitAccessProto(blk, blockToken, slotId, tracing stuff)
          

          2. On the DataNode, in DataXceiver, we read the OpRequestShortCircuitAccessProto that the client sent. We call DataNode#requestShortCircuitFdsForRead to load the file descriptors. If that succeeded, we send back a BlockOpResponseProto with status SUCCESS.

          3. Back in the DFSClient, we read the BlockOpResponseProto.

          4. If it contains a SUCCESS response, the DFSClient calls sock.recvFileInputStreams. This reads a single byte and also passes the new file descriptor to us (the DFSClient.)

          The problem is that if the DFSClient closes the socket after step #3, but before step #4, the DataNode thinks that the transfer was successful and never unregisters the slot. This is what led to the unit test failures earlier. It seems that there is a buffer in the UNIX domain socket that we are writing to, which lets the DataNode's write succeed immediately even before the DFSClient actually reads the data.

          To fix this, we can add a step #5: the DFSClient writes a byte for the DataNode to receive. And step #6: the datanode reads it. That way, if a socket close or other error happens before step #5, we know that the FD didn't get sent.

          This can be done compatibly by adding a new boolean to the protobuf which indicates to the DataNode that the client supports "receipt verification." New datanodes will set this bit and old ones will not. Neither the datanode nor the dfsclient will attempt to do receipt verification unless the other party supports it.

          Show
          cmccabe Colin P. McCabe added a comment - I found another problem here. To explain it, I need to explain how the communication happens now. 1. In the BlockReaderFactory , the DFSClient initiates the file descriptor request by sending: [2- byte ] 28 [DATA_TRANSFER_VERSION] [1- byte ] 87 [REQUEST_SHORT_CIRCUIT_FDS] [ var ] OpRequestShortCircuitAccessProto(blk, blockToken, slotId, tracing stuff) 2. On the DataNode, in DataXceiver , we read the OpRequestShortCircuitAccessProto that the client sent. We call DataNode#requestShortCircuitFdsForRead to load the file descriptors. If that succeeded, we send back a BlockOpResponseProto with status SUCCESS . 3. Back in the DFSClient, we read the BlockOpResponseProto . 4. If it contains a SUCCESS response, the DFSClient calls sock.recvFileInputStreams . This reads a single byte and also passes the new file descriptor to us (the DFSClient.) The problem is that if the DFSClient closes the socket after step #3, but before step #4, the DataNode thinks that the transfer was successful and never unregisters the slot. This is what led to the unit test failures earlier. It seems that there is a buffer in the UNIX domain socket that we are writing to, which lets the DataNode's write succeed immediately even before the DFSClient actually reads the data. To fix this, we can add a step #5: the DFSClient writes a byte for the DataNode to receive. And step #6: the datanode reads it. That way, if a socket close or other error happens before step #5, we know that the FD didn't get sent. This can be done compatibly by adding a new boolean to the protobuf which indicates to the DataNode that the client supports "receipt verification." New datanodes will set this bit and old ones will not. Neither the datanode nor the dfsclient will attempt to do receipt verification unless the other party supports it.
          Hide
          cmccabe Colin P. McCabe added a comment -

          cnauroth asked: Thanks for the patch, Colin. The change looks good. In the test, is the Visitor indirection necessary, or would it be easier to add 2 VisibleForTesting getters that return the segments and slots directly to the test code?

          The problem is locking. If there is a getter for these hash tables, is the caller going to take the appropriate locks when accessing them? If not, we get findbugs warnings and possibly actual test bugs. If so, it adds a lot of coupling between the unit test and the registry code. In contrast, the visitor interface lets the unit test see a single consistent snapshot of what is going on in the ShortCircuitRegistry.

          Show
          cmccabe Colin P. McCabe added a comment - cnauroth asked: Thanks for the patch, Colin. The change looks good. In the test, is the Visitor indirection necessary, or would it be easier to add 2 VisibleForTesting getters that return the segments and slots directly to the test code? The problem is locking. If there is a getter for these hash tables, is the caller going to take the appropriate locks when accessing them? If not, we get findbugs warnings and possibly actual test bugs. If so, it adds a lot of coupling between the unit test and the registry code. In contrast, the visitor interface lets the unit test see a single consistent snapshot of what is going on in the ShortCircuitRegistry .
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12704248/HDFS-7915.004.patch
          against trunk revision 863079b.

          -1 patch. Trunk compilation may be broken.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9862//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12704248/HDFS-7915.004.patch against trunk revision 863079b. -1 patch . Trunk compilation may be broken. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9862//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12704248/HDFS-7915.004.patch
          against trunk revision 863079b.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.TestAppendSnapshotTruncate

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9863//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9863//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12704248/HDFS-7915.004.patch against trunk revision 863079b. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestAppendSnapshotTruncate Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9863//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9863//console This message is automatically generated.
          Hide
          yzhangal Yongjun Zhang added a comment -

          HI Colin,

          Nice find of the new problem!

          I did a review of rev04, sorry for a long post here:

          1. I think we should look harder in logging a reason when having to unregister a slot for better supportability (e.g., we want to find out the root cause). I agree that to make it 100% right would result in too complex logic though. I would propose the following:

          We don't have to capture runtime exception, instead, we can just care about IOException. Since bld already records reason for the different scenarios, we can add one more try/catch for the remaining:

          We can add code to remember whatever exception recorded to bld as stage1Exception, and the other part stage2Exception.

              IOException stage1Exception = null;
              IOException stage2Exception = null;
                  ...
              // add code to record the stage1Exception
              ...
              try {
                  bld.build().writeDelimitedTo(socketOut);
                  if (fis != null) {
                    ......
                    if (supportsReceiptVerification) {
                      LOG.trace("Sending receipt verification byte for " + slotId);
                      int val = sock.getInputStream().read();
                      if (val < 0) {
                        throw new EOFException("No verification byte received"); <== Add a message to this exception
                      }
                    } else {
                      LOG.trace("Receipt verification is not enabled on the DataNode.  " +
                          "Not verifying " + slotId);
                    }
                    success = true;
                  }
              } catch (IOException e) {
                 otherException = e;
                 throw e;
              }
          

          Notice I also added a message to the EOFException thrown there.

                if ((!success) && (registeredSlotId != null)) {
                  String errMsg =
                      (bld.getStatus() != SUCCESS)? bld.getMessage() :
                        ((stage2Exception != null)?
                            stage2Exception.getMessage() : "unknown");
                  LOG.info("Unregistering " + registeredSlotId + " because the " +
                      "requestShortCircuitFdsForRead operation failed (" + errMsg + ")");
                  datanode.shortCircuitRegistry.unregisterSlot(registeredSlotId);
                  if (LOG.isDebugEnabled()) {
                      if (stage1Exception != null) {
                        LOG.debug("requestShortCircuitFds stage1Exception: " + StringUtils.stringifyException(stage1Exception));
                  }
                  if (stage2Exception != null) {
                        LOG.debug("requestShortCircuitFds stage2Exception: " + StringUtils.stringifyException(stage2Exception));
                  }
                  }
                }
          

          ?

          We can actually use a single exception variable for stage1 and stage2 because only one would be assigned at the time of logging. I just want to throw some thoughts here.

          2. question: change in BlockReaderFactory.java to move
          "return new ShortCircuitReplicaInfo(replica);" to within the try block
          is not important, I mean, it's ok not to move it, correct?

          3. About the receipt verification:

                  // writer
                  if (buf[0] == SUPPORTS_RECEIPT_VERIFICATION.getNumber()) {
                    LOG.trace("Sending receipt verification byte for slot " + slot);
                    sock.getOutputStream().write((byte)0); 
                  }
                  =======================
                  // reader
                  int val = sock.getInputStream().read();
                  if (val < 0) {
                      throw new EOFException();
                    }
          
          • suggest to change sock.getOutputStream().write((byte).. to sock.getOutputStream().write((int), since we are using {{DomainSocket#public void write(int val) throws IOException }} API.
          • Should we define "0" as an constant somewhere and check equivalence instead of "val < 0" at the reader?

          4.

           LOG.trace("Sending receipt verification byte for " + slotId);
                    int val = sock.getInputStream().read();
          

          Looks to me that the message should be "Reading receipt byte for ...". right?

          Thanks.

          Show
          yzhangal Yongjun Zhang added a comment - HI Colin, Nice find of the new problem! I did a review of rev04, sorry for a long post here: 1. I think we should look harder in logging a reason when having to unregister a slot for better supportability (e.g., we want to find out the root cause). I agree that to make it 100% right would result in too complex logic though. I would propose the following: We don't have to capture runtime exception, instead, we can just care about IOException. Since bld already records reason for the different scenarios, we can add one more try/catch for the remaining: We can add code to remember whatever exception recorded to bld as stage1Exception, and the other part stage2Exception. IOException stage1Exception = null ; IOException stage2Exception = null ; ... // add code to record the stage1Exception ... try { bld.build().writeDelimitedTo(socketOut); if (fis != null ) { ...... if (supportsReceiptVerification) { LOG.trace( "Sending receipt verification byte for " + slotId); int val = sock.getInputStream().read(); if (val < 0) { throw new EOFException( "No verification byte received" ); <== Add a message to this exception } } else { LOG.trace( "Receipt verification is not enabled on the DataNode. " + "Not verifying " + slotId); } success = true ; } } catch (IOException e) { otherException = e; throw e; } Notice I also added a message to the EOFException thrown there. if ((!success) && (registeredSlotId != null )) { String errMsg = (bld.getStatus() != SUCCESS)? bld.getMessage() : ((stage2Exception != null )? stage2Exception.getMessage() : "unknown" ); LOG.info( "Unregistering " + registeredSlotId + " because the " + "requestShortCircuitFdsForRead operation failed (" + errMsg + ")" ); datanode.shortCircuitRegistry.unregisterSlot(registeredSlotId); if (LOG.isDebugEnabled()) { if (stage1Exception != null ) { LOG.debug( "requestShortCircuitFds stage1Exception: " + StringUtils.stringifyException(stage1Exception)); } if (stage2Exception != null ) { LOG.debug( "requestShortCircuitFds stage2Exception: " + StringUtils.stringifyException(stage2Exception)); } } } ? We can actually use a single exception variable for stage1 and stage2 because only one would be assigned at the time of logging. I just want to throw some thoughts here. 2. question: change in BlockReaderFactory.java to move "return new ShortCircuitReplicaInfo(replica);" to within the try block is not important, I mean, it's ok not to move it, correct? 3. About the receipt verification: // writer if (buf[0] == SUPPORTS_RECEIPT_VERIFICATION.getNumber()) { LOG.trace( "Sending receipt verification byte for slot " + slot); sock.getOutputStream().write(( byte )0); } ======================= // reader int val = sock.getInputStream().read(); if (val < 0) { throw new EOFException(); } suggest to change sock.getOutputStream().write((byte).. to sock.getOutputStream().write((int) , since we are using {{DomainSocket#public void write(int val) throws IOException }} API. Should we define "0" as an constant somewhere and check equivalence instead of "val < 0" at the reader? 4. LOG.trace( "Sending receipt verification byte for " + slotId); int val = sock.getInputStream().read(); Looks to me that the message should be "Reading receipt byte for ...". right? Thanks.
          Hide
          cmccabe Colin P. McCabe added a comment -

          1. I think we should look harder in logging a reason when having to unregister a slot for better supportability (e.g., we want to find out the root cause). I agree that to make it 100% right would result in too complex logic though. I would propose the following:

          I understand your concerns, but every log I've looked at does display the reason why the fd passing failed, including the full exception. It simply is logged in a catch block further up in the DataXceiver. Logging it again in this function would just be repetitious. Sorry if that was unclear.

          2. question: change in BlockReaderFactory.java to move "return new ShortCircuitReplicaInfo(replica);" to within the try block is not important, I mean, it's ok not to move it, correct?

          Yes, it is OK not to move it, because currently the ShortCircuitReplicaInfo can't fail (never throws). But it is better to have it in the catch block in case the constructor later has a throw... added to it. It is safer.

          suggest to change sock.getOutputStream().write((byte).. to sock.getOutputStream().write((int), since we are using {{DomainSocket#public void write(int val) throws IOException }} API.

          OK

          Should we define "0" as an constant somewhere and check equivalence instead of "val < 0" at the reader?

          It's not necessary. We don't care what the value is. Adding checks is actually bad because it means we can't decide to use it later for some other purpose.

          Looks to me that the message should be "Reading receipt byte for ...". right?

          thanks, fixed

          Show
          cmccabe Colin P. McCabe added a comment - 1. I think we should look harder in logging a reason when having to unregister a slot for better supportability (e.g., we want to find out the root cause). I agree that to make it 100% right would result in too complex logic though. I would propose the following: I understand your concerns, but every log I've looked at does display the reason why the fd passing failed, including the full exception. It simply is logged in a catch block further up in the DataXceiver. Logging it again in this function would just be repetitious. Sorry if that was unclear. 2. question: change in BlockReaderFactory.java to move "return new ShortCircuitReplicaInfo(replica);" to within the try block is not important, I mean, it's ok not to move it, correct? Yes, it is OK not to move it, because currently the ShortCircuitReplicaInfo can't fail (never throws). But it is better to have it in the catch block in case the constructor later has a throw... added to it. It is safer. suggest to change sock.getOutputStream().write((byte).. to sock.getOutputStream().write((int), since we are using {{DomainSocket#public void write(int val) throws IOException }} API. OK Should we define "0" as an constant somewhere and check equivalence instead of "val < 0" at the reader? It's not necessary. We don't care what the value is. Adding checks is actually bad because it means we can't decide to use it later for some other purpose. Looks to me that the message should be "Reading receipt byte for ...". right? thanks, fixed
          Hide
          cmccabe Colin P. McCabe added a comment -

          updated patch

          Show
          cmccabe Colin P. McCabe added a comment - updated patch
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12704525/HDFS-7915.005.patch
          against trunk revision 6fdef76.

          -1 patch. Trunk compilation may be broken.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9879//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12704525/HDFS-7915.005.patch against trunk revision 6fdef76. -1 patch . Trunk compilation may be broken. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9879//console This message is automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          I'm repeating my earlier comment in case it got lost while focusing on the admittedly much more important discussion about the actual bug.

          Thanks for the patch, Colin. The change looks good. In the test, is the Visitor indirection necessary, or would it be easier to add 2 VisibleForTesting getters that return the segments and slots directly to the test code?

          Show
          cnauroth Chris Nauroth added a comment - I'm repeating my earlier comment in case it got lost while focusing on the admittedly much more important discussion about the actual bug. Thanks for the patch, Colin. The change looks good. In the test, is the Visitor indirection necessary, or would it be easier to add 2 VisibleForTesting getters that return the segments and slots directly to the test code?
          Hide
          Rosa Rosa Ali added a comment -

          Please tell me how i can use batch file HDFS-1783
          https://issues.apache.org/jira/browse/HDFS-1783
          Please

          Show
          Rosa Rosa Ali added a comment - Please tell me how i can use batch file HDFS-1783 https://issues.apache.org/jira/browse/HDFS-1783 Please
          Hide
          cmccabe Colin P. McCabe added a comment -

          I did answer that comment, here: https://issues.apache.org/jira/browse/HDFS-7915?focusedCommentId=14359377&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14359377

          The summary is that it would not be easier because we'd have to start worrying about locking. Both the hash maps and the objects inside them are mutable and you need a lock to access them. The visitor hides this detail, but we'd have to worry about it with accessors.

          thanks

          Show
          cmccabe Colin P. McCabe added a comment - I did answer that comment, here: https://issues.apache.org/jira/browse/HDFS-7915?focusedCommentId=14359377&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14359377 The summary is that it would not be easier because we'd have to start worrying about locking. Both the hash maps and the objects inside them are mutable and you need a lock to access them. The visitor hides this detail, but we'd have to worry about it with accessors. thanks
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12704525/HDFS-7915.005.patch
          against trunk revision 6fdef76.

          -1 patch. Trunk compilation may be broken.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9881//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12704525/HDFS-7915.005.patch against trunk revision 6fdef76. -1 patch . Trunk compilation may be broken. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9881//console This message is automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          Thanks, Colin. I had missed that the first time.

          Show
          cnauroth Chris Nauroth added a comment - Thanks, Colin. I had missed that the first time.
          Hide
          yzhangal Yongjun Zhang added a comment -

          Hi Colin P. McCabe,

          Thanks for the updated patch. +1 on rev 6 pending jenkins.

          Would you please commit and let's create follow-up jira for better logging and possible improvements.

          Thanks.

          Show
          yzhangal Yongjun Zhang added a comment - Hi Colin P. McCabe , Thanks for the updated patch. +1 on rev 6 pending jenkins. Would you please commit and let's create follow-up jira for better logging and possible improvements. Thanks.
          Hide
          cmccabe Colin P. McCabe added a comment -

          committed. thanks, guys.

          I will file a follow-up to look into if we can do more logging. Note that in the specific case where we caught this bug (writeArray failing), we actually got as much logging as possible from the DataNode. Everything we needed was logged there, including the failed domain socket I/O stack traces. Similarly, I can't think of any DFSClient logs we needed and didn't get. We got the domain socket I/O stack traces there was well. What we don't know is why the write failed, but we logged as much information as the kernel gave us (it returned EAGAIN, which means timeout).

          In general socket reads and writes can fail, and HDFS needs to be able to handle that. The cause of the timeout in the case we saw is outside the scope of this JIRA.

          Show
          cmccabe Colin P. McCabe added a comment - committed. thanks, guys. I will file a follow-up to look into if we can do more logging. Note that in the specific case where we caught this bug (writeArray failing), we actually got as much logging as possible from the DataNode. Everything we needed was logged there, including the failed domain socket I/O stack traces. Similarly, I can't think of any DFSClient logs we needed and didn't get. We got the domain socket I/O stack traces there was well. What we don't know is why the write failed, but we logged as much information as the kernel gave us (it returned EAGAIN, which means timeout). In general socket reads and writes can fail, and HDFS needs to be able to handle that. The cause of the timeout in the case we saw is outside the scope of this JIRA.
          Hide
          cmccabe Colin P. McCabe added a comment -

          oops, I just saw that jenkins didn't run on v6 yet. sigh...

          Show
          cmccabe Colin P. McCabe added a comment - oops, I just saw that jenkins didn't run on v6 yet. sigh...
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7322 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7322/)
          HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe) (cmccabe: rev 5aa892ed486d42ae6b94c4866b92cd2b382ea640)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7322 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7322/ ) HDFS-7915 . The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe) (cmccabe: rev 5aa892ed486d42ae6b94c4866b92cd2b382ea640) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7323 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7323/)
          Revert "HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe)" (jenkins didn't run yet) (cmccabe: rev 32741cf3d25d85a92e3deb11c302cc2a718d71dd)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7323 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7323/ ) Revert " HDFS-7915 . The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe)" (jenkins didn't run yet) (cmccabe: rev 32741cf3d25d85a92e3deb11c302cc2a718d71dd) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12704540/HDFS-7915.006.patch
          against trunk revision 6fdef76.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9884//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9884//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12704540/HDFS-7915.006.patch against trunk revision 6fdef76. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9884//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9884//console This message is automatically generated.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #132 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/132/)
          HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe) (cmccabe: rev 5aa892ed486d42ae6b94c4866b92cd2b382ea640)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java
            Revert "HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe)" (jenkins didn't run yet) (cmccabe: rev 32741cf3d25d85a92e3deb11c302cc2a718d71dd)
          • hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #132 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/132/ ) HDFS-7915 . The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe) (cmccabe: rev 5aa892ed486d42ae6b94c4866b92cd2b382ea640) hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java Revert " HDFS-7915 . The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe)" (jenkins didn't run yet) (cmccabe: rev 32741cf3d25d85a92e3deb11c302cc2a718d71dd) hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #866 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/866/)
          HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe) (cmccabe: rev 5aa892ed486d42ae6b94c4866b92cd2b382ea640)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto
            Revert "HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe)" (jenkins didn't run yet) (cmccabe: rev 32741cf3d25d85a92e3deb11c302cc2a718d71dd)
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #866 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/866/ ) HDFS-7915 . The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe) (cmccabe: rev 5aa892ed486d42ae6b94c4866b92cd2b382ea640) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto Revert " HDFS-7915 . The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe)" (jenkins didn't run yet) (cmccabe: rev 32741cf3d25d85a92e3deb11c302cc2a718d71dd) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2064 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2064/)
          HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe) (cmccabe: rev 5aa892ed486d42ae6b94c4866b92cd2b382ea640)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
            Revert "HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe)" (jenkins didn't run yet) (cmccabe: rev 32741cf3d25d85a92e3deb11c302cc2a718d71dd)
          • hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2064 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2064/ ) HDFS-7915 . The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe) (cmccabe: rev 5aa892ed486d42ae6b94c4866b92cd2b382ea640) hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java Revert " HDFS-7915 . The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe)" (jenkins didn't run yet) (cmccabe: rev 32741cf3d25d85a92e3deb11c302cc2a718d71dd) hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #123 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/123/)
          HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe) (cmccabe: rev 5aa892ed486d42ae6b94c4866b92cd2b382ea640)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java
            Revert "HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe)" (jenkins didn't run yet) (cmccabe: rev 32741cf3d25d85a92e3deb11c302cc2a718d71dd)
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #123 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/123/ ) HDFS-7915 . The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe) (cmccabe: rev 5aa892ed486d42ae6b94c4866b92cd2b382ea640) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java Revert " HDFS-7915 . The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe)" (jenkins didn't run yet) (cmccabe: rev 32741cf3d25d85a92e3deb11c302cc2a718d71dd) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #132 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/132/)
          HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe) (cmccabe: rev 5aa892ed486d42ae6b94c4866b92cd2b382ea640)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java
            Revert "HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe)" (jenkins didn't run yet) (cmccabe: rev 32741cf3d25d85a92e3deb11c302cc2a718d71dd)
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #132 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/132/ ) HDFS-7915 . The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe) (cmccabe: rev 5aa892ed486d42ae6b94c4866b92cd2b382ea640) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java Revert " HDFS-7915 . The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe)" (jenkins didn't run yet) (cmccabe: rev 32741cf3d25d85a92e3deb11c302cc2a718d71dd) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2082 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2082/)
          HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe) (cmccabe: rev 5aa892ed486d42ae6b94c4866b92cd2b382ea640)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java
            Revert "HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe)" (jenkins didn't run yet) (cmccabe: rev 32741cf3d25d85a92e3deb11c302cc2a718d71dd)
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2082 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2082/ ) HDFS-7915 . The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe) (cmccabe: rev 5aa892ed486d42ae6b94c4866b92cd2b382ea640) hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java Revert " HDFS-7915 . The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe)" (jenkins didn't run yet) (cmccabe: rev 32741cf3d25d85a92e3deb11c302cc2a718d71dd) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7328 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7328/)
          HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe) (cmccabe: rev bc9cb3e271b22069a15ca110cd60c860250aaab2)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7328 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7328/ ) HDFS-7915 . The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe) (cmccabe: rev bc9cb3e271b22069a15ca110cd60c860250aaab2) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #133 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/133/)
          HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe) (cmccabe: rev bc9cb3e271b22069a15ca110cd60c860250aaab2)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #133 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/133/ ) HDFS-7915 . The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe) (cmccabe: rev bc9cb3e271b22069a15ca110cd60c860250aaab2) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #867 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/867/)
          HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe) (cmccabe: rev bc9cb3e271b22069a15ca110cd60c860250aaab2)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #867 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/867/ ) HDFS-7915 . The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe) (cmccabe: rev bc9cb3e271b22069a15ca110cd60c860250aaab2) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2065 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2065/)
          HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe) (cmccabe: rev bc9cb3e271b22069a15ca110cd60c860250aaab2)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2065 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2065/ ) HDFS-7915 . The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe) (cmccabe: rev bc9cb3e271b22069a15ca110cd60c860250aaab2) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #124 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/124/)
          HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe) (cmccabe: rev bc9cb3e271b22069a15ca110cd60c860250aaab2)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #124 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/124/ ) HDFS-7915 . The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe) (cmccabe: rev bc9cb3e271b22069a15ca110cd60c860250aaab2) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #133 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/133/)
          HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe) (cmccabe: rev bc9cb3e271b22069a15ca110cd60c860250aaab2)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #133 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/133/ ) HDFS-7915 . The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe) (cmccabe: rev bc9cb3e271b22069a15ca110cd60c860250aaab2) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2083 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2083/)
          HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe) (cmccabe: rev bc9cb3e271b22069a15ca110cd60c860250aaab2)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2083 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2083/ ) HDFS-7915 . The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe) (cmccabe: rev bc9cb3e271b22069a15ca110cd60c860250aaab2) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java
          Hide
          shv Konstantin Shvachko added a comment -

          Did you mean to commit this to 2.7.0 or is the "Fix Version" lying?

          Show
          shv Konstantin Shvachko added a comment - Did you mean to commit this to 2.7.0 or is the "Fix Version" lying?
          Hide
          cmccabe Colin P. McCabe added a comment -

          Good catch. I committed to branch-2 but not branch-2.7, which was incorrect. Fixed.

          Show
          cmccabe Colin P. McCabe added a comment - Good catch. I committed to branch-2 but not branch-2.7, which was incorrect. Fixed.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #7659 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7659/)
          HDFS-8070. Pre-HDFS-7915 DFSClient cannot use short circuit on post-HDFS-7915 DataNode (cmccabe) (cmccabe: rev a8898445dc9b5cdb7230e2e23a57393c9f378ff0)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #7659 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7659/ ) HDFS-8070 . Pre- HDFS-7915 DFSClient cannot use short circuit on post- HDFS-7915 DataNode (cmccabe) (cmccabe: rev a8898445dc9b5cdb7230e2e23a57393c9f378ff0) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2105 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2105/)
          HDFS-8070. Pre-HDFS-7915 DFSClient cannot use short circuit on post-HDFS-7915 DataNode (cmccabe) (cmccabe: rev a8898445dc9b5cdb7230e2e23a57393c9f378ff0)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2105 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2105/ ) HDFS-8070 . Pre- HDFS-7915 DFSClient cannot use short circuit on post- HDFS-7915 DataNode (cmccabe) (cmccabe: rev a8898445dc9b5cdb7230e2e23a57393c9f378ff0) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #164 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/164/)
          HDFS-8070. Pre-HDFS-7915 DFSClient cannot use short circuit on post-HDFS-7915 DataNode (cmccabe) (cmccabe: rev a8898445dc9b5cdb7230e2e23a57393c9f378ff0)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #164 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/164/ ) HDFS-8070 . Pre- HDFS-7915 DFSClient cannot use short circuit on post- HDFS-7915 DataNode (cmccabe) (cmccabe: rev a8898445dc9b5cdb7230e2e23a57393c9f378ff0) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #173 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/173/)
          HDFS-8070. Pre-HDFS-7915 DFSClient cannot use short circuit on post-HDFS-7915 DataNode (cmccabe) (cmccabe: rev a8898445dc9b5cdb7230e2e23a57393c9f378ff0)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #173 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/173/ ) HDFS-8070 . Pre- HDFS-7915 DFSClient cannot use short circuit on post- HDFS-7915 DataNode (cmccabe) (cmccabe: rev a8898445dc9b5cdb7230e2e23a57393c9f378ff0) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #907 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/907/)
          HDFS-8070. Pre-HDFS-7915 DFSClient cannot use short circuit on post-HDFS-7915 DataNode (cmccabe) (cmccabe: rev a8898445dc9b5cdb7230e2e23a57393c9f378ff0)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #907 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/907/ ) HDFS-8070 . Pre- HDFS-7915 DFSClient cannot use short circuit on post- HDFS-7915 DataNode (cmccabe) (cmccabe: rev a8898445dc9b5cdb7230e2e23a57393c9f378ff0) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #174 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/174/)
          HDFS-8070. Pre-HDFS-7915 DFSClient cannot use short circuit on post-HDFS-7915 DataNode (cmccabe) (cmccabe: rev a8898445dc9b5cdb7230e2e23a57393c9f378ff0)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #174 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/174/ ) HDFS-8070 . Pre- HDFS-7915 DFSClient cannot use short circuit on post- HDFS-7915 DataNode (cmccabe) (cmccabe: rev a8898445dc9b5cdb7230e2e23a57393c9f378ff0) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2123 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2123/)
          HDFS-8070. Pre-HDFS-7915 DFSClient cannot use short circuit on post-HDFS-7915 DataNode (cmccabe) (cmccabe: rev a8898445dc9b5cdb7230e2e23a57393c9f378ff0)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2123 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2123/ ) HDFS-8070 . Pre- HDFS-7915 DFSClient cannot use short circuit on post- HDFS-7915 DataNode (cmccabe) (cmccabe: rev a8898445dc9b5cdb7230e2e23a57393c9f378ff0) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java
          Hide
          ajisakaa Akira Ajisaka added a comment -

          HADOOP-11802 depends on this issue. If we are going to cherry-pick HADOOP-11802, we need to cherry-pick this issue first. Attaching a patch for branch-2.6.

          Show
          ajisakaa Akira Ajisaka added a comment - HADOOP-11802 depends on this issue. If we are going to cherry-pick HADOOP-11802 , we need to cherry-pick this issue first. Attaching a patch for branch-2.6.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Thanks Akira Ajisaka. Pulled this into 2.6.1 after running compilation and TestShortCircuitCache.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Thanks Akira Ajisaka . Pulled this into 2.6.1 after running compilation and TestShortCircuitCache.
          Hide
          fanqiushi fanqiushi added a comment -

          Hi Colin P. McCabe , I find a similar error on hadoop 2.5.2, i wonder if it is the same with this jira:
          when I use hbase on hadoop, sometimes the read and write on hdfs become very slow, and some errors are found on both hbase log and datanode log.
          hbase log:
          2017-06-28 04:59:16,206 WARN org.apache.hadoop.hdfs.BlockReaderFactory: BlockReaderFactory(fileName=/hyperbase1/data/default/DW_TB_PROTOCOL_201726/693313b31f0db7ffaa7b818a82f75f05/d/baa8cfbf62ac49578829a90c1f8b9603, block=BP-673187784-166.0.8.10-1489391820899:blk_1265649740_191912424): I/O error requesting file descriptors. Disabling domain socket DomainSocket(fd=3830,path=/var/run/hdfs1/dn_socket)
          java.net.SocketTimeoutException: read(2) error: Resource temporarily unavailable
          at org.apache.hadoop.net.unix.DomainSocket.readArray0(Native Method)
          at org.apache.hadoop.net.unix.DomainSocket.access$000(DomainSocket.java:45)
          at org.apache.hadoop.net.unix.DomainSocket$DomainInputStream.read(DomainSocket.java:532)
          at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:1998)
          at org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager$EndpointShmManager.requestNewShm(DfsClientShmManager.java:169)
          at org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager$EndpointShmManager.allocSlot(DfsClientShmManager.java:261)
          at org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager.allocSlot(DfsClientShmManager.java:432)
          at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.allocShmSlot(ShortCircuitCache.java:1015)
          at org.apache.hadoop.hdfs.BlockReaderFactory.createShortCircuitReplicaInfo(BlockReaderFactory.java:450)
          at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.create(ShortCircuitCache.java:782)
          at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.fetchOrCreate(ShortCircuitCache.java:716)
          at org.apache.hadoop.hdfs.BlockReaderFactory.getBlockReaderLocal(BlockReaderFactory.java:396)
          at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:304)
          at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:609)
          at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:832)
          at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:881)
          at java.io.DataInputStream.read(DataInputStream.java:149)
          at org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:563)
          at org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1215)
          at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1430)
          at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1312)
          at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:387)
          at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:642)
          at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.next(HFileReaderV2.java:1102)
          at org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:273)
          at org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:173)
          at org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:55)
          at org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:313)
          at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:257)
          at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:697)
          at org.apache.hadoop.hbase.regionserver.StoreScanner.seekAsDirection(StoreScanner.java:683)
          at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:533)
          at org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:223)
          at org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:76)
          at org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:109)
          at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1131)
          at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1528)
          at org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:494)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:745)
          2017-06-28 04:59:16,207 WARN org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache: ShortCircuitCache(0x20a5dc77): failed to load 1265649740_BP-673187784-166.0.8.10-1489391820899
          2017-06-28 04:59:16,208 DEBUG org.apache.hadoop.hbase.regionserver.compactions.Compactor: Compaction progress: 129941931/6104366 (2128.67%), rate=2293.77 kB/sec
          2017-06-28 04:59:22,875 INFO org.apache.hadoop.hbase.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1680ms
          GC pool 'ParNew' had collection(s): count=1 time=1846ms
          2017-06-28 04:59:31,969 ERROR org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache: ShortCircuitCache(0x20a5dc77): failed to release short-circuit shared memory slot Slot(slotIdx=56, shm=DfsClientShm(36b4864aeb6f612be0f3420acc48c2af)) by sending ReleaseShortCircuitAccessRequestProto to /var/run/hdfs1/dn_socket. Closing shared memory segment.
          java.io.IOException: ERROR_INVALID: there is no shared memory segment registered with shmId 36b4864aeb6f612be0f3420acc48c2af
          at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache$SlotReleaser.run(ShortCircuitCache.java:208)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
          at java.util.concurrent.FutureTask.run(FutureTask.java:262)
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:745)
          2017-06-28 04:59:31,969 WARN org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager: EndpointShmManager(166.0.8.33:50010, parent=ShortCircuitShmManager(0993730b)): error shutting down shm: got IOException calling shutdown(SHUT_RDWR)
          java.nio.channels.ClosedChannelException
          at org.apache.hadoop.util.CloseableReferenceCount.reference(CloseableReferenceCount.java:57)
          at org.apache.hadoop.net.unix.DomainSocket.shutdown(DomainSocket.java:387)
          at org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager$EndpointShmManager.shutdown(DfsClientShmManager.java:378)
          at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache$SlotReleaser.run(ShortCircuitCache.java:223)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
          at java.util.concurrent.FutureTask.run(FutureTask.java:262)
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:745)
          2017-06-28 04:59:32,368 ERROR org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache: ShortCircuitCache(0x20a5dc77): failed to release short-circuit shared memory slot Slot(slotIdx=27, shm=DfsClientShm(c59bc3c02472ae764b297761a88984c7)) by sending ReleaseShortCircuitAccessRequestProto to /var/run/hdfs1/dn_socket. Closing shared memory segment.
          java.io.IOException: ERROR_INVALID: there is no shared memory segment registered with shmId c59bc3c02472ae764b297761a88984c7
          at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache$SlotReleaser.run(ShortCircuitCache.java:208)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
          at java.util.concurrent.FutureTask.run(FutureTask.java:262)
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:745)
          2017-06-28 04:59:32,368 WARN org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager: EndpointShmManager(166.0.8.33:50010, parent=ShortCircuitShmManager(0993730b)): error shutting down shm: got IOException calling shutdown(SHUT_RDWR)
          java.nio.channels.ClosedChannelException
          at org.apache.hadoop.util.CloseableReferenceCount.reference(CloseableReferenceCount.java:57)
          at org.apache.hadoop.net.unix.DomainSocket.shutdown(DomainSocket.java:387)
          at org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager$EndpointShmManager.shutdown(DfsClientShmManager.java:378)
          at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache$SlotReleaser.run(ShortCircuitCache.java:223)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
          at java.util.concurrent.FutureTask.run(FutureTask.java:262)
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:745)
          2017-06-28 04:59:36,948 INFO org.mortbay.log: org.mortbay.io.nio.SelectorManager$SelectSet@21941784 JVM BUG(s) - injecting delay2 times
          2017-06-28 04:59:36,948 INFO org.mortbay.log: org.mortbay.io.nio.SelectorManager$SelectSet@21941784 JVM BUG(s) - recreating selector 2 times, canceled keys 52 times

          datanode log:
          2017-06-28 04:48:27,157 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Exception for BP-673187784-166.0.8.10-1489391820899:blk_1266878404_193141813
          java.net.SocketTimeoutException: 120000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/166.0.8.33:50010 remote=/166.0.8.33:52712]
          at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
          at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
          at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
          at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
          at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
          at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
          at java.io.DataInputStream.read(DataInputStream.java:149)
          at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:192)
          at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
          at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134)
          at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
          at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:453)
          at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:734)
          at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:741)
          at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:124)
          at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71)
          at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:234)
          at java.lang.Thread.run(Thread.java:745)
          2017-06-28 04:48:27,158 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-673187784-166.0.8.10-1489391820899:blk_1266878404_193141813, type=HAS_DOWNSTREAM_IN_PIPELINE
          java.io.EOFException: Premature EOF: no length prefix available
          at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2000)
          at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176)
          at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1115)
          at java.lang.Thread.run(Thread.java:745)
          2017-06-28 04:48:27,208 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Exception for BP-673187784-166.0.8.10-1489391820899:blk_1266878380_193141789

          I have two questions:
          (1) How did this error happen?
          (2) Can patch on this jira solve this problem?

          looking forward to your reply

          Show
          fanqiushi fanqiushi added a comment - Hi Colin P. McCabe , I find a similar error on hadoop 2.5.2, i wonder if it is the same with this jira: when I use hbase on hadoop, sometimes the read and write on hdfs become very slow, and some errors are found on both hbase log and datanode log. hbase log: 2017-06-28 04:59:16,206 WARN org.apache.hadoop.hdfs.BlockReaderFactory: BlockReaderFactory(fileName=/hyperbase1/data/default/DW_TB_PROTOCOL_201726/693313b31f0db7ffaa7b818a82f75f05/d/baa8cfbf62ac49578829a90c1f8b9603, block=BP-673187784-166.0.8.10-1489391820899:blk_1265649740_191912424): I/O error requesting file descriptors. Disabling domain socket DomainSocket(fd=3830,path=/var/run/hdfs1/dn_socket) java.net.SocketTimeoutException: read(2) error: Resource temporarily unavailable at org.apache.hadoop.net.unix.DomainSocket.readArray0(Native Method) at org.apache.hadoop.net.unix.DomainSocket.access$000(DomainSocket.java:45) at org.apache.hadoop.net.unix.DomainSocket$DomainInputStream.read(DomainSocket.java:532) at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:1998) at org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager$EndpointShmManager.requestNewShm(DfsClientShmManager.java:169) at org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager$EndpointShmManager.allocSlot(DfsClientShmManager.java:261) at org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager.allocSlot(DfsClientShmManager.java:432) at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.allocShmSlot(ShortCircuitCache.java:1015) at org.apache.hadoop.hdfs.BlockReaderFactory.createShortCircuitReplicaInfo(BlockReaderFactory.java:450) at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.create(ShortCircuitCache.java:782) at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.fetchOrCreate(ShortCircuitCache.java:716) at org.apache.hadoop.hdfs.BlockReaderFactory.getBlockReaderLocal(BlockReaderFactory.java:396) at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:304) at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:609) at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:832) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:881) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:563) at org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1215) at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1430) at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1312) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:387) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:642) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.next(HFileReaderV2.java:1102) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:273) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:173) at org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:55) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:313) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:257) at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:697) at org.apache.hadoop.hbase.regionserver.StoreScanner.seekAsDirection(StoreScanner.java:683) at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:533) at org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:223) at org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:76) at org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:109) at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1131) at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1528) at org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:494) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2017-06-28 04:59:16,207 WARN org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache: ShortCircuitCache(0x20a5dc77): failed to load 1265649740_BP-673187784-166.0.8.10-1489391820899 2017-06-28 04:59:16,208 DEBUG org.apache.hadoop.hbase.regionserver.compactions.Compactor: Compaction progress: 129941931/6104366 (2128.67%), rate=2293.77 kB/sec 2017-06-28 04:59:22,875 INFO org.apache.hadoop.hbase.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1680ms GC pool 'ParNew' had collection(s): count=1 time=1846ms 2017-06-28 04:59:31,969 ERROR org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache: ShortCircuitCache(0x20a5dc77): failed to release short-circuit shared memory slot Slot(slotIdx=56, shm=DfsClientShm(36b4864aeb6f612be0f3420acc48c2af)) by sending ReleaseShortCircuitAccessRequestProto to /var/run/hdfs1/dn_socket. Closing shared memory segment. java.io.IOException: ERROR_INVALID: there is no shared memory segment registered with shmId 36b4864aeb6f612be0f3420acc48c2af at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache$SlotReleaser.run(ShortCircuitCache.java:208) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2017-06-28 04:59:31,969 WARN org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager: EndpointShmManager(166.0.8.33:50010, parent=ShortCircuitShmManager(0993730b)): error shutting down shm: got IOException calling shutdown(SHUT_RDWR) java.nio.channels.ClosedChannelException at org.apache.hadoop.util.CloseableReferenceCount.reference(CloseableReferenceCount.java:57) at org.apache.hadoop.net.unix.DomainSocket.shutdown(DomainSocket.java:387) at org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager$EndpointShmManager.shutdown(DfsClientShmManager.java:378) at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache$SlotReleaser.run(ShortCircuitCache.java:223) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2017-06-28 04:59:32,368 ERROR org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache: ShortCircuitCache(0x20a5dc77): failed to release short-circuit shared memory slot Slot(slotIdx=27, shm=DfsClientShm(c59bc3c02472ae764b297761a88984c7)) by sending ReleaseShortCircuitAccessRequestProto to /var/run/hdfs1/dn_socket. Closing shared memory segment. java.io.IOException: ERROR_INVALID: there is no shared memory segment registered with shmId c59bc3c02472ae764b297761a88984c7 at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache$SlotReleaser.run(ShortCircuitCache.java:208) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2017-06-28 04:59:32,368 WARN org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager: EndpointShmManager(166.0.8.33:50010, parent=ShortCircuitShmManager(0993730b)): error shutting down shm: got IOException calling shutdown(SHUT_RDWR) java.nio.channels.ClosedChannelException at org.apache.hadoop.util.CloseableReferenceCount.reference(CloseableReferenceCount.java:57) at org.apache.hadoop.net.unix.DomainSocket.shutdown(DomainSocket.java:387) at org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager$EndpointShmManager.shutdown(DfsClientShmManager.java:378) at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache$SlotReleaser.run(ShortCircuitCache.java:223) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2017-06-28 04:59:36,948 INFO org.mortbay.log: org.mortbay.io.nio.SelectorManager$SelectSet@21941784 JVM BUG(s) - injecting delay2 times 2017-06-28 04:59:36,948 INFO org.mortbay.log: org.mortbay.io.nio.SelectorManager$SelectSet@21941784 JVM BUG(s) - recreating selector 2 times, canceled keys 52 times datanode log: 2017-06-28 04:48:27,157 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Exception for BP-673187784-166.0.8.10-1489391820899:blk_1266878404_193141813 java.net.SocketTimeoutException: 120000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel [connected local=/166.0.8.33:50010 remote=/166.0.8.33:52712] at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) at java.io.BufferedInputStream.read1(BufferedInputStream.java:275) at java.io.BufferedInputStream.read(BufferedInputStream.java:334) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:192) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:453) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:734) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:741) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:124) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:234) at java.lang.Thread.run(Thread.java:745) 2017-06-28 04:48:27,158 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-673187784-166.0.8.10-1489391820899:blk_1266878404_193141813, type=HAS_DOWNSTREAM_IN_PIPELINE java.io.EOFException: Premature EOF: no length prefix available at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2000) at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1115) at java.lang.Thread.run(Thread.java:745) 2017-06-28 04:48:27,208 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Exception for BP-673187784-166.0.8.10-1489391820899:blk_1266878380_193141789 I have two questions: (1) How did this error happen? (2) Can patch on this jira solve this problem? looking forward to your reply

            People

            • Assignee:
              cmccabe Colin P. McCabe
              Reporter:
              cmccabe Colin P. McCabe
            • Votes:
              0 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development