Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-6695

Investigate using Java 7's nonblocking file I/O in BlockReaderLocal to implement read timeouts

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      In BlockReaderLocal, the "read" system call could block for a long time if the disk drive is having problems, or there is a huge amount of I/O contention. This might cause poor latency performance.

      In the remote block readers, we have implemented a read timeout, but we don't have one for the local block reader, since FileChannel#read doesn't support this.

      Once we move to JDK 7, we should investigate the java.nio.file nonblocking file I/O package to see if it could be used to implement read timeouts.

        Issue Links

          Activity

          Hide
          Colin Patrick McCabe added a comment -

          There are some docs here: http://docs.oracle.com/javase/7/docs/api/java/nio/channels/AsynchronousFileChannel.html

          It's a little unclear what the actual implementation of AsynchronousFileChannel is. If it's just a thread pool, then it might not be high-performance enough to be useful here. If it uses the underlying asynchronous I/O primitives provided by the operating system, then it would probably be useful for implementing these timeouts.

          Another way we could do this on Linux is to read from a pipe instead of a file, and use the splice system call to move pages from a local disk file into the pipe in a high-performance fashion. But I guess this doesn't work with position reads or alternative operating systems, so maybe it's not that useful....

          Show
          Colin Patrick McCabe added a comment - There are some docs here: http://docs.oracle.com/javase/7/docs/api/java/nio/channels/AsynchronousFileChannel.html It's a little unclear what the actual implementation of AsynchronousFileChannel is. If it's just a thread pool, then it might not be high-performance enough to be useful here. If it uses the underlying asynchronous I/O primitives provided by the operating system, then it would probably be useful for implementing these timeouts. Another way we could do this on Linux is to read from a pipe instead of a file, and use the splice system call to move pages from a local disk file into the pipe in a high-performance fashion. But I guess this doesn't work with position reads or alternative operating systems, so maybe it's not that useful....
          Hide
          Andrew Purtell added a comment -

          If it uses the underlying asynchronous I/O primitives provided by the operating system, then it would probably be useful for implementing these timeouts.

          Some grepping around the 7u forest doesn't reveal any native AIO support for Linux. Search wasn't exhaustive though.

          Show
          Andrew Purtell added a comment - If it uses the underlying asynchronous I/O primitives provided by the operating system, then it would probably be useful for implementing these timeouts. Some grepping around the 7u forest doesn't reveal any native AIO support for Linux. Search wasn't exhaustive though.
          Hide
          Liang Xie added a comment -

          I did not look through AsynchronousfileChannel impl, but per the above link saying: " This class also defines read and write methods that initiate asynchronous operations, returning a Future to represent the pending result of the operation. The Future may be used to check if the operation has completed, wait for its completion, and retrieve the result. ", seems does the same idea like my pseudo code at https://issues.apache.org/jira/browse/HDFS-6286?focusedCommentId=14008621&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14008621

          Show
          Liang Xie added a comment - I did not look through AsynchronousfileChannel impl, but per the above link saying: " This class also defines read and write methods that initiate asynchronous operations, returning a Future to represent the pending result of the operation. The Future may be used to check if the operation has completed, wait for its completion, and retrieve the result. ", seems does the same idea like my pseudo code at https://issues.apache.org/jira/browse/HDFS-6286?focusedCommentId=14008621&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14008621
          Hide
          Chris Nauroth added a comment -

          I'm linking this to HADOOP-9590, which was created a while back as a sort of umbrella jira to track ideas for migration to improved JDK7 APIs.

          Show
          Chris Nauroth added a comment - I'm linking this to HADOOP-9590 , which was created a while back as a sort of umbrella jira to track ideas for migration to improved JDK7 APIs.
          Hide
          Colin Patrick McCabe added a comment -

          I did not look through AsynchronousfileChannel impl, but per the above link saying: " This class also defines read and write methods that initiate asynchronous operations, returning a Future to represent the pending result of the operation. The Future may be used to check if the operation has completed, wait for its completion, and retrieve the result. ", seems does the same idea like my pseudo code...

          Yes, we could definitely set up a thread pool ourselves, but we don't want to do that because it would be slow. Passing large amounts of data between threads can mean moving it between CPU cores, which really kills performance. It also introduces latency from the context switches. I was hoping that Java7 had support for the kernel's native AIO interface... I guess we'll have to see.

          We might be able to do something interesting by having a thread pool where certain threads were locked to certain physical CPU cores. Of course, that requires JNI to pull off, at the moment, and the overheads of that might wipe out any gain...

          Show
          Colin Patrick McCabe added a comment - I did not look through AsynchronousfileChannel impl, but per the above link saying: " This class also defines read and write methods that initiate asynchronous operations, returning a Future to represent the pending result of the operation. The Future may be used to check if the operation has completed, wait for its completion, and retrieve the result. ", seems does the same idea like my pseudo code... Yes, we could definitely set up a thread pool ourselves, but we don't want to do that because it would be slow. Passing large amounts of data between threads can mean moving it between CPU cores, which really kills performance. It also introduces latency from the context switches. I was hoping that Java7 had support for the kernel's native AIO interface... I guess we'll have to see. We might be able to do something interesting by having a thread pool where certain threads were locked to certain physical CPU cores. Of course, that requires JNI to pull off, at the moment, and the overheads of that might wipe out any gain...
          Hide
          Zhe Zhang added a comment -

          It seems AsynchronousFileChannel in both Java 7 and 8 is "faked" with thread.

          Show
          Zhe Zhang added a comment - It seems AsynchronousFileChannel in both Java 7 and 8 is "faked" with thread.
          Hide
          Colin Patrick McCabe added a comment -

          And sending an Interrupt to a thread that is reading using blocking I/O is "faked" by closing the FD.

          Show
          Colin Patrick McCabe added a comment - And sending an Interrupt to a thread that is reading using blocking I/O is "faked" by closing the FD.

            People

            • Assignee:
              Unassigned
              Reporter:
              Colin Patrick McCabe
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:

                Development