Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.6.0
    • Labels:
      None

      Description

      HDFS-2080 improved the CPU efficiency of the read path by using native SSE-enabled code for CRC verification. Benchmarks of the write path show that it's often CPU bound by checksums as well, so we should make the same improvement there.

        Issue Links

        There are no Sub-Tasks for this issue.

          Activity

          Todd Lipcon created issue -
          Jeff Hammerbacher made changes -
          Field Original Value New Value
          Link This issue is related to HDFS-2080 [ HDFS-2080 ]
          Jeff Hammerbacher made changes -
          Link This issue relates to HDFS-3529 [ HDFS-3529 ]
          Radim Kolar made changes -
          Link This issue is part of HDFS-4225 [ HDFS-4225 ]
          Colin Patrick McCabe made changes -
          Link This issue relates to HDFS-5395 [ HDFS-5395 ]
          Hide
          James Thomas added a comment -

          Preliminary work, no performance testing done yet. We avoid rewriting code to use ByteBuffers by using the GetPrimitiveArrayCritical method to access byte arrays without copying if possible.

          Show
          James Thomas added a comment - Preliminary work, no performance testing done yet. We avoid rewriting code to use ByteBuffers by using the GetPrimitiveArrayCritical method to access byte arrays without copying if possible.
          James Thomas made changes -
          Attachment HDFS-3528.patch [ 12649929 ]
          Todd Lipcon made changes -
          Assignee Todd Lipcon [ tlipcon ] James Thomas [ james.thomas ]
          Hide
          Todd Lipcon added a comment -

          Cool, thanks for starting on this. For the full improvement on the write path we also will need a "bulk_calculate_crc32" equivalent to run on the client side. Might make sense to split it up, though.

          Show
          Todd Lipcon added a comment - Cool, thanks for starting on this. For the full improvement on the write path we also will need a "bulk_calculate_crc32" equivalent to run on the client side. Might make sense to split it up, though.
          James Thomas made changes -
          Attachment HDFS-3528.patch [ 12649929 ]
          Hide
          James Thomas added a comment -

          Moved patch to DN subtask.

          Show
          James Thomas added a comment - Moved patch to DN subtask.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Is the native library faster than the Java implementation only for CRC32C but not CRC32?

          Show
          Tsz Wo Nicholas Sze added a comment - Is the native library faster than the Java implementation only for CRC32C but not CRC32?
          Hide
          James Thomas added a comment -

          Tsz Wo Nicholas Sze No, it is faster for both cases – see my comment at HDFS-6560.

          Show
          James Thomas added a comment - Tsz Wo Nicholas Sze No, it is faster for both cases – see my comment at HDFS-6560 .
          Hide
          Tsz Wo Nicholas Sze added a comment -

          It turns out that java.util.zip.CRC32 is the fastest in Java 7; see HADOOP-10778.

          Show
          Tsz Wo Nicholas Sze added a comment - It turns out that java.util.zip.CRC32 is the fastest in Java 7; see HADOOP-10778 .
          Hide
          Todd Lipcon added a comment -

          James Thomas – I think we should probably file a JIRA in the HADOOP project to add the relevant APIs to DataChecksum. They aren't HDFS-specific – I just filed MAPREDUCE-5962 for one example of a place where we can use them from within MR.

          Perhaps it makes sense to even file two:
          1) support verifyChunkedSums on a byte array
          2) native code to calculate chunked sums (on both byte array and byte buffer)

          which would allow us to break up the work a bit better for easy review. Feel free to ping me for review when patches are ready.

          Show
          Todd Lipcon added a comment - James Thomas – I think we should probably file a JIRA in the HADOOP project to add the relevant APIs to DataChecksum. They aren't HDFS-specific – I just filed MAPREDUCE-5962 for one example of a place where we can use them from within MR. Perhaps it makes sense to even file two: 1) support verifyChunkedSums on a byte array 2) native code to calculate chunked sums (on both byte array and byte buffer) which would allow us to break up the work a bit better for easy review. Feel free to ping me for review when patches are ready.
          Todd Lipcon made changes -
          Link This issue relates to MAPREDUCE-5962 [ MAPREDUCE-5962 ]
          Hide
          Todd Lipcon added a comment -

          hey James Thomas – now that HDFS-6865 is committed, are we missing any pieces here? Or are we all-native-all-the-time throughout the write pipeline now?

          Show
          Todd Lipcon added a comment - hey James Thomas – now that HDFS-6865 is committed, are we missing any pieces here? Or are we all-native-all-the-time throughout the write pipeline now?
          Hide
          James Thomas added a comment -

          Yep, think so – for both ByteBuffers and byte arrays we both compute and verify checksums natively if possible. That's HADOOP-10838, HADOOP-10975, and HDFS-6865.

          Show
          James Thomas added a comment - Yep, think so – for both ByteBuffers and byte arrays we both compute and verify checksums natively if possible. That's HADOOP-10838 , HADOOP-10975 , and HDFS-6865 .
          Hide
          Todd Lipcon added a comment -

          Cool, I'll mark this as resolved then

          Show
          Todd Lipcon added a comment - Cool, I'll mark this as resolved then
          Todd Lipcon made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Fix Version/s 2.6.0 [ 12327181 ]
          Resolution Fixed [ 1 ]
          Steve Loughran made changes -
          Link This issue breaks HADOOP-11064 [ HADOOP-11064 ]
          Hide
          Steve Loughran added a comment -

          This appears to have has broken all hadoop 2.2 applications on a hadoop 2.6 cluster ... filed HADOOP-11064 to highlight this

          Show
          Steve Loughran added a comment - This appears to have has broken all hadoop 2.2 applications on a hadoop 2.6 cluster ... filed HADOOP-11064 to highlight this
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Resolved Resolved
          807d 2h 28m 1 Todd Lipcon 29/Aug/14 01:51
          Resolved Resolved Closed Closed
          94d 2h 17m 1 Arun C Murthy 01/Dec/14 03:09

            People

            • Assignee:
              James Thomas
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              25 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development