Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-6148

Implement a pure Java CRC32 calculator

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: performance, util
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      We've seen a reducer writing 200MB to HDFS with replication = 1 spending a long time in crc calculation. In particular, it was spending 5 seconds in crc calculation out of a total of 6 for the write. I suspect that it is the java-jni border that is causing us grief.

        Attachments

        1. benchmarks20090714.txt
          9 kB
          Tsz Wo Nicholas Sze
        2. benchmarks20090715.txt
          12 kB
          Tsz Wo Nicholas Sze
        3. crc32-results.txt
          3 kB
          Todd Lipcon
        4. hadoop-5598.txt
          7 kB
          Todd Lipcon
        5. hadoop-5598.txt
          7 kB
          Todd Lipcon
        6. hadoop-5598-evil.txt
          8 kB
          Todd Lipcon
        7. hadoop-5598-hybrid.txt
          11 kB
          Todd Lipcon
        8. hadoop-6148.txt
          25 kB
          Todd Lipcon
        9. hadoop-6148.txt
          25 kB
          Todd Lipcon
        10. hadoop-6148.txt
          25 kB
          Todd Lipcon
        11. hdfs-297.txt
          19 kB
          Todd Lipcon
        12. PureJavaCrc32.java
          18 kB
          Tsz Wo Nicholas Sze
        13. PureJavaCrc32.java
          17 kB
          Scott Carey
        14. PureJavaCrc32.java
          2 kB
          Scott Carey
        15. PureJavaCrc32.java
          2 kB
          Scott Carey
        16. PureJavaCrc32New.java
          16 kB
          Scott Carey
        17. PureJavaCrc32NewInner.java
          16 kB
          Scott Carey
        18. PureJavaCrc32NewLoop.java
          16 kB
          Scott Carey
        19. TestCrc32Performance.java
          3 kB
          Scott Carey
        20. TestCrc32Performance.java
          2 kB
          Scott Carey
        21. TestCrc32Performance.java
          2 kB
          Todd Lipcon
        22. TestCrc32Performance.java
          1 kB
          Todd Lipcon
        23. TestPureJavaCrc32.java
          3 kB
          Scott Carey

          Issue Links

            Activity

              People

              • Assignee:
                scott_carey Scott Carey
                Reporter:
                owen.omalley Owen O'Malley
              • Votes:
                0 Vote for this issue
                Watchers:
                20 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: