Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: performance, util
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      We've seen a reducer writing 200MB to HDFS with replication = 1 spending a long time in crc calculation. In particular, it was spending 5 seconds in crc calculation out of a total of 6 for the write. I suspect that it is the java-jni border that is causing us grief.

      1. hadoop-6148.txt
        25 kB
        Todd Lipcon
      2. hadoop-6148.txt
        25 kB
        Todd Lipcon
      3. hadoop-6148.txt
        25 kB
        Todd Lipcon
      4. TestCrc32Performance.java
        3 kB
        Scott Carey
      5. PureJavaCrc32NewLoop.java
        16 kB
        Scott Carey
      6. PureJavaCrc32NewInner.java
        16 kB
        Scott Carey
      7. PureJavaCrc32New.java
        16 kB
        Scott Carey
      8. benchmarks20090715.txt
        12 kB
        Tsz Wo Nicholas Sze
      9. benchmarks20090714.txt
        9 kB
        Tsz Wo Nicholas Sze
      10. PureJavaCrc32.java
        18 kB
        Tsz Wo Nicholas Sze
      11. hdfs-297.txt
        19 kB
        Todd Lipcon
      12. TestCrc32Performance.java
        2 kB
        Scott Carey
      13. TestPureJavaCrc32.java
        3 kB
        Scott Carey
      14. PureJavaCrc32.java
        17 kB
        Scott Carey
      15. PureJavaCrc32.java
        2 kB
        Scott Carey
      16. PureJavaCrc32.java
        2 kB
        Scott Carey
      17. hadoop-5598.txt
        7 kB
        Todd Lipcon
      18. hadoop-5598-evil.txt
        8 kB
        Todd Lipcon
      19. TestCrc32Performance.java
        2 kB
        Todd Lipcon
      20. hadoop-5598-hybrid.txt
        11 kB
        Todd Lipcon
      21. crc32-results.txt
        3 kB
        Todd Lipcon
      22. TestCrc32Performance.java
        1 kB
        Todd Lipcon
      23. hadoop-5598.txt
        7 kB
        Todd Lipcon

        Issue Links

          Activity

            People

            • Assignee:
              Scott Carey
              Reporter:
              Owen O'Malley
            • Votes:
              0 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development