Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3561 [Umbrella ticket] Performance issues in YARN+MR
  3. MAPREDUCE-3534

Compression benchmark run-time increased by 13% in 0.23

        Details

        • Type: Sub-task Sub-task
        • Status: Closed
        • Priority: Blocker Blocker
        • Resolution: Fixed
        • Affects Version/s: 0.23.1
        • Fix Version/s: 0.23.1
        • Component/s: mrv2
        • Labels:
          None

          Description

          Compression runtime is increased by 13% as well as throughput is decreased by 24% in 0.23 when compared to 0.20.204 on
          a 350 node size cluster.

            Issue Links

            is blocked by

            Sub-task - The sub-task of the issue MAPREDUCE-3641 CapacityScheduler should be more conservative assigning off-switch requests

            • Blocker - Blocks development and/or testing work, production could not run
            • Closed

              Activity

              Ascending order - Click to sort in descending order
              Hide
              Permalink
              Todd Lipcon added a comment -

              What's the test/benchmark used for this result?

              Show
              Todd Lipcon added a comment - What's the test/benchmark used for this result?
              Hide
              Permalink
              Vinay Kumar Thota added a comment -

              Ran compression benchmark with following environments.

              Codec type : BLOCK
              Codec : com.hadoop.compression.lzo.LzoCodec

              Show
              Vinay Kumar Thota added a comment - Ran compression benchmark with following environments. Codec type : BLOCK Codec : com.hadoop.compression.lzo.LzoCodec
              Hide
              Permalink
              Scott Carey added a comment -

              Is this also a sub-task of MAPREDUCE-3561?

              Show
              Scott Carey added a comment - Is this also a sub-task of MAPREDUCE-3561 ?
              Hide
              Permalink
              Vinod Kumar Vavilapalli added a comment -

              Is this also a sub-task of MAPREDUCE-3561?

              Yeah, I couldn't find this ticket before.

              But my guess is most of those sub-tasks are related and most likely will be fixed in one shot. Let's see.

              Show
              Vinod Kumar Vavilapalli added a comment - Is this also a sub-task of MAPREDUCE-3561 ? Yeah, I couldn't find this ticket before. But my guess is most of those sub-tasks are related and most likely will be fixed in one shot. Let's see.
              Hide
              Permalink
              Amol Kekre added a comment -

              We would post new compression numbers post MAPREDUCE-3641

              Show
              Amol Kekre added a comment - We would post new compression numbers post MAPREDUCE-3641
              Hide
              Permalink
              Arun C Murthy added a comment -

              This was mostly due to misconfiguration, but MAPREDUCE-3641 does help a lot.

              Show
              Arun C Murthy added a comment - This was mostly due to misconfiguration, but MAPREDUCE-3641 does help a lot.
              Hide
              Permalink
              Amol Kekre added a comment -

              Compression is now 10.4% faster in .23 (compared to .205/1.0). Please close this jira

              Show
              Amol Kekre added a comment - Compression is now 10.4% faster in .23 (compared to .205/1.0). Please close this jira
              Hide
              Permalink
              Vinod Kumar Vavilapalli added a comment -

              Thanks for the update Amol. Thanks to Sid and Vinay who've been running this and verifying it.

              This is fixed now after the recent commits tracked at MAPREDUCE-3561.

              Show
              Vinod Kumar Vavilapalli added a comment - Thanks for the update Amol. Thanks to Sid and Vinay who've been running this and verifying it. This is fixed now after the recent commits tracked at MAPREDUCE-3561 .

                People

                • Assignee:
                  Vinod Kumar Vavilapalli
                  Reporter:
                  Vinay Kumar Thota
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  4 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Development