Hadoop Common
  1. Hadoop Common
  2. HADOOP-7255

Performance regression bug caused by locking code

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.20.203.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      While testing the 0.20.203 branch we have discovered quite a severe performance regression caused by the following checkin:

      http://svn.apache.org/viewvc?view=revision&revision=1079260

      This locking seems to cause a pause of a few seconds between tasks, example with DFSIOTest:

      CURRENT CODE

      11/05/02 21:16:28 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read
      11/05/02 21:16:28 INFO fs.TestDFSIO:            Date & time: Mon May 02 21:16:28 EDT 2011
      11/05/02 21:16:28 INFO fs.TestDFSIO:        Number of files: 5
      11/05/02 21:16:28 INFO fs.TestDFSIO: Total MBytes processed: 500
      11/05/02 21:16:28 INFO fs.TestDFSIO:      Throughput mb/sec: 303.3980582524272
      11/05/02 21:16:28 INFO fs.TestDFSIO: Average IO rate mb/sec: 327.69122314453125
      11/05/02 21:16:28 INFO fs.TestDFSIO:  IO rate std deviation: 78.95767116969083
      11/05/02 21:16:28 INFO fs.TestDFSIO:     Test exec time sec: 19.476
      11/05/02 21:16:28 INFO fs.TestDFSIO: 
      

      WITH PATCH REMOVED

      11/05/02 21:19:03 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read
      11/05/02 21:19:03 INFO fs.TestDFSIO:            Date & time: Mon May 02 21:19:03 EDT 2011
      11/05/02 21:19:03 INFO fs.TestDFSIO:        Number of files: 5
      11/05/02 21:19:03 INFO fs.TestDFSIO: Total MBytes processed: 500
      11/05/02 21:19:03 INFO fs.TestDFSIO:      Throughput mb/sec: 366.03221083455344
      11/05/02 21:19:03 INFO fs.TestDFSIO: Average IO rate mb/sec: 366.35528564453125
      11/05/02 21:19:03 INFO fs.TestDFSIO:  IO rate std deviation: 11.020931463080379
      11/05/02 21:19:03 INFO fs.TestDFSIO:     Test exec time sec: 3.543
      11/05/02 21:19:03 INFO fs.TestDFSIO: 
      

        Activity

        Hide
        Bharath Mundlapudi added a comment -

        Thanks for posting performance data. Can you please let us know the test setup details? Like how big is your cluster?
        It is interesting to see high Std Deviation without the patch. Did you try multiple runs?

        Show
        Bharath Mundlapudi added a comment - Thanks for posting performance data. Can you please let us know the test setup details? Like how big is your cluster? It is interesting to see high Std Deviation without the patch. Did you try multiple runs?
        Hide
        Bharath Mundlapudi added a comment -

        Correction: Std deviation is high with patch.

        Show
        Bharath Mundlapudi added a comment - Correction: Std deviation is high with patch.
        Hide
        T Jake Luciani added a comment -

        For simplicity this is a single node from a fresh checkout. Yes I did run this multiple times... the stddev varies on read from 78 - 30... writes are fine in terms of stddev.

        If you build the test jar you should be able to reproduce locally.

        Show
        T Jake Luciani added a comment - For simplicity this is a single node from a fresh checkout. Yes I did run this multiple times... the stddev varies on read from 78 - 30... writes are fine in terms of stddev. If you build the test jar you should be able to reproduce locally.

          People

          • Assignee:
            Unassigned
            Reporter:
            T Jake Luciani
          • Votes:
            1 Vote for this issue
            Watchers:
            14 Start watching this issue

            Dates

            • Created:
              Updated:

              Development