Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4798

TestJobHistoryServer fails some times with 'java.lang.AssertionError: Address already in use'

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 1.0.3
    • Fix Version/s: 1.1.2
    • Component/s: jobhistoryserver, test
    • Labels:
    • Environment:

      Red Hat Ent Server 6.2

      Description

      UT Failure in IHC 1.0.3: org.apache.hadoop.mapred.TestJobHistoryServer. This UT fails sometimes.

      The error message is:
      'Testcase: testHistoryServerStandalone took 5.376 sec
      Caused an ERROR
      Address already in use
      java.lang.AssertionError: Address already in use
      at org.apache.hadoop.mapred.TestJobHistoryServer.testHistoryServerStandalone(TestJobHistoryServer.java:113)'

        Issue Links

          Activity

          sam liu created issue -
          Hide
          sam liu added a comment -

          Current issue is different from the existing TestJobHistoryServer related JIRAs, because it has a special failure reason: 'java.lang.AssertionError: Address already in use'. BTW, in my testing, this unit test has 1 failure in 6 executions.

          Show
          sam liu added a comment - Current issue is different from the existing TestJobHistoryServer related JIRAs, because it has a special failure reason: 'java.lang.AssertionError: Address already in use'. BTW, in my testing, this unit test has 1 failure in 6 executions.
          Hide
          sam liu added a comment -

          The failures always happen in the step 'String redirectUrl = getRedirectUrl(job.getTrackingURL())' of testHistoryServerStandalone(). In the failure case, when executing into the method getRedirectUrl(job.getTrackingURL()), it always fails to execute 'Assert.assertEquals(status, HttpURLConnection.HTTP_MOVED_TEMP)' because the status value now is 200, but the HttpURLConnection.HTTP_MOVED_TEMP value is 302. In the successful cases., the status value will be 302.
          If add a break(sleep 10 sec) between 'int status = client.executeMethod(method)' and 'method.setFollowRedirects(false)', this case will be stable. I tried: 10 executions with 0 failure.

          Show
          sam liu added a comment - The failures always happen in the step 'String redirectUrl = getRedirectUrl(job.getTrackingURL())' of testHistoryServerStandalone(). In the failure case, when executing into the method getRedirectUrl(job.getTrackingURL()), it always fails to execute 'Assert.assertEquals(status, HttpURLConnection.HTTP_MOVED_TEMP)' because the status value now is 200, but the HttpURLConnection.HTTP_MOVED_TEMP value is 302. In the successful cases., the status value will be 302. If add a break(sleep 10 sec) between 'int status = client.executeMethod(method)' and 'method.setFollowRedirects(false)', this case will be stable. I tried: 10 executions with 0 failure.
          sam liu made changes -
          Field Original Value New Value
          Status Open [ 1 ] Patch Available [ 10002 ]
          Target Version/s 1.0.3 [ 12320250 ]
          Labels patch
          Fix Version/s 1.0.3 [ 12320250 ]
          sam liu made changes -
          Attachment MAPREDUCE-4798.patch [ 12553609 ]
          Hide
          sam liu added a comment -

          Just attached the patch now, but need the review of committer

          Show
          sam liu added a comment - Just attached the patch now, but need the review of committer
          sam liu made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Eric Yang added a comment -

          Could we indent the if block properly? In addition, could we do more to check the URL redirect method in a while loop with a maximum of 4 retries? We can add synthetic logic to pause 1, 2, 3, and 4 seconds upon each retry. Hence, the worst case would be delayed by 10 seconds, but it can recover earlier than 10 seconds in most cases. This could reduce the amount of time required to execute this test case.

          Show
          Eric Yang added a comment - Could we indent the if block properly? In addition, could we do more to check the URL redirect method in a while loop with a maximum of 4 retries? We can add synthetic logic to pause 1, 2, 3, and 4 seconds upon each retry. Hence, the worst case would be delayed by 10 seconds, but it can recover earlier than 10 seconds in most cases. This could reduce the amount of time required to execute this test case.
          sam liu made changes -
          Attachment MAPREDUCE-4798.patch [ 12553609 ]
          sam liu made changes -
          Attachment MAPREDUCE-4798.patch [ 12554123 ]
          sam liu made changes -
          Attachment MAPREDUCE-4798.patch [ 12554123 ]
          sam liu made changes -
          Attachment MAPREDUCE-4798.patch [ 12554127 ]
          Hide
          sam liu added a comment -

          Hi Eric, according to your suggestions, I updated the patch:

          • Indent the code properly
          • Set 4 retries of checking the URL redirect method, and add synthetic logic to pause 1, 2, 3, and 4 seconds upon each retry
          Show
          sam liu added a comment - Hi Eric, according to your suggestions, I updated the patch: Indent the code properly Set 4 retries of checking the URL redirect method, and add synthetic logic to pause 1, 2, 3, and 4 seconds upon each retry
          Hide
          Eric Yang added a comment -

          The indentation still has problem. In addition, could you generate a patch for branch-1, and trunk? Thanks

          Show
          Eric Yang added a comment - The indentation still has problem. In addition, could you generate a patch for branch-1, and trunk? Thanks
          sam liu made changes -
          Attachment MAPREDUCE-4798.patch [ 12554127 ]
          sam liu made changes -
          Attachment MAPREDUCE-4798.patch [ 12554618 ]
          Hide
          sam liu added a comment -

          Eric, I updated the patch. Thanks!

          Show
          sam liu added a comment - Eric, I updated the patch. Thanks!
          sam liu made changes -
          Attachment MAPREDUCE-4798.patch [ 12554618 ]
          sam liu made changes -
          Attachment MAPREDUCE-4798.patch [ 12555142 ]
          Attachment MAPREDUCE-4798_branch-1.patch [ 12555143 ]
          Hide
          sam liu added a comment -

          Eric,
          I updated the patch for 1.0.3 and uploaded the patch for branch-1.
          For trunk, there is no TestJobHistoryServer.java, so I did not generate another patch for trunk.
          Thanks!

          Show
          sam liu added a comment - Eric, I updated the patch for 1.0.3 and uploaded the patch for branch-1. For trunk, there is no TestJobHistoryServer.java, so I did not generate another patch for trunk. Thanks!
          Eric Yang made changes -
          Fix Version/s 1.0.3 [ 12320250 ]
          Target Version/s 1.0.3 [ 12320250 ] 1.0.4, 1.2.0, 1.1.2 [ 12323325, 12321661, 12323594 ]
          Hide
          Eric Yang added a comment -

          +1 looks good.

          Show
          Eric Yang added a comment - +1 looks good.
          Eric Yang made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Eric Yang added a comment -

          I just committed this to 1.x branches. JobHistoryServer doesn't exist in trunk, hence no action there.

          Show
          Eric Yang added a comment - I just committed this to 1.x branches. JobHistoryServer doesn't exist in trunk, hence no action there.
          Eric Yang made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Eric Yang added a comment -

          Thanks Sam.

          Show
          Eric Yang added a comment - Thanks Sam.
          Hide
          sam liu added a comment -

          Thanks Eric! Pls let me know if there is any further action

          Show
          sam liu added a comment - Thanks Eric! Pls let me know if there is any further action
          Matt Foley made changes -
          Fix Version/s 1.1.2 [ 12323594 ]
          Matt Foley made changes -
          Target Version/s 1.0.4, 1.2.0, 1.1.2 [ 12323325, 12321661, 12323594 ] 1.0.4, 1.1.2 [ 12323325, 12323594 ]
          Hide
          Matt Foley added a comment -

          Closed upon successful release of 1.1.2.

          Show
          Matt Foley added a comment - Closed upon successful release of 1.1.2.
          Matt Foley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Ravi Prakash made changes -
          Link This issue duplicates MAPREDUCE-3135 [ MAPREDUCE-3135 ]
          sam liu made changes -
          Assignee sam liu [ sam liu ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Patch Available Patch Available Open Open
          1m 31s 1 sam liu 15/Nov/12 03:01
          Open Open Patch Available Patch Available
          17d 21h 21m 2 Eric Yang 02/Dec/12 00:32
          Patch Available Patch Available Resolved Resolved
          1m 57s 1 Eric Yang 02/Dec/12 00:34
          Resolved Resolved Closed Closed
          94d 9h 21m 1 Matt Foley 06/Mar/13 09:55

            People

            • Assignee:
              sam liu
              Reporter:
              sam liu
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 3h
                3h
                Remaining:
                Remaining Estimate - 3h
                3h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development