Bigtop
  1. Bigtop
  2. BIGTOP-721

improve the package daemon status check, check twice by some delay if status doesn't match expected value

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.5.0
    • Fix Version/s: 0.5.0
    • Component/s: tests
    • Labels:
      None

      Description

      the current package test check the daemon can be correctly started, stopped, restarted and so on. It first changes the daemon status (start it, or stop it), then sleep 3001, then use function checkThat to check if the daemon status satisfy the matcher.

      However, there is a case that daemon status update speed is slightly different: some daemon status change a little bit slower than others. In this case, we want the test can give a delay and check the daemon status again if it doesn't match the matcher at the first place.

      Example is:

      service flume-ng-agent start
      __EOT__
      12/09/28 14:10:19 TRACE shell.Shell: 
      <stdout>
      Flume NG agent is not running..failed
      Starting Flume NG agent daemon (flume-ng-agent): ..done
      </stdout>
      12/09/28 14:10:22 TRACE shell.Shell: /bin/bash -s << __EOT__
      service flume-ng-agent status
      __EOT__
      12/09/28 14:10:22 TRACE shell.Shell: return: 3
      12/09/28 14:10:22 TRACE shell.Shell: 
      <stdout>
      Flume NG agent is not running..failed
      </stdout>
      12/09/28 14:10:25 TRACE shell.Shell: /bin/bash -s << __EOT__
      service flume-ng-agent status
      __EOT__
      12/09/28 14:10:25 TRACE shell.Shell: 
      <stdout>
      Flume NG agent is running..done
      

      in above example (real case) is in SLES11, the flume-ng-agent daemon status update is a little bit slower, after sleep 3001, the daemon doesn't shown as "running" yet, then we give another 3001 delay and check again, and it is shown as "running".

      1. BIGTOP_721.txt
        4 kB
        Johnny Zhang
      2. BIGTOP_721-1.txt
        4 kB
        Johnny Zhang

        Activity

        Johnny Zhang created issue -
        Johnny Zhang made changes -
        Field Original Value New Value
        Status Open [ 1 ] Patch Available [ 10002 ]
        Johnny Zhang made changes -
        Attachment BIGTOP_721.txt [ 12547052 ]
        Hide
        Johnny Zhang added a comment -

        of course, for case daemon status doesn't match expected by 3rd try, test will record the failure as usual

        service hadoop-yarn-proxyserver status
        __EOT__
        12/09/28 14:17:33 TRACE shell.Shell: return: 1
        12/09/28 14:17:33 TRACE shell.Shell: 
        <stdout>
        Hadoop proxyserver is dead and pid file exists ... failed!
        </stdout>
        12/09/28 14:17:36 TRACE shell.Shell: /bin/bash -s << __EOT__
        service hadoop-yarn-proxyserver status
        __EOT__
        12/09/28 14:17:36 TRACE shell.Shell: return: 1
        12/09/28 14:17:36 TRACE shell.Shell: 
        <stdout>
        Hadoop proxyserver is dead and pid file exists ... failed!
        </stdout>
        12/09/28 14:17:39 TRACE shell.Shell: /bin/bash -s << __EOT__
        service hadoop-yarn-proxyserver status
        __EOT__
        12/09/28 14:17:39 TRACE shell.Shell: return: 1
        12/09/28 14:17:39 TRACE shell.Shell: 
        <stdout>
        Hadoop proxyserver is dead and pid file exists ... failed!
        

        I have been tested both the positive and negative case already!

        Show
        Johnny Zhang added a comment - of course, for case daemon status doesn't match expected by 3rd try, test will record the failure as usual service hadoop-yarn-proxyserver status __EOT__ 12/09/28 14:17:33 TRACE shell.Shell: return: 1 12/09/28 14:17:33 TRACE shell.Shell: <stdout> Hadoop proxyserver is dead and pid file exists ... failed! </stdout> 12/09/28 14:17:36 TRACE shell.Shell: /bin/bash -s << __EOT__ service hadoop-yarn-proxyserver status __EOT__ 12/09/28 14:17:36 TRACE shell.Shell: return: 1 12/09/28 14:17:36 TRACE shell.Shell: <stdout> Hadoop proxyserver is dead and pid file exists ... failed! </stdout> 12/09/28 14:17:39 TRACE shell.Shell: /bin/bash -s << __EOT__ service hadoop-yarn-proxyserver status __EOT__ 12/09/28 14:17:39 TRACE shell.Shell: return: 1 12/09/28 14:17:39 TRACE shell.Shell: <stdout> Hadoop proxyserver is dead and pid file exists ... failed! I have been tested both the positive and negative case already!
        Hide
        Johnny Zhang added a comment -

        found some room to improve the patch, will submit a new one later

        Show
        Johnny Zhang added a comment - found some room to improve the patch, will submit a new one later
        Johnny Zhang made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Johnny Zhang made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Johnny Zhang added a comment -

        this is a better patch.
        Please let me know if you think the function checkEquals() in PackageTestErrorProxy.java is necessary (we can call method matches() directly in PackageTestCommon.groovy). I am fine with both.

        Show
        Johnny Zhang added a comment - this is a better patch. Please let me know if you think the function checkEquals() in PackageTestErrorProxy.java is necessary (we can call method matches() directly in PackageTestCommon.groovy). I am fine with both.
        Johnny Zhang made changes -
        Attachment BIGTOP_721-1.txt [ 12550572 ]
        Hide
        Roman Shaposhnik added a comment -

        +1 and committed!

        Show
        Roman Shaposhnik added a comment - +1 and committed!
        Roman Shaposhnik made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Roman Shaposhnik made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Patch Available Patch Available Open Open
        24d 22h 4m 1 Johnny Zhang 23/Oct/12 20:40
        Open Open Patch Available Patch Available
        5h 28m 2 Johnny Zhang 24/Oct/12 02:09
        Patch Available Patch Available Resolved Resolved
        20d 23h 29m 1 Roman Shaposhnik 14/Nov/12 00:38
        Resolved Resolved Closed Closed
        219d 23h 16m 1 Roman Shaposhnik 22/Jun/13 00:55

          People

          • Assignee:
            Johnny Zhang
            Reporter:
            Johnny Zhang
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development