Accumulo
  1. Accumulo
  2. ACCUMULO-2005

Provide a scaling factor for mapred.task.timeout inside functional test's RunTests.java

    Details

    • Type: Test Test
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 1.4.4, 1.5.0
    • Fix Version/s: 1.4.5, 1.5.2, 1.6.0
    • Component/s: test
    • Labels:
      None

      Description

      The MR option for running functional tests ($ACCUMULO_HOME/test/system/auto) fails with a mapred timeout currently set at 480 secs (8*60*1000). A flag or argument for scaling mapred.task.timeout inside RunTests.java (ie, a number that influence the 8 in 8*60*1000) needs to be in place to allow scaling up or down mapred.task.timeout as needed. This is similar to ACCUMULO-1829.

      1. ACCUMULO-2005.v4.patch.txt
        4 kB
        Hung Pham
      2. ACCUMULO-2005.v4-1.4.5-SNAPSHOT.patch.txt
        5 kB
        Sean Busbey
      3. ACCUMULO-2005.v7.patch.txt
        4 kB
        Hung Pham
      4. ACCUMULO-2005.v7-1.5.1-SNAPSHOT.patch.txt
        5 kB
        Sean Busbey

        Issue Links

          Activity

          Hide
          ASF subversion and git services added a comment -

          Commit 4ee0f6494ea0d2265d930bce962ebcc3c7dbfe5c in accumulo's branch refs/heads/master from Christopher Tubbs
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=4ee0f64 ]

          Fix recent warnings introduced by several tickets

          ACCUMULO-2005
          ACCUMULO-2182
          ACCUMULO-2224
          ACCUMULO-2234
          ACCUMULO-2539

          Show
          ASF subversion and git services added a comment - Commit 4ee0f6494ea0d2265d930bce962ebcc3c7dbfe5c in accumulo's branch refs/heads/master from Christopher Tubbs [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=4ee0f64 ] Fix recent warnings introduced by several tickets ACCUMULO-2005 ACCUMULO-2182 ACCUMULO-2224 ACCUMULO-2234 ACCUMULO-2539
          Hide
          ASF subversion and git services added a comment -

          Commit 4ee0f6494ea0d2265d930bce962ebcc3c7dbfe5c in accumulo's branch refs/heads/1.6.0-SNAPSHOT from Christopher Tubbs
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=4ee0f64 ]

          Fix recent warnings introduced by several tickets

          ACCUMULO-2005
          ACCUMULO-2182
          ACCUMULO-2224
          ACCUMULO-2234
          ACCUMULO-2539

          Show
          ASF subversion and git services added a comment - Commit 4ee0f6494ea0d2265d930bce962ebcc3c7dbfe5c in accumulo's branch refs/heads/1.6.0-SNAPSHOT from Christopher Tubbs [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=4ee0f64 ] Fix recent warnings introduced by several tickets ACCUMULO-2005 ACCUMULO-2182 ACCUMULO-2224 ACCUMULO-2234 ACCUMULO-2539
          Hide
          ASF subversion and git services added a comment -

          Commit 4ee0f6494ea0d2265d930bce962ebcc3c7dbfe5c in accumulo's branch refs/heads/1.5.2-SNAPSHOT from Christopher Tubbs
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=4ee0f64 ]

          Fix recent warnings introduced by several tickets

          ACCUMULO-2005
          ACCUMULO-2182
          ACCUMULO-2224
          ACCUMULO-2234
          ACCUMULO-2539

          Show
          ASF subversion and git services added a comment - Commit 4ee0f6494ea0d2265d930bce962ebcc3c7dbfe5c in accumulo's branch refs/heads/1.5.2-SNAPSHOT from Christopher Tubbs [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=4ee0f64 ] Fix recent warnings introduced by several tickets ACCUMULO-2005 ACCUMULO-2182 ACCUMULO-2224 ACCUMULO-2234 ACCUMULO-2539
          Hide
          ASF subversion and git services added a comment -

          Commit 4ee0f6494ea0d2265d930bce962ebcc3c7dbfe5c in accumulo's branch refs/heads/1.4.5-SNAPSHOT from Christopher Tubbs
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=4ee0f64 ]

          Fix recent warnings introduced by several tickets

          ACCUMULO-2005
          ACCUMULO-2182
          ACCUMULO-2224
          ACCUMULO-2234
          ACCUMULO-2539

          Show
          ASF subversion and git services added a comment - Commit 4ee0f6494ea0d2265d930bce962ebcc3c7dbfe5c in accumulo's branch refs/heads/1.4.5-SNAPSHOT from Christopher Tubbs [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=4ee0f64 ] Fix recent warnings introduced by several tickets ACCUMULO-2005 ACCUMULO-2182 ACCUMULO-2224 ACCUMULO-2234 ACCUMULO-2539
          Hide
          ASF subversion and git services added a comment -

          Commit 1577778 from busbey@apache.org in branch 'site/trunk'
          [ https://svn.apache.org/r1577778 ]

          Added Hung Pham to contributor list (ACCUMULO-2005)

          Show
          ASF subversion and git services added a comment - Commit 1577778 from busbey@apache.org in branch 'site/trunk' [ https://svn.apache.org/r1577778 ] Added Hung Pham to contributor list ( ACCUMULO-2005 )
          Hide
          Hung Pham added a comment -

          Just like everyone else, ie:

          username name organization timezone
          hpham Hung Pham Cloudera ET (-5 / -4)

          Show
          Hung Pham added a comment - Just like everyone else, ie: username name organization timezone hpham Hung Pham Cloudera ET (-5 / -4)
          Hide
          Sean Busbey added a comment -

          Thanks for the contribution Hung Pham. How would you like to be listed on the contributors page? (namely organization and timezone, if either)

          Show
          Sean Busbey added a comment - Thanks for the contribution Hung Pham . How would you like to be listed on the contributors page ? (namely organization and timezone, if either)
          Hide
          Sean Busbey added a comment -

          Attaching the final rebased versions of Hung Pham's patches.

          Show
          Sean Busbey added a comment - Attaching the final rebased versions of Hung Pham 's patches.
          Hide
          ASF subversion and git services added a comment -

          Commit bc9cee0c02952a0ba6d4c96f71204e84a5acb701 in accumulo's branch refs/heads/master from Hung Pham
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=bc9cee0 ]

          ACCUMULO-2005: Add a 3rd argument to functional test RunTests.java to allow a timeout factor for mapred.task.timeout and run.py -f flag

          (reimplemented for 1.5.x branch)

          Signed-off-by: Sean Busbey <busbey@cloudera.com>

          Show
          ASF subversion and git services added a comment - Commit bc9cee0c02952a0ba6d4c96f71204e84a5acb701 in accumulo's branch refs/heads/master from Hung Pham [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=bc9cee0 ] ACCUMULO-2005 : Add a 3rd argument to functional test RunTests.java to allow a timeout factor for mapred.task.timeout and run.py -f flag (reimplemented for 1.5.x branch) Signed-off-by: Sean Busbey <busbey@cloudera.com>
          Hide
          ASF subversion and git services added a comment -

          Commit 6faac421b560c9ce92f1d4db3c0b60d830498a71 in accumulo's branch refs/heads/master from Hung Pham
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=6faac42 ]

          ACCUMULO-2005: Add a 3rd argument to functional test RunTests.java to allow a timeout factor for mapred.task.timeout and run.py -f flag

          Signed-off-by: Sean Busbey <busbey@cloudera.com>

          Show
          ASF subversion and git services added a comment - Commit 6faac421b560c9ce92f1d4db3c0b60d830498a71 in accumulo's branch refs/heads/master from Hung Pham [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=6faac42 ] ACCUMULO-2005 : Add a 3rd argument to functional test RunTests.java to allow a timeout factor for mapred.task.timeout and run.py -f flag Signed-off-by: Sean Busbey <busbey@cloudera.com>
          Hide
          ASF subversion and git services added a comment -

          Commit bc9cee0c02952a0ba6d4c96f71204e84a5acb701 in accumulo's branch refs/heads/1.6.0-SNAPSHOT from Hung Pham
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=bc9cee0 ]

          ACCUMULO-2005: Add a 3rd argument to functional test RunTests.java to allow a timeout factor for mapred.task.timeout and run.py -f flag

          (reimplemented for 1.5.x branch)

          Signed-off-by: Sean Busbey <busbey@cloudera.com>

          Show
          ASF subversion and git services added a comment - Commit bc9cee0c02952a0ba6d4c96f71204e84a5acb701 in accumulo's branch refs/heads/1.6.0-SNAPSHOT from Hung Pham [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=bc9cee0 ] ACCUMULO-2005 : Add a 3rd argument to functional test RunTests.java to allow a timeout factor for mapred.task.timeout and run.py -f flag (reimplemented for 1.5.x branch) Signed-off-by: Sean Busbey <busbey@cloudera.com>
          Hide
          ASF subversion and git services added a comment -

          Commit 6faac421b560c9ce92f1d4db3c0b60d830498a71 in accumulo's branch refs/heads/1.6.0-SNAPSHOT from Hung Pham
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=6faac42 ]

          ACCUMULO-2005: Add a 3rd argument to functional test RunTests.java to allow a timeout factor for mapred.task.timeout and run.py -f flag

          Signed-off-by: Sean Busbey <busbey@cloudera.com>

          Show
          ASF subversion and git services added a comment - Commit 6faac421b560c9ce92f1d4db3c0b60d830498a71 in accumulo's branch refs/heads/1.6.0-SNAPSHOT from Hung Pham [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=6faac42 ] ACCUMULO-2005 : Add a 3rd argument to functional test RunTests.java to allow a timeout factor for mapred.task.timeout and run.py -f flag Signed-off-by: Sean Busbey <busbey@cloudera.com>
          Hide
          ASF subversion and git services added a comment -

          Commit bc9cee0c02952a0ba6d4c96f71204e84a5acb701 in accumulo's branch refs/heads/1.5.2-SNAPSHOT from Hung Pham
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=bc9cee0 ]

          ACCUMULO-2005: Add a 3rd argument to functional test RunTests.java to allow a timeout factor for mapred.task.timeout and run.py -f flag

          (reimplemented for 1.5.x branch)

          Signed-off-by: Sean Busbey <busbey@cloudera.com>

          Show
          ASF subversion and git services added a comment - Commit bc9cee0c02952a0ba6d4c96f71204e84a5acb701 in accumulo's branch refs/heads/1.5.2-SNAPSHOT from Hung Pham [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=bc9cee0 ] ACCUMULO-2005 : Add a 3rd argument to functional test RunTests.java to allow a timeout factor for mapred.task.timeout and run.py -f flag (reimplemented for 1.5.x branch) Signed-off-by: Sean Busbey <busbey@cloudera.com>
          Hide
          ASF subversion and git services added a comment -

          Commit 6faac421b560c9ce92f1d4db3c0b60d830498a71 in accumulo's branch refs/heads/1.5.2-SNAPSHOT from Hung Pham
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=6faac42 ]

          ACCUMULO-2005: Add a 3rd argument to functional test RunTests.java to allow a timeout factor for mapred.task.timeout and run.py -f flag

          Signed-off-by: Sean Busbey <busbey@cloudera.com>

          Show
          ASF subversion and git services added a comment - Commit 6faac421b560c9ce92f1d4db3c0b60d830498a71 in accumulo's branch refs/heads/1.5.2-SNAPSHOT from Hung Pham [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=6faac42 ] ACCUMULO-2005 : Add a 3rd argument to functional test RunTests.java to allow a timeout factor for mapred.task.timeout and run.py -f flag Signed-off-by: Sean Busbey <busbey@cloudera.com>
          Hide
          ASF subversion and git services added a comment -

          Commit 6faac421b560c9ce92f1d4db3c0b60d830498a71 in accumulo's branch refs/heads/1.4.5-SNAPSHOT from Hung Pham
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=6faac42 ]

          ACCUMULO-2005: Add a 3rd argument to functional test RunTests.java to allow a timeout factor for mapred.task.timeout and run.py -f flag

          Signed-off-by: Sean Busbey <busbey@cloudera.com>

          Show
          ASF subversion and git services added a comment - Commit 6faac421b560c9ce92f1d4db3c0b60d830498a71 in accumulo's branch refs/heads/1.4.5-SNAPSHOT from Hung Pham [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=6faac42 ] ACCUMULO-2005 : Add a 3rd argument to functional test RunTests.java to allow a timeout factor for mapred.task.timeout and run.py -f flag Signed-off-by: Sean Busbey <busbey@cloudera.com>
          Hide
          Sean Busbey added a comment -

          I've already got the rebase done. Just getting a test run finished.

          Show
          Sean Busbey added a comment - I've already got the rebase done. Just getting a test run finished.
          Hide
          Hung Pham added a comment -

          Hi Sean,

          Sorry, I've been late on emails. I've on-site with no internet access
          until now. Just caught up and saw this. Yes I'd like to go through
          rebasing exercise myself (to learn), but I probably won't be able to touch
          this til this weekend. Is that ok? If yes, please show me the steps.

          Thanks,
          Hung

          Show
          Hung Pham added a comment - Hi Sean, Sorry, I've been late on emails. I've on-site with no internet access until now. Just caught up and saw this. Yes I'd like to go through rebasing exercise myself (to learn), but I probably won't be able to touch this til this weekend. Is that ok? If yes, please show me the steps. Thanks, Hung
          Hide
          Sean Busbey added a comment -

          Unless someone objects, I'm rebasing these onto current 1.4.x and 1.5.x. I'm presuming v4 is for 1.4.5-SNAPSHOT and v7 is for 1.5.1-SNAPSHOT.

          Hung Pham, if you'd like to do that rebasing yourself please let me know.

          Show
          Sean Busbey added a comment - Unless someone objects, I'm rebasing these onto current 1.4.x and 1.5.x. I'm presuming v4 is for 1.4.5-SNAPSHOT and v7 is for 1.5.1-SNAPSHOT. Hung Pham , if you'd like to do that rebasing yourself please let me know.
          Hide
          Bill Havanki added a comment - - edited

          Hung Pham, my commits for ACCUMULO-2380 will likely conflict when you go to rebase /push. Feel free to call on me to help when the time comes.

          Show
          Bill Havanki added a comment - - edited Hung Pham , my commits for ACCUMULO-2380 will likely conflict when you go to rebase /push . Feel free to call on me to help when the time comes.
          Hide
          Hung Pham added a comment -

          ACCUMULO-2005.v7.patch.txt includes following:

          1) Annotation Parameter strTimeoutFactor was changed from String to Integer, intTimeoutFactor with a default value of 1.
          2) static private Integer timeoutFactor was eliminated and opts.intTimeoutFactor was used to assigned directly to static final String TIMEOUT_FACTOR.
          3) NumberFormatException was no longer needed.

          Show
          Hung Pham added a comment - ACCUMULO-2005 .v7.patch.txt includes following: 1) Annotation Parameter strTimeoutFactor was changed from String to Integer, intTimeoutFactor with a default value of 1. 2) static private Integer timeoutFactor was eliminated and opts.intTimeoutFactor was used to assigned directly to static final String TIMEOUT_FACTOR. 3) NumberFormatException was no longer needed.
          Hide
          Hung Pham added a comment -

          This patch is for accumulo-1.5.1-SNAPSHOT

          Show
          Hung Pham added a comment - This patch is for accumulo-1.5.1-SNAPSHOT
          Hide
          Sean Busbey added a comment -

          The patch applies well to 1.4.5-SNAPSHOT. Unfortunately, argument handling changed substantially in 1.5.x, so the changes won't impact there.

          Could you checkout 1.5.1-SNAPSHOT and take a look at making your changes work with that version of the code? You'd just add another patch for that branch (e.g. ACCUMULO-2005.v4-1.5.1-SNAPSHOT.patch).

          If you don't think you'll have time, just drop a note here and we'll get someone to handle it when they have time.

          Show
          Sean Busbey added a comment - The patch applies well to 1.4.5-SNAPSHOT. Unfortunately, argument handling changed substantially in 1.5.x, so the changes won't impact there. Could you checkout 1.5.1-SNAPSHOT and take a look at making your changes work with that version of the code? You'd just add another patch for that branch (e.g. ACCUMULO-2005 .v4-1.5.1-SNAPSHOT.patch). If you don't think you'll have time, just drop a note here and we'll get someone to handle it when they have time.
          Hide
          Hung Pham added a comment -

          Add a 3rd argument to functional test RunTests.java to allow a timeout factor for mapred.task.timeout and run.py -f flag

          Show
          Hung Pham added a comment - Add a 3rd argument to functional test RunTests.java to allow a timeout factor for mapred.task.timeout and run.py -f flag
          Hide
          Hung Pham added a comment -

          Add a 3rd argument to functional test RunTests.java to allow a timeout factor for mapred.task.timeout and run.py -f flag

          Show
          Hung Pham added a comment - Add a 3rd argument to functional test RunTests.java to allow a timeout factor for mapred.task.timeout and run.py -f flag
          Hide
          Mike Drob added a comment -

          Hung Pham - I've updated your JIRA permissions so that you can assign Accumulo issues now. Thanks for volunteering to contribute!

          Show
          Mike Drob added a comment - Hung Pham - I've updated your JIRA permissions so that you can assign Accumulo issues now. Thanks for volunteering to contribute!
          Hide
          Hung Pham added a comment -

          I will work on this ticket. Could someone with admin privs assign it to me. Thanks!

          Show
          Hung Pham added a comment - I will work on this ticket. Could someone with admin privs assign it to me. Thanks!
          Hide
          Sean Busbey added a comment -

          we should also make sure that such a timeout scaling argument also results in the same factor being passed to the individual tests.

          Show
          Sean Busbey added a comment - we should also make sure that such a timeout scaling argument also results in the same factor being passed to the individual tests.

            People

            • Assignee:
              Hung Pham
              Reporter:
              Hung Pham
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development