Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.14.0
    • Component/s: None
    • Labels:
    • Release Note:
      Hide
      hive.exec.submit.local.task.via.child

      Determines whether local tasks (typically mapjoin hashtable generation phase) runs in separate JVM ("true", recommended) or not. Avoids the overhead of spawning new JVM, but can lead to out-of-memory issues.
      Show
      hive.exec.submit.local.task.via.child Determines whether local tasks (typically mapjoin hashtable generation phase) runs in separate JVM ("true", recommended) or not. Avoids the overhead of spawning new JVM, but can lead to out-of-memory issues.

      Description

      Did some experiments to see if there's a way to speed up unit tests. TestCliDriver seemed to take a lot of time just spinning up/tearing down JVMs. I was also curious to see if running everything on a ram disk would help.

      Results (I ran tests up to authorization_2):

      • Current setup: 40 minutes
      • Single JVM (not using child JVM to run all queries): 8 minutes
      • Single JVM + ram disk: 7 minutes

      So the ram disk didn't help that much. But running tests in single JVM seems worthwhile doing.

      1. HIVE-7271.1.patch
        10 kB
        Gunther Hagleitner
      2. HIVE-7271.2.patch
        33 kB
        Gunther Hagleitner
      3. HIVE-7271.3.patch
        32 kB
        Gunther Hagleitner
      4. HIVE-7271.4.patch
        32 kB
        Gunther Hagleitner
      5. HIVE-7271.5.patch
        32 kB
        Gunther Hagleitner
      6. HIVE-7271.6.patch
        35 kB
        Gunther Hagleitner
      7. HIVE-7271.7.patch
        36 kB
        Gunther Hagleitner

        Issue Links

          Activity

          Hide
          Lefty Leverenz added a comment -

          Doc note: hive.exec.submit.local.task.via.child is documented in two places in the wiki.

          Please review the doc. If it's okay, the TODOC14 label can be removed from this issue. If changes are needed, just let me know.

          Show
          Lefty Leverenz added a comment - Doc note: hive.exec.submit.local.task.via.child is documented in two places in the wiki. Unit Test Parallel Execution – Configuration Configuration Properties – Test Properties – hive.exec.submit.local.task.via.child Please review the doc. If it's okay, the TODOC14 label can be removed from this issue. If changes are needed, just let me know.
          Hide
          Thejas M Nair added a comment -

          This has been fixed in 0.14 release. Please open new jira if you see any issues.

          Show
          Thejas M Nair added a comment - This has been fixed in 0.14 release. Please open new jira if you see any issues.
          Hide
          Lefty Leverenz added a comment -

          Perhaps this should be documented here too:

          Show
          Lefty Leverenz added a comment - Perhaps this should be documented here too: Unit Test Parallel Execution
          Hide
          Lefty Leverenz added a comment -

          Gunther Hagleitner, the wiki has a section for test parameters (hive.test.mode, etc.) so this can be documented with them. I recommend that the release note & description explicitly advise people not to set this parameter to true except when running unit tests. Of course the wiki will link to this Jira so they'll be able to find the information, but why make them hunt for it?

          When HIVE-6037 gets committed, all parameter descriptions will be in HiveConf.java and then hive-default.xml.template will be generated from HiveConf.java. You can have an empty description if you prefer, but the parameter will still appear in hive-default.xml.template at that point.

          Show
          Lefty Leverenz added a comment - Gunther Hagleitner , the wiki has a section for test parameters (hive.test.mode, etc.) so this can be documented with them. I recommend that the release note & description explicitly advise people not to set this parameter to true except when running unit tests. Of course the wiki will link to this Jira so they'll be able to find the information, but why make them hunt for it? When HIVE-6037 gets committed, all parameter descriptions will be in HiveConf.java and then hive-default.xml.template will be generated from HiveConf.java. You can have an empty description if you prefer, but the parameter will still appear in hive-default.xml.template at that point. Configuration Properties – Test Properties
          Hide
          Gunther Hagleitner added a comment -

          Lefty Leverenz Just a thought: This parameter is useful to run unit tests quicker, but in general not something i would recommend anyone to set to true outside of that. It basically runs the hashtable generation for mapjoins in the CLI process - something that could give you memory issues. I'll put that in the release notes, but in general I think parameters like this needn't be exposed to users. In these cases should we just leave it out of the xml?

          Show
          Gunther Hagleitner added a comment - Lefty Leverenz Just a thought: This parameter is useful to run unit tests quicker, but in general not something i would recommend anyone to set to true outside of that. It basically runs the hashtable generation for mapjoins in the CLI process - something that could give you memory issues. I'll put that in the release notes, but in general I think parameters like this needn't be exposed to users. In these cases should we just leave it out of the xml?
          Hide
          Lefty Leverenz added a comment -

          This adds configuration parameter hive.exec.submit.local.task.via.child to HiveConf.java, so it needs a description in hive-default.xml.template.

          Or you can put the description in a release note, and I'll make sure it gets into the new version of HiveConf.java (which will include descriptions) after HIVE-6037 gets committed.

          Show
          Lefty Leverenz added a comment - This adds configuration parameter hive.exec.submit.local.task.via.child to HiveConf.java, so it needs a description in hive-default.xml.template. Or you can put the description in a release note, and I'll make sure it gets into the new version of HiveConf.java (which will include descriptions) after HIVE-6037 gets committed.
          Hide
          Gunther Hagleitner added a comment -

          Committed to trunk. Thanks Brock Noland and Szehon Ho!

          Show
          Gunther Hagleitner added a comment - Committed to trunk. Thanks Brock Noland and Szehon Ho !
          Hide
          Gunther Hagleitner added a comment -

          Test failures are unrelated. (ran schemeAuthority locally a couple times no problems).

          Show
          Gunther Hagleitner added a comment - Test failures are unrelated. (ran schemeAuthority locally a couple times no problems).
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12652249/HIVE-7271.7.patch

          ERROR: -1 due to 3 failed/errored test(s), 5669 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority2
          org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/582/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/582/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-582/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 3 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12652249

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12652249/HIVE-7271.7.patch ERROR: -1 due to 3 failed/errored test(s), 5669 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority2 org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/582/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/582/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-582/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed This message is automatically generated. ATTACHMENT ID: 12652249
          Hide
          Szehon Ho added a comment -

          Sounds good, thanks.

          Show
          Szehon Ho added a comment - Sounds good, thanks.
          Hide
          Gunther Hagleitner added a comment -

          The other failures are unrelated. Szehon Ho - = I did not analyze all of them in detail, but yes, the ones I looked at ran OOM, so I switched them to out-of-process.

          Show
          Gunther Hagleitner added a comment - The other failures are unrelated. Szehon Ho - = I did not analyze all of them in detail, but yes, the ones I looked at ran OOM, so I switched them to out-of-process.
          Hide
          Gunther Hagleitner added a comment -

          Smaller size causes MTQueries to fail. In the latest patch I've made a small change to run the multithreaded query test starting child VM (makes sense that multiple join/group by queries in process would eventually run out of memory).

          Show
          Gunther Hagleitner added a comment - Smaller size causes MTQueries to fail. In the latest patch I've made a small change to run the multithreaded query test starting child VM (makes sense that multiple join/group by queries in process would eventually run out of memory).
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12652124/HIVE-7271.6.patch

          ERROR: -1 due to 4 failed/errored test(s), 5654 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample8
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
          org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1
          org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/572/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/572/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-572/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 4 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12652124

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12652124/HIVE-7271.6.patch ERROR: -1 due to 4 failed/errored test(s), 5654 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample8 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/572/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/572/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-572/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 4 tests failed This message is automatically generated. ATTACHMENT ID: 12652124
          Hide
          Szehon Ho added a comment -

          +1, thanks a lot for this Gunther. Just for my understanding, the tests for which you set submitviachild=true and local.task.via.child=true were because of OOM at 2GB?

          Show
          Szehon Ho added a comment - +1, thanks a lot for this Gunther. Just for my understanding, the tests for which you set submitviachild=true and local.task.via.child=true were because of OOM at 2GB?
          Hide
          Brock Noland added a comment -

          +1 LGTM

          Regardless of the memory item, I updated our instance types since the c3 have a faster CPU.

          Show
          Brock Noland added a comment - +1 LGTM Regardless of the memory item, I updated our instance types since the c3 have a faster CPU.
          Hide
          Gunther Hagleitner added a comment -

          .6 fixes test failures (golden files again). Also includes the renamed methods Szehon Ho asked for.

          Show
          Gunther Hagleitner added a comment - .6 fixes test failures (golden files again). Also includes the renamed methods Szehon Ho asked for.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12651931/HIVE-7271.5.patch

          ERROR: -1 due to 6 failed/errored test(s), 5669 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dyn_part3
          org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/558/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/558/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-558/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 6 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12651931

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12651931/HIVE-7271.5.patch ERROR: -1 due to 6 failed/errored test(s), 5669 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dyn_part3 org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/558/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/558/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-558/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 6 tests failed This message is automatically generated. ATTACHMENT ID: 12651931
          Hide
          Gunther Hagleitner added a comment -

          Maybe we don't have to switch instance type after all. I just tried to run some of the larger tests with 2gb. Looks good. I might have just run into a mem leak earlier. Before this patch we ran 2 1gb jvms per query. Now it's 1 2gb. If that works mem requirements should be the same.

          Show
          Gunther Hagleitner added a comment - Maybe we don't have to switch instance type after all. I just tried to run some of the larger tests with 2gb. Looks good. I might have just run into a mem leak earlier. Before this patch we ran 2 1gb jvms per query. Now it's 1 2gb. If that works mem requirements should be the same.
          Hide
          Brock Noland added a comment -

          To use the new instance type we need to upgrade jclouds. I created HIVE-7274.

          Show
          Brock Noland added a comment - To use the new instance type we need to upgrade jclouds. I created HIVE-7274 .
          Hide
          Brock Noland added a comment -

          I just checked and we run 3 threads per c1.xlarge (3 * 4GB max heap). I think we should change to c3* instance types. It looks like spot prices for c3.2xlarge, which has 15GB of ram, are similar to c1.xlarge.

          Show
          Brock Noland added a comment - I just checked and we run 3 threads per c1.xlarge (3 * 4GB max heap). I think we should change to c3* instance types. It looks like spot prices for c3.2xlarge, which has 15GB of ram, are similar to c1.xlarge.
          Hide
          Brock Noland added a comment -

          We specify the number of q files to include in a "test batch" and that number is much less than the number of q file tests. Thus we probably aren't seeing as much benefit. After this change we can probably increase the batch size.

          Show
          Brock Noland added a comment - We specify the number of q files to include in a "test batch" and that number is much less than the number of q file tests. Thus we probably aren't seeing as much benefit. After this change we can probably increase the batch size.
          Hide
          Gunther Hagleitner added a comment -

          Also - checked some build/test times in the precommit q. seems like the cli portion of the tests went from 25 hrs to more like 8 hrs. Although run in parallel it seems to not hit that hard - something else the critical path?

          Show
          Gunther Hagleitner added a comment - Also - checked some build/test times in the precommit q. seems like the cli portion of the tests went from 25 hrs to more like 8 hrs. Although run in parallel it seems to not hit that hard - something else the critical path?
          Hide
          Gunther Hagleitner added a comment -

          missed two golden files in .2. other than that looks like no new failures.

          Show
          Gunther Hagleitner added a comment - missed two golden files in .2. other than that looks like no new failures.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12651897/HIVE-7271.2.patch

          ERROR: -1 due to 7 failed/errored test(s), 5625 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_explain
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_10_0
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dyn_part3
          org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/552/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/552/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-552/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 7 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12651897

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12651897/HIVE-7271.2.patch ERROR: -1 due to 7 failed/errored test(s), 5625 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_explain org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_10_0 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dyn_part3 org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/552/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/552/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-552/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 7 tests failed This message is automatically generated. ATTACHMENT ID: 12651897
          Hide
          Gunther Hagleitner added a comment -

          Szehon Ho I think this is already fixed. I remember having to clean that up for tez container reuse. I think the IOContext is properly reinitialized for each file now. Also, the unit tests should show if that was still an issue.

          Show
          Gunther Hagleitner added a comment - Szehon Ho I think this is already fixed. I remember having to clean that up for tez container reuse. I think the IOContext is properly reinitialized for each file now. Also, the unit tests should show if that was still an issue.
          Hide
          Gunther Hagleitner added a comment -

          .2 fixes test failures and reigns in mem to 4g.

          Show
          Gunther Hagleitner added a comment - .2 fixes test failures and reigns in mem to 4g.
          Hide
          Szehon Ho added a comment -

          Oh yea I hit that once, there's some statics in ExecMapper that don't get reset after the MR job, and result in some error if you run more than one query in localJobRunner, hence they forced the localJobRunner jobs to run out of process. Not sure, but it may still be an issue that needs to be fixed, for this to work.

          Show
          Szehon Ho added a comment - Oh yea I hit that once, there's some statics in ExecMapper that don't get reset after the MR job, and result in some error if you run more than one query in localJobRunner, hence they forced the localJobRunner jobs to run out of process. Not sure, but it may still be an issue that needs to be fixed, for this to work.
          Hide
          Gunther Hagleitner added a comment -

          We shouldn't remove the hadoopShims check of local mode, in MapRedTask right? Shouldn't user still able to set this via the mr-specific flags?

          I think this was done to "fix" something broken in the unit tests. The user can always choose in or out of process via the hive conf var (submit via child). This is independent of using local job runner or not (right now we always run out of process in local mode.)

          Not related to this change, but can this method name be refactored as it's confusing (now its executing in process and not in child jvm)

          Yes, this is cheesy. I'll fix that.

          Show
          Gunther Hagleitner added a comment - We shouldn't remove the hadoopShims check of local mode, in MapRedTask right? Shouldn't user still able to set this via the mr-specific flags? I think this was done to "fix" something broken in the unit tests. The user can always choose in or out of process via the hive conf var (submit via child). This is independent of using local job runner or not (right now we always run out of process in local mode.) Not related to this change, but can this method name be refactored as it's confusing (now its executing in process and not in child jvm) Yes, this is cheesy. I'll fix that.
          Hide
          Szehon Ho added a comment -

          Today the ptest host instance is size C1.xlarge (7gb).

          Glad somebody is looking at it, also had minor comments.

          1. We shouldn't remove the hadoopShims check of local mode, in MapRedTask right? Shouldn't user still able to set this via the mr-specific flags?

          1. Not related to this change, but can this method name be refactored as it's confusing (now its executing in process and not in child jvm)

          +    if (!conf.getBoolVar(HiveConf.ConfVars.SUBMITLOCALTASKVIACHILD)) {
          +      // execute in process
          +      return executeFromChildJVM(driverContext);
          +    }
          
          Show
          Szehon Ho added a comment - Today the ptest host instance is size C1.xlarge (7gb). Glad somebody is looking at it, also had minor comments. 1. We shouldn't remove the hadoopShims check of local mode, in MapRedTask right? Shouldn't user still able to set this via the mr-specific flags? 1. Not related to this change, but can this method name be refactored as it's confusing (now its executing in process and not in child jvm) + if (!conf.getBoolVar(HiveConf.ConfVars.SUBMITLOCALTASKVIACHILD)) { + // execute in process + return executeFromChildJVM(driverContext); + }
          Hide
          Gunther Hagleitner added a comment -

          I removed the asserts because it turns out that they are firing all the time in our unit tests. However since we execute the queries in a separate jvm that doesn't have assertions turned on they are ignored. Finally - even when they fire they result in correct query execution. Seems they were useful some time ago, but don't have any meaning anymore.

          In other words: In today's hive, "assert XXX" will only fire if you write an actual unit test (not .q files). We should probably switch to throwing assertion error or use guava's preconditions to encode certain conditions in hive.

          Brock Noland/Szehon Ho it might be possible to run with less than 8GB. I added that to run hashtable gen in process with all the other stuff. What instance type are we using currently?

          Show
          Gunther Hagleitner added a comment - I removed the asserts because it turns out that they are firing all the time in our unit tests. However since we execute the queries in a separate jvm that doesn't have assertions turned on they are ignored. Finally - even when they fire they result in correct query execution. Seems they were useful some time ago, but don't have any meaning anymore. In other words: In today's hive, "assert XXX" will only fire if you write an actual unit test (not .q files). We should probably switch to throwing assertion error or use guava's preconditions to encode certain conditions in hive. Brock Noland / Szehon Ho it might be possible to run with less than 8GB. I added that to run hashtable gen in process with all the other stuff. What instance type are we using currently?
          Hide
          Brock Noland added a comment -

          Also be curious why we are removing the assertions in StandardStructObjectInspector.

          Show
          Brock Noland added a comment - Also be curious why we are removing the assertions in StandardStructObjectInspector.
          Hide
          Brock Noland added a comment -

          It looks like this increases the heap of the tests to 8GB. We'd have to change the instance type we are using for PTest to allow for this.

          FYI Szehon Ho

          Show
          Brock Noland added a comment - It looks like this increases the heap of the tests to 8GB. We'd have to change the instance type we are using for PTest to allow for this. FYI Szehon Ho
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12651843/HIVE-7271.1.patch

          ERROR: -1 due to 43 failed/errored test(s), 5607 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_excludeHadoop20
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_explain
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_opt_vectorization
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_optimization
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_fetch_aggregation
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_bucket_sort_convert_join
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_hook
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nonmr_fetch
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_analyze
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample10
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_parquet
          org.apache.hadoop.hive.cli.TestContribCliDriver.testCliDriver_dboutput
          org.apache.hadoop.hive.cli.TestContribNegativeCliDriver.testNegativeCliDriver_case_with_row_sequence
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_cachingprintstream
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_compute_stats_long
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dyn_part3
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dyn_part_max_per_node
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_index_compact_entry_limit
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_index_compact_size_limit
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_local_mapred_error_cache
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe2
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe3
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_error
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_serde_regex2
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_stats_aggregator_error_2
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_stats_publisher_error_1
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_stats_publisher_error_2
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udf_assert_true
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udf_assert_true2
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udf_reflect_neg
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udf_test_error
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udf_test_error_reduce
          org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapPlan1
          org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapPlan2
          org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan1
          org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan2
          org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan3
          org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan4
          org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan5
          org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan6
          org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/551/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/551/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-551/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 43 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12651843

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12651843/HIVE-7271.1.patch ERROR: -1 due to 43 failed/errored test(s), 5607 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_excludeHadoop20 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_explain org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_opt_vectorization org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_optimization org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_fetch_aggregation org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_bucket_sort_convert_join org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_hook org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nonmr_fetch org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_analyze org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample10 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_parquet org.apache.hadoop.hive.cli.TestContribCliDriver.testCliDriver_dboutput org.apache.hadoop.hive.cli.TestContribNegativeCliDriver.testNegativeCliDriver_case_with_row_sequence org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_cachingprintstream org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_compute_stats_long org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dyn_part3 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dyn_part_max_per_node org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_index_compact_entry_limit org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_index_compact_size_limit org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_local_mapred_error_cache org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe2 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe3 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_error org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_serde_regex2 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_stats_aggregator_error_2 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_stats_publisher_error_1 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_stats_publisher_error_2 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udf_assert_true org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udf_assert_true2 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udf_reflect_neg org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udf_test_error org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udf_test_error_reduce org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapPlan1 org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapPlan2 org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan1 org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan2 org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan3 org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan4 org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan5 org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan6 org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/551/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/551/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-551/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 43 tests failed This message is automatically generated. ATTACHMENT ID: 12651843

            People

            • Assignee:
              Gunther Hagleitner
              Reporter:
              Gunther Hagleitner
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development