Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.1.0
    • Component/s: Spark
    • Labels:
      None

      Description

      This jira is to enable most of the tests below. If tests don't pass because of some unsupported feature, ensure that a JIRA exists and move on.

      limit_pushdown.q,\
        load_dyn_part1.q,\
        load_dyn_part2.q,\
        load_dyn_part3.q,\
        mapjoin_mapjoin.q,\
        mapreduce1.q,\
        mapreduce2.q,\
        merge1.q,\
        merge2.q,\
        metadata_only_queries.q,\
        optimize_nullscan.q,\
        orc_analyze.q,\
        orc_merge1.q,\
        orc_merge2.q,\
        orc_merge3.q,\
        orc_merge4.q,\
      
      1. HIVE-7792.1-spark.patch
        260 kB
        Venki Korukanti

        Issue Links

          Activity

          Hide
          brocknoland Brock Noland added a comment -

          Excellent work Venki! I have committed this to spark!

          Show
          brocknoland Brock Noland added a comment - Excellent work Venki! I have committed this to spark!
          Hide
          hiveqa Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12663980/HIVE-7792.1-spark.patch

          ERROR: -1 due to 4 failed/errored test(s), 6253 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fs_default_name2
          org.apache.hive.hcatalog.pig.TestOrcHCatLoader.testReadDataPrimitiveTypes
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/91/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/91/console
          Test logs: http://ec2-54-176-176-199.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-91/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 4 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12663980

          Show
          hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12663980/HIVE-7792.1-spark.patch ERROR: -1 due to 4 failed/errored test(s), 6253 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fs_default_name2 org.apache.hive.hcatalog.pig.TestOrcHCatLoader.testReadDataPrimitiveTypes Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/91/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/91/console Test logs: http://ec2-54-176-176-199.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-91/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 4 tests failed This message is automatically generated. ATTACHMENT ID: 12663980
          Hide
          vkorukanti Venki Korukanti added a comment -

          For merge[1-2].q, stat (number of rows) differences are in intermediate tables created in test and inserted data using "insert overwrite". In MR/Tez case stats are calculated during insert overwrite, but no in Spark case. Thats why there are differences in query plan when the same intermediate table is used in a query.

          Show
          vkorukanti Venki Korukanti added a comment - For merge [1-2] .q, stat (number of rows) differences are in intermediate tables created in test and inserted data using "insert overwrite". In MR/Tez case stats are calculated during insert overwrite, but no in Spark case. Thats why there are differences in query plan when the same intermediate table is used in a query.
          Hide
          vkorukanti Venki Korukanti added a comment -

          Added merge[1-2].q tests to existing patch. merge[1-2].q test outputs are different from MR, but same as Tez as merge related properties are applicable to only MR. Rest of the tests orc_merge[1-4].q have TRANSFORM. These tests run fine, but the output data is different. Issues could be same as HIVE-7799.

          Show
          vkorukanti Venki Korukanti added a comment - Added merge [1-2] .q tests to existing patch. merge [1-2] .q test outputs are different from MR, but same as Tez as merge related properties are applicable to only MR. Rest of the tests orc_merge [1-4] .q have TRANSFORM. These tests run fine, but the output data is different. Issues could be same as HIVE-7799 .
          Hide
          vkorukanti Venki Korukanti added a comment -

          Attached patch enables following tests:

           * metadata_only_queries.q
           * load_dyn_part2.q
           * load_dyn_part3.q
           * mapreduce1.q
           * mapreduce2.q
           * limit_pushdown.q (Order of results is different from MR, but order is deterministic)
          
           * load_dyn_part1.q - Failure - tracked by HIVE-7842
           * mapjoin_mapjoin.q: Results are wrong, probably because MapJoin is not  supported yet (HIVE-7613).
           * optimize_nullscan.q: Differences in table serdes and plan. Looks like the plan is not optimized for limit 0 cases - tracked by HIVE-7844
           * orc_analyze.q - Failure - tracked by HIVE-7843
          

          Stats differences such as below for rest of the q files.

          < Statistics: Num rows: 1000 Data size: 94000 Basic stats: COMPLETE Column stats: NONE
          ---
          > Statistics: Num rows: 46 Data size: 4920 Basic stats: COMPLETE Column stats: NONE
          

          Not sure if it is because we don't have stats collection from Spark job yet. Still investigating.

           * orc_merge1.q
           * orc_merge2.q
           * orc_merge3.q
           * orc_merge4.q
           * merge1.q
           * merge2.q
          
          Show
          vkorukanti Venki Korukanti added a comment - Attached patch enables following tests: * metadata_only_queries.q * load_dyn_part2.q * load_dyn_part3.q * mapreduce1.q * mapreduce2.q * limit_pushdown.q (Order of results is different from MR, but order is deterministic) * load_dyn_part1.q - Failure - tracked by HIVE-7842 * mapjoin_mapjoin.q: Results are wrong, probably because MapJoin is not supported yet (HIVE-7613). * optimize_nullscan.q: Differences in table serdes and plan. Looks like the plan is not optimized for limit 0 cases - tracked by HIVE-7844 * orc_analyze.q - Failure - tracked by HIVE-7843 Stats differences such as below for rest of the q files. < Statistics: Num rows: 1000 Data size: 94000 Basic stats: COMPLETE Column stats: NONE --- > Statistics: Num rows: 46 Data size: 4920 Basic stats: COMPLETE Column stats: NONE Not sure if it is because we don't have stats collection from Spark job yet. Still investigating. * orc_merge1.q * orc_merge2.q * orc_merge3.q * orc_merge4.q * merge1.q * merge2.q

            People

            • Assignee:
              vkorukanti Venki Korukanti
              Reporter:
              brocknoland Brock Noland
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development