Hive
  1. Hive
  2. HIVE-6979

Hadoop-2 test failures related to quick stats not being populated correctly

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.14.0
    • Fix Version/s: 0.14.0
    • Component/s: None
    • Labels:
      None

      Description

      The test failures that are currently reported by Hive QA running on hadoop-2 (https://issues.apache.org/jira/browse/HIVE-6968?focusedCommentId=13980570&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13980570) are related to difference in the way hadoop FileSystem.globStatus() api behaves. For a directory structure like below

      dir1/file1
      dir1/file2
      

      Two level of path pattern like dir1// will return both files in hadoop 1.x but will return empty result in hadoop 2.x (in fact it will say no such file or directory and return empty file status array). Hadoop 2.x seems to be compliant to linux behaviour (ls dir1//) but hadoop 1.x is not.

      As a result of this, the fast statistics (NUM_FILES and TOTAL_SIZE) are populated wrongly causing diffs in qfile tests for hadoop-1 and hadoop-2.

      1. HIVE-6979.1.patch
        64 kB
        Prasanth Jayachandran
      2. HIVE-6979.2.patch
        197 kB
        Prasanth Jayachandran
      3. HIVE-6979.3.patch
        199 kB
        Prasanth Jayachandran

        Issue Links

          Activity

          Hide
          Thejas M Nair added a comment -

          This has been fixed in 0.14 release. Please open new jira if you see any issues.

          Show
          Thejas M Nair added a comment - This has been fixed in 0.14 release. Please open new jira if you see any issues.
          Hide
          Ashutosh Chauhan added a comment -

          Committed this to trunk. Thanks, Prasanth! Lets track remaining failures in follow-up.

          Show
          Ashutosh Chauhan added a comment - Committed this to trunk. Thanks, Prasanth! Lets track remaining failures in follow-up.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12642181/HIVE-6979.3.patch

          ERROR: -1 due to 5 failed/errored test(s), 5420 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby2_map_skew
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_partscan_1_23
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dynamic_partitions_with_whitelist
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_stats_partialscan_autogether
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/62/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/62/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 5 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12642181

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12642181/HIVE-6979.3.patch ERROR: -1 due to 5 failed/errored test(s), 5420 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby2_map_skew org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_partscan_1_23 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dynamic_partitions_with_whitelist org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_stats_partialscan_autogether Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/62/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/62/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 5 tests failed This message is automatically generated. ATTACHMENT ID: 12642181
          Hide
          Ashutosh Chauhan added a comment -

          +1

          Show
          Ashutosh Chauhan added a comment - +1
          Hide
          Prasanth Jayachandran added a comment -

          union_remove_25 was failing as it wasn't able to find partition. So increased the limit count so that the specific partition value is always present. Updated stats_partscan_1_23 test as well. Other tests seems to pass in my local setup (Mac OS X).

          Show
          Prasanth Jayachandran added a comment - union_remove_25 was failing as it wasn't able to find partition. So increased the limit count so that the specific partition value is always present. Updated stats_partscan_1_23 test as well. Other tests seems to pass in my local setup (Mac OS X).
          Hide
          Ashutosh Chauhan added a comment - - edited

          union_remove_25 wasn't expected to fail. Prashant, can you take a look?

          Show
          Ashutosh Chauhan added a comment - - edited union_remove_25 wasn't expected to fail. Prashant, can you take a look?
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12642117/HIVE-6979.2.patch

          ERROR: -1 due to 5 failed/errored test(s), 5419 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby2_map_skew
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_partscan_1_23
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_25
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dynamic_partitions_with_whitelist
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/59/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/59/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 5 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12642117

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12642117/HIVE-6979.2.patch ERROR: -1 due to 5 failed/errored test(s), 5419 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby2_map_skew org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_partscan_1_23 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_25 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dynamic_partitions_with_whitelist Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/59/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/59/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 5 tests failed This message is automatically generated. ATTACHMENT ID: 12642117
          Hide
          Prasanth Jayachandran added a comment -

          Addressed Ashutosh Chauhan's review comments. Ashutosh Chauhan I fixed the recent test failures. Can you please take a look at the changes in RB?

          Show
          Prasanth Jayachandran added a comment - Addressed Ashutosh Chauhan 's review comments. Ashutosh Chauhan I fixed the recent test failures. Can you please take a look at the changes in RB?
          Hide
          Ashutosh Chauhan added a comment -

          Seems like a progress all union_remove tests are now passing. But seems like some new failures got introduced. I cannot repro some, but able to repro few others, like create_like.q, database_drop.q etc.

          Show
          Ashutosh Chauhan added a comment - Seems like a progress all union_remove tests are now passing. But seems like some new failures got introduced. I cannot repro some, but able to repro few others, like create_like.q, database_drop.q etc.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12642049/HIVE-6979.1.patch

          ERROR: -1 due to 32 failed/errored test(s), 5418 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_7
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer15
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_like
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_database_drop
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_exim_10_external_managed
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_exim_12_external_location
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_exim_13_managed_location
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_exim_14_managed_location_over_existing
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_numeric
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby2_map_skew
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_overwrite_local_directory_1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_fs
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_fs_overwrite
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_test_outer
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullformatCTAS
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_create_table_alter
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_tblproperties
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_noscan_2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_partscan_1_23
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_unset_table_view_property
          org.apache.hadoop.hive.cli.TestHBaseMinimrCliDriver.testCliDriver_hbase_bulk
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_external_table_with_space_in_location_path
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_file_with_header_footer
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dynamic_partitions_with_whitelist
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_unset_table_property
          org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_file_with_header_footer_negative
          org.apache.hadoop.hive.ql.security.TestFolderPermissions.testExternalTable
          org.apache.hive.hcatalog.api.TestHCatClient.testCreateTableLike
          org.apache.hive.hcatalog.pig.TestHCatLoader.testConvertBooleanToInt
          org.apache.hive.hcatalog.pig.TestHCatLoaderStorer.testReadWrite
          org.apache.hive.hcatalog.pig.TestOrcHCatLoader.testConvertBooleanToInt
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/54/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/54/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 32 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12642049

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12642049/HIVE-6979.1.patch ERROR: -1 due to 32 failed/errored test(s), 5418 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer15 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_like org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_database_drop org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_exim_10_external_managed org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_exim_12_external_location org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_exim_13_managed_location org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_exim_14_managed_location_over_existing org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_numeric org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby2_map_skew org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_overwrite_local_directory_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_fs org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_fs_overwrite org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_test_outer org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullformatCTAS org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_create_table_alter org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_tblproperties org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_noscan_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_partscan_1_23 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_unset_table_view_property org.apache.hadoop.hive.cli.TestHBaseMinimrCliDriver.testCliDriver_hbase_bulk org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_external_table_with_space_in_location_path org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_file_with_header_footer org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dynamic_partitions_with_whitelist org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_unset_table_property org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_file_with_header_footer_negative org.apache.hadoop.hive.ql.security.TestFolderPermissions.testExternalTable org.apache.hive.hcatalog.api.TestHCatClient.testCreateTableLike org.apache.hive.hcatalog.pig.TestHCatLoader.testConvertBooleanToInt org.apache.hive.hcatalog.pig.TestHCatLoaderStorer.testReadWrite org.apache.hive.hcatalog.pig.TestOrcHCatLoader.testConvertBooleanToInt Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/54/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/54/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 32 tests failed This message is automatically generated. ATTACHMENT ID: 12642049
          Hide
          Ashutosh Chauhan added a comment -

          +1 Left couple of comments on RB, if you prefer to fix them in follow-up, thats fine with me.

          Show
          Ashutosh Chauhan added a comment - +1 Left couple of comments on RB, if you prefer to fix them in follow-up, thats fine with me.
          Hide
          Prasanth Jayachandran added a comment -

          Ashutosh Chauhan/Jason Dere Can anyone take a look at the test failures fix?

          Show
          Prasanth Jayachandran added a comment - Ashutosh Chauhan / Jason Dere Can anyone take a look at the test failures fix?
          Hide
          Prasanth Jayachandran added a comment -

          attaching RB link

          Show
          Prasanth Jayachandran added a comment - attaching RB link

            People

            • Assignee:
              Prasanth Jayachandran
              Reporter:
              Prasanth Jayachandran
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development