Hive
  1. Hive
  2. HIVE-6572

Use shimmed version of hadoop conf names for mapred.{min,max}.split.size{.*}

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.13.0, 0.14.0
    • Fix Version/s: 0.13.0
    • Component/s: None
    • Labels:
      None

      Description

      HadoopShims has a method to fetch config parameters by name so that they return the appropriate config param name for the appropriate hadoop version. We need to be consistent about using these versions.

      For eg:. mapred.min.split.size is deprecated with hadoop 2.x, and is instead called mapreduce.input.fileinputformat.split.minsize .

      Also, there is a bug in Hadoop23Shims, Hadoop20SShims and Hadoop20Shims that defines MAPREDMINSPLITSIZEPERNODE as mapred.min.split.size.per.rack and MAPREDMINSPLITSIZEPERRACK as mapred.min.split.size.per.node. This is wrong and confusing.

      1. HIVE-6572.patch
        13 kB
        Sushanth Sowmyan

        Issue Links

          Activity

          Hide
          Ashutosh Chauhan added a comment -

          Committed to trunk & 0.13 Thanks, Sushanth!

          Show
          Ashutosh Chauhan added a comment - Committed to trunk & 0.13 Thanks, Sushanth!
          Hide
          Ashutosh Chauhan added a comment -

          +1

          Show
          Ashutosh Chauhan added a comment - +1
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12633249/HIVE-6572.patch

          ERROR: -1 due to 2 failed/errored test(s), 5373 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_merge
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1654/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1654/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 2 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12633249

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12633249/HIVE-6572.patch ERROR: -1 due to 2 failed/errored test(s), 5373 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_merge org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1654/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1654/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed This message is automatically generated. ATTACHMENT ID: 12633249
          Hide
          Sushanth Sowmyan added a comment -

          Ashutosh Chauhan, I'd appreciate your eyes on this for a review as well.

          Show
          Sushanth Sowmyan added a comment - Ashutosh Chauhan , I'd appreciate your eyes on this for a review as well.
          Hide
          Sushanth Sowmyan added a comment -
          Show
          Sushanth Sowmyan added a comment - Review board link : https://reviews.apache.org/r/18878/
          Hide
          Sushanth Sowmyan added a comment -

          Patch attached.

          Owen O'Malley, could you please check this patch real quick?

          Harish Butani, I would like this patch to also be included in the hive-0.13 release, since it fixes some erroneous config usage introduced in the 0.13 timeframe, and we don't want to have a release with them, which might make this behaviour something we have to support in the longer run.

          Show
          Sushanth Sowmyan added a comment - Patch attached. Owen O'Malley , could you please check this patch real quick? Harish Butani , I would like this patch to also be included in the hive-0.13 release, since it fixes some erroneous config usage introduced in the 0.13 timeframe, and we don't want to have a release with them, which might make this behaviour something we have to support in the longer run.

            People

            • Assignee:
              Sushanth Sowmyan
              Reporter:
              Sushanth Sowmyan
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development