Hive
  1. Hive
  2. HIVE-4843

Refactoring MapRedTask and ExecDriver for better re-usability (for tez) and readability

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.12.0, tez-branch
    • Fix Version/s: 0.12.0
    • Component/s: None
    • Labels:
      None

      Description

      Currently, there are static apis in multiple locations in ExecDriver and MapRedTask that can be leveraged if put in the already existing utility class in the exec package. This would help making the code more maintainable, readable and also re-usable by other run-time infra such as tez.

      1. HIVE-4843.5.patch
        44 kB
        Vikram Dixit K
      2. HIVE-4843.4.patch
        42 kB
        Vikram Dixit K
      3. HIVE-4843.3.patch
        37 kB
        Vikram Dixit K
      4. HIVE-4843.2.patch
        29 kB
        Vikram Dixit K
      5. HIVE-4843.1.patch
        32 kB
        Vikram Dixit K

        Issue Links

          Activity

          Hide
          Gunther Hagleitner added a comment -

          can you create a review on rb or phabricator please?

          Show
          Gunther Hagleitner added a comment - can you create a review on rb or phabricator please?
          Show
          Vikram Dixit K added a comment - https://reviews.apache.org/r/12476/
          Hide
          Vikram Dixit K added a comment -

          New iteration of the changes. The first patch was a work in progress.

          Show
          Vikram Dixit K added a comment - New iteration of the changes. The first patch was a work in progress.
          Hide
          Vikram Dixit K added a comment -

          Latest patch based on trunk.

          Show
          Vikram Dixit K added a comment - Latest patch based on trunk.
          Hide
          Vikram Dixit K added a comment -

          Latest iteration after addressing comments.

          Show
          Vikram Dixit K added a comment - Latest iteration after addressing comments.
          Hide
          Edward Capriolo added a comment -
          List<Path> inputPaths = Utilities.getInputPaths(newJob, selectTask.getWork().getMapWork(), emptyScratchDir.toString(), ctx);
          

          Can we remove any Path/File toString() and just pass the Path if possible?

          Show
          Edward Capriolo added a comment - List<Path> inputPaths = Utilities.getInputPaths(newJob, selectTask.getWork().getMapWork(), emptyScratchDir.toString(), ctx); Can we remove any Path/File toString() and just pass the Path if possible?
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12594872/HIVE-4843.4.patch

          ERROR: -1 due to 18 failed/errored test(s), 2736 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_escape1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_8
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_12
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part14
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppr_pushdown
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_11
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_7
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_view
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_special_char
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_6
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_merge_2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_combine2_hadoop20
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_decode_name
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_4
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_partition_coltype
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_serde_user_properties
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_escape2
          

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/235/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/235/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests failed with: TestsFailedException: 18 tests failed
          

          This message is automatically generated.

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12594872/HIVE-4843.4.patch ERROR: -1 due to 18 failed/errored test(s), 2736 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_escape1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part14 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppr_pushdown org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_11 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_view org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_special_char org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_merge_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_combine2_hadoop20 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_decode_name org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_partition_coltype org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_serde_user_properties org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_escape2 Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/235/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/235/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests failed with: TestsFailedException: 18 tests failed This message is automatically generated.
          Hide
          Vikram Dixit K added a comment -

          Addressed Ed's comments.

          Show
          Vikram Dixit K added a comment - Addressed Ed's comments.
          Hide
          Vikram Dixit K added a comment -

          Hi Edward Capriolo

          Thanks for the feedback, took care of it. Strings are only used now when we update the pathToAlias etc datastructures, because those use strings themselves.

          Thanks
          Vikram.

          Show
          Vikram Dixit K added a comment - Hi Edward Capriolo Thanks for the feedback, took care of it. Strings are only used now when we update the pathToAlias etc datastructures, because those use strings themselves. Thanks Vikram.
          Hide
          Edward Capriolo added a comment -

          Thanks, its nice to have methods with java doc and comments

          Show
          Edward Capriolo added a comment - Thanks, its nice to have methods with java doc and comments
          Hide
          Hive QA added a comment -

          Overall: +1 all checks pass

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12595094/HIVE-4843.5.patch

          SUCCESS: +1 2748 tests passed

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/254/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/254/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          

          This message is automatically generated.

          Show
          Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12595094/HIVE-4843.5.patch SUCCESS: +1 2748 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/254/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/254/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated.
          Hide
          Gunther Hagleitner added a comment -

          Looks good to me: +1

          Planning to commit tomorrow.

          Show
          Gunther Hagleitner added a comment - Looks good to me: +1 Planning to commit tomorrow.
          Hide
          Gunther Hagleitner added a comment -

          Committed to trunk. Thanks Vikram!

          Show
          Gunther Hagleitner added a comment - Committed to trunk. Thanks Vikram!
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hive-trunk-hadoop2-ptest #39 (See https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/39/)
          HIVE-4843: Refactoring MapRedTask and ExecDriver for better re-usability and readability (Vikram Dixit K via Gunther Hagleitner) (gunther: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1509081)

          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Context.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java
          • /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/TestSymlinkTextInputFormat.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hive-trunk-hadoop2-ptest #39 (See https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/39/ ) HIVE-4843 : Refactoring MapRedTask and ExecDriver for better re-usability and readability (Vikram Dixit K via Gunther Hagleitner) (gunther: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1509081 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Context.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/TestSymlinkTextInputFormat.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hive-trunk-hadoop1-ptest #111 (See https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/111/)
          HIVE-4843: Refactoring MapRedTask and ExecDriver for better re-usability and readability (Vikram Dixit K via Gunther Hagleitner) (gunther: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1509081)

          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Context.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java
          • /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/TestSymlinkTextInputFormat.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hive-trunk-hadoop1-ptest #111 (See https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/111/ ) HIVE-4843 : Refactoring MapRedTask and ExecDriver for better re-usability and readability (Vikram Dixit K via Gunther Hagleitner) (gunther: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1509081 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Context.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/TestSymlinkTextInputFormat.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hive-trunk-h0.21 #2237 (See https://builds.apache.org/job/Hive-trunk-h0.21/2237/)
          HIVE-4843: Refactoring MapRedTask and ExecDriver for better re-usability and readability (Vikram Dixit K via Gunther Hagleitner) (gunther: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1509081)

          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Context.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java
          • /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/TestSymlinkTextInputFormat.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hive-trunk-h0.21 #2237 (See https://builds.apache.org/job/Hive-trunk-h0.21/2237/ ) HIVE-4843 : Refactoring MapRedTask and ExecDriver for better re-usability and readability (Vikram Dixit K via Gunther Hagleitner) (gunther: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1509081 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Context.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/TestSymlinkTextInputFormat.java
          Hide
          Hudson added a comment -

          ABORTED: Integrated in Hive-trunk-hadoop2 #320 (See https://builds.apache.org/job/Hive-trunk-hadoop2/320/)
          HIVE-4843: Refactoring MapRedTask and ExecDriver for better re-usability and readability (Vikram Dixit K via Gunther Hagleitner) (gunther: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1509081)

          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Context.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java
          • /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/TestSymlinkTextInputFormat.java
          Show
          Hudson added a comment - ABORTED: Integrated in Hive-trunk-hadoop2 #320 (See https://builds.apache.org/job/Hive-trunk-hadoop2/320/ ) HIVE-4843 : Refactoring MapRedTask and ExecDriver for better re-usability and readability (Vikram Dixit K via Gunther Hagleitner) (gunther: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1509081 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Context.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/TestSymlinkTextInputFormat.java
          Hide
          Ashutosh Chauhan added a comment -

          This issue has been fixed and released as part of 0.12 release. If you find further issues, please create a new jira and link it to this one.

          Show
          Ashutosh Chauhan added a comment - This issue has been fixed and released as part of 0.12 release. If you find further issues, please create a new jira and link it to this one.

            People

            • Assignee:
              Vikram Dixit K
              Reporter:
              Vikram Dixit K
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development