Hive
  1. Hive
  2. HIVE-5666

use Path instead of String for IOContext.inputPath

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.13.0
    • Component/s: None
    • Labels:
      None

      Description

      Path is converted to string in HiveContextAwareRecordReader to be stored in IOContext.inputPath, then in MapOperator normalizePath gets called on it which converts it back to Path.
      Path creation is expensive, so it is better to use Path instead of string through the call stack.

      This is also a step towards HIVE-3616.

      1. HIVE-5666.1.patch
        10 kB
        Thejas M Nair

        Activity

        Hide
        Hive QA added a comment -

        Overall: +1 all checks pass

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12610514/HIVE-5666.1.patch

        SUCCESS: +1 4502 tests passed

        Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1267/testReport
        Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1267/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        

        This message is automatically generated.

        Show
        Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12610514/HIVE-5666.1.patch SUCCESS: +1 4502 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1267/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1267/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated.
        Hide
        Ashutosh Chauhan added a comment -

        +1

        Show
        Ashutosh Chauhan added a comment - +1
        Hide
        Ashutosh Chauhan added a comment -

        Committed to trunk. Thanks, Thejas!

        Show
        Ashutosh Chauhan added a comment - Committed to trunk. Thanks, Thejas!
        Hide
        Hudson added a comment -

        ABORTED: Integrated in Hive-trunk-hadoop2 #526 (See https://builds.apache.org/job/Hive-trunk-hadoop2/526/)
        HIVE-5666 : use Path instead of String for IOContext.inputPath (Thejas Nair via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1536478)

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecMapperContext.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveContextAwareRecordReader.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/IOContext.java
        Show
        Hudson added a comment - ABORTED: Integrated in Hive-trunk-hadoop2 #526 (See https://builds.apache.org/job/Hive-trunk-hadoop2/526/ ) HIVE-5666 : use Path instead of String for IOContext.inputPath (Thejas Nair via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1536478 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecMapperContext.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveContextAwareRecordReader.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/IOContext.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hive-trunk-hadoop1-ptest #217 (See https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/217/)
        HIVE-5666 : use Path instead of String for IOContext.inputPath (Thejas Nair via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1536478)

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecMapperContext.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveContextAwareRecordReader.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/IOContext.java
        Show
        Hudson added a comment - SUCCESS: Integrated in Hive-trunk-hadoop1-ptest #217 (See https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/217/ ) HIVE-5666 : use Path instead of String for IOContext.inputPath (Thejas Nair via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1536478 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecMapperContext.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveContextAwareRecordReader.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/IOContext.java
        Hide
        Hudson added a comment -

        ABORTED: Integrated in Hive-trunk-h0.21 #2427 (See https://builds.apache.org/job/Hive-trunk-h0.21/2427/)
        HIVE-5666 : use Path instead of String for IOContext.inputPath (Thejas Nair via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1536478)

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecMapperContext.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveContextAwareRecordReader.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/IOContext.java
        Show
        Hudson added a comment - ABORTED: Integrated in Hive-trunk-h0.21 #2427 (See https://builds.apache.org/job/Hive-trunk-h0.21/2427/ ) HIVE-5666 : use Path instead of String for IOContext.inputPath (Thejas Nair via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1536478 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecMapperContext.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveContextAwareRecordReader.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/IOContext.java

          People

          • Assignee:
            Thejas M Nair
            Reporter:
            Thejas M Nair
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development