Hive
  1. Hive
  2. HIVE-2413

BlockMergeTask ignores client-specified jars

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: Query Processor
    • Labels:
      None

      Description

      User-specified jars are not added to the hadoop tasks while executing a BlockMergeTask resulting in a ClassNotFoundException.

      1. HIVE-2413.v0.patch
        2 kB
        Krishna Kumar
      2. HIVE-2413.v1.patch
        2 kB
        Krishna Kumar

        Activity

        Krishna Kumar created issue -
        Hide
        Krishna Kumar added a comment -

        Here is a simple fix which works for me, though there may be better ways of handling this (refactor mapredtask & blockmerge task to use a common routine / blockmerge as an mr operator).

        No automated tests added.

        Show
        Krishna Kumar added a comment - Here is a simple fix which works for me, though there may be better ways of handling this (refactor mapredtask & blockmerge task to use a common routine / blockmerge as an mr operator). No automated tests added.
        Krishna Kumar made changes -
        Field Original Value New Value
        Attachment HIVE-2413.v0.patch [ 12491739 ]
        Krishna Kumar made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Assignee Krishna Kumar [ n_krishna_kumar ]
        Hide
        He Yongqiang added a comment -

        +1, will commit after tests pass.

        Show
        He Yongqiang added a comment - +1, will commit after tests pass.
        Hide
        He Yongqiang added a comment -

        [junit] java.lang.IllegalArgumentException: Can not create a Path from an empty string
        [junit] at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
        [junit] at org.apache.hadoop.fs.Path.<init>(Path.java:90)
        [junit] at org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:602)
        [junit] at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)
        [junit] at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
        [junit] at org.apache.hadoop.hive.ql.io.rcfile.merge.BlockMergeTask.execute(BlockMergeTask.java:203)
        [junit] at org.apache.hadoop.hive.ql.exec.DDLTask.mergeFiles(DDLTask.java:410)
        [junit] at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:366)
        [junit] at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:132)
        [junit] at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
        [junit] at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1343)
        [junit] at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1134)
        [junit] at org.apache.hadoop.hive.ql.Driver.run(Driver.java:943)
        [junit] at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:253)
        [junit] at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:210)
        [junit] at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:401)
        [junit] at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:336)
        [junit] at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:638)
        [junit] at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_concatenate_indexed_table(TestCliDriver.java:1190)
        [junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        [junit] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        [junit] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

        I got these error with a bunch of testcases. Here are some of them: rcfile_merge3.q, load_fs.q, alter_merge.q etc

        can u take a look?

        Show
        He Yongqiang added a comment - [junit] java.lang.IllegalArgumentException: Can not create a Path from an empty string [junit] at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82) [junit] at org.apache.hadoop.fs.Path.<init>(Path.java:90) [junit] at org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:602) [junit] at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761) [junit] at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730) [junit] at org.apache.hadoop.hive.ql.io.rcfile.merge.BlockMergeTask.execute(BlockMergeTask.java:203) [junit] at org.apache.hadoop.hive.ql.exec.DDLTask.mergeFiles(DDLTask.java:410) [junit] at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:366) [junit] at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:132) [junit] at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57) [junit] at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1343) [junit] at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1134) [junit] at org.apache.hadoop.hive.ql.Driver.run(Driver.java:943) [junit] at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:253) [junit] at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:210) [junit] at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:401) [junit] at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:336) [junit] at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:638) [junit] at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_concatenate_indexed_table(TestCliDriver.java:1190) [junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [junit] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [junit] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) I got these error with a bunch of testcases. Here are some of them: rcfile_merge3.q, load_fs.q, alter_merge.q etc can u take a look?
        Hide
        Krishna Kumar added a comment -

        Empty string not handled correctly in JC so handling it here...

        Show
        Krishna Kumar added a comment - Empty string not handled correctly in JC so handling it here...
        Krishna Kumar made changes -
        Attachment HIVE-2413.v1.patch [ 12492613 ]
        Hide
        He Yongqiang added a comment -

        committed, thanks Krishna Kumar!

        Show
        He Yongqiang added a comment - committed, thanks Krishna Kumar!
        He Yongqiang made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-h0.21 #930 (See https://builds.apache.org/job/Hive-trunk-h0.21/930/)
        HIVE-2413: BlockMergeTask ignores client-specified jars (Krishna Kumar via He Yongqiang)

        heyongqiang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1164656
        Files :

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java
        Show
        Hudson added a comment - Integrated in Hive-trunk-h0.21 #930 (See https://builds.apache.org/job/Hive-trunk-h0.21/930/ ) HIVE-2413 : BlockMergeTask ignores client-specified jars (Krishna Kumar via He Yongqiang) heyongqiang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1164656 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java
        Carl Steinbach made changes -
        Fix Version/s 0.9.0 [ 12317742 ]
        Carl Steinbach made changes -
        Fix Version/s 0.8.0 [ 12316178 ]
        Carl Steinbach made changes -
        Fix Version/s 0.9.0 [ 12317742 ]
        Carl Steinbach made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Krishna Kumar
            Reporter:
            Krishna Kumar
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development