Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.9.0
    • Component/s: None
    • Labels:
      None

      Description

      In our load statement we are allowed to prefix the table name with "hbase://" but when we call
      store it throws an exception unless we remove hbase:// from the table
      name:

      this works:
      store raw into 'piggytest2' USING
      org.apache.pig.backend.hadoop.hbase.HBaseStorage('content2:field1
      anchor2:field1a anchor2:field2a');

      this won't
      store raw into 'hbase://piggytest2'

      Exception:
      Caused by: java.lang.IllegalArgumentException:
      java.net.URISyntaxException: Relative path in absolute URI:
      hbase://piggytest2_logs

      Would be nice to be able to prefix the store with hbase:// so it's consistent with the load syntax

      1. PIG_1769.patch
        4 kB
        Dmitriy V. Ryaboy

        Activity

        Hide
        Daniel Dai added a comment -

        Definitely a new issue. A closed ticket cannot be reopened.

        Show
        Daniel Dai added a comment - Definitely a new issue. A closed ticket cannot be reopened.
        Hide
        Nicolas ROUYER added a comment -

        This issue was corrected for habase storage, but still the same problem remains for Swift object storage :

        In a PIG script I've tried to store data into openstack swift object storage. The following error occurs :
        ERROR 2017: Internal error creating job configuration.
        (...)
        Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: swift://hdfs2swift.bigswift_logs

        map reduce store output data into Swift works well, but within pig script it fails.

        My PIG script last line :
        STORE sum INTO 'swift://hdfs2swift.bigswift/';

        Any known workaround for the issue ?
        Should we open a new issue ?

        Full PIG stack trace :
        Pig Stack Trace
        ---------------
        ERROR 2017: Internal error creating job configuration.

        org.apache.pig.backend.hadoop.executionengine.JobCreationException: ERROR 2017: Internal error creating job configur
        ation.
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler
        .java:998)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompile
        r.java:323)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLaunche
        r.java:196)
        at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:280)
        at org.apache.pig.PigServer.launchPlan(PigServer.java:1390)
        at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1375)
        at org.apache.pig.PigServer.execute(PigServer.java:1364)
        at org.apache.pig.PigServer.executeBatch(PigServer.java:415)
        at org.apache.pig.PigServer.executeBatch(PigServer.java:398)
        at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:171)
        at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:234)
        at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205)
        at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
        at org.apache.pig.Main.run(Main.java:495)
        at org.apache.pig.Main.main(Main.java:170)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
        Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: swift://h
        dfs2swift.bigswift_logs
        at org.apache.hadoop.fs.Path.initialize(Path.java:206)
        at org.apache.hadoop.fs.Path.<init>(Path.java:116)
        at org.apache.hadoop.fs.Path.<init>(Path.java:89)
        at org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil.setupStreamingDirsConfSingle(MapRedUtil.jav
        a:193)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler
        .java:777)
        ... 20 more
        Caused by: java.net.URISyntaxException: Relative path in absolute URI: swift://hdfs2swift.bigswift_logs
        at java.net.URI.checkPath(URI.java:1804)
        at java.net.URI.<init>(URI.java:752)
        at org.apache.hadoop.fs.Path.initialize(Path.java:203)
        ... 24 more

        Show
        Nicolas ROUYER added a comment - This issue was corrected for habase storage, but still the same problem remains for Swift object storage : In a PIG script I've tried to store data into openstack swift object storage. The following error occurs : ERROR 2017: Internal error creating job configuration. (...) Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: swift://hdfs2swift.bigswift_logs map reduce store output data into Swift works well, but within pig script it fails. My PIG script last line : STORE sum INTO 'swift://hdfs2swift.bigswift/'; Any known workaround for the issue ? Should we open a new issue ? Full PIG stack trace : Pig Stack Trace --------------- ERROR 2017: Internal error creating job configuration. org.apache.pig.backend.hadoop.executionengine.JobCreationException: ERROR 2017: Internal error creating job configur ation. at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler .java:998) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompile r.java:323) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLaunche r.java:196) at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:280) at org.apache.pig.PigServer.launchPlan(PigServer.java:1390) at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1375) at org.apache.pig.PigServer.execute(PigServer.java:1364) at org.apache.pig.PigServer.executeBatch(PigServer.java:415) at org.apache.pig.PigServer.executeBatch(PigServer.java:398) at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:171) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:234) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205) at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81) at org.apache.pig.Main.run(Main.java:495) at org.apache.pig.Main.main(Main.java:170) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: swift://h dfs2swift.bigswift_logs at org.apache.hadoop.fs.Path.initialize(Path.java:206) at org.apache.hadoop.fs.Path.<init>(Path.java:116) at org.apache.hadoop.fs.Path.<init>(Path.java:89) at org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil.setupStreamingDirsConfSingle(MapRedUtil.jav a:193) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler .java:777) ... 20 more Caused by: java.net.URISyntaxException: Relative path in absolute URI: swift://hdfs2swift.bigswift_logs at java.net.URI.checkPath(URI.java:1804) at java.net.URI.<init>(URI.java:752) at org.apache.hadoop.fs.Path.initialize(Path.java:203) ... 24 more
        Hide
        Dmitriy V. Ryaboy added a comment -

        committed.

        Show
        Dmitriy V. Ryaboy added a comment - committed.
        Hide
        Alan Gates added a comment -

        [exec] +1 overall.
        [exec]
        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] +1 tests included. The patch appears to include 3 new or modified tests.
        [exec]
        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
        [exec]
        [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
        [exec]
        [exec]
        [exec]

        Show
        Alan Gates added a comment - [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] [exec]
        Hide
        Alan Gates added a comment -

        +1, changes look good. I'll run the test-patch and unit tests on it.

        Show
        Alan Gates added a comment - +1, changes look good. I'll run the test-patch and unit tests on it.
        Hide
        Dmitriy V. Ryaboy added a comment -

        Submitting as a fix fo 0.9 although this can easily be applied to 0.8 as well.

        Show
        Dmitriy V. Ryaboy added a comment - Submitting as a fix fo 0.9 although this can easily be applied to 0.8 as well.
        Hide
        Dmitriy V. Ryaboy added a comment -

        The attached patch fixes the issue – the problem was that we try to set the location of the _logs directory for Stores regardless of whether we are writing into HDFS or not. Instead I check for path starting with hdfs:// or not containing "://" at all, and if it does, set the _logs directory to a temp location.

        It's a bit hacky... if someone else has a better idea, please jump in.

        Show
        Dmitriy V. Ryaboy added a comment - The attached patch fixes the issue – the problem was that we try to set the location of the _logs directory for Stores regardless of whether we are writing into HDFS or not. Instead I check for path starting with hdfs:// or not containing "://" at all, and if it does, set the _logs directory to a temp location. It's a bit hacky... if someone else has a better idea, please jump in.
        Hide
        Dmitriy V. Ryaboy added a comment -

        Hm. setStoreLocation does the right thing but I probably need to do something in relativeToAbsolutePathForStoreLocation.

        Complete stack trace:

        ================================================================================
        Pig Stack Trace
        ---------------
        ERROR 2017: Internal error creating job configuration.

        org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable to store alias 1
        at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1569)
        at org.apache.pig.PigServer.registerQuery(PigServer.java:523)
        at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:868)
        at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:388)
        at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
        at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141)
        at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:76)
        at org.apache.pig.Main.run(Main.java:465)
        at org.apache.pig.Main.main(Main.java:107)
        Caused by: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobCreationException: ERROR 2017: Internal error creating job configuration.
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:667)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:256)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:147)
        at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:378)
        at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1198)
        at org.apache.pig.PigServer.execute(PigServer.java:1190)
        at org.apache.pig.PigServer.access$100(PigServer.java:128)
        at org.apache.pig.PigServer$Graph.execute(PigServer.java:1517)
        at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1564)
        ... 8 more
        Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: hbase://foo_logs
        at org.apache.hadoop.fs.Path.initialize(Path.java:140)
        at org.apache.hadoop.fs.Path.<init>(Path.java:71)
        at org.apache.hadoop.fs.Path.<init>(Path.java:45)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:470)
        ... 16 more
        Caused by: java.net.URISyntaxException: Relative path in absolute URI: hbase://foo_logs
        at java.net.URI.checkPath(URI.java:1787)
        at java.net.URI.<init>(URI.java:735)
        at org.apache.hadoop.fs.Path.initialize(Path.java:137)
        ... 19 more

        Show
        Dmitriy V. Ryaboy added a comment - Hm. setStoreLocation does the right thing but I probably need to do something in relativeToAbsolutePathForStoreLocation. Complete stack trace: ================================================================================ Pig Stack Trace --------------- ERROR 2017: Internal error creating job configuration. org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable to store alias 1 at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1569) at org.apache.pig.PigServer.registerQuery(PigServer.java:523) at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:868) at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:388) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141) at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:76) at org.apache.pig.Main.run(Main.java:465) at org.apache.pig.Main.main(Main.java:107) Caused by: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobCreationException: ERROR 2017: Internal error creating job configuration. at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:667) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:256) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:147) at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:378) at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1198) at org.apache.pig.PigServer.execute(PigServer.java:1190) at org.apache.pig.PigServer.access$100(PigServer.java:128) at org.apache.pig.PigServer$Graph.execute(PigServer.java:1517) at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1564) ... 8 more Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: hbase://foo_logs at org.apache.hadoop.fs.Path.initialize(Path.java:140) at org.apache.hadoop.fs.Path.<init>(Path.java:71) at org.apache.hadoop.fs.Path.<init>(Path.java:45) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:470) ... 16 more Caused by: java.net.URISyntaxException: Relative path in absolute URI: hbase://foo_logs at java.net.URI.checkPath(URI.java:1787) at java.net.URI.<init>(URI.java:735) at org.apache.hadoop.fs.Path.initialize(Path.java:137) ... 19 more

          People

          • Assignee:
            Dmitriy V. Ryaboy
            Reporter:
            Corbin Hoenes
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development