Details

    • Type: Bug Bug
    • Status: Reopened
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.11.1
    • Fix Version/s: 0.11.1
    • Component/s: None
    • Labels:
      None
    • Environment:

      hadoop 1.0.3, hbase 0.94.1

      Description

      I am trying to run a pig script in Java program, I get the following error sometimes but not all the time. Here is the snippet of the program and the exception I've got. I have "/user/root" directory created in hdfs.

      ----------------------

      URL path = getClass().getClassLoader().getResource("cfg/concatall.py");

      LOG.info("CDNResolve2Hbase: reading concatall.py file from " + path.toString());
      pigServer.getPigContext().getProperties().setProperty(PigContext.JOB_NAME,
      "CDNResolve2Hbase");
      pigServer.registerQuery("A = load '" + inputPath + "' using PigStorage('\t') as (ip:chararray, do:chararray, cn:chararray, cdn:chararray, firsttime:chararray, updatetime:chararray);");
      pigServer.registerCode(path.toString(),"jython","myfunc");
      pigServer.registerQuery("B = foreach A generate myfunc.concatall('"+ extractTimestamp(inputPath)+"',ip,do,cn), cdn, SUBSTRING(firsttime,0,8);");
      outputTable = "hbase://" + outputTable;
      ExecJob job = pigServer.store("B",outputTable,"org.apache.pig.backend.hadoop.hbase.HBaseStorage('d:cdn d:dtime')");

      ---------------------
      org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000: Error during parsing. Unable to check name hdfs://DC-001:9000/user/root
      at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1607)
      at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1546)
      at org.apache.pig.PigServer.registerQuery(PigServer.java:516)
      at org.apache.pig.PigServer.registerQuery(PigServer.java:529)
      at com.hugedata.cdnserver.datanalysis.CDNResolve2Hbase.execute(Unknown Source)
      at com.hugedata.cdnserver.DatAnalysis.cdnResolve2Hbase(Unknown Source)
      at com.hugedata.cdnserver.task.HandleDomainNameLogTask.execute(Unknown Source)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:273)
      at org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean$MethodInvokingJob.executeInternal(MethodInvokingJobDetailFactoryBean.java:264)
      at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:86)
      at org.quartz.core.JobRunShell.run(JobRunShell.java:203)
      at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520)
      Caused by: Failed to parse: Pig script failed to parse:
      <line 6, column 4> pig script failed to validate: org.apache.pig.backend.datastorage.DataStorageException: ERROR 6007: Unable to check name hdfs://DC-001:9000/user/root
      at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:191)
      at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1599)
      ... 15 more
      Caused by:
      <line 6, column 4> pig script failed to validate: org.apache.pig.backend.datastorage.DataStorageException: ERROR 6007: Unable to check name hdfs://DC-001:9000/user/root
      at org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:835)
      at org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3236)
      at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1315)
      at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:799)
      at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:517)
      at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:392)
      at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:184)
      ... 16 more
      Caused by: org.apache.pig.backend.datastorage.DataStorageException: ERROR 6007: Unable to check name hdfs://DC-001:9000/user/root
      at org.apache.pig.backend.hadoop.datastorage.HDataStorage.isContainer(HDataStorage.java:207)
      at org.apache.pig.backend.hadoop.datastorage.HDataStorage.asElement(HDataStorage.java:128)
      at org.apache.pig.backend.hadoop.datastorage.HDataStorage.asElement(HDataStorage.java:138)
      at org.apache.pig.parser.QueryParserUtils.getCurrentDir(QueryParserUtils.java:91)
      at org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:827)
      ... 22 more
      Caused by: java.io.IOException: Filesystem closed
      at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:264)
      at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:873)
      at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:513)
      at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:768)
      at org.apache.pig.backend.hadoop.datastorage.HDataStorage.isContainer(HDataStorage.java:200)
      ... 26 more

        Activity

        Hide
        zhenyingshan added a comment -

        It turns out that because my static PigServer is not thread-safe, I use ThreadLocal and the problem does not appear anymore.

        ----------------------
        private static ThreadLocal<PigServer> pigServer = new ThreadLocal<PigServer>();

        public static PigServer getServer() {
        if (pigServer.get() == null) {
        try

        { printClassPath(); Properties prop = SystemUtils.getCfg(); pigServer.set(new PigServer(ExecType.MAPREDUCE, prop)); return pigServer.get(); }

        catch (Exception e)

        { LOG.error("error in starting PigServer:", e); return null; }

        }
        return pigServer.get();
        }

        Show
        zhenyingshan added a comment - It turns out that because my static PigServer is not thread-safe, I use ThreadLocal and the problem does not appear anymore. ---------------------- private static ThreadLocal<PigServer> pigServer = new ThreadLocal<PigServer>(); public static PigServer getServer() { if (pigServer.get() == null) { try { printClassPath(); Properties prop = SystemUtils.getCfg(); pigServer.set(new PigServer(ExecType.MAPREDUCE, prop)); return pigServer.get(); } catch (Exception e) { LOG.error("error in starting PigServer:", e); return null; } } return pigServer.get(); }
        Hide
        zhenyingshan added a comment -

        It turns out that the previous fix still does not work, the problem appears again, please help

        Show
        zhenyingshan added a comment - It turns out that the previous fix still does not work, the problem appears again, please help

          People

          • Assignee:
            Unassigned
            Reporter:
            zhenyingshan
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:

              Development