Pig
  1. Pig
  2. PIG-4003

Error is thrown by JobStats.getOutputSize() when storing to a Hive table

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.14.0
    • Component/s: None
    • Labels:
      None

      Description

      Here is an example of stack trace printed to console output. Technically, this is a warning message and does not make the job fail. However, this is certainly not user-friendly.

      4/06/09 16:20:28 WARN pigstats.JobStats: unable to find the output file
      java.io.FileNotFoundException: File hdfs://10.61.10.185:9000/user/cheolsoop/prodhive.benchmark.unittest_vhs_bitrate_asn_sum_stg_test2 does not exist.
      	at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:654)
      	at org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:102)
      	at org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:712)
      	at org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:708)
      	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
      	at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:708)
      	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.FileBasedOutputSizeReader.getOutputSize(FileBasedOutputSizeReader.java:65)
      	at org.apache.pig.tools.pigstats.JobStats.getOutputSize(JobStats.java:352)
      

      The issue is that FileBasedOutputSizeReader mis-interprets hive table name as hdfs path.

      @Override
      public boolean supports(POStore sto, Configuration conf) {
          return UriUtil.isHDFSFileOrLocalOrS3N(getLocationUri(sto), conf);
      }
      
      1. PIG-4003-5.patch
        5 kB
        Cheolsoo Park
      2. PIG-4003-4.patch
        4 kB
        Cheolsoo Park
      3. PIG-4003-3.patch
        3 kB
        Cheolsoo Park
      4. PIG-4003-2.patch
        1 kB
        Cheolsoo Park
      5. PIG-4003-1.patch
        3 kB
        Cheolsoo Park

        Activity

        Cheolsoo Park created issue -
        Cheolsoo Park made changes -
        Field Original Value New Value
        Attachment PIG-4003-1.patch [ 12649467 ]
        Cheolsoo Park made changes -
        Attachment PIG-4003-2.patch [ 12649597 ]
        Cheolsoo Park made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Cheolsoo Park made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Cheolsoo Park made changes -
        Attachment PIG-4003-3.patch [ 12652260 ]
        Cheolsoo Park made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Cheolsoo Park made changes -
        Attachment PIG-4003-4.patch [ 12652268 ]
        Cheolsoo Park made changes -
        Attachment PIG-4003-5.patch [ 12652285 ]
        Cheolsoo Park made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Daniel Dai made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Cheolsoo Park
            Reporter:
            Cheolsoo Park
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development