Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-4003

Error is thrown by JobStats.getOutputSize() when storing to a Hive table

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.14.0
    • Component/s: None
    • Labels:
      None

      Description

      Here is an example of stack trace printed to console output. Technically, this is a warning message and does not make the job fail. However, this is certainly not user-friendly.

      4/06/09 16:20:28 WARN pigstats.JobStats: unable to find the output file
      java.io.FileNotFoundException: File hdfs://10.61.10.185:9000/user/cheolsoop/prodhive.benchmark.unittest_vhs_bitrate_asn_sum_stg_test2 does not exist.
      	at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:654)
      	at org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:102)
      	at org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:712)
      	at org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:708)
      	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
      	at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:708)
      	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.FileBasedOutputSizeReader.getOutputSize(FileBasedOutputSizeReader.java:65)
      	at org.apache.pig.tools.pigstats.JobStats.getOutputSize(JobStats.java:352)
      

      The issue is that FileBasedOutputSizeReader mis-interprets hive table name as hdfs path.

      @Override
      public boolean supports(POStore sto, Configuration conf) {
          return UriUtil.isHDFSFileOrLocalOrS3N(getLocationUri(sto), conf);
      }
      

        Attachments

        1. PIG-4003-1.patch
          3 kB
          Cheolsoo Park
        2. PIG-4003-2.patch
          1 kB
          Cheolsoo Park
        3. PIG-4003-3.patch
          3 kB
          Cheolsoo Park
        4. PIG-4003-4.patch
          4 kB
          Cheolsoo Park
        5. PIG-4003-5.patch
          5 kB
          Cheolsoo Park

          Activity

            People

            • Assignee:
              cheolsoo Cheolsoo Park
              Reporter:
              cheolsoo Cheolsoo Park
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: