[PIG-3796] PigStats output bytes written not collected for relative paths

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.13.0
• Assignee: Rohini Palaniswamy
• Reporter: Rohini Palaniswamy
• Created: 04/Mar/14 22:09
• Updated: 07/Jul/14 18:07
• Resolved: 30/Apr/14 14:01

Description

PIG-2924 added support for custom stats reader. But the FileBasedOutputSizeReader only checks for

public static boolean isHDFSFileOrLocalOrS3N(String uri){
        if(uri == null)
            return false;
        if(uri.startsWith("/") || uri.matches("[A-Za-z]:.*") || uri.startsWith("hdfs:")
                || uri.startsWith("viewfs:") || uri.startsWith("file:") || uri.startsWith("s3n:")) {
            return true;
        }
        return false;
    }

Better to change this to UriUtil.hasFileSystemImpl which will automatically filter out hbase://. This would still not solve cases like HCatStorer which does not have a scheme. Will also write a default stats reader that checks for known StoreFuncInterface implementations that are not file based like HCatStorer. More standard ones can be added later. AccumuloStorage should not be a problem as it has scheme accumulo://.

Issue Links

• is part of: PIG-3672 Pig should not check for hardcoded file system implementations