Hadoop Common
  1. Hadoop Common
  2. HADOOP-7817

RawLocalFileSystem.append() should give FSDataOutputStream with accurate .getPos()

    Details

    • Type: Bug Bug
    • Status: Patch Available
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: 1.0.0
    • Fix Version/s: None
    • Component/s: fs
    • Labels:
      None

      Description

      When RawLocalFileSyste.append() is called it returns an FSDataOutputStream whose .getPos() returns 0.
      getPos() should return position in the file where appends will start writing.

      1. HADOOP-7817.patch
        4 kB
        Kristofer Tomasette

        Activity

        Hide
        Kristofer Tomasette added a comment -

        Is anyone looking at this? I have another patch that depends on this and would like to get some feedback.

        Thanks!

        Show
        Kristofer Tomasette added a comment - Is anyone looking at this? I have another patch that depends on this and would like to get some feedback. Thanks!
        Hide
        Kristofer Tomasette added a comment -

        I think the -1 javadoc mentioned above is not related to my patch.

        Show
        Kristofer Tomasette added a comment - I think the -1 javadoc mentioned above is not related to my patch.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12506157/HADOOP-7817.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 4 new or modified tests.

        -1 javadoc. The javadoc tool appears to have generated 9 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/437//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/437//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12506157/HADOOP-7817.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 9 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/437//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/437//console This message is automatically generated.
        Hide
        Kristofer Tomasette added a comment -

        FWIW:
        DFSClient.append does:

        return new FSDataOutputStream(out, statistics, out.getInitialLen());

        Was planning on submitting a patch on RawLocalFileSystem w/

        public FSDataOutputStream append(Path f, int bufferSize,
        Progressable progress) throws IOException {
        if (!exists(f))

        { throw new FileNotFoundException("File " + f + " not found"); }

        FileStatus status = getFileStatus(f);
        if (status.isDirectory())

        { throw new IOException("Cannot append to a diretory (=" + f + " )"); }

        return new FSDataOutputStream(new BufferedOutputStream(
        new LocalFSFileOutputStream(f, true), bufferSize), statistics,status.getLen());
        }

        Show
        Kristofer Tomasette added a comment - FWIW: DFSClient.append does: return new FSDataOutputStream(out, statistics, out.getInitialLen()); Was planning on submitting a patch on RawLocalFileSystem w/ public FSDataOutputStream append(Path f, int bufferSize, Progressable progress) throws IOException { if (!exists(f)) { throw new FileNotFoundException("File " + f + " not found"); } FileStatus status = getFileStatus(f); if (status.isDirectory()) { throw new IOException("Cannot append to a diretory (=" + f + " )"); } return new FSDataOutputStream(new BufferedOutputStream( new LocalFSFileOutputStream(f, true), bufferSize), statistics,status.getLen()); }
        Hide
        Uma Maheswara Rao G added a comment -

        Yes, the position will come as zero, because it directly opens the FileOutPutstream in append mode. This will ensures that the data to go at the end of the file.
        fixing this would be simple, we can set the position on fileChannel position to the length of the file. But i am not sure this is really a bug or not, because the author intention is to leave as it is equal behavior with FileOutPutStream when opening in append mode....?

        Show
        Uma Maheswara Rao G added a comment - Yes, the position will come as zero, because it directly opens the FileOutPutstream in append mode. This will ensures that the data to go at the end of the file. fixing this would be simple, we can set the position on fileChannel position to the length of the file. But i am not sure this is really a bug or not, because the author intention is to leave as it is equal behavior with FileOutPutStream when opening in append mode....?
        Hide
        Kristofer Tomasette added a comment -

        patch forthcoming

        Show
        Kristofer Tomasette added a comment - patch forthcoming

          People

          • Assignee:
            Unassigned
            Reporter:
            Kristofer Tomasette
          • Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:

              Time Tracking

              Estimated:
              Original Estimate - 1h
              1h
              Remaining:
              Remaining Estimate - 1h
              1h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development