Hadoop Common
  1. Hadoop Common
  2. HADOOP-7937

Forward port SequenceFile#syncFs and friends from Hadoop 1.x

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0, 0.23.1
    • Fix Version/s: 0.22.1
    • Component/s: io
    • Labels:
    • Hadoop Flags:
      Reviewed
    • Target Version/s:

      Description

      HDFS-200 added a new public API SequenceFile#syncFs, we need to forward port this for compatibility. Looks like it might have introduced other APIs that need forward porting as well (eg LocaltedBlocks#setFileLength, and DataNode#getBlockInfo).

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #951 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/951/)
          HADOOP-7937. Forward port SequenceFile#syncFs and friends from Hadoop 1.x.

          tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1228291
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SequenceFile.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #951 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/951/ ) HADOOP-7937 . Forward port SequenceFile#syncFs and friends from Hadoop 1.x. tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1228291 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SequenceFile.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Build #153 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/153/)
          Merge -r 1228290:1228291 from trunk to branch-0.23. Fixes: HADOOP-7937

          tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1228292
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SequenceFile.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #153 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/153/ ) Merge -r 1228290:1228291 from trunk to branch-0.23. Fixes: HADOOP-7937 tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1228292 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SequenceFile.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #918 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/918/)
          HADOOP-7937. Forward port SequenceFile#syncFs and friends from Hadoop 1.x.

          tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1228291
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SequenceFile.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #918 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/918/ ) HADOOP-7937 . Forward port SequenceFile#syncFs and friends from Hadoop 1.x. tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1228291 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SequenceFile.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #131 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/131/)
          Merge -r 1228290:1228291 from trunk to branch-0.23. Fixes: HADOOP-7937

          tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1228292
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SequenceFile.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #131 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/131/ ) Merge -r 1228290:1228291 from trunk to branch-0.23. Fixes: HADOOP-7937 tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1228292 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SequenceFile.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-22-branch #104 (See https://builds.apache.org/job/Hadoop-Common-22-branch/104/)
          Merge -r 1228290:1228291 from trunk to branch-0.22. Fixes: HADOOP-7937

          tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1228295
          Files :

          • /hadoop/common/branches/branch-0.22/common/CHANGES.txt
          • /hadoop/common/branches/branch-0.22/common/src/java/org/apache/hadoop/io/SequenceFile.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-22-branch #104 (See https://builds.apache.org/job/Hadoop-Common-22-branch/104/ ) Merge -r 1228290:1228291 from trunk to branch-0.22. Fixes: HADOOP-7937 tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1228295 Files : /hadoop/common/branches/branch-0.22/common/CHANGES.txt /hadoop/common/branches/branch-0.22/common/src/java/org/apache/hadoop/io/SequenceFile.java
          Tom White made changes -
          Link This issue relates to HDFS-2763 [ HDFS-2763 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1529 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1529/)
          HADOOP-7937. Forward port SequenceFile#syncFs and friends from Hadoop 1.x.

          tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1228291
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SequenceFile.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1529 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1529/ ) HADOOP-7937 . Forward port SequenceFile#syncFs and friends from Hadoop 1.x. tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1228291 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SequenceFile.java
          Hide
          Eli Collins added a comment -

          I see what you mean about reducing the visibility of the other methods in 1.x - really this should be done by applying the @Private annotations to the HDFS codebase (I think that the annotations are available in the 1 branch now at least).

          Cool, file a jira, link here and assign to me?

          Show
          Eli Collins added a comment - I see what you mean about reducing the visibility of the other methods in 1.x - really this should be done by applying the @Private annotations to the HDFS codebase (I think that the annotations are available in the 1 branch now at least). Cool, file a jira, link here and assign to me?
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Commit #361 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/361/)
          Merge -r 1228290:1228291 from trunk to branch-0.23. Fixes: HADOOP-7937

          tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1228292
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SequenceFile.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #361 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/361/ ) Merge -r 1228290:1228291 from trunk to branch-0.23. Fixes: HADOOP-7937 tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1228292 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SequenceFile.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-0.23-Commit #350 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/350/)
          Merge -r 1228290:1228291 from trunk to branch-0.23. Fixes: HADOOP-7937

          tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1228292
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SequenceFile.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #350 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/350/ ) Merge -r 1228290:1228291 from trunk to branch-0.23. Fixes: HADOOP-7937 tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1228292 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SequenceFile.java
          Tom White made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Target Version/s 0.23.1, 0.22.0 [ 12318884, 12314296 ] 0.22.0 [ 12314296 ]
          Fix Version/s 0.22.1 [ 12319240 ]
          Resolution Fixed [ 1 ]
          Hide
          Tom White added a comment -

          I just committed this to 0.22, 0.23 and trunk.

          I see what you mean about reducing the visibility of the other methods in 1.x - really this should be done by applying the @Private annotations to the HDFS codebase (I think that the annotations are available in the 1 branch now at least).

          Show
          Tom White added a comment - I just committed this to 0.22, 0.23 and trunk. I see what you mean about reducing the visibility of the other methods in 1.x - really this should be done by applying the @Private annotations to the HDFS codebase (I think that the annotations are available in the 1 branch now at least).
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1509 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1509/)
          HADOOP-7937. Forward port SequenceFile#syncFs and friends from Hadoop 1.x.

          tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1228291
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SequenceFile.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1509 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1509/ ) HADOOP-7937 . Forward port SequenceFile#syncFs and friends from Hadoop 1.x. tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1228291 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SequenceFile.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1582 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1582/)
          HADOOP-7937. Forward port SequenceFile#syncFs and friends from Hadoop 1.x.

          tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1228291
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SequenceFile.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1582 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1582/ ) HADOOP-7937 . Forward port SequenceFile#syncFs and friends from Hadoop 1.x. tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1228291 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SequenceFile.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Commit #340 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/340/)
          Merge -r 1228290:1228291 from trunk to branch-0.23. Fixes: HADOOP-7937

          tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1228292
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SequenceFile.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #340 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/340/ ) Merge -r 1228290:1228291 from trunk to branch-0.23. Fixes: HADOOP-7937 tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1228292 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SequenceFile.java
          Hide
          Eli Collins added a comment -

          Should DFSClient in trunk/23 use these methods, given that the append implementation is different to the one in 20? If so then that will need fixing in an HDFS JIRA.

          I don't think so, which is why I'm thinking we should reduce their visibility in branch-1 rather than forward port them to trunk. Make sense?

          Show
          Eli Collins added a comment - Should DFSClient in trunk/23 use these methods, given that the append implementation is different to the one in 20? If so then that will need fixing in an HDFS JIRA. I don't think so, which is why I'm thinking we should reduce their visibility in branch-1 rather than forward port them to trunk. Make sense?
          Hide
          Tom White added a comment -

          The javac warning is because sync() is deprecated.

          > Given that they're just used via DFSClient

          Should DFSClient in trunk/23 use these methods, given that the append implementation is different to the one in 20? If so then that will need fixing in an HDFS JIRA.

          I'll go ahead and commit this.

          Show
          Tom White added a comment - The javac warning is because sync() is deprecated. > Given that they're just used via DFSClient Should DFSClient in trunk/23 use these methods, given that the append implementation is different to the one in 20? If so then that will need fixing in an HDFS JIRA. I'll go ahead and commit this.
          Hide
          Eli Collins added a comment -

          +1 (assuming the javac warning is unrelated, should be)

          Looking at the other HDFS changes, agree that we probably don't need to forward port these. Given that they're just used via DFSClient, perhaps we should restrict the visibility in branch-1 to be more limited or javadoc that they won't be compatible?

          Show
          Eli Collins added a comment - +1 (assuming the javac warning is unrelated, should be) Looking at the other HDFS changes, agree that we probably don't need to forward port these. Given that they're just used via DFSClient, perhaps we should restrict the visibility in branch-1 to be more limited or javadoc that they won't be compatible?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12509348/HADOOP-7937.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 javadoc. The javadoc tool appears to have generated 5 warning messages.

          -1 javac. The applied patch generated 1044 javac compiler warnings (more than the trunk's current 1043 warnings).

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/494//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/494//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12509348/HADOOP-7937.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated 5 warning messages. -1 javac. The applied patch generated 1044 javac compiler warnings (more than the trunk's current 1043 warnings). +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/494//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/494//console This message is automatically generated.
          Tom White made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Target Version/s 0.23.1, 0.22.0 [ 12318884, 12314296 ] 0.22.0, 0.23.1 [ 12314296, 12318884 ]
          Assignee Tom White [ tomwhite ]
          Tom White made changes -
          Attachment HADOOP-7937.patch [ 12509348 ]
          Hide
          Tom White added a comment -

          Here's a patch that adds the missing method to SequenceFile.Writer. No test is included (there isn't one in 1.0.0).

          It's not clear that the other changes need to be ported, since the append implementation is different. If they do, they can be done in an HDFS JIRA.

          Show
          Tom White added a comment - Here's a patch that adds the missing method to SequenceFile.Writer. No test is included (there isn't one in 1.0.0). It's not clear that the other changes need to be ported, since the append implementation is different. If they do, they can be done in an HDFS JIRA.
          Bruno Mahé made changes -
          Field Original Value New Value
          Labels bigtop
          Eli Collins created issue -

            People

            • Assignee:
              Tom White
              Reporter:
              Eli Collins
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development