Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0-alpha
    • Component/s: None
    • Labels:
      None
    1. hdfs.proto
      5 kB
      Sanjay Radia
    2. hdfs.proto
      4 kB
      Sanjay Radia
    3. hdfs.proto
      4 kB
      Sanjay Radia
    4. pbClientTypes1.patch
      216 kB
      Sanjay Radia
    5. pbClientTypes2.patch
      236 kB
      Sanjay Radia
    6. pbClientTypes3.patch
      261 kB
      Sanjay Radia
    7. pbClientTypes4.patch
      294 kB
      Sanjay Radia
    8. pbClientTypes5.patch
      304 kB
      Sanjay Radia

      Activity

      Hide
      Sanjay Radia added a comment -

      Early patch of pb types. Note:

      • I have made some of the fields of HdfsFileStatus to be optional since they only apply to a file. Thoughts?
      • Rather than define a new HdfsLocatedFileStatus (as done with writables), I made locations optional. Thoughts?
      Show
      Sanjay Radia added a comment - Early patch of pb types. Note: I have made some of the fields of HdfsFileStatus to be optional since they only apply to a file. Thoughts? Rather than define a new HdfsLocatedFileStatus (as done with writables), I made locations optional. Thoughts?
      Hide
      Sanjay Radia added a comment -

      For convenience here is the .proto file (note this is not a new file, i have simply added some new types).

      Show
      Sanjay Radia added a comment - For convenience here is the .proto file (note this is not a new file, i have simply added some new types).
      Hide
      Sanjay Radia added a comment -

      Another option is to create a new type for HdfsLocatedFileStatus as

      HdfsLocatedFileStatusProto {
        HdfsFileStatusProto fs = 1;
        LocatedBlocksProto locations = 2;
      }
      
      Show
      Sanjay Radia added a comment - Another option is to create a new type for HdfsLocatedFileStatus as HdfsLocatedFileStatusProto { HdfsFileStatusProto fs = 1; LocatedBlocksProto locations = 2; }
      Hide
      Hadoop QA added a comment -

      -1 overall. Here are the results of testing the latest attachment
      http://issues.apache.org/jira/secure/attachment/12500782/pbClientTypes4.patch
      against trunk revision .

      +1 @author. The patch does not contain any @author tags.

      -1 tests included. The patch doesn't appear to include any new or modified tests.
      Please justify why no new tests are needed for this patch.
      Also please list what manual steps were performed to verify this patch.

      +1 javadoc. The javadoc tool did not generate any warning messages.

      +1 javac. The applied patch does not increase the total number of javac compiler warnings.

      +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

      +1 release audit. The applied patch does not increase the total number of release audit warnings.

      -1 core tests. The patch failed these unit tests:
      org.apache.hadoop.hdfs.TestFSInputChecker
      org.apache.hadoop.tools.TestJMXGet
      org.apache.hadoop.hdfs.TestHDFSTrash
      org.apache.hadoop.hdfs.TestSmallBlock
      org.apache.hadoop.hdfs.TestListPathServlet
      org.apache.hadoop.cli.TestHDFSCLI
      org.apache.hadoop.security.TestRefreshUserMappings
      org.apache.hadoop.hdfs.TestFileCreationClient
      org.apache.hadoop.security.TestPermission
      org.apache.hadoop.hdfs.TestHftpFileSystem
      org.apache.hadoop.hdfs.TestSetrepIncreasing

      +1 contrib tests. The patch passed contrib unit tests.

      Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1448//testReport/
      Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1448//console

      This message is automatically generated.

      Show
      Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12500782/pbClientTypes4.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestFSInputChecker org.apache.hadoop.tools.TestJMXGet org.apache.hadoop.hdfs.TestHDFSTrash org.apache.hadoop.hdfs.TestSmallBlock org.apache.hadoop.hdfs.TestListPathServlet org.apache.hadoop.cli.TestHDFSCLI org.apache.hadoop.security.TestRefreshUserMappings org.apache.hadoop.hdfs.TestFileCreationClient org.apache.hadoop.security.TestPermission org.apache.hadoop.hdfs.TestHftpFileSystem org.apache.hadoop.hdfs.TestSetrepIncreasing +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1448//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1448//console This message is automatically generated.
      Hide
      Suresh Srinivas added a comment -

      +1 for the patch.

      Minor comments:

      1. There are some tab files, some extra spaces. It would also be good to have all the members of message ordered numerically.
      2. UpgradeStatusReportProto is missing documentation
      3. I am not sure what this comment means {{ required uint64 numBytes = 3; // for historical reasons - doesn't belong here}}
      Show
      Suresh Srinivas added a comment - +1 for the patch. Minor comments: There are some tab files, some extra spaces. It would also be good to have all the members of message ordered numerically. UpgradeStatusReportProto is missing documentation I am not sure what this comment means {{ required uint64 numBytes = 3; // for historical reasons - doesn't belong here}}
      Hide
      Sanjay Radia added a comment -

      Update patch that addresses Suresh's feedback.

      Show
      Sanjay Radia added a comment - Update patch that addresses Suresh's feedback.
      Hide
      Sanjay Radia added a comment -

      The failures are unrelated to the patch.

      Show
      Sanjay Radia added a comment - The failures are unrelated to the patch.
      Hide
      Sanjay Radia added a comment -

      Tests runs fine for me
      Running org.apache.hadoop.hdfs.TestFSInputChecker
      Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.542 sec
      Running org.apache.hadoop.tools.TestJMXGet
      Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.941 sec
      Running org.apache.hadoop.hdfs.TestSmallBlock
      Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.209 sec
      Running org.apache.hadoop.hdfs.TestListPathServlet
      Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.34 sec
      Running org.apache.hadoop.cli.TestHDFSCLI
      Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 66.918 sec
      Running org.apache.hadoop.security.TestRefreshUserMappings
      Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.788 sec
      Running org.apache.hadoop.hdfs.TestFileCreationClient
      Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.487 sec
      Running org.apache.hadoop.security.TestPermission
      Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.289 sec
      Running org.apache.hadoop.hdfs.TestSetrepDecreasing
      Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.13 sec

      TestHftpFileSystem a failure due to locking

      Show
      Sanjay Radia added a comment - Tests runs fine for me Running org.apache.hadoop.hdfs.TestFSInputChecker Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.542 sec Running org.apache.hadoop.tools.TestJMXGet Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.941 sec Running org.apache.hadoop.hdfs.TestSmallBlock Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.209 sec Running org.apache.hadoop.hdfs.TestListPathServlet Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.34 sec Running org.apache.hadoop.cli.TestHDFSCLI Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 66.918 sec Running org.apache.hadoop.security.TestRefreshUserMappings Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.788 sec Running org.apache.hadoop.hdfs.TestFileCreationClient Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.487 sec Running org.apache.hadoop.security.TestPermission Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.289 sec Running org.apache.hadoop.hdfs.TestSetrepDecreasing Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.13 sec TestHftpFileSystem a failure due to locking
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Mapreduce-trunk #875 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/875/)
      HDFS-2479 HDFS Client Data Types in Protocol Buffers (sanjay)

      sradia : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190127
      Files :

      • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
      • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsProtoUtil.java
      • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/proto/HdfsProtos.java
      • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/proto/hdfs.proto
      Show
      Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #875 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/875/ ) HDFS-2479 HDFS Client Data Types in Protocol Buffers (sanjay) sradia : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190127 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsProtoUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/proto/HdfsProtos.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/proto/hdfs.proto
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Mapreduce-trunk-Commit #1200 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1200/)
      HDFS-2479 HDFS Client Data Types in Protocol Buffers (sanjay)

      sradia : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190127
      Files :

      • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
      • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsProtoUtil.java
      • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/proto/HdfsProtos.java
      • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/proto/hdfs.proto
      Show
      Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1200 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1200/ ) HDFS-2479 HDFS Client Data Types in Protocol Buffers (sanjay) sradia : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190127 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsProtoUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/proto/HdfsProtos.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/proto/hdfs.proto
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Hdfs-trunk-Commit #1259 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1259/)
      HDFS-2479 HDFS Client Data Types in Protocol Buffers (sanjay)

      sradia : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190127
      Files :

      • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
      • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsProtoUtil.java
      • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/proto/HdfsProtos.java
      • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/proto/hdfs.proto
      Show
      Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1259 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1259/ ) HDFS-2479 HDFS Client Data Types in Protocol Buffers (sanjay) sradia : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190127 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsProtoUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/proto/HdfsProtos.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/proto/hdfs.proto
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Common-trunk-Commit #1182 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1182/)
      HDFS-2479 HDFS Client Data Types in Protocol Buffers (sanjay)

      sradia : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190127
      Files :

      • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
      • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsProtoUtil.java
      • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/proto/HdfsProtos.java
      • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/proto/hdfs.proto
      Show
      Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1182 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1182/ ) HDFS-2479 HDFS Client Data Types in Protocol Buffers (sanjay) sradia : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190127 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsProtoUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/proto/HdfsProtos.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/proto/hdfs.proto
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Hdfs-trunk #846 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/846/)
      HDFS-2479 HDFS Client Data Types in Protocol Buffers (sanjay)

      sradia : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190127
      Files :

      • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
      • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsProtoUtil.java
      • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/proto/HdfsProtos.java
      • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/proto/hdfs.proto
      Show
      Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #846 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/846/ ) HDFS-2479 HDFS Client Data Types in Protocol Buffers (sanjay) sradia : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190127 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsProtoUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/proto/HdfsProtos.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/proto/hdfs.proto
      Hide
      Tsz Wo Nicholas Sze added a comment -

      I have merged this to 0.23.

      Show
      Tsz Wo Nicholas Sze added a comment - I have merged this to 0.23.

        People

        • Assignee:
          Sanjay Radia
          Reporter:
          Sanjay Radia
        • Votes:
          0 Vote for this issue
          Watchers:
          3 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved:

            Development