Hadoop Common
  1. Hadoop Common
  2. HADOOP-7899

Generate proto java files as part of the build

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.24.0
    • Fix Version/s: 2.0.0-alpha
    • Component/s: build
    • Labels:
      None

      Description

      currently the generated java files are precompiled and checked in into the source.

      1. HADOOP-7899v1.patch
        2 kB
        Alejandro Abdelnur
      2. HADOOP-7899v1.sh
        1 kB
        Alejandro Abdelnur

        Issue Links

          Activity

          Hide
          Alejandro Abdelnur added a comment -

          Run script from src root then apply patch.

          The script requires one parameter fs|svn (fs for GIT, svn for SVN).

          The script deletes the generated JAVA files and moves src/proto to src/main/proto

          The generated files are found by IDEs as they are in target/generated-sources/java.

          Show
          Alejandro Abdelnur added a comment - Run script from src root then apply patch. The script requires one parameter fs|svn (fs for GIT, svn for SVN). The script deletes the generated JAVA files and moves src/proto to src/main/proto The generated files are found by IDEs as they are in target/generated-sources/java.
          Hide
          Alejandro Abdelnur added a comment -

          tested by doing clean build, including testcases.

          Show
          Alejandro Abdelnur added a comment - tested by doing clean build, including testcases.
          Hide
          Tom White added a comment -

          +1

          Show
          Tom White added a comment - +1
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12506670/HADOOP-7899v1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The patch appears to cause tar ant target to fail.

          -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed the unit tests build

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/458//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/458//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12506670/HADOOP-7899v1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed the unit tests build +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/458//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/458//console This message is automatically generated.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1393 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1393/)
          HADOOP-7899. Generate proto java files as part of the build. (tucu)

          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1212615
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/protobuf/HadoopRpcProtos.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/protobuf/package-info.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/proto
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/proto
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/protobuf/TestProtos.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/protobuf/TestRpcServiceProtos.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1393 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1393/ ) HADOOP-7899 . Generate proto java files as part of the build. (tucu) tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1212615 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/protobuf/HadoopRpcProtos.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/protobuf/package-info.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/proto /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/proto /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/protobuf/TestProtos.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/protobuf/TestRpcServiceProtos.java
          Hide
          Suresh Srinivas added a comment -

          Alejandro, please respond to -1s from Jenkins before committing the patch.

          Show
          Suresh Srinivas added a comment - Alejandro, please respond to -1s from Jenkins before committing the patch.
          Hide
          Alejandro Abdelnur added a comment -

          committed to trunk, but don't see the proto files in branch-0.23, seems that the patch that introduced it has not been backported.

          Leaving this JIRA open til get some clarification on what to do in branch-0.23

          Show
          Alejandro Abdelnur added a comment - committed to trunk, but don't see the proto files in branch-0.23, seems that the patch that introduced it has not been backported. Leaving this JIRA open til get some clarification on what to do in branch-0.23
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1467 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1467/)
          HADOOP-7899. Generate proto java files as part of the build. (tucu)

          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1212615
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/protobuf/HadoopRpcProtos.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/protobuf/package-info.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/proto
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/proto
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/protobuf/TestProtos.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/protobuf/TestRpcServiceProtos.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1467 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1467/ ) HADOOP-7899 . Generate proto java files as part of the build. (tucu) tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1212615 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/protobuf/HadoopRpcProtos.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/protobuf/package-info.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/proto /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/proto /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/protobuf/TestProtos.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/protobuf/TestRpcServiceProtos.java
          Hide
          Suresh Srinivas added a comment -

          This is not gone into 0.23 yet. Can you please comment on my previous comment?

          Show
          Suresh Srinivas added a comment - This is not gone into 0.23 yet. Can you please comment on my previous comment?
          Hide
          Alejandro Abdelnur added a comment -

          Suresh, apologies, I've missed your comment before.

          The -1 from jenkins do not apply, this patch does not introduce/modify any java code, actually it deletes autogenerated java code and generate/compiles those files as part of the build.

          Regarding your last comment that this has not gone in 0.23 yet. Will it go into 0.23? If so, when that happens we have to do this patch there.

          Show
          Alejandro Abdelnur added a comment - Suresh, apologies, I've missed your comment before. The -1 from jenkins do not apply, this patch does not introduce/modify any java code, actually it deletes autogenerated java code and generate/compiles those files as part of the build. Regarding your last comment that this has not gone in 0.23 yet. Will it go into 0.23? If so, when that happens we have to do this patch there.
          Hide
          Suresh Srinivas added a comment -

          This will go in along with the related protobuf changes. However it does require lot of testing. Will port it along with other patches.

          Show
          Suresh Srinivas added a comment - This will go in along with the related protobuf changes. However it does require lot of testing. Will port it along with other patches.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1415 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1415/)
          HADOOP-7899. Generate proto java files as part of the build. (tucu)

          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1212615
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/protobuf/HadoopRpcProtos.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/protobuf/package-info.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/proto
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/proto
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/protobuf/TestProtos.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/protobuf/TestRpcServiceProtos.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1415 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1415/ ) HADOOP-7899 . Generate proto java files as part of the build. (tucu) tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1212615 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/protobuf/HadoopRpcProtos.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/protobuf/package-info.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/proto /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/proto /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/protobuf/TestProtos.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/protobuf/TestRpcServiceProtos.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #890 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/890/)
          HADOOP-7899. Generate proto java files as part of the build. (tucu)

          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1212615
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/protobuf/HadoopRpcProtos.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/protobuf/package-info.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/proto
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/proto
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/protobuf/TestProtos.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/protobuf/TestRpcServiceProtos.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #890 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/890/ ) HADOOP-7899 . Generate proto java files as part of the build. (tucu) tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1212615 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/protobuf/HadoopRpcProtos.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/protobuf/package-info.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/proto /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/proto /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/protobuf/TestProtos.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/protobuf/TestRpcServiceProtos.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #923 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/923/)
          HADOOP-7899. Generate proto java files as part of the build. (tucu)

          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1212615
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/protobuf/HadoopRpcProtos.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/protobuf/package-info.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/proto
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/proto
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/protobuf/TestProtos.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/protobuf/TestRpcServiceProtos.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #923 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/923/ ) HADOOP-7899 . Generate proto java files as part of the build. (tucu) tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1212615 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/protobuf/HadoopRpcProtos.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/protobuf/package-info.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/proto /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/proto /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/protobuf/TestProtos.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/protobuf/TestRpcServiceProtos.java
          Hide
          Uma Maheswara Rao G added a comment -

          Hi Alejandro,

          Looks build is failing for the windows dev environments.

          [ERROR] \Trunk-Common\hadoop-common-project\hadoop-common\src\main\java\org\apac
          he\hadoop\ipc\ProtobufRpcEngine.java:[425,8] cannot find symbol
          symbol : class HadoopRpcResponseProto
          location: class org.apache.hadoop.ipc.ProtobufRpcEngine.Server.ProtoBufRpcInvoke
          r
          [INFO] 29 errors
          [INFO] -------------------------------------------------------------
          [INFO] ------------------------------------------------------------------------
          [INFO] Reactor Summary:
          [INFO]
          [INFO] Apache Hadoop Main ................................ SUCCESS [1.971s]
          [INFO] Apache Hadoop Project POM ......................... SUCCESS [0.641s]
          [INFO] Apache Hadoop Annotations ......................... SUCCESS [1.862s]
          [INFO] Apache Hadoop Project Dist POM .................... SUCCESS [0.469s]
          [INFO] Apache Hadoop Assemblies .......................... SUCCESS [0.250s]
          [INFO] Apache Hadoop Auth ................................ SUCCESS [3.176s]
          [INFO] Apache Hadoop Auth Examples ....................... SUCCESS [2.018s]
          [INFO] Apache Hadoop Common .............................. FAILURE [11.404s]
          [INFO] Apache Hadoop Common Project ...................... SKIPPED
          [INFO] Apache Hadoop HDFS ................................ SKIPPED
          [INFO] Apache Hadoop HttpFS .............................. SKIPPED

          I think it is working fine for the Linux environments.
          Since yours is a familiar hand here, could you please take a look?
          Am i missing something here?

          Thanks
          Uma

          Show
          Uma Maheswara Rao G added a comment - Hi Alejandro, Looks build is failing for the windows dev environments. [ERROR] \Trunk-Common\hadoop-common-project\hadoop-common\src\main\java\org\apac he\hadoop\ipc\ProtobufRpcEngine.java: [425,8] cannot find symbol symbol : class HadoopRpcResponseProto location: class org.apache.hadoop.ipc.ProtobufRpcEngine.Server.ProtoBufRpcInvoke r [INFO] 29 errors [INFO] ------------------------------------------------------------- [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Apache Hadoop Main ................................ SUCCESS [1.971s] [INFO] Apache Hadoop Project POM ......................... SUCCESS [0.641s] [INFO] Apache Hadoop Annotations ......................... SUCCESS [1.862s] [INFO] Apache Hadoop Project Dist POM .................... SUCCESS [0.469s] [INFO] Apache Hadoop Assemblies .......................... SUCCESS [0.250s] [INFO] Apache Hadoop Auth ................................ SUCCESS [3.176s] [INFO] Apache Hadoop Auth Examples ....................... SUCCESS [2.018s] [INFO] Apache Hadoop Common .............................. FAILURE [11.404s] [INFO] Apache Hadoop Common Project ...................... SKIPPED [INFO] Apache Hadoop HDFS ................................ SKIPPED [INFO] Apache Hadoop HttpFS .............................. SKIPPED I think it is working fine for the Linux environments. Since yours is a familiar hand here, could you please take a look? Am i missing something here? Thanks Uma
          Hide
          Ravi Teja Ch N V added a comment -

          Hi Alejandro,

          As Uma has mentioned,it works on Linux env, but its broken in windows env. Can you please look into the issue as its kinda blocking as we unable to build the jars.

          Thanks

          Show
          Ravi Teja Ch N V added a comment - Hi Alejandro, As Uma has mentioned,it works on Linux env, but its broken in windows env. Can you please look into the issue as its kinda blocking as we unable to build the jars. Thanks
          Hide
          Alejandro Abdelnur added a comment -

          Ravi, Uma,

          I don't have a WIN environment, I have to setup a VM. I'll try do this TUE. In the mean time, if you can try to make the script cygwin friendly it would be great.

          Show
          Alejandro Abdelnur added a comment - Ravi, Uma, I don't have a WIN environment, I have to setup a VM. I'll try do this TUE. In the mean time, if you can try to make the script cygwin friendly it would be great.
          Hide
          Alejandro Abdelnur added a comment -

          Patch avail at HADOOP-7917

          Show
          Alejandro Abdelnur added a comment - Patch avail at HADOOP-7917
          Hide
          Uma Maheswara Rao G added a comment -

          Thanks a lot, Alejandro for the patch, I will verify it on HADOOP-7917.

          Show
          Uma Maheswara Rao G added a comment - Thanks a lot, Alejandro for the patch, I will verify it on HADOOP-7917 .
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I have merged this to 0.23.

          Show
          Tsz Wo Nicholas Sze added a comment - I have merged this to 0.23.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          This was committed some time ago. Resolving.

          Show
          Tsz Wo Nicholas Sze added a comment - This was committed some time ago. Resolving.
          Hide
          Suresh Srinivas added a comment -

          When proto files cannot be compiled, the build still continues. Filing a bug a for that.

          Show
          Suresh Srinivas added a comment - When proto files cannot be compiled, the build still continues. Filing a bug a for that.

            People

            • Assignee:
              Alejandro Abdelnur
              Reporter:
              Alejandro Abdelnur
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development