Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2211

Build does not pass along properties to contrib builds

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.0
    • Component/s: build
    • Labels:
      None
    • Environment:

      RHEL 6.1 & Ubuntu 11.04
      Sun JRE 1.6
      Ant 1.8.2

    • Hadoop Flags:
      Reviewed

      Description

      Subant call to compile contribs do not pass along parameters from parent build.
      Properties such as hadoop-common.version, asfrepo, offline, etc. are not passed along.
      Result is that build not connected to Internet fails, hdfs proxy refuses to build against own recently built common but rather downloads 0.22-SNAPSHOT from apache again.

      1. HDFS-2211.patch
        4 kB
        Joep Rottinghuis

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          20h 40m 1 Joep Rottinghuis 28/Jul/11 18:31
          Patch Available Patch Available Resolved Resolved
          9d 7h 20m 1 Konstantin Boudnik 07/Aug/11 01:52
          Resolved Resolved Closed Closed
          127d 5h 27m 1 Konstantin Shvachko 12/Dec/11 06:19
          Konstantin Shvachko made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Joep Rottinghuis made changes -
          Assignee Joep Rottinghuis [ jrottinghuis ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-22-branch #73 (See https://builds.apache.org/job/Hadoop-Hdfs-22-branch/73/)
          HDFS-2211. Build does not pass along properties to contrib builds. Contributed by Joep Rottinghuis.

          cos : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1154630
          Files :

          • /hadoop/common/branches/branch-0.22/hdfs/src/contrib/build-contrib.xml
          • /hadoop/common/branches/branch-0.22/hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.22/hdfs/src/contrib/build.xml
          • /hadoop/common/branches/branch-0.22/hdfs/build.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-22-branch #73 (See https://builds.apache.org/job/Hadoop-Hdfs-22-branch/73/ ) HDFS-2211 . Build does not pass along properties to contrib builds. Contributed by Joep Rottinghuis. cos : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1154630 Files : /hadoop/common/branches/branch-0.22/hdfs/src/contrib/build-contrib.xml /hadoop/common/branches/branch-0.22/hdfs/CHANGES.txt /hadoop/common/branches/branch-0.22/hdfs/src/contrib/build.xml /hadoop/common/branches/branch-0.22/hdfs/build.xml
          Konstantin Boudnik made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Fix Version/s 0.22.0 [ 12314241 ]
          Resolution Fixed [ 1 ]
          Hide
          Konstantin Boudnik added a comment -

          I have committed it. Thanks Joep!

          Show
          Konstantin Boudnik added a comment - I have committed it. Thanks Joep!
          Hide
          Joep Rottinghuis added a comment -

          Hi,
          Thanks for your mail. I'll be out of the office until Wednesday August 17th.
          During this time I will not be checking my e-mail.
          Thanks,
          Joep

          Show
          Joep Rottinghuis added a comment - Hi, Thanks for your mail. I'll be out of the office until Wednesday August 17th. During this time I will not be checking my e-mail. Thanks, Joep
          Hide
          Konstantin Boudnik added a comment -

          I have applied the patch to hdfs 0.22 and tried to build with and without network connection. Everything seems to be working properly. Contrib is building and I can create package.

          +1 on the patch. I am going to commit it shortly.

          I am not clear if HDFS-1935 needs to be in as well?

          Show
          Konstantin Boudnik added a comment - I have applied the patch to hdfs 0.22 and tried to build with and without network connection. Everything seems to be working properly. Contrib is building and I can create package. +1 on the patch. I am going to commit it shortly. I am not clear if HDFS-1935 needs to be in as well?
          Hide
          Konstantin Boudnik added a comment -

          I have applied the patch to hdfs 0.22 and tried to build with and without network connection. Everything seems to be working properly. Contrib is building and I can create package.

          +1 on the patch. I am going to commit it shortly.

          I am not clear if HDFS-1935 needs to be in as well?

          Show
          Konstantin Boudnik added a comment - I have applied the patch to hdfs 0.22 and tried to build with and without network connection. Everything seems to be working properly. Contrib is building and I can create package. +1 on the patch. I am going to commit it shortly. I am not clear if HDFS-1935 needs to be in as well?
          Konstantin Shvachko made changes -
          Priority Minor [ 4 ] Blocker [ 1 ]
          Hide
          Konstantin Shvachko added a comment -

          Build problem, marking as blocker.

          Show
          Konstantin Shvachko added a comment - Build problem, marking as blocker.
          Joep Rottinghuis made changes -
          Link This issue relates to MAPREDUCE-2752 [ MAPREDUCE-2752 ]
          Joep Rottinghuis made changes -
          Link This issue relates to HDFS-2214 [ HDFS-2214 ]
          Konstantin Shvachko made changes -
          Link This issue incorporates HDFS-1935 [ HDFS-1935 ]
          Hide
          Joep Rottinghuis added a comment -

          Should have used code markup as pound comment sign was translated to 1.'s.

          Show
          Joep Rottinghuis added a comment - Should have used code markup as pound comment sign was translated to 1.'s.
          Hide
          Joep Rottinghuis added a comment -

          To build hdfs without Internet connectivity:
          1) First do a build with connectivity or copy ~/.ivy2 from a machine where such build did succeed.
          2) Set up a hand-full of files in a local repo to avoid download from Maven repo:
          and have three files available:
          /home/user/buildrepo/dist/commons/daemon/binaries/1.0.2/linux/commons-daemon-1.0.2-bin-linux-i386.tar.gz
          /home/user/buildrepo/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar
          /home/user/buildrepo/org/apache/maven/maven-ant-tasks/2.0.10/maven-ant-tasks-2.0.10.jar
          3) Before the build run these commands to copy the jars in place for ant to resolve (otherwise you have a bootstrap problem):
          cp $

          {HOME}/buildrepo/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar ${WORKSPACE}/hdfs/ivy/
          cp ${HOME}

          /buildrepo/org/apache/maven/maven-ant-tasks/2.0.10/maven-ant-tasks-2.0.10.jar $

          {WORKSPACE}

          /hdfs/ivy/

          3) Do a build of common (using similar tricks to this if needed) on common. Exectute the mvn-install target to publish your own version to the local Maven repo (~/.m2/repository).
          4) Set these properties in your build.properties (version can be passed in from Jenkins as a parameter instead when this patch is applied. Build.properties can sit in user's home directory or in hdfs directory.

          1. Ivy dependency resolution instruction:
            resolvers=internal

          #you can increment this number as you see fit and/or pass the $

          {BUILD_NUMBER}

          from Jenkins
          build.number=0
          version=0.22-coolname-$

          {build.number}

          project.version=$

          {version}
          hadoop.version=${version}
          1. Note that hadoop-core from 0.20* branches is renamed to hadoop-common
            hadoop-common.version=$ {version}
            hadoop-hdfs.version=${version}
          1. Specify to not download ivy. When used, must provide ivy jar ourselves.
            offline=true
          1. Instead of reaching out to Internet, pull ivy jar from local
            mvnrepo=file:$ {user.home}/buildrepo
            # User by hadoop-common/hdfs and hadoop-common/mapreduce
            mvn.repo=file:${user.home}

            /buildrepo

          Show
          Joep Rottinghuis added a comment - To build hdfs without Internet connectivity: 1) First do a build with connectivity or copy ~/.ivy2 from a machine where such build did succeed. 2) Set up a hand-full of files in a local repo to avoid download from Maven repo: and have three files available: /home/user/buildrepo/dist/commons/daemon/binaries/1.0.2/linux/commons-daemon-1.0.2-bin-linux-i386.tar.gz /home/user/buildrepo/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar /home/user/buildrepo/org/apache/maven/maven-ant-tasks/2.0.10/maven-ant-tasks-2.0.10.jar 3) Before the build run these commands to copy the jars in place for ant to resolve (otherwise you have a bootstrap problem): cp $ {HOME}/buildrepo/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar ${WORKSPACE}/hdfs/ivy/ cp ${HOME} /buildrepo/org/apache/maven/maven-ant-tasks/2.0.10/maven-ant-tasks-2.0.10.jar $ {WORKSPACE} /hdfs/ivy/ 3) Do a build of common (using similar tricks to this if needed) on common. Exectute the mvn-install target to publish your own version to the local Maven repo (~/.m2/repository). 4) Set these properties in your build.properties (version can be passed in from Jenkins as a parameter instead when this patch is applied. Build.properties can sit in user's home directory or in hdfs directory. Ivy dependency resolution instruction: resolvers=internal #you can increment this number as you see fit and/or pass the $ {BUILD_NUMBER} from Jenkins build.number=0 version=0.22-coolname-$ {build.number} project.version=$ {version} hadoop.version=${version} Note that hadoop-core from 0.20* branches is renamed to hadoop-common hadoop-common.version=$ {version} hadoop-hdfs.version=${version} Specify to not download ivy. When used, must provide ivy jar ourselves. offline=true Instead of reaching out to Internet, pull ivy jar from local mvnrepo= file:$ {user.home}/buildrepo # User by hadoop-common/hdfs and hadoop-common/mapreduce mvn.repo= file:$ {user.home} /buildrepo
          Joep Rottinghuis made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Joep Rottinghuis made changes -
          Field Original Value New Value
          Attachment HDFS-2211.patch [ 12488051 ]
          Hide
          Joep Rottinghuis added a comment -

          The subant calls in src/contrib/build.xml and src/contrib/build-contrib.xml suffer from the same problem. There is one antcall in build-contrib with the same.

          Show
          Joep Rottinghuis added a comment - The subant calls in src/contrib/build.xml and src/contrib/build-contrib.xml suffer from the same problem. There is one antcall in build-contrib with the same.
          Hide
          Joep Rottinghuis added a comment -

          Make that HDFS-1935

          Show
          Joep Rottinghuis added a comment - Make that HDFS-1935
          Hide
          Joep Rottinghuis added a comment -

          This is related to MAPREDUCE-1935

          Show
          Joep Rottinghuis added a comment - This is related to MAPREDUCE-1935
          Joep Rottinghuis created issue -

            People

            • Assignee:
              Joep Rottinghuis
              Reporter:
              Joep Rottinghuis
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development