Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11843

Make setting up the build environment easier

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: build
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Includes a docker based solution for setting up a build environment with minimal effort.

      Description

      ( As discussed with Allen Wittenauer )
      In AVRO-1537 a docker based solution was created to setup all the tools for doing a full build. This enables much easier reproduction of any issues and getting up and running for new developers.

      This issue is to 'copy/port' that setup into the hadoop project in preparation for the bug squash.

      1. HADOOP-11843-2015-04-17-1612.patch
        5 kB
        Niels Basjes
      2. HADOOP-11843-2015-04-17-2226.patch
        7 kB
        Niels Basjes
      3. HADOOP-11843-2015-04-17-2308.patch
        7 kB
        Niels Basjes
      4. HADOOP-11843-2015-04-19-2206.patch
        10 kB
        Niels Basjes
      5. HADOOP-11843-2015-04-19-2232.patch
        10 kB
        Niels Basjes
      6. HADOOP-11843-2015-04-22-1122.patch
        11 kB
        Niels Basjes
      7. HADOOP-11843-2015-04-23-1000.patch
        11 kB
        Niels Basjes

        Issue Links

          Activity

          Hide
          nielsbasjes Niels Basjes added a comment -

          This patch offers an start-build-env.sh that creates a fully functional environment with (as far as I've been able to test) all the tools needed to build the Hadoop source tree.

          Jenkins won't be able to test this.

          Show
          nielsbasjes Niels Basjes added a comment - This patch offers an start-build-env.sh that creates a fully functional environment with (as far as I've been able to test) all the tools needed to build the Hadoop source tree. Jenkins won't be able to test this.
          Hide
          nielsbasjes Niels Basjes added a comment -

          Because this patch only has new files I expect this patch can be applied to any branch without modification.

          Show
          nielsbasjes Niels Basjes added a comment - Because this patch only has new files I expect this patch can be applied to any branch without modification.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 0m 00s Pre-patch trunk compilation is healthy.
          +1 @author 0m 00s The patch does not contain any @author tags.
          -1 whitespace 0m 00s The patch has 7 line(s) that end in whitespace.
          -1 release audit 0m 04s The applied patch generated 1 release audit warnings.
          -1 shellcheck 0m 02s The applied patch generated 12 new shellcheck issues (total was 25, now 37).
              0m 14s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12726165/HADOOP-11843-2015-04-17-1612.patch
          Optional Tests shellcheck
          git revision trunk / b9b832a
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/fake/artifact/patchprocess/whitespace.txt
          Release Audit https://builds.apache.org/job/PreCommit-HADOOP-Build/fake/artifact/patchprocess/patchReleaseAuditProblems.txt
          shellcheck https://builds.apache.org/job/PreCommit-HADOOP-Build/fake/artifact/patchprocess/diffpatchshellcheck.txt
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/fake//console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 0m 00s Pre-patch trunk compilation is healthy. +1 @author 0m 00s The patch does not contain any @author tags. -1 whitespace 0m 00s The patch has 7 line(s) that end in whitespace. -1 release audit 0m 04s The applied patch generated 1 release audit warnings. -1 shellcheck 0m 02s The applied patch generated 12 new shellcheck issues (total was 25, now 37).     0m 14s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12726165/HADOOP-11843-2015-04-17-1612.patch Optional Tests shellcheck git revision trunk / b9b832a whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/fake/artifact/patchprocess/whitespace.txt Release Audit https://builds.apache.org/job/PreCommit-HADOOP-Build/fake/artifact/patchprocess/patchReleaseAuditProblems.txt shellcheck https://builds.apache.org/job/PreCommit-HADOOP-Build/fake/artifact/patchprocess/diffpatchshellcheck.txt Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/fake//console This message was automatically generated.
          Hide
          aw Allen Wittenauer added a comment -

          just me testing the new test-patch.sh script against your patch

          Show
          aw Allen Wittenauer added a comment - just me testing the new test-patch.sh script against your patch
          Hide
          aw Allen Wittenauer added a comment -

          This is awesome. I'll start playing with it here in a bit! A quick view of the patch does seem to indicate a few things to fix already though:

          • motd should have an apache license on it.
          • Dockerfile says "for Avro" still
          • Hadoop's dependencies are actually smaller. off the top, no autotools, subversion, boost, doxygen, forrest, or ant.
          • Add shellcheck and findbugs, if possible, for the new test-patch.sh code that's about to get committed...
          • Let's clean up the extra whitespace on the end of lines. This makes git happy.

          Much thanks!

          Show
          aw Allen Wittenauer added a comment - This is awesome. I'll start playing with it here in a bit! A quick view of the patch does seem to indicate a few things to fix already though: motd should have an apache license on it. Dockerfile says "for Avro" still Hadoop's dependencies are actually smaller. off the top, no autotools, subversion, boost, doxygen, forrest, or ant. Add shellcheck and findbugs, if possible, for the new test-patch.sh code that's about to get committed... Let's clean up the extra whitespace on the end of lines. This makes git happy. Much thanks!
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12726165/HADOOP-11843-2015-04-17-1612.patch
          against trunk revision 76e7264.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          -1 release audit. The applied patch generated 1 release audit warnings.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/6117//testReport/
          Release audit warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/6117//artifact/patchprocess/patchReleaseAuditProblems.txt
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/6117//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12726165/HADOOP-11843-2015-04-17-1612.patch against trunk revision 76e7264. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. -1 release audit . The applied patch generated 1 release audit warnings. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/6117//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/6117//artifact/patchprocess/patchReleaseAuditProblems.txt Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/6117//console This message is automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          Great idea! Could we please update BUILDING.txt so that people know this is here as an option for a quick dev environment?

          Show
          cnauroth Chris Nauroth added a comment - Great idea! Could we please update BUILDING.txt so that people know this is here as an option for a quick dev environment?
          Hide
          cnauroth Chris Nauroth added a comment -

          I recommend adding libcurl4-openssl-dev. libwebhdfs needs the libcurl headers. BTW, I think I'm seeing a build failure in libwebhdfs, but I'll follow up separately on that.

          I think we can drop libprotobuf-dev. We just need the protoc binary provided by protobuf-compiler.

          I don't believe we need asciidoc, source-highlight, flex, or bison. (I have no objection though if someone wants to keep these for some reason.) This is in addition to what Allen already mentioned in his earlier comment.

          Show
          cnauroth Chris Nauroth added a comment - I recommend adding libcurl4-openssl-dev. libwebhdfs needs the libcurl headers. BTW, I think I'm seeing a build failure in libwebhdfs, but I'll follow up separately on that. I think we can drop libprotobuf-dev. We just need the protoc binary provided by protobuf-compiler. I don't believe we need asciidoc, source-highlight, flex, or bison. (I have no objection though if someone wants to keep these for some reason.) This is in addition to what Allen already mentioned in his earlier comment.
          Hide
          nielsbasjes Niels Basjes added a comment -
          • Fixed shellcheck warnings
          • Fixed the motd point and added a filter to not show it on the screen.
          • Cleaned up whitespace
          • "for Hadoop" ...
          • Removed a lot of dependencies. I added most of these because they are explicitly mentioned as needed for specific parts of the build (like -Pnative) in BUILDING.TXT. Someone should update that. I do not have enough knowledge at this moment to do that.
          • Added a section to the BUILDING.TXT to point out this new feature.
          • Added all requested dependencies (shellcheck, findbugs and libcurl4-openssl-dev).
          Show
          nielsbasjes Niels Basjes added a comment - Fixed shellcheck warnings Fixed the motd point and added a filter to not show it on the screen. Cleaned up whitespace "for Hadoop" ... Removed a lot of dependencies. I added most of these because they are explicitly mentioned as needed for specific parts of the build (like -Pnative) in BUILDING.TXT. Someone should update that. I do not have enough knowledge at this moment to do that. Added a section to the BUILDING.TXT to point out this new feature. Added all requested dependencies (shellcheck, findbugs and libcurl4-openssl-dev).
          Hide
          nielsbasjes Niels Basjes added a comment -

          This patch succesfully ran

          mvn package -Pdist -Pdocs -Psrc -Pnative -Dtar -Preleasedocs -DskipTests=true

          So everything except I didn't run the tests because that takes very long.

          Show
          nielsbasjes Niels Basjes added a comment - This patch succesfully ran mvn package -Pdist -Pdocs -Psrc -Pnative -Dtar -Preleasedocs -DskipTests= true So everything except I didn't run the tests because that takes very long.
          Hide
          nielsbasjes Niels Basjes added a comment -

          P.S. I left Forrest in because I found several references in the code base.

          Show
          nielsbasjes Niels Basjes added a comment - P.S. I left Forrest in because I found several references in the code base.
          Hide
          cnauroth Chris Nauroth added a comment -

          I added most of these because they are explicitly mentioned as needed for specific parts of the build (like -Pnative) in BUILDING.TXT.

          Thanks for pointing that out. I took a look, and I think libprotobuf-dev is the only extraneous thing mentioned in BUILDING.txt. Would you mind removing that since you're already touching this file?

          Show
          cnauroth Chris Nauroth added a comment - I added most of these because they are explicitly mentioned as needed for specific parts of the build (like -Pnative) in BUILDING.TXT. Thanks for pointing that out. I took a look, and I think libprotobuf-dev is the only extraneous thing mentioned in BUILDING.txt. Would you mind removing that since you're already touching this file?
          Hide
          nielsbasjes Niels Basjes added a comment -

          Removed reference to libprotobuf-dev in BUILDING.txt

          Show
          nielsbasjes Niels Basjes added a comment - Removed reference to libprotobuf-dev in BUILDING.txt
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12726259/HADOOP-11843-2015-04-17-2226.patch
          against trunk revision d573f09.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/6119//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/6119//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12726259/HADOOP-11843-2015-04-17-2226.patch against trunk revision d573f09. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/6119//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/6119//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12726263/HADOOP-11843-2015-04-17-2308.patch
          against trunk revision d573f09.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/6120//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/6120//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12726263/HADOOP-11843-2015-04-17-2308.patch against trunk revision d573f09. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/6120//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/6120//console This message is automatically generated.
          Hide
          aw Allen Wittenauer added a comment -

          P.S. I left Forrest in because I found several references in the code base.

          Ugh. We'll need to hunt those out at some point, but this isn't the place.

          I'm still running some tests, but so far so good. So excited!

          Show
          aw Allen Wittenauer added a comment - P.S. I left Forrest in because I found several references in the code base. Ugh. We'll need to hunt those out at some point, but this isn't the place. I'm still running some tests, but so far so good. So excited!
          Hide
          nielsbasjes Niels Basjes added a comment -

          @Allen Wittenauer The (relevant) references I found are in:

          hadoop/dev-support/test-patch.sh

          (Which I understand you are rewriting ... I assume without Forrest)

          And in all of these documentation files:

          hadoop/hadoop-common-project/hadoop-common/src/main/docs/status.xml
          hadoop/hadoop-common-project/hadoop-common/src/main/docs/src/documentation/classes/CatalogManager.properties
          hadoop/hadoop-common-project/hadoop-common/src/main/docs/src/documentation/README.txt
          hadoop/hadoop-common-project/hadoop-common/src/main/docs/src/documentation/content/xdocs/index.xml
          hadoop/hadoop-common-project/hadoop-common/src/main/docs/src/documentation/content/xdocs/site.xml
          hadoop/hadoop-common-project/hadoop-common/src/main/docs/src/documentation/content/xdocs/tabs.xml
          hadoop/hadoop-common-project/hadoop-common/src/main/docs/src/documentation/skinconf.xml
          hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/docs/status.xml
          hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/docs/src/documentation/classes/CatalogManager.properties
          hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/docs/src/documentation/README.txt
          hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/docs/src/documentation/content/xdocs/index.xml
          hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/docs/src/documentation/content/xdocs/webhdfs.xml
          hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/docs/src/documentation/content/xdocs/site.xml
          hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/docs/src/documentation/content/xdocs/tabs.xml
          hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/docs/src/documentation/skinconf.xml
          

          I didn't find any references in pom.xml files so it seems that indeed the references I found are all in 'dead documentation'.
          If you say "take forrest out" I'll take it out of my patch. In that case I assume you'll create a new Jira to get rid of these obsolete files.

          Show
          nielsbasjes Niels Basjes added a comment - @ Allen Wittenauer The (relevant) references I found are in: hadoop/dev-support/test-patch.sh (Which I understand you are rewriting ... I assume without Forrest) And in all of these documentation files: hadoop/hadoop-common-project/hadoop-common/src/main/docs/status.xml hadoop/hadoop-common-project/hadoop-common/src/main/docs/src/documentation/classes/CatalogManager.properties hadoop/hadoop-common-project/hadoop-common/src/main/docs/src/documentation/README.txt hadoop/hadoop-common-project/hadoop-common/src/main/docs/src/documentation/content/xdocs/index.xml hadoop/hadoop-common-project/hadoop-common/src/main/docs/src/documentation/content/xdocs/site.xml hadoop/hadoop-common-project/hadoop-common/src/main/docs/src/documentation/content/xdocs/tabs.xml hadoop/hadoop-common-project/hadoop-common/src/main/docs/src/documentation/skinconf.xml hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/docs/status.xml hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/docs/src/documentation/classes/CatalogManager.properties hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/docs/src/documentation/README.txt hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/docs/src/documentation/content/xdocs/index.xml hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/docs/src/documentation/content/xdocs/webhdfs.xml hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/docs/src/documentation/content/xdocs/site.xml hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/docs/src/documentation/content/xdocs/tabs.xml hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/docs/src/documentation/skinconf.xml I didn't find any references in pom.xml files so it seems that indeed the references I found are all in 'dead documentation'. If you say "take forrest out" I'll take it out of my patch. In that case I assume you'll create a new Jira to get rid of these obsolete files.
          Hide
          cnauroth Chris Nauroth added a comment -

          I ran into a few problems while testing this. First, I got an "invalid ENV format" error. I was able to fix that by removing the '=' from the ENV line. The next problem I hit was that the volumes mounted from the host showed as owned by root within the container, which made them inaccessible to my logged-in user.

          For both problems, I believe the root cause was that I was running an older version of Docker (1.0.0). I upgraded to 1.6.0, and that solved both problems. Let's document the minimum required version inside BUILDING.txt.

          Otherwise, it looks good. I've run a few mvn commands successfully. I do see that TestNativeIO is failing, perhaps due to some quirk to file permissions handling within the container. I don't think this needs to stop check-in though. I think we can work towards getting this committed, and then address individual test failures as follow-ups.

          I'm testing a few more build commands. I'll report back again with the results, probably later tonight.

          Show
          cnauroth Chris Nauroth added a comment - I ran into a few problems while testing this. First, I got an "invalid ENV format" error. I was able to fix that by removing the '=' from the ENV line. The next problem I hit was that the volumes mounted from the host showed as owned by root within the container, which made them inaccessible to my logged-in user. For both problems, I believe the root cause was that I was running an older version of Docker (1.0.0). I upgraded to 1.6.0, and that solved both problems. Let's document the minimum required version inside BUILDING.txt. Otherwise, it looks good. I've run a few mvn commands successfully. I do see that TestNativeIO is failing, perhaps due to some quirk to file permissions handling within the container. I don't think this needs to stop check-in though. I think we can work towards getting this committed, and then address individual test failures as follow-ups. I'm testing a few more build commands. I'll report back again with the results, probably later tonight.
          Hide
          nielsbasjes Niels Basjes added a comment -

          I looking into verifying a minimal version of Docker.
          I found this http://stackoverflow.com/a/4023881
          Can you guys please test if this works on your Macs ? Thanks.
          Command:

          printf '2.4.5\n2.8\n2.4.5.1\n' | sort -V
          

          Expected output:

          2.4.5
          2.4.5.1
          2.8
          Show
          nielsbasjes Niels Basjes added a comment - I looking into verifying a minimal version of Docker. I found this http://stackoverflow.com/a/4023881 Can you guys please test if this works on your Macs ? Thanks. Command: printf '2.4.5\n2.8\n2.4.5.1\n' | sort -V Expected output: 2.4.5 2.4.5.1 2.8
          Hide
          cnauroth Chris Nauroth added a comment -

          It looks like the -V option is not supported for sort on Mac/BSD:

          > printf '2.4.5\n2.8\n2.4.5.1\n' | sort -V
          sort: invalid option -- V
          Try `sort --help' for more information.
          

          In general, I'm seeing very sluggish performance. Builds take approximately 10x longer to finish than what I'm used to seeing. I left TestMiniMRClientCluster running for hours and it never finished. This might just be a weakness of Docker on Mac, where everything needs to work through the boot2docker VM. I'm currently exploring trying to allocate more RAM to the boot2docker VM and setting the MAVEN_OPTS environment variable with a high -Xmx option.

          I did get a successful (but slow) run of this command, which covers a lot of the build process:

          mvn clean install -Pnative -Pdist -Dtar -DskipTests -Drequire.bzip2 -Drequire.openssl -Drequire.snappy -Drequire.fuse
          

          That's a good sign that the setup is complete as far as getting the required tools.

          Show
          cnauroth Chris Nauroth added a comment - It looks like the -V option is not supported for sort on Mac/BSD: > printf '2.4.5\n2.8\n2.4.5.1\n' | sort -V sort: invalid option -- V Try `sort --help' for more information. In general, I'm seeing very sluggish performance. Builds take approximately 10x longer to finish than what I'm used to seeing. I left TestMiniMRClientCluster running for hours and it never finished. This might just be a weakness of Docker on Mac, where everything needs to work through the boot2docker VM. I'm currently exploring trying to allocate more RAM to the boot2docker VM and setting the MAVEN_OPTS environment variable with a high -Xmx option. I did get a successful (but slow) run of this command, which covers a lot of the build process: mvn clean install -Pnative -Pdist -Dtar -DskipTests -Drequire.bzip2 -Drequire.openssl -Drequire.snappy -Drequire.fuse That's a good sign that the setup is complete as far as getting the required tools.
          Hide
          aw Allen Wittenauer added a comment -

          In general, I'm seeing very sluggish performance.

          Me too. Switching to Oracle's JDK gave a bit of a boost.

          Show
          aw Allen Wittenauer added a comment - In general, I'm seeing very sluggish performance. Me too. Switching to Oracle's JDK gave a bit of a boost.
          Hide
          d4rr3ll Darrell Taylor added a comment -

          I've been using this to get going with HADOOP-11813 and found I needed to add python2.7 into the apt-get package list. releasedocmaker.py was throwing errors with python3 (which appears to be installed by default). It may even need the 'python' packing installed as well because this still doesn't give a a 'python' executable, I have to run 'python2.7'.

          Show
          d4rr3ll Darrell Taylor added a comment - I've been using this to get going with HADOOP-11813 and found I needed to add python2.7 into the apt-get package list. releasedocmaker.py was throwing errors with python3 (which appears to be installed by default). It may even need the 'python' packing installed as well because this still doesn't give a a 'python' executable, I have to run 'python2.7'.
          Hide
          nielsbasjes Niels Basjes added a comment -
          • docker version: I think the easiest and most portable sanity check for the right docker version is to check if you become root when inside the container.
            Good to hear we are getting close to having the right set of tools.
          • jdk: How can I automate the download of the Oracle JDK ? Last time I did that you needed to accept the licence and then there was a (unique looking) download url. I propose to stick to the openjdk, perhaps with some additional settings. What was the memory setting you used for the boot2docker vm? I can check for available memory when entering the container and warn if it is too low.
          • python: What I understand is that it is I should install python 2.7 and create an alias/symlink so you can simply run 'python'. Should python 3 be uninstalled?
          Show
          nielsbasjes Niels Basjes added a comment - docker version: I think the easiest and most portable sanity check for the right docker version is to check if you become root when inside the container. Good to hear we are getting close to having the right set of tools. jdk: How can I automate the download of the Oracle JDK ? Last time I did that you needed to accept the licence and then there was a (unique looking) download url. I propose to stick to the openjdk, perhaps with some additional settings. What was the memory setting you used for the boot2docker vm? I can check for available memory when entering the container and warn if it is too low. python: What I understand is that it is I should install python 2.7 and create an alias/symlink so you can simply run 'python'. Should python 3 be uninstalled?
          Hide
          nielsbasjes Niels Basjes added a comment -

          Can you give me an exact build command and how much time it should take?
          That will allow me to verify if I also see slowness (I don't have any Macs).

          Show
          nielsbasjes Niels Basjes added a comment - Can you give me an exact build command and how much time it should take? That will allow me to verify if I also see slowness (I don't have any Macs).
          Hide
          aw Allen Wittenauer added a comment -

          forrest

          So I did some digging. From what I can tell, all of that stuff under src/docs is basically abandoned. But we can leave forrest in for now simply because the main site is still written in forrest and it might be useful for us committers to have there until we can convert it.

          How can I automate the download of the Oracle JDK ?

          Just change the dockerfile line to be

          FROM dockerfile/java:oracle-java7
          
          time mvn clean site site:stage -DskipTests -Dmaven.javadoc.skip=true  > ${home}/logfile 2>&1
          

          Takes about 2 minutes on my Mac. Via boot2docker, this actually fails after about a minute because it can't find Apache commons. So weird.

          I'm also missing a 'python' in the path, but 'python3' is definitely there.

          Show
          aw Allen Wittenauer added a comment - forrest So I did some digging. From what I can tell, all of that stuff under src/docs is basically abandoned. But we can leave forrest in for now simply because the main site is still written in forrest and it might be useful for us committers to have there until we can convert it. How can I automate the download of the Oracle JDK ? Just change the dockerfile line to be FROM dockerfile/java:oracle-java7 time mvn clean site site:stage -DskipTests -Dmaven.javadoc.skip= true > ${home}/logfile 2>&1 Takes about 2 minutes on my Mac. Via boot2docker, this actually fails after about a minute because it can't find Apache commons. So weird. I'm also missing a 'python' in the path, but 'python3' is definitely there.
          Hide
          d4rr3ll Darrell Taylor added a comment -

          I can confirm that the packages required to make releasedocmaker.py work are :

          # Install dependencies from packages
          RUN apt-get update && apt-get install --no-install-recommends -y \
            git curl ant make maven \
            cmake gcc g++ protobuf-compiler \
            build-essential libtool \
            zlib1g-dev pkg-config libssl-dev \
            snappy libsnappy-dev \
            bzip2 libbz2-dev \
            libjansson-dev \
            fuse libfuse-dev \
            libcurl4-openssl-dev \
            python python2.7
          

          This does leave python3 in there, but just running 'python' now defaults to the 2.7 version.

          Show
          d4rr3ll Darrell Taylor added a comment - I can confirm that the packages required to make releasedocmaker.py work are : # Install dependencies from packages RUN apt-get update && apt-get install --no-install-recommends -y \ git curl ant make maven \ cmake gcc g++ protobuf-compiler \ build-essential libtool \ zlib1g-dev pkg-config libssl-dev \ snappy libsnappy-dev \ bzip2 libbz2-dev \ libjansson-dev \ fuse libfuse-dev \ libcurl4-openssl-dev \ python python2.7 This does leave python3 in there, but just running 'python' now defaults to the 2.7 version.
          Hide
          nielsbasjes Niels Basjes added a comment -

          Changes:

          • Changed to Oracle JDK. Left the OpenJDK command as a comment in there to allow easy changing back if a Dev wants to.
          • Fixed the ENV '=' warning
          • Added python dependencies
          • Replaced the motd with a set of environment checks: No root user and more than 2 GiB memory (Is this the right value?)
          • Added note about docker version to the BUILDING.TXT. I had it working with docker 1.4.1.

          I ran inside the latest container in my CentOS 6.6 Virtual (16GB RAM, VirtualBox on Windows, Spinning disk) these commands:

          Build and install (Same command as Chris ran):

          time mvn clean install -Pnative -Pdist -Dtar -DskipTests -Drequire.bzip2 -Drequire.openssl -Drequire.snappy -Drequire.fuse

          The output

          [INFO] ------------------------------------------------------------------------
          [INFO] BUILD SUCCESS
          [INFO] ------------------------------------------------------------------------
          [INFO] Total time: 12:35.202s
          [INFO] Finished at: Sun Apr 19 20:03:52 UTC 2015
          [INFO] Final Memory: 238M/1586M
          [INFO] ------------------------------------------------------------------------
          
          real	12m37.851s
          user	14m24.508s
          sys	3m21.735s
          

          Allen's command:

          time mvn clean site site:stage -DskipTests -Dmaven.javadoc.skip=true > logfile2 2>&1

          Also fails on my system with an error relating to apache commons.

          [ERROR] Failed to execute goal org.apache.maven.plugins:maven-site-plugin:3.4:stage (default-cli) on project hadoop-main: Execution default-cli of goal org.apache.maven.plugins:maven-site-plugin:3.4:stage failed: A required class was missing while executing org.apache.maven.plugins:maven-site-plugin:3.4:stage: org/apache/commons/lang/StringUtils
          Show
          nielsbasjes Niels Basjes added a comment - Changes: Changed to Oracle JDK. Left the OpenJDK command as a comment in there to allow easy changing back if a Dev wants to. Fixed the ENV '=' warning Added python dependencies Replaced the motd with a set of environment checks: No root user and more than 2 GiB memory (Is this the right value?) Added note about docker version to the BUILDING.TXT. I had it working with docker 1.4.1. I ran inside the latest container in my CentOS 6.6 Virtual (16GB RAM, VirtualBox on Windows, Spinning disk) these commands: Build and install (Same command as Chris ran): time mvn clean install -Pnative -Pdist -Dtar -DskipTests -Drequire.bzip2 -Drequire.openssl -Drequire.snappy -Drequire.fuse The output [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 12:35.202s [INFO] Finished at: Sun Apr 19 20:03:52 UTC 2015 [INFO] Final Memory: 238M/1586M [INFO] ------------------------------------------------------------------------ real 12m37.851s user 14m24.508s sys 3m21.735s Allen's command: time mvn clean site site:stage -DskipTests -Dmaven.javadoc.skip= true > logfile2 2>&1 Also fails on my system with an error relating to apache commons. [ERROR] Failed to execute goal org.apache.maven.plugins:maven-site-plugin:3.4:stage ( default -cli) on project hadoop-main: Execution default -cli of goal org.apache.maven.plugins:maven-site-plugin:3.4:stage failed: A required class was missing while executing org.apache.maven.plugins:maven-site-plugin:3.4:stage: org/apache/commons/lang/StringUtils
          Hide
          nielsbasjes Niels Basjes added a comment -

          Fixing commons problem ...

          Show
          nielsbasjes Niels Basjes added a comment - Fixing commons problem ...
          Hide
          nielsbasjes Niels Basjes added a comment -

          Cause of the commons problem:
          http://wiki.apache.org/commons/VfsProblems

          Show
          nielsbasjes Niels Basjes added a comment - Cause of the commons problem: http://wiki.apache.org/commons/VfsProblems
          Hide
          nielsbasjes Niels Basjes added a comment -
          mvn clean site site:stage -DskipTests -Dmaven.javadoc.skip=true  

          Results in

          [INFO] ------------------------------------------------------------------------
          [INFO] BUILD SUCCESS
          [INFO] ------------------------------------------------------------------------
          [INFO] Total time: 2:25.106s
          [INFO] Finished at: Sun Apr 19 20:33:09 UTC 2015
          [INFO] Final Memory: 272M/1699M
          [INFO] ------------------------------------------------------------------------
          
          Show
          nielsbasjes Niels Basjes added a comment - mvn clean site site:stage -DskipTests -Dmaven.javadoc.skip= true Results in [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 2:25.106s [INFO] Finished at: Sun Apr 19 20:33:09 UTC 2015 [INFO] Final Memory: 272M/1699M [INFO] ------------------------------------------------------------------------
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12726466/HADOOP-11843-2015-04-19-2206.patch
          against trunk revision 5459b24.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/6124//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/6124//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12726466/HADOOP-11843-2015-04-19-2206.patch against trunk revision 5459b24. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/6124//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/6124//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12726468/HADOOP-11843-2015-04-19-2232.patch
          against trunk revision 5459b24.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/6125//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/6125//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12726468/HADOOP-11843-2015-04-19-2232.patch against trunk revision 5459b24. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/6125//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/6125//console This message is automatically generated.
          Hide
          d4rr3ll Darrell Taylor added a comment -

          Some food for thought. I just wanted to run

          dev-support/test-patch.sh /path/to/my.patch
          

          As recommended on the wiki (http://wiki.apache.org/hadoop/HowToContribute)

          But I have my patch outside of the source directory that is 'mounted' by the container (i.e. ~/hadoop) so can't access the .patch file from within the container. I could copy the file into the ~/hadoop directory I guess but that feels a little bit dirty. Any recommendations on what to do here?

          Show
          d4rr3ll Darrell Taylor added a comment - Some food for thought. I just wanted to run dev-support/test-patch.sh /path/to/my.patch As recommended on the wiki ( http://wiki.apache.org/hadoop/HowToContribute ) But I have my patch outside of the source directory that is 'mounted' by the container (i.e. ~/hadoop) so can't access the .patch file from within the container. I could copy the file into the ~/hadoop directory I guess but that feels a little bit dirty. Any recommendations on what to do here?
          Hide
          cnauroth Chris Nauroth added a comment -

          After some digging, I found that Docker has a known issue with poor performance of volume mounts on Mac/boot2docker:

          https://github.com/boot2docker/boot2docker/issues/593

          This issue has been resolved as a duplicate, and they point to a new feature for utilizing NFS mounts as the proposed solution:

          https://github.com/boot2docker/boot2docker/issues/64

          This issue is still open.

          As it stands, I don't believe this is usable on Mac in its current form. I just retested to check the performance boost reported by Allen when using the Oracle JDK, but I still don't think the performance is anywhere near acceptable. That's unfortunate, since a lot of developers like Macs.

          Something else I can try is "real" Docker (not boot2docker) running in a Linux VM hosted by VirtualBox. I do get acceptable disk performance from my VirtualBox VMs, so maybe one extra layer of indirection from Docker in there won't hurt.

          Show
          cnauroth Chris Nauroth added a comment - After some digging, I found that Docker has a known issue with poor performance of volume mounts on Mac/boot2docker: https://github.com/boot2docker/boot2docker/issues/593 This issue has been resolved as a duplicate, and they point to a new feature for utilizing NFS mounts as the proposed solution: https://github.com/boot2docker/boot2docker/issues/64 This issue is still open. As it stands, I don't believe this is usable on Mac in its current form. I just retested to check the performance boost reported by Allen when using the Oracle JDK, but I still don't think the performance is anywhere near acceptable. That's unfortunate, since a lot of developers like Macs. Something else I can try is "real" Docker (not boot2docker) running in a Linux VM hosted by VirtualBox. I do get acceptable disk performance from my VirtualBox VMs, so maybe one extra layer of indirection from Docker in there won't hurt.
          Hide
          nielsbasjes Niels Basjes added a comment -

          I don't have a Mac so I'm unable to develop/test this boot2docker/NFS solution.
          At home I run everything as Windows-7(VirtualBox(CentOS-6.6(Docker( HADOOP-11843 )))) and I run IntelliJ inside the same CentOS which performs fine.
          At work I have a system that runs CentOS natively on the hardware which also performs fine.

          Although being a real pain I propose I simply add some documentation in the BUILDING.TXT pointing to these workarounds for the time these problems exist.

          Show
          nielsbasjes Niels Basjes added a comment - I don't have a Mac so I'm unable to develop/test this boot2docker/NFS solution. At home I run everything as Windows-7(VirtualBox(CentOS-6.6(Docker( HADOOP-11843 )))) and I run IntelliJ inside the same CentOS which performs fine. At work I have a system that runs CentOS natively on the hardware which also performs fine. Although being a real pain I propose I simply add some documentation in the BUILDING.TXT pointing to these workarounds for the time these problems exist.
          Hide
          arpitagarwal Arpit Agarwal added a comment - - edited

          This is a great idea. Got the container started on Mac with the following steps.

          $ brew install docker boot2docker
          $ boot2docker init -m 4096
          $ boot2docker start
          $ $(boot2docker shellinit)
          $ ./start-build-env.sh
          

          As Chris already pointed out the performance with boot2docker makes it impractical. For now we can go with Niels's recommendation of a linux VM. I often build in Centos/Ubuntu VMs on OS X and the performance is not significantly worse than building natively.

          Show
          arpitagarwal Arpit Agarwal added a comment - - edited This is a great idea. Got the container started on Mac with the following steps. $ brew install docker boot2docker $ boot2docker init -m 4096 $ boot2docker start $ $(boot2docker shellinit) $ ./start-build-env.sh As Chris already pointed out the performance with boot2docker makes it impractical. For now we can go with Niels's recommendation of a linux VM. I often build in Centos/Ubuntu VMs on OS X and the performance is not significantly worse than building natively.
          Hide
          aw Allen Wittenauer added a comment -

          I could copy the file into the ~/hadoop directory I guess but that feels a little bit dirty. Any recommendations on what to do here?

          That's exactly what I've been doing + using the newer test-patch.sh (HADOOP-11746) which supports HTTP URLs to test currently uploaded patches.

          Show
          aw Allen Wittenauer added a comment - I could copy the file into the ~/hadoop directory I guess but that feels a little bit dirty. Any recommendations on what to do here? That's exactly what I've been doing + using the newer test-patch.sh ( HADOOP-11746 ) which supports HTTP URLs to test currently uploaded patches.
          Hide
          nielsbasjes Niels Basjes added a comment -
          • Added additional documentation about the Mac/Boot2docker problems and possible solutions.
          • Minor code layout fixes (indentation, empty lines)
          Show
          nielsbasjes Niels Basjes added a comment - Added additional documentation about the Mac/Boot2docker problems and possible solutions. Minor code layout fixes (indentation, empty lines)
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 0m 0s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 release audit 0m 15s The applied patch does not increase the total number of release audit warnings.
          +1 shellcheck 0m 0s There were no new shellcheck (v) issues.
              0m 17s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12727228/HADOOP-11843-2015-04-22-1122.patch
          Optional Tests shellcheck
          git revision trunk / b08908a
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6147//console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 0m 0s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 release audit 0m 15s The applied patch does not increase the total number of release audit warnings. +1 shellcheck 0m 0s There were no new shellcheck (v) issues.     0m 17s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12727228/HADOOP-11843-2015-04-22-1122.patch Optional Tests shellcheck git revision trunk / b08908a Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6147//console This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          I tried running Docker in an Ubuntu VM with 4 GB RAM. For the full build command I described earlier, I got a runtime of ~16 minutes. I think that's acceptable.

          I tried TestMiniMRClientCluster again, and the test timed out. This is another manifestation of a problem we've consistently had with configuring test timeout values. It's very hard to predict the performance of the build environment, so setting a static timeout value is unlikely to work perfectly for everyone. I don't think this is solvable in the scope of this patch. We just might find that certain tests aren't working well in the Docker container, and we can improve on that over time.

          I have one more suggestion. Please set the MAVEN_OPTS environment variable to allocate a larger heap than the default to mvn processes. Here is what we document in BUILDING.txt:

          export MAVEN_OPTS="-Xms256m -Xmx512m"
          

          I'll be +1 after that small change. Thanks so much for your work on this, Nils!

          Show
          cnauroth Chris Nauroth added a comment - I tried running Docker in an Ubuntu VM with 4 GB RAM. For the full build command I described earlier, I got a runtime of ~16 minutes. I think that's acceptable. I tried TestMiniMRClientCluster again, and the test timed out. This is another manifestation of a problem we've consistently had with configuring test timeout values. It's very hard to predict the performance of the build environment, so setting a static timeout value is unlikely to work perfectly for everyone. I don't think this is solvable in the scope of this patch. We just might find that certain tests aren't working well in the Docker container, and we can improve on that over time. I have one more suggestion. Please set the MAVEN_OPTS environment variable to allocate a larger heap than the default to mvn processes. Here is what we document in BUILDING.txt: export MAVEN_OPTS= "-Xms256m -Xmx512m" I'll be +1 after that small change. Thanks so much for your work on this, Nils!
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Hi Niels, re Mac instructions - brew i.e. the Homebrew package manager is not installed by default on OS X. Perhaps adding a link or note would avoid confusion.

          Show
          arpitagarwal Arpit Agarwal added a comment - Hi Niels, re Mac instructions - brew i.e. the Homebrew package manager is not installed by default on OS X. Perhaps adding a link or note would avoid confusion.
          Hide
          nielsbasjes Niels Basjes added a comment -

          Added :

          • "First make sure Homebrew has been installed ( http://brew.sh/ )"
          • ENV MAVEN_OPTS -Xms256m -Xmx512m
          Show
          nielsbasjes Niels Basjes added a comment - Added : "First make sure Homebrew has been installed ( http://brew.sh/ )" ENV MAVEN_OPTS -Xms256m -Xmx512m
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 0m 0s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 release audit 0m 15s The applied patch does not increase the total number of release audit warnings.
          0 shellcheck 0m 15s Shellcheck was not available.
              0m 23s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12727559/HADOOP-11843-2015-04-23-1000.patch
          Optional Tests shellcheck
          git revision trunk / d9bcf99
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6165/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 0m 0s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 release audit 0m 15s The applied patch does not increase the total number of release audit warnings. 0 shellcheck 0m 15s Shellcheck was not available.     0m 23s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12727559/HADOOP-11843-2015-04-23-1000.patch Optional Tests shellcheck git revision trunk / d9bcf99 Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6165/console This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          +1 for the latest patch. Thanks again, Nils. I'll hold off committing this until tomorrow in case there is any remaining feedback from the other watchers.

          Show
          cnauroth Chris Nauroth added a comment - +1 for the latest patch. Thanks again, Nils. I'll hold off committing this until tomorrow in case there is any remaining feedback from the other watchers.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          +1 from me too. I was able to get an environment on an Ubuntu 14 VM and build with mvn install package -Pnative. This is very cool!

          Show
          arpitagarwal Arpit Agarwal added a comment - +1 from me too. I was able to get an environment on an Ubuntu 14 VM and build with mvn install package -Pnative . This is very cool!
          Hide
          cnauroth Chris Nauroth added a comment -

          I have committed this to trunk and branch-2. Niels Basjes, thank you for the helpful contribution. Allen, Arpit and Darrell, thank you for your testing and code reviews.

          Show
          cnauroth Chris Nauroth added a comment - I have committed this to trunk and branch-2. Niels Basjes , thank you for the helpful contribution. Allen, Arpit and Darrell, thank you for your testing and code reviews.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7667 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7667/)
          HADOOP-11843. Make setting up the build environment easier. Contributed by Niels Basjes. (cnauroth: rev 80935268f5fdd358070c6b68f89e8bd699785c54)

          • dev-support/docker/hadoop_env_checks.sh
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • BUILDING.txt
          • dev-support/docker/Dockerfile
          • start-build-env.sh
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7667 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7667/ ) HADOOP-11843 . Make setting up the build environment easier. Contributed by Niels Basjes. (cnauroth: rev 80935268f5fdd358070c6b68f89e8bd699785c54) dev-support/docker/hadoop_env_checks.sh hadoop-common-project/hadoop-common/CHANGES.txt BUILDING.txt dev-support/docker/Dockerfile start-build-env.sh
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #174 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/174/)
          HADOOP-11843. Make setting up the build environment easier. Contributed by Niels Basjes. (cnauroth: rev 80935268f5fdd358070c6b68f89e8bd699785c54)

          • dev-support/docker/Dockerfile
          • start-build-env.sh
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • BUILDING.txt
          • dev-support/docker/hadoop_env_checks.sh
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #174 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/174/ ) HADOOP-11843 . Make setting up the build environment easier. Contributed by Niels Basjes. (cnauroth: rev 80935268f5fdd358070c6b68f89e8bd699785c54) dev-support/docker/Dockerfile start-build-env.sh hadoop-common-project/hadoop-common/CHANGES.txt BUILDING.txt dev-support/docker/hadoop_env_checks.sh
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #908 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/908/)
          HADOOP-11843. Make setting up the build environment easier. Contributed by Niels Basjes. (cnauroth: rev 80935268f5fdd358070c6b68f89e8bd699785c54)

          • BUILDING.txt
          • start-build-env.sh
          • dev-support/docker/hadoop_env_checks.sh
          • dev-support/docker/Dockerfile
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #908 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/908/ ) HADOOP-11843 . Make setting up the build environment easier. Contributed by Niels Basjes. (cnauroth: rev 80935268f5fdd358070c6b68f89e8bd699785c54) BUILDING.txt start-build-env.sh dev-support/docker/hadoop_env_checks.sh dev-support/docker/Dockerfile hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2106 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2106/)
          HADOOP-11843. Make setting up the build environment easier. Contributed by Niels Basjes. (cnauroth: rev 80935268f5fdd358070c6b68f89e8bd699785c54)

          • start-build-env.sh
          • dev-support/docker/hadoop_env_checks.sh
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • BUILDING.txt
          • dev-support/docker/Dockerfile
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2106 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2106/ ) HADOOP-11843 . Make setting up the build environment easier. Contributed by Niels Basjes. (cnauroth: rev 80935268f5fdd358070c6b68f89e8bd699785c54) start-build-env.sh dev-support/docker/hadoop_env_checks.sh hadoop-common-project/hadoop-common/CHANGES.txt BUILDING.txt dev-support/docker/Dockerfile
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #165 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/165/)
          HADOOP-11843. Make setting up the build environment easier. Contributed by Niels Basjes. (cnauroth: rev 80935268f5fdd358070c6b68f89e8bd699785c54)

          • start-build-env.sh
          • dev-support/docker/hadoop_env_checks.sh
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • dev-support/docker/Dockerfile
          • BUILDING.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #165 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/165/ ) HADOOP-11843 . Make setting up the build environment easier. Contributed by Niels Basjes. (cnauroth: rev 80935268f5fdd358070c6b68f89e8bd699785c54) start-build-env.sh dev-support/docker/hadoop_env_checks.sh hadoop-common-project/hadoop-common/CHANGES.txt dev-support/docker/Dockerfile BUILDING.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #175 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/175/)
          HADOOP-11843. Make setting up the build environment easier. Contributed by Niels Basjes. (cnauroth: rev 80935268f5fdd358070c6b68f89e8bd699785c54)

          • dev-support/docker/Dockerfile
          • start-build-env.sh
          • dev-support/docker/hadoop_env_checks.sh
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • BUILDING.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #175 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/175/ ) HADOOP-11843 . Make setting up the build environment easier. Contributed by Niels Basjes. (cnauroth: rev 80935268f5fdd358070c6b68f89e8bd699785c54) dev-support/docker/Dockerfile start-build-env.sh dev-support/docker/hadoop_env_checks.sh hadoop-common-project/hadoop-common/CHANGES.txt BUILDING.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2124 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2124/)
          HADOOP-11843. Make setting up the build environment easier. Contributed by Niels Basjes. (cnauroth: rev 80935268f5fdd358070c6b68f89e8bd699785c54)

          • dev-support/docker/hadoop_env_checks.sh
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • dev-support/docker/Dockerfile
          • start-build-env.sh
          • BUILDING.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2124 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2124/ ) HADOOP-11843 . Make setting up the build environment easier. Contributed by Niels Basjes. (cnauroth: rev 80935268f5fdd358070c6b68f89e8bd699785c54) dev-support/docker/hadoop_env_checks.sh hadoop-common-project/hadoop-common/CHANGES.txt dev-support/docker/Dockerfile start-build-env.sh BUILDING.txt

            People

            • Assignee:
              nielsbasjes Niels Basjes
              Reporter:
              nielsbasjes Niels Basjes
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development