Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1582

Remove auto-generated native build files

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.22.0, 0.23.0
    • Component/s: libhdfs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      The native build run when from trunk now requires autotools, libtool and openssl dev libraries.

      Description

      The repo currently includes the automake and autoconf generated files for the native build. Per discussion on HADOOP-6421 let's remove them and use the host's automake and autoconf. We should also do this for libhdfs and fuse-dfs.

      1. HDFS-1582.diff
        35 kB
        Roman Shaposhnik
      2. HADOOP-6436.patch
        1.12 MB
        Roman Shaposhnik

        Issue Links

          Activity

          Roman Shaposhnik created issue -
          Roman Shaposhnik made changes -
          Field Original Value New Value
          Link This issue is part of HADOOP-6436 [ HADOOP-6436 ]
          Hide
          Roman Shaposhnik added a comment -

          I'm attaching a first cut at the patch. Please let me know what do you think about the approach taken. I've incorporate the feedback from HADOOP-6436

          Show
          Roman Shaposhnik added a comment - I'm attaching a first cut at the patch. Please let me know what do you think about the approach taken. I've incorporate the feedback from HADOOP-6436
          Roman Shaposhnik made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Fix Version/s 0.23.0 [ 12315571 ]
          Roman Shaposhnik made changes -
          Attachment HADOOP-6436.patch [ 12468203 ]
          Hide
          Eli Collins added a comment -

          Thanks for taking this on! Approach looks good.

          Couple things:

          • Why modify the package target to depend on the libhdfs build?
          • Testing? I get the following when running ant -Dcompile.c++=true -Dlibhdfs=true compile and this patch on a fresh tree.
            create-libhdfs-configure:
                [mkdir] Created dir: /home/eli/src/hdfs4/build/c++-build/Linux-amd64-64/libhdfs
                 [exec] configure: error: cannot find install-sh, install.sh, or shtool in config "/home/eli/src/hdfs4/src/c++/libhdfs"/config
            
          Show
          Eli Collins added a comment - Thanks for taking this on! Approach looks good. Couple things: Why modify the package target to depend on the libhdfs build? Testing? I get the following when running ant -Dcompile.c++=true -Dlibhdfs=true compile and this patch on a fresh tree. create-libhdfs-configure: [mkdir] Created dir: /home/eli/src/hdfs4/build/c++-build/Linux-amd64-64/libhdfs [exec] configure: error: cannot find install-sh, install.sh, or shtool in config "/home/eli/src/hdfs4/src/c++/libhdfs"/config
          Hide
          Eli Collins added a comment -

          Also worth noting is that the build now depends on libtool.

          Show
          Eli Collins added a comment - Also worth noting is that the build now depends on libtool.
          Hide
          Roman Shaposhnik added a comment -

          Eli,

          Why modify the package target to depend on the libhdfs build?

          The logic here is that we want those files to end up in the tarball before a release. As was noted by Todd on the original issue we want those files to be bundled for platforms that lack autotools.

          Also worth noting is that the build now depends on libtool.

          It is not libtool. The build now depends on the entire autoconf (since it uses autoreconf). Now, do you want me to handle
          that dependency somehow? Code? Docs?

          Testing? I get the following when running ant -Dcompile.c++=true -Dlibhdfs=true compile and this patch on a fresh tree.

          See above – project now depend on autoconf and it seems like you don't have it installed. Here's what I get:

          $ svn co http://svn.apache.org/repos/asf/hadoop/hdfs/trunk HDFS
          $ cd HDFS
          $ wget https://issues.apache.org/jira/secure/attachment/12468203/HADOOP-6436.patch
          $ patch -p0 < HADOOP-6436.patch
          $ ant -Dcompile.c++=true -Dlibhdfs=true compile
          ....
          BUILD SUCCESSFUL
          Total time: 57 seconds

          Are you talking about a graceful message when autoreconf is not present?

          Show
          Roman Shaposhnik added a comment - Eli, Why modify the package target to depend on the libhdfs build? The logic here is that we want those files to end up in the tarball before a release. As was noted by Todd on the original issue we want those files to be bundled for platforms that lack autotools. Also worth noting is that the build now depends on libtool. It is not libtool. The build now depends on the entire autoconf (since it uses autoreconf). Now, do you want me to handle that dependency somehow? Code? Docs? Testing? I get the following when running ant -Dcompile.c++=true -Dlibhdfs=true compile and this patch on a fresh tree. See above – project now depend on autoconf and it seems like you don't have it installed. Here's what I get: $ svn co http://svn.apache.org/repos/asf/hadoop/hdfs/trunk HDFS $ cd HDFS $ wget https://issues.apache.org/jira/secure/attachment/12468203/HADOOP-6436.patch $ patch -p0 < HADOOP-6436 .patch $ ant -Dcompile.c++=true -Dlibhdfs=true compile .... BUILD SUCCESSFUL Total time: 57 seconds Are you talking about a graceful message when autoreconf is not present?
          Hide
          Eli Collins added a comment -

          The logic here is that we want those files to end up in the tarball before a release. As was noted by Todd on the original issue we want those files to be bundled for platforms that lack autotools.

          Makes sense.

          It is not libtool. The build now depends on the entire autoconf.

          Yes, both autoconf and libtool (which autoconf does not depend on). On trunk my host (ubuntu 10.10) with autoconf but not libtool installed I could compile libhdfs, I had to install libtool after applying this patch. Wrt the new dependencies we just need to update the release notes field on this jira indicating they add this dependency.

          Are you talking about a graceful message when autoreconf is not present?

          Nope, I have autoconf (v 2.67) and this libhdfs build fails. From the message below it looks like it's looking for install-sh in the wrong directory, it exists in src/c++/libhdfs, but the path below looks odd, eg the quoting is off and there's no config directory.

                [exec] configure: error: cannot find install-sh, install.sh, or shtool in config "/home/eli/src/hdfs4/src/c++/libhdfs"/config
          
          Show
          Eli Collins added a comment - The logic here is that we want those files to end up in the tarball before a release. As was noted by Todd on the original issue we want those files to be bundled for platforms that lack autotools. Makes sense. It is not libtool. The build now depends on the entire autoconf. Yes, both autoconf and libtool (which autoconf does not depend on). On trunk my host (ubuntu 10.10) with autoconf but not libtool installed I could compile libhdfs, I had to install libtool after applying this patch. Wrt the new dependencies we just need to update the release notes field on this jira indicating they add this dependency. Are you talking about a graceful message when autoreconf is not present? Nope, I have autoconf (v 2.67) and this libhdfs build fails. From the message below it looks like it's looking for install-sh in the wrong directory, it exists in src/c++/libhdfs, but the path below looks odd, eg the quoting is off and there's no config directory. [exec] configure: error: cannot find install-sh, install.sh, or shtool in config "/home/eli/src/hdfs4/src/c++/libhdfs"/config
          Hide
          Eli Collins added a comment -

          Forgot to mention, on trunk w/o the patch I applied I can compile libhdfs.

          Show
          Eli Collins added a comment - Forgot to mention, on trunk w/o the patch I applied I can compile libhdfs.
          Hide
          Eli Collins added a comment -

          Hey Roman,

          My bad, looks like this error was due to a partially updated libtool package on my system, after rebooting the build with your patch it compiles as expected.

          Thanks,
          Eli

          Show
          Eli Collins added a comment - Hey Roman, My bad, looks like this error was due to a partially updated libtool package on my system, after rebooting the build with your patch it compiles as expected. Thanks, Eli
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12468203/HADOOP-6436.patch
          against trunk revision 1057414.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.fs.permission.TestStickyBit
          org.apache.hadoop.hdfs.security.TestDelegationToken
          org.apache.hadoop.hdfs.server.common.TestDistributedUpgrade
          org.apache.hadoop.hdfs.server.datanode.TestBlockReport
          org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
          org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
          org.apache.hadoop.hdfs.server.namenode.TestBackupNode
          org.apache.hadoop.hdfs.server.namenode.TestBlocksWithNotEnoughRacks
          org.apache.hadoop.hdfs.server.namenode.TestBlockTokenWithDFS
          org.apache.hadoop.hdfs.server.namenode.TestCheckpoint
          org.apache.hadoop.hdfs.server.namenode.TestFsck
          org.apache.hadoop.hdfs.server.namenode.TestNameEditsConfigs
          org.apache.hadoop.hdfs.server.namenode.TestStorageRestore
          org.apache.hadoop.hdfs.TestCrcCorruption
          org.apache.hadoop.hdfs.TestDatanodeBlockScanner
          org.apache.hadoop.hdfs.TestDatanodeDeath
          org.apache.hadoop.hdfs.TestDFSClientRetries
          org.apache.hadoop.hdfs.TestDFSFinalize
          org.apache.hadoop.hdfs.TestDFSRollback
          org.apache.hadoop.hdfs.TestDFSShell
          org.apache.hadoop.hdfs.TestDFSStartupVersions
          org.apache.hadoop.hdfs.TestDFSStorageStateRecovery
          org.apache.hadoop.hdfs.TestDFSUpgradeFromImage
          org.apache.hadoop.hdfs.TestDFSUpgrade
          org.apache.hadoop.hdfs.TestDistributedFileSystem
          org.apache.hadoop.hdfs.TestFileAppend2
          org.apache.hadoop.hdfs.TestFileAppend3
          org.apache.hadoop.hdfs.TestFileAppend4
          org.apache.hadoop.hdfs.TestFileAppend
          org.apache.hadoop.hdfs.TestFileConcurrentReader
          org.apache.hadoop.hdfs.TestFileCreationNamenodeRestart
          org.apache.hadoop.hdfs.TestFileCreation
          org.apache.hadoop.hdfs.TestHDFSFileSystemContract
          org.apache.hadoop.hdfs.TestHDFSTrash
          org.apache.hadoop.hdfs.TestPread
          org.apache.hadoop.hdfs.TestQuota
          org.apache.hadoop.hdfs.TestReplication
          org.apache.hadoop.hdfs.TestRestartDFS
          org.apache.hadoop.hdfs.TestSetrepDecreasing
          org.apache.hadoop.hdfs.TestSetrepIncreasing
          org.apache.hadoop.hdfs.TestWriteConfigurationToDFS

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/101//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/101//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/101//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12468203/HADOOP-6436.patch against trunk revision 1057414. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.fs.permission.TestStickyBit org.apache.hadoop.hdfs.security.TestDelegationToken org.apache.hadoop.hdfs.server.common.TestDistributedUpgrade org.apache.hadoop.hdfs.server.datanode.TestBlockReport org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics org.apache.hadoop.hdfs.server.namenode.TestBackupNode org.apache.hadoop.hdfs.server.namenode.TestBlocksWithNotEnoughRacks org.apache.hadoop.hdfs.server.namenode.TestBlockTokenWithDFS org.apache.hadoop.hdfs.server.namenode.TestCheckpoint org.apache.hadoop.hdfs.server.namenode.TestFsck org.apache.hadoop.hdfs.server.namenode.TestNameEditsConfigs org.apache.hadoop.hdfs.server.namenode.TestStorageRestore org.apache.hadoop.hdfs.TestCrcCorruption org.apache.hadoop.hdfs.TestDatanodeBlockScanner org.apache.hadoop.hdfs.TestDatanodeDeath org.apache.hadoop.hdfs.TestDFSClientRetries org.apache.hadoop.hdfs.TestDFSFinalize org.apache.hadoop.hdfs.TestDFSRollback org.apache.hadoop.hdfs.TestDFSShell org.apache.hadoop.hdfs.TestDFSStartupVersions org.apache.hadoop.hdfs.TestDFSStorageStateRecovery org.apache.hadoop.hdfs.TestDFSUpgradeFromImage org.apache.hadoop.hdfs.TestDFSUpgrade org.apache.hadoop.hdfs.TestDistributedFileSystem org.apache.hadoop.hdfs.TestFileAppend2 org.apache.hadoop.hdfs.TestFileAppend3 org.apache.hadoop.hdfs.TestFileAppend4 org.apache.hadoop.hdfs.TestFileAppend org.apache.hadoop.hdfs.TestFileConcurrentReader org.apache.hadoop.hdfs.TestFileCreationNamenodeRestart org.apache.hadoop.hdfs.TestFileCreation org.apache.hadoop.hdfs.TestHDFSFileSystemContract org.apache.hadoop.hdfs.TestHDFSTrash org.apache.hadoop.hdfs.TestPread org.apache.hadoop.hdfs.TestQuota org.apache.hadoop.hdfs.TestReplication org.apache.hadoop.hdfs.TestRestartDFS org.apache.hadoop.hdfs.TestSetrepDecreasing org.apache.hadoop.hdfs.TestSetrepIncreasing org.apache.hadoop.hdfs.TestWriteConfigurationToDFS -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/101//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/101//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/101//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12468203/HADOOP-6436.patch
          against trunk revision 1059508.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.hdfs.server.namenode.TestStorageRestore

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/115//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/115//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/115//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12468203/HADOOP-6436.patch against trunk revision 1059508. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.server.namenode.TestStorageRestore -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/115//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/115//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/115//console This message is automatically generated.
          Hide
          Konstantin Boudnik added a comment -

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.hdfs.server.namenode.TestStorageRestore

          Show
          Konstantin Boudnik added a comment - -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.server.namenode.TestStorageRestore
          Hide
          Konstantin Boudnik added a comment -

          too fast with my buttons

          What I meant to say is that the test failure is well known and documented in https://issues.apache.org/jira/browse/HDFS-1496

          Show
          Konstantin Boudnik added a comment - too fast with my buttons What I meant to say is that the test failure is well known and documented in https://issues.apache.org/jira/browse/HDFS-1496
          Roman Shaposhnik made changes -
          Assignee Roman Shaposhnik [ rvs ]
          Hide
          Eli Collins added a comment -

          Patch looks good. What testing has been done to check the native part of the build, eg run libhdfs or fuse-dfs?

          Show
          Eli Collins added a comment - Patch looks good. What testing has been done to check the native part of the build, eg run libhdfs or fuse-dfs?
          Hide
          Roman Shaposhnik added a comment -

          testing for libhdfs was manually running hdfs_test on a Linux box. hope this suffices since fuse-dfs seems to be broken ATM.

          Show
          Roman Shaposhnik added a comment - testing for libhdfs was manually running hdfs_test on a Linux box. hope this suffices since fuse-dfs seems to be broken ATM.
          Hide
          Eli Collins added a comment -

          +1

          I verified ant -Dcompile.c++=true -Dlibhdfs=true compile builds correctly on 64-bit Centos 5 and 64-bit Ubuntu Maverick. And that ant create-libhdfs-configure -Dneed.libhdfs.configure=true can be used to regenerate the configure file.

          I will commit this to trunk and branch 22. Thanks Roman!

          Show
          Eli Collins added a comment - +1 I verified ant -Dcompile.c++=true -Dlibhdfs=true compile builds correctly on 64-bit Centos 5 and 64-bit Ubuntu Maverick. And that ant create-libhdfs-configure -Dneed.libhdfs.configure=true can be used to regenerate the configure file. I will commit this to trunk and branch 22. Thanks Roman!
          Eli Collins made changes -
          Hadoop Flags [Reviewed]
          Release Note The native build run when from trunk now requires autotools, libtool and openssl dev libraries.
          Eli Collins made changes -
          Fix Version/s 0.22.0 [ 12314241 ]
          Eli Collins made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Eli Collins added a comment -

          I've committed this.

          Show
          Eli Collins added a comment - I've committed this.
          Eli Collins made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Roman Shaposhnik made changes -
          Attachment HDFS-1582.diff [ 12470647 ]
          Hide
          Roman Shaposhnik added a comment -

          Thanks for committing the patch! It looks like one file was accidentally left in place. src/c++/libhdfs/aclocal.m4 was removed in the original patch but it is still present in trunk. I'm attaching a patch for your convenience that gets rid of it.

          Show
          Roman Shaposhnik added a comment - Thanks for committing the patch! It looks like one file was accidentally left in place. src/c++/libhdfs/aclocal.m4 was removed in the original patch but it is still present in trunk. I'm attaching a patch for your convenience that gets rid of it.
          Hide
          Eli Collins added a comment -

          +1

          Thanks Roman, I'll commit this to trunk and branch 22.

          Show
          Eli Collins added a comment - +1 Thanks Roman, I'll commit this to trunk and branch 22.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #539 (See https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/539/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #539 (See https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/539/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #643 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/643/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #643 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/643/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-22-branch #35 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-22-branch/35/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-22-branch #35 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-22-branch/35/ )

            People

            • Assignee:
              Roman Shaposhnik
              Reporter:
              Roman Shaposhnik
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 24h
                24h
                Remaining:
                Remaining Estimate - 24h
                24h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development