Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-423

Unbreak FUSE build and fuse_dfs_wrapper.sh

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: fuse-dfs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      fuse-dfs depends on libhdfs, and fuse-dfs build.xml still points to the libhfds/libhdfs.so location but libhdfs now is build in a different location
      please take a look at this bug for the location details

      https://issues.apache.org/jira/browse/HADOOP-3344

      Thanks,
      Giri

      1. patch-4922.v1.txt
        3 kB
        Craig Macdonald
      2. hdfs-423-4.patch
        5 kB
        Eli Collins
      3. hdfs-423-3.patch
        5 kB
        Eli Collins
      4. hdfs-423-2.patch
        5 kB
        Eli Collins
      5. hdfs423.patch
        3 kB
        Eli Collins

        Issue Links

          Activity

          Giridharan Kesavan created issue -
          Craig Macdonald made changes -
          Field Original Value New Value
          Component/s contrib/fuse-dfs [ 12312376 ]
          Hide
          Craig Macdonald added a comment -

          Here's a work in progress patch.

          fuse_dfs_wrapper.sh could do with knowing where the libhdfs lib is - perhaps we need to have fuse_dfs_wrapper.sh built from a template at make time?

          Show
          Craig Macdonald added a comment - Here's a work in progress patch. fuse_dfs_wrapper.sh could do with knowing where the libhdfs lib is - perhaps we need to have fuse_dfs_wrapper.sh built from a template at make time?
          Craig Macdonald made changes -
          Attachment patch-4922.v1.txt [ 12397230 ]
          Owen O'Malley made changes -
          Project Hadoop Common [ 12310240 ] HDFS [ 12310942 ]
          Key HADOOP-4922 HDFS-423
          Component/s contrib/fuse-dfs [ 12312913 ]
          Component/s contrib/fuse-dfs [ 12312376 ]
          Giridharan Kesavan made changes -
          Link This issue is blocked by MAPREDUCE-665 [ MAPREDUCE-665 ]
          Hide
          Eli Collins added a comment -

          Made a minor fix to the earlier patch, this one works for me.

          Built fuse-dfs with:

          • ant compile -Dcompile.c++=true -Dlibhdfs=true
          • ant compile-contrib -Dlibhdfs=1 -Dfusedfs=1

          Note that sun.arch.data.model is not supported on all platforms, doesn't seem to be an easy platform agnostic way to set that property.

          Show
          Eli Collins added a comment - Made a minor fix to the earlier patch, this one works for me. Built fuse-dfs with: ant compile -Dcompile.c++=true -Dlibhdfs=true ant compile-contrib -Dlibhdfs=1 -Dfusedfs=1 Note that sun.arch.data.model is not supported on all platforms, doesn't seem to be an easy platform agnostic way to set that property.
          Eli Collins made changes -
          Attachment hdfs423.patch [ 12422411 ]
          Eli Collins made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Assignee Eli Collins [ eli ]
          Hide
          Eli Collins added a comment -

          fuse_dfs_wrapper.sh could do with knowing where the libhdfs lib is - perhaps we need to have fuse_dfs_wrapper.sh built from a template at make time?

          We could also link the libhdfs build dir to build/libhdfs after building libhdfs.

          Show
          Eli Collins added a comment - fuse_dfs_wrapper.sh could do with knowing where the libhdfs lib is - perhaps we need to have fuse_dfs_wrapper.sh built from a template at make time? We could also link the libhdfs build dir to build/libhdfs after building libhdfs.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12422411/hdfs423.patch
          against trunk revision 826149.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/74/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/74/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/74/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/74/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12422411/hdfs423.patch against trunk revision 826149. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/74/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/74/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/74/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/74/console This message is automatically generated.
          Hide
          Eli Collins added a comment -

          The test failure (org.apache.hadoop.hdfs.TestFileAppend2.testComplexAppend) doesn't seem related.

          Show
          Eli Collins added a comment - The test failure (org.apache.hadoop.hdfs.TestFileAppend2.testComplexAppend) doesn't seem related.
          Hide
          Eli Collins added a comment -

          Here's an updated patch that gets fuse dfs building on trunk and fuse_dfs_wrapper.sh working post-project split. I tested basic operations (read, write, creat, rm etc) manually on a fuse mount. TestFuseDFS no longer runs, I filed HDFS-780 to revive it, but let's do that as a separate checkin and get the code compiling first.

          Show
          Eli Collins added a comment - Here's an updated patch that gets fuse dfs building on trunk and fuse_dfs_wrapper.sh working post-project split. I tested basic operations (read, write, creat, rm etc) manually on a fuse mount. TestFuseDFS no longer runs, I filed HDFS-780 to revive it, but let's do that as a separate checkin and get the code compiling first.
          Eli Collins made changes -
          Attachment hdfs-423-2.patch [ 12425531 ]
          Hide
          Konstantin Boudnik added a comment -

          Ran patched version of HDFS on a Linux machine
          Linux everyelectric.corp.yahoo.com 2.6.18-164.2.1.el5PAE #1 SMP Mon Sep 21 04:45:05 EDT 2009 i686 i686 i386 GNU/Linux

          Running

           % ant compile -Dcompile.c++=true -Dlibhdfs=true
           % ant compile-contrib -Dlibhdfs=1 -Dfusedfs=1
          

          have ran into this issue:

               [exec] fuse_dfs.c: In function 'main':
               [exec] fuse_dfs.c:73: error: variable 'args' has initializer but incomplete type
               [exec] fuse_dfs.c:73: warning: implicit declaration of function 'FUSE_ARGS_INIT'
               [exec] fuse_dfs.c:73: error: storage size of 'args' isn't known
               [exec] fuse_dfs.c:83: warning: implicit declaration of function 'fuse_opt_parse'
               [exec] fuse_dfs.c:90: warning: implicit declaration of function 'fuse_opt_add_arg'
               [exec] fuse_dfs.c:132: warning: implicit declaration of function 'fuse_main'
               [exec] fuse_dfs.c:137: warning: implicit declaration of function 'fuse_opt_free_args'
               [exec] fuse_dfs.c:73: warning: unused variable 'args'
               [exec] make[1]: Leaving directory `/homes/cos/work/Hdfs.trunk/src/contrib/fuse-dfs/src'
               [exec] make[1]: *** [fuse_dfs.o] Error 1
               [exec] make: *** [all-recursive] Error 1
          
          BUILD FAILED
          /homes/xxx/work/Hdfs.trunk/build.xml:321: The following error occurred while executing this line:
          /homes/xxx/work/Hdfs.trunk/src/contrib/build.xml:30: The following error occurred while executing this line:
          /homes/xxx/work/Hdfs.trunk/src/contrib/fuse-dfs/build.xml:57: exec returned: 2
          

          Also, it seems that running
          {{% ant test-contrib -Dcompile.c++=true -Dlibhdfs=true -Dfusedfs=true }}
          doesn't detect the problem above. Won't it make sense to have test-contrib to depend on compile-contrib ?

          Show
          Konstantin Boudnik added a comment - Ran patched version of HDFS on a Linux machine Linux everyelectric.corp.yahoo.com 2.6.18-164.2.1.el5PAE #1 SMP Mon Sep 21 04:45:05 EDT 2009 i686 i686 i386 GNU/Linux Running % ant compile -Dcompile.c++=true -Dlibhdfs=true % ant compile-contrib -Dlibhdfs=1 -Dfusedfs=1 have ran into this issue: [exec] fuse_dfs.c: In function 'main': [exec] fuse_dfs.c:73: error: variable 'args' has initializer but incomplete type [exec] fuse_dfs.c:73: warning: implicit declaration of function 'FUSE_ARGS_INIT' [exec] fuse_dfs.c:73: error: storage size of 'args' isn't known [exec] fuse_dfs.c:83: warning: implicit declaration of function 'fuse_opt_parse' [exec] fuse_dfs.c:90: warning: implicit declaration of function 'fuse_opt_add_arg' [exec] fuse_dfs.c:132: warning: implicit declaration of function 'fuse_main' [exec] fuse_dfs.c:137: warning: implicit declaration of function 'fuse_opt_free_args' [exec] fuse_dfs.c:73: warning: unused variable 'args' [exec] make[1]: Leaving directory `/homes/cos/work/Hdfs.trunk/src/contrib/fuse-dfs/src' [exec] make[1]: *** [fuse_dfs.o] Error 1 [exec] make: *** [all-recursive] Error 1 BUILD FAILED /homes/xxx/work/Hdfs.trunk/build.xml:321: The following error occurred while executing this line: /homes/xxx/work/Hdfs.trunk/src/contrib/build.xml:30: The following error occurred while executing this line: /homes/xxx/work/Hdfs.trunk/src/contrib/fuse-dfs/build.xml:57: exec returned: 2 Also, it seems that running {{% ant test-contrib -Dcompile.c++=true -Dlibhdfs=true -Dfusedfs=true }} doesn't detect the problem above. Won't it make sense to have test-contrib to depend on compile-contrib ?
          Hide
          Eli Collins added a comment -

          Do you have the fuse-dev package?

          Won't it make sense to have test-contrib to depend on compile-contrib ?

          That seems reasonable. Btw I run the libhdfs test with ant compile -Dcompile.c++=true -Dlibhdfs=true test-c++-libhdfs

          Show
          Eli Collins added a comment - Do you have the fuse-dev package? Won't it make sense to have test-contrib to depend on compile-contrib ? That seems reasonable. Btw I run the libhdfs test with ant compile -Dcompile.c++=true -Dlibhdfs=true test-c++-libhdfs
          Hide
          Konstantin Boudnik added a comment -

          Right, installing fuse-dev helped a lot
          Another problem: I've ran the command line with the patch in place two or three times in a row. I got this:

               [exec] make[1]: Nothing to be done for `install-data-am'.
               [exec] make[1]: Leaving directory `/homes/cos/work/Hdfs.trunk/build/c++-build/Linux-i386-32/libhdfs'
            [symlink] ln -s /homes/cos/work/Hdfs.trunk/build/c++/Linux-i386-32/lib /homes/cos/work/Hdfs.trunk/build/c++/lib
            [symlink] ln: creating symbolic link `/homes/cos/work/Hdfs.trunk/build/c++/lib/lib' to `/homes/cos/work/Hdfs.trunk/build/c++/Linux-i386-32/lib': File exists
          

          Symlink creation has to be conditional or it might break the build. Seems to be working otherwise.

          Show
          Konstantin Boudnik added a comment - Right, installing fuse-dev helped a lot Another problem: I've ran the command line with the patch in place two or three times in a row. I got this: [exec] make[1]: Nothing to be done for `install-data-am'. [exec] make[1]: Leaving directory `/homes/cos/work/Hdfs.trunk/build/c++-build/Linux-i386-32/libhdfs' [symlink] ln -s /homes/cos/work/Hdfs.trunk/build/c++/Linux-i386-32/lib /homes/cos/work/Hdfs.trunk/build/c++/lib [symlink] ln: creating symbolic link `/homes/cos/work/Hdfs.trunk/build/c++/lib/lib' to `/homes/cos/work/Hdfs.trunk/build/c++/Linux-i386-32/lib': File exists Symlink creation has to be conditional or it might break the build. Seems to be working otherwise.
          Hide
          Eli Collins added a comment -

          Thanks Cos. Uploaded a patch with a fix and tested with back to back runs of the libhdfs test.

          Show
          Eli Collins added a comment - Thanks Cos. Uploaded a patch with a fix and tested with back to back runs of the libhdfs test.
          Eli Collins made changes -
          Attachment hdfs-423-3.patch [ 12426489 ]
          Eli Collins made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Eli Collins made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12426489/hdfs-423-3.patch
          against trunk revision 885658.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/128/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/128/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/128/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/128/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12426489/hdfs-423-3.patch against trunk revision 885658. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/128/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/128/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/128/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/128/console This message is automatically generated.
          Hide
          Konstantin Boudnik added a comment -

          Test failure seems to be related to HDFS-792

          Show
          Konstantin Boudnik added a comment - Test failure seems to be related to HDFS-792
          Eli Collins made changes -
          Summary libhdfs.so is pushed to a new location , hence fuds-dfs has to updated to point to the new location of libhdfs Unbreak FUSE build and fuse_dfs_wrapper.sh
          Hide
          Eli Collins added a comment -

          There's an ant option to overwrite the existing symlink, that's cleaner than deleting it and re-creating.

          Show
          Eli Collins added a comment - There's an ant option to overwrite the existing symlink, that's cleaner than deleting it and re-creating.
          Eli Collins made changes -
          Attachment hdfs-423-4.patch [ 12426697 ]
          Hide
          Konstantin Boudnik added a comment -

          +1 on the patch.
          I've checked and verified the latest version of the it. All looks good and I'm gonna wait for a day and then will commit it.

          Show
          Konstantin Boudnik added a comment - +1 on the patch. I've checked and verified the latest version of the it. All looks good and I'm gonna wait for a day and then will commit it.
          Hide
          Konstantin Boudnik added a comment -

          I've just committed this. Thanks Todd.

          Show
          Konstantin Boudnik added a comment - I've just committed this. Thanks Todd.
          Konstantin Boudnik made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Fix Version/s 0.22.0 [ 12314241 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #133 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/133/)
          . Unbreak FUSE build and fuse_dfs_wrapper.sh. Contributed by Todd Lipcon.

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #133 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/133/ ) . Unbreak FUSE build and fuse_dfs_wrapper.sh. Contributed by Todd Lipcon.
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #135 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/135/)
          . Unbreak FUSE build and fuse_dfs_wrapper.sh. Contributed by Todd Lipcon.

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #135 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/135/ ) . Unbreak FUSE build and fuse_dfs_wrapper.sh. Contributed by Todd Lipcon.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #164 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/164/)
          . Unbreak FUSE build and fuse_dfs_wrapper.sh. Contributed by Todd Lipcon.

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #164 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/164/ ) . Unbreak FUSE build and fuse_dfs_wrapper.sh. Contributed by Todd Lipcon.
          Hide
          Konstantin Boudnik added a comment -

          Sorry, Eli - my bad. I've updated CHANGES.txt with correct information. Thank you for the patch!

          Show
          Konstantin Boudnik added a comment - Sorry, Eli - my bad. I've updated CHANGES.txt with correct information. Thank you for the patch!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #137 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/137/)
          . Unbreak FUSE build and fuse_dfs_wrapper.sh. Contributed by Eli Collins (fixing contributor name).

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #137 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/137/ ) . Unbreak FUSE build and fuse_dfs_wrapper.sh. Contributed by Eli Collins (fixing contributor name).
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #137 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/137/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #137 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/137/ )
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #83 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/83/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #83 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/83/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #170 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/170/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #170 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/170/ )
          Tom White made changes -
          Fix Version/s 0.21.0 [ 12314046 ]
          Fix Version/s 0.22.0 [ 12314241 ]
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Eli Collins
              Reporter:
              Giridharan Kesavan
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development