Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9268

fuse_dfs chown crashes when uid is passed as -1

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:

      Description

      JVM crashes when users attempt to use vi to update a file on fuse file system with insufficient permission. (I use CDH's hadoop-fuse-dfs wrapper script to generate the bug, but the same bug is reproducible in trunk)

      The root cause is a segfault in a dfs-fuse method

      To reproduce it do as follows:
      mkdir /mnt/fuse
      chmod 777 /mnt/fuse
      ulimit -c unlimited # to enable coredump
      hadoop-fuse-dfs -odebug hdfs://localhost:9000/fuse /mnt/fuse
      touch /mnt/fuse/y
      chmod 600 /mnt/fuse/y
      vim /mnt/fuse/y
      (in vim, :w to save the file)

      #

      1. A fatal error has been detected by the Java Runtime Environment:
        #
      2. SIGSEGV (0xb) at pc=0x0000003b82f27ad6, pid=26606, tid=140079005689600
        #
      3. JRE version: Java(TM) SE Runtime Environment (7.0_79-b15) (build 1.7.0_79-b15)
      4. Java VM: Java HotSpot(TM) 64-Bit Server VM (24.79-b02 mixed mode linux-amd64 compressed oops)
      5. Problematic frame:
      6. C [libc.so.6+0x127ad6] __tls_get_addr@@GLIBC_2.3+0x127ad6
        #
      7. Core dump written. Default location: /home/weichiu/core or core.26606
        #
      8. An error report file with more information is saved as:
      9. /home/weichiu/hs_err_pid26606.log
        #
      10. If you would like to submit a bug report, please visit:
      11. http://bugreport.java.com/bugreport/crash.jsp
      12. The crash happened outside the Java Virtual Machine in native code.
      13. See problematic frame for where to report the bug.
        #
        /usr/bin/hadoop-fuse-dfs: line 29: 26606 Aborted (core dumped) env CLASSPATH="$ {CLASSPATH}

        " $

        {HADOOP_HOME}

        /bin/fuse_dfs $@

      ===========
      The coredump shows the segfault comes from
      (gdb) bt
      #0 0x0000003b82e328e5 in raise () from /lib64/libc.so.6
      #1 0x0000003b82e340c5 in abort () from /lib64/libc.so.6
      #2 0x00007f66fc924d75 in os::abort(bool) () from /etc/alternatives/jre/jre/lib/amd64/server/libjvm.so
      #3 0x00007f66fcaa76d7 in VMError::report_and_die() () from /etc/alternatives/jre/jre/lib/amd64/server/libjvm.so
      #4 0x00007f66fc929c8f in JVM_handle_linux_signal () from /etc/alternatives/jre/jre/lib/amd64/server/libjvm.so
      #5 <signal handler called>
      #6 0x0000003b82f27ad6 in __strcmp_sse42 () from /lib64/libc.so.6
      #7 0x00000000004039a0 in hdfsConnTree_RB_FIND ()
      #8 0x0000000000403e8f in fuseConnect ()
      #9 0x00000000004046db in dfs_chown ()
      #10 0x00007f66fcf8f6d2 in ?? () from /lib64/libfuse.so.2
      #11 0x00007f66fcf940d1 in ?? () from /lib64/libfuse.so.2
      #12 0x00007f66fcf910ef in ?? () from /lib64/libfuse.so.2
      #13 0x0000003b83207851 in start_thread () from /lib64/libpthread.so.0
      #14 0x0000003b82ee894d in clone () from /lib64/libc.so.6

      1. HDFS-9268.001.patch
        2 kB
        Wei-Chiu Chuang
      2. HDFS-9268.002.patch
        0.9 kB
        Colin P. McCabe

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #588 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/588/)
          HDFS-9268. fuse_dfs chown crashes when uid is passed as -1 (cmccabe) (cmccabe: rev 2f1eb2bceb1df5f27649a514246b38b9ccf60cba)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/fuse-dfs/fuse_impls_chown.c
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #588 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/588/ ) HDFS-9268 . fuse_dfs chown crashes when uid is passed as -1 (cmccabe) (cmccabe: rev 2f1eb2bceb1df5f27649a514246b38b9ccf60cba) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/fuse-dfs/fuse_impls_chown.c
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #540 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/540/)
          HDFS-9268. fuse_dfs chown crashes when uid is passed as -1 (cmccabe) (cmccabe: rev 2f1eb2bceb1df5f27649a514246b38b9ccf60cba)

          • hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/fuse-dfs/fuse_impls_chown.c
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #540 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/540/ ) HDFS-9268 . fuse_dfs chown crashes when uid is passed as -1 (cmccabe) (cmccabe: rev 2f1eb2bceb1df5f27649a514246b38b9ccf60cba) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/fuse-dfs/fuse_impls_chown.c hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2530 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2530/)
          HDFS-9268. fuse_dfs chown crashes when uid is passed as -1 (cmccabe) (cmccabe: rev 2f1eb2bceb1df5f27649a514246b38b9ccf60cba)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/fuse-dfs/fuse_impls_chown.c
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2530 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2530/ ) HDFS-9268 . fuse_dfs chown crashes when uid is passed as -1 (cmccabe) (cmccabe: rev 2f1eb2bceb1df5f27649a514246b38b9ccf60cba) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/fuse-dfs/fuse_impls_chown.c
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2477 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2477/)
          HDFS-9268. fuse_dfs chown crashes when uid is passed as -1 (cmccabe) (cmccabe: rev 2f1eb2bceb1df5f27649a514246b38b9ccf60cba)

          • hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/fuse-dfs/fuse_impls_chown.c
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2477 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2477/ ) HDFS-9268 . fuse_dfs chown crashes when uid is passed as -1 (cmccabe) (cmccabe: rev 2f1eb2bceb1df5f27649a514246b38b9ccf60cba) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/fuse-dfs/fuse_impls_chown.c hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #599 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/599/)
          HDFS-9268. fuse_dfs chown crashes when uid is passed as -1 (cmccabe) (cmccabe: rev 2f1eb2bceb1df5f27649a514246b38b9ccf60cba)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/fuse-dfs/fuse_impls_chown.c
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #599 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/599/ ) HDFS-9268 . fuse_dfs chown crashes when uid is passed as -1 (cmccabe) (cmccabe: rev 2f1eb2bceb1df5f27649a514246b38b9ccf60cba) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/fuse-dfs/fuse_impls_chown.c
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1323 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1323/)
          HDFS-9268. fuse_dfs chown crashes when uid is passed as -1 (cmccabe) (cmccabe: rev 2f1eb2bceb1df5f27649a514246b38b9ccf60cba)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/fuse-dfs/fuse_impls_chown.c
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1323 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1323/ ) HDFS-9268 . fuse_dfs chown crashes when uid is passed as -1 (cmccabe) (cmccabe: rev 2f1eb2bceb1df5f27649a514246b38b9ccf60cba) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/fuse-dfs/fuse_impls_chown.c
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8710 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8710/)
          HDFS-9268. fuse_dfs chown crashes when uid is passed as -1 (cmccabe) (cmccabe: rev 2f1eb2bceb1df5f27649a514246b38b9ccf60cba)

          • hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/fuse-dfs/fuse_impls_chown.c
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8710 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8710/ ) HDFS-9268 . fuse_dfs chown crashes when uid is passed as -1 (cmccabe) (cmccabe: rev 2f1eb2bceb1df5f27649a514246b38b9ccf60cba) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/fuse-dfs/fuse_impls_chown.c hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          cmccabe Colin P. McCabe added a comment -

          Thanks, Zhe Zhang. I'll open a follow-on JIRA for making fuseConnect private to fuse_connect.c. Committing to 2.8

          Show
          cmccabe Colin P. McCabe added a comment - Thanks, Zhe Zhang . I'll open a follow-on JIRA for making fuseConnect private to fuse_connect.c . Committing to 2.8
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Colin P. McCabe for the clarification. Please see if you want to update the patch and make fuseConnect private. Otherwise +1 on the latest patch.

          Show
          zhz Zhe Zhang added a comment - Thanks Colin P. McCabe for the clarification. Please see if you want to update the patch and make fuseConnect private. Otherwise +1 on the latest patch.
          Hide
          cmccabe Colin P. McCabe added a comment -

          I don't think fuseConnect is used anywhere but in fuse_connect.c at this point, so it could be made private to that file.

          Show
          cmccabe Colin P. McCabe added a comment - I don't think fuseConnect is used anywhere but in fuse_connect.c at this point, so it could be made private to that file.
          Hide
          zhz Zhe Zhang added a comment -

          The patch LGTM. One minor suggestion is maybe we can fold fuseConnect into fuseConnectAsThreadUid to avoid bugs of this kind in the future? Seems we should always call fuseConnect with the thread UID anyway.

          Show
          zhz Zhe Zhang added a comment - The patch LGTM. One minor suggestion is maybe we can fold fuseConnect into fuseConnectAsThreadUid to avoid bugs of this kind in the future? Seems we should always call fuseConnect with the thread UID anyway.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          By the way, I tested your patch and it does work under the scenario. Thanks

          On Tuesday, October 20, 2015, Colin Patrick McCabe (JIRA) <jira@apache.org>


          Newbie Clouderan

          Show
          jojochuang Wei-Chiu Chuang added a comment - By the way, I tested your patch and it does work under the scenario. Thanks On Tuesday, October 20, 2015, Colin Patrick McCabe (JIRA) <jira@apache.org> – Newbie Clouderan
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Hi Colin P. McCabe, I was going to create a test case for this, but then noticed "mvn test" does not run any unit tests of fuse_dfs. Is that right? I understand unit-testing this may be hard, but the code base does appear to have test code ( I did not run it though)

          Show
          jojochuang Wei-Chiu Chuang added a comment - Hi Colin P. McCabe , I was going to create a test case for this, but then noticed "mvn test" does not run any unit tests of fuse_dfs. Is that right? I understand unit-testing this may be hard, but the code base does appear to have test code ( I did not run it though)
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 5m 42s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 8m 54s There were no new javac warning messages.
          +1 release audit 0m 27s The applied patch does not increase the total number of release audit warnings.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 49s mvn install still works.
          +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse.
          +1 native 1m 13s Pre-build of native portion
          +1 hdfs tests 0m 51s Tests passed in hadoop-hdfs-native-client.
              19m 35s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12767653/HDFS-9268.002.patch
          Optional Tests javac unit
          git revision trunk / 01b103f
          hadoop-hdfs-native-client test log https://builds.apache.org/job/PreCommit-HDFS-Build/13090/artifact/patchprocess/testrun_hadoop-hdfs-native-client.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13090/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13090/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 5m 42s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 8m 54s There were no new javac warning messages. +1 release audit 0m 27s The applied patch does not increase the total number of release audit warnings. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 49s mvn install still works. +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse. +1 native 1m 13s Pre-build of native portion +1 hdfs tests 0m 51s Tests passed in hadoop-hdfs-native-client.     19m 35s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12767653/HDFS-9268.002.patch Optional Tests javac unit git revision trunk / 01b103f hadoop-hdfs-native-client test log https://builds.apache.org/job/PreCommit-HDFS-Build/13090/artifact/patchprocess/testrun_hadoop-hdfs-native-client.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13090/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13090/console This message was automatically generated.
          Hide
          cmccabe Colin P. McCabe added a comment -

          I posted a patch which fixes the root of the problem, I think. We should be using fuseConnectAsThreadUid instead of fuseConnect.

          Show
          cmccabe Colin P. McCabe added a comment - I posted a patch which fixes the root of the problem, I think. We should be using fuseConnectAsThreadUid instead of fuseConnect .
          Hide
          cmccabe Colin P. McCabe added a comment -

          Hi Wei-Chiu Chuang, good job debugging this. However, I think you may have misinterpreted the man page. Your change makes it so that nothing is changed if either uid or gid is -1. But in fact, only the ID which is -1 should be left unchanged.

          Show
          cmccabe Colin P. McCabe added a comment - Hi Wei-Chiu Chuang , good job debugging this. However, I think you may have misinterpreted the man page. Your change makes it so that nothing is changed if either uid or gid is -1. But in fact, only the ID which is -1 should be left unchanged.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Reproducible as long as the file permission is not the default (e.g. 600, 642, 640)
          Bug found: vim creates a temporary file and then attempts to change its owner chown(-1,99)
          The hdfs-fuse implementation does not handle this case. The Linux manual for chown states: "If the owner or group is specified as -1, then that ID is not changed."

          Rev1 fix is uploaded. I need to make a test case as well.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Reproducible as long as the file permission is not the default (e.g. 600, 642, 640) Bug found: vim creates a temporary file and then attempts to change its owner chown(-1,99) The hdfs-fuse implementation does not handle this case. The Linux manual for chown states: "If the owner or group is specified as -1, then that ID is not changed." Rev1 fix is uploaded. I need to make a test case as well.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          The segfault is triggered by strcmp(), so likely a NULL pointer or a corrupted pointer was passed in.
          This only happens using vim to edit the file. Other text editors including pico does not trigger segfault. Looking at the debugging message of the fuse process, the last few lines are:

          unique: 59, opcode: LOOKUP (1), nodeid: 1, insize: 43
          LOOKUP /y~
          getattr /y~
          unique: 59, error: -2 (No such file or directory), outsize: 16
          unique: 60, opcode: LOOKUP (1), nodeid: 1, insize: 43
          LOOKUP /y~
          getattr /y~
          unique: 60, error: -2 (No such file or directory), outsize: 16
          unique: 61, opcode: LOOKUP (1), nodeid: 1, insize: 43
          LOOKUP /y~
          getattr /y~
          unique: 61, error: -2 (No such file or directory), outsize: 16
          unique: 62, opcode: CREATE (35), nodeid: 1, insize: 59
          create flags: 0x280c1 /y~ 0100600 umask=0022
          create[140079219900624] flags: 0x281c1 /y~
          getattr /y~
          NODEID: 7
          unique: 62, success, outsize: 160
          unique: 63, opcode: GETATTR (3), nodeid: 1, insize: 56
          getattr /
          unique: 63, success, outsize: 120
          unique: 64, opcode: SETATTR (4), nodeid: 7, insize: 128
          getattr /y~
          unique: 64, success, outsize: 120
          unique: 65, opcode: SETATTR (4), nodeid: 7, insize: 128
          chown /y~ 4294967295 99

          So when vi saves the file, it create a temporary file y~ and attempt to chown() it. There is a red-black tree structure in fuse where a bad pointer is found traversing the tree.

          Show
          jojochuang Wei-Chiu Chuang added a comment - The segfault is triggered by strcmp(), so likely a NULL pointer or a corrupted pointer was passed in. This only happens using vim to edit the file. Other text editors including pico does not trigger segfault. Looking at the debugging message of the fuse process, the last few lines are: unique: 59, opcode: LOOKUP (1), nodeid: 1, insize: 43 LOOKUP /y~ getattr /y~ unique: 59, error: -2 (No such file or directory), outsize: 16 unique: 60, opcode: LOOKUP (1), nodeid: 1, insize: 43 LOOKUP /y~ getattr /y~ unique: 60, error: -2 (No such file or directory), outsize: 16 unique: 61, opcode: LOOKUP (1), nodeid: 1, insize: 43 LOOKUP /y~ getattr /y~ unique: 61, error: -2 (No such file or directory), outsize: 16 unique: 62, opcode: CREATE (35), nodeid: 1, insize: 59 create flags: 0x280c1 /y~ 0100600 umask=0022 create [140079219900624] flags: 0x281c1 /y~ getattr /y~ NODEID: 7 unique: 62, success, outsize: 160 unique: 63, opcode: GETATTR (3), nodeid: 1, insize: 56 getattr / unique: 63, success, outsize: 120 unique: 64, opcode: SETATTR (4), nodeid: 7, insize: 128 getattr /y~ unique: 64, success, outsize: 120 unique: 65, opcode: SETATTR (4), nodeid: 7, insize: 128 chown /y~ 4294967295 99 So when vi saves the file, it create a temporary file y~ and attempt to chown() it. There is a red-black tree structure in fuse where a bad pointer is found traversing the tree.

            People

            • Assignee:
              cmccabe Colin P. McCabe
              Reporter:
              jojochuang Wei-Chiu Chuang
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development