Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1249

with fuse-dfs, chown which only has owner (or only group) argument fails with Input/output error.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.20.1, 0.20.2
    • Fix Version/s: 2.0.2-alpha
    • Component/s: fuse-dfs
    • Labels:
    • Environment:

      x86 linux (ubuntu 10.04)

    • Hadoop Flags:
      Reviewed

      Description

      with fuse-dfs, chown which only has owner (or only group) argument fails with Input/output error.
      ----------
      /mnt/hdfs/tmp# chown root file1
      chown: changing ownership of `file1': Input/output error
      /mnt/hdfs/tmp# chown root:root file1
      /mnt/hdfs/tmp# chown :root file1
      chown: changing group of `file1': Input/output error
      ----------
      I think it should be treated as unchanged for missing part(owner or group) instead of returning an error.

      I took fuse_dfs log and it is saying
      ----------
      unique: 25, opcode: SETATTR (4), nodeid: 14, insize: 128
      chown /tmp/file1 0 4294967295
      could not lookup group -1
      unique: 25, error: -5 (Input/output error), outsize: 16
      unique: 26, opcode: SETATTR (4), nodeid: 14, insize: 128
      chown /tmp/file1 0 0
      getattr /tmp/file1
      unique: 26, success, outsize: 120
      unique: 27, opcode: SETATTR (4), nodeid: 14, insize: 128
      chown /tmp/file1 4294967295 0
      could not lookup userid -1
      unique: 27, error: -5 (Input/output error), outsize: 16
      ----------

      therefore this should happen because dfs_chown() in src/contrib/fuse-dfs/src/fuse_impls_chown.c has following
      ----------
      ...
      user = getUsername(uid);
      if (NULL == user)

      { syslog(LOG_ERR,"Could not lookup the user id string %d\n",(int)uid); fprintf(stderr, "could not lookup userid %d\n", (int)uid); ret = -EIO; }

      if (0 == ret) {
      group = getGroup(gid);
      if (group == NULL)

      { syslog(LOG_ERR,"Could not lookup the group id string %d\n",(int)gid); fprintf(stderr, "could not lookup group %d\n", (int)gid); ret = -EIO; }


      }
      ...
      ----------

      but actually, hdfsChown() in src/c++/libhdfs/hdfs.c has this
      ----------
      ...
      if (owner == NULL && group == NULL)

      { fprintf(stderr, "Both owner and group cannot be null in chown"); errno = EINVAL; return -1; }

      ...
      ----------

      and also, setOwner seems allowing NULL
      ----------
      username - If it is null, the original username remains unchanged.
      groupname - If it is null, the original groupname remains unchanged.
      ----------
      according to the api document.

      therefore, I think fuse_impls_chown.c should not treat only user(or only group) lookup fail as an error.

      1. HDFS1249.1
        0.9 kB
        matsusaka kentaro
      2. HDFS-1249.001.patch
        5 kB
        Colin Patrick McCabe
      3. HDFS-1249.002.patch
        6 kB
        Colin Patrick McCabe

        Activity

        Hide
        matsusaka kentaro added a comment -

        I think it should behave like this attached diff.

        Show
        matsusaka kentaro added a comment - I think it should behave like this attached diff.
        Hide
        Colin Patrick McCabe added a comment -

        Hi matsuska,

        Thanks for reporting this! I think you are right that fuse-dfs should allow you to set the owner or group separately, without requiring both to be set at once. I have a patch that fixes this and adds a unit test, which I'll post shortly.

        Show
        Colin Patrick McCabe added a comment - Hi matsuska, Thanks for reporting this! I think you are right that fuse-dfs should allow you to set the owner or group separately, without requiring both to be set at once. I have a patch that fixes this and adds a unit test, which I'll post shortly.
        Hide
        Colin Patrick McCabe added a comment -
        • fuse_impls_chown.c: handle the case where one or both of the uid/gid are -1, which FUSE uses to indicate "no change."
        • fuse_impls_chown.c: be more verbose about error codes. Avoid checking errno after potentially clearing it by doing something else.
        • hdfsChown: don't treat owner == NULL && group == NULL (no change) as an error. It just means no operation was requested, so return success.
        • Do not call NewStringUTF on a NULL argument. This is not guaranteed to work, although it may with some JVM versions.
        • hdfs.h: fix header file documentation. We never did anything special with the empty string in hdfsChown-- NULL was always the special value indicating "no change."
        Show
        Colin Patrick McCabe added a comment - fuse_impls_chown.c: handle the case where one or both of the uid/gid are -1, which FUSE uses to indicate "no change." fuse_impls_chown.c: be more verbose about error codes. Avoid checking errno after potentially clearing it by doing something else. hdfsChown: don't treat owner == NULL && group == NULL (no change) as an error. It just means no operation was requested, so return success. Do not call NewStringUTF on a NULL argument. This is not guaranteed to work, although it may with some JVM versions. hdfs.h: fix header file documentation. We never did anything special with the empty string in hdfsChown-- NULL was always the special value indicating "no change."
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12536789/HDFS-1249.001.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.TestCrcCorruption

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2842//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2842//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12536789/HDFS-1249.001.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestCrcCorruption +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2842//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2842//console This message is automatically generated.
        Hide
        Colin Patrick McCabe added a comment -

        the CRC test failure is unrelated

        Show
        Colin Patrick McCabe added a comment - the CRC test failure is unrelated
        Hide
        Andy Isaacson added a comment -
        -    ret = (errno > 0) ? -errno : -EIO;
        +    ret = errno;
        +    ERROR("Could not chown %s to %d:%d: error %d", path, (int)uid, gid, ret);
        +    if (ret < 0)
        +      ret = -EIO;
        +    else
        +      ret = -ret;
        

        the if test should be if (ret <= 0), or even better,

        if (ret > 0)
            ret = -ret;
        else
            ret = -EIO;
        

        otherwise we get the wrong answer when errno == 0.

        Something went wrong with indentation on the prototype of hdfsChown. Since the line isn't very long, just keep it on one line rather than rewrapping it.

        Thank you for adding the testcase for "only change group"!

        Please add testcases for "only change user" and "change both user and group".

        Show
        Andy Isaacson added a comment - - ret = (errno > 0) ? -errno : -EIO; + ret = errno; + ERROR( "Could not chown %s to %d:%d: error %d" , path, ( int )uid, gid, ret); + if (ret < 0) + ret = -EIO; + else + ret = -ret; the if test should be if (ret <= 0) , or even better, if (ret > 0) ret = -ret; else ret = -EIO; otherwise we get the wrong answer when errno == 0. Something went wrong with indentation on the prototype of hdfsChown. Since the line isn't very long, just keep it on one line rather than rewrapping it. Thank you for adding the testcase for "only change group"! Please add testcases for "only change user" and "change both user and group".
        Hide
        Colin Patrick McCabe added a comment -
        • handle case where hdfsChown fails with errno = 0 (although that's not "supposed" to happen)
        • fix indentation on hdfs.h a bit (but still need to wrap due to 80-column limit, sorry)
        • unit test: add test for set user only, set user+group
        Show
        Colin Patrick McCabe added a comment - handle case where hdfsChown fails with errno = 0 (although that's not "supposed" to happen) fix indentation on hdfs.h a bit (but still need to wrap due to 80-column limit, sorry) unit test: add test for set user only, set user+group
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12536936/HDFS-1249.002.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2850//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2850//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12536936/HDFS-1249.002.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2850//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2850//console This message is automatically generated.
        Hide
        Eli Collins added a comment -

        +1 looks great

        Show
        Eli Collins added a comment - +1 looks great
        Hide
        Eli Collins added a comment -

        I've committed this and merged to branch-2. Thanks Colin.

        Show
        Eli Collins added a comment - I've committed this and merged to branch-2. Thanks Colin.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2567 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2567/)
        HDFS-1249. With fuse-dfs, chown which only has owner (or only group) argument fails with Input/output error. Contributed by Colin Patrick McCabe (Revision 1363466)

        Result = SUCCESS
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363466
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/fuse-dfs/fuse_impls_chown.c
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.c
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.h
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/test_libhdfs_threaded.c
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2567 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2567/ ) HDFS-1249 . With fuse-dfs, chown which only has owner (or only group) argument fails with Input/output error. Contributed by Colin Patrick McCabe (Revision 1363466) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363466 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/fuse-dfs/fuse_impls_chown.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.h /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/test_libhdfs_threaded.c
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2502 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2502/)
        HDFS-1249. With fuse-dfs, chown which only has owner (or only group) argument fails with Input/output error. Contributed by Colin Patrick McCabe (Revision 1363466)

        Result = SUCCESS
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363466
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/fuse-dfs/fuse_impls_chown.c
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.c
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.h
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/test_libhdfs_threaded.c
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2502 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2502/ ) HDFS-1249 . With fuse-dfs, chown which only has owner (or only group) argument fails with Input/output error. Contributed by Colin Patrick McCabe (Revision 1363466) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363466 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/fuse-dfs/fuse_impls_chown.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.h /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/test_libhdfs_threaded.c
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2523 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2523/)
        HDFS-1249. With fuse-dfs, chown which only has owner (or only group) argument fails with Input/output error. Contributed by Colin Patrick McCabe (Revision 1363466)

        Result = FAILURE
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363466
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/fuse-dfs/fuse_impls_chown.c
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.c
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.h
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/test_libhdfs_threaded.c
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2523 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2523/ ) HDFS-1249 . With fuse-dfs, chown which only has owner (or only group) argument fails with Input/output error. Contributed by Colin Patrick McCabe (Revision 1363466) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363466 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/fuse-dfs/fuse_impls_chown.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.h /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/test_libhdfs_threaded.c
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1142 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1142/)
        HDFS-1249. With fuse-dfs, chown which only has owner (or only group) argument fails with Input/output error. Contributed by Colin Patrick McCabe (Revision 1363466)

        Result = FAILURE
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363466
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/fuse-dfs/fuse_impls_chown.c
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.c
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.h
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/test_libhdfs_threaded.c
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1142 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1142/ ) HDFS-1249 . With fuse-dfs, chown which only has owner (or only group) argument fails with Input/output error. Contributed by Colin Patrick McCabe (Revision 1363466) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363466 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/fuse-dfs/fuse_impls_chown.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.h /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/test_libhdfs_threaded.c

          People

          • Assignee:
            Colin Patrick McCabe
            Reporter:
            matsusaka kentaro
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development