Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1000

libhdfs needs to be updated to use the new UGI

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.21.0, 0.22.0
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None

      Description

      libhdfs needs to be updated w.r.t the APIs in the new UGI.

      1. fs-javadoc.patch
        0.8 kB
        Devaraj Das
      2. hdfs-1000-bp20.3.patch
        12 kB
        Devaraj Das
      3. hdfs-1000-bp20.4.patch
        12 kB
        Devaraj Das
      4. hdfs-1000-bp20.patch
        14 kB
        Devaraj Das
      5. hdfs-1000-trunk.1.patch
        18 kB
        Devaraj Das

        Issue Links

          Activity

          Hide
          Devaraj Das added a comment -

          This is a preliminary patch. Not passing the libhdfs tests yet (and has some debug messages).

          Show
          Devaraj Das added a comment - This is a preliminary patch. Not passing the libhdfs tests yet (and has some debug messages).
          Hide
          Eli Collins added a comment -

          I've got a patch on HDFS-939 that gets libhdfs running again, let's unbreak the test before committing any additional code. Also, I think this bug is a dupe of HDFS-940.

          Thanks,
          Eli

          Show
          Eli Collins added a comment - I've got a patch on HDFS-939 that gets libhdfs running again, let's unbreak the test before committing any additional code. Also, I think this bug is a dupe of HDFS-940 . Thanks, Eli
          Hide
          Eli Collins added a comment -

          Minor comments:

          • Needs additional tests, you already know that
          • Need to check return values, eg cURI below. Know this is old code but good to fix it while you're at it.
            +    cURI = malloc(strlen(host)+16);
            +    sprintf(cURI, "hdfs://%s:%d", host, (int)(port));
            
          • You should be able to remove HADOOP_UNIX_USER_GROUP_INFO now
          • Some lines are > 80 cols
          • Can remove some of the debug statements you added, eg:
            +    System.out.println("GETCURRENTUSER CALLED");
            
          • Have you checked that this doesn't introduce additional leaks, eg by looking at memory usage?
          Show
          Eli Collins added a comment - Minor comments: Needs additional tests, you already know that Need to check return values, eg cURI below. Know this is old code but good to fix it while you're at it. + cURI = malloc(strlen(host)+16); + sprintf(cURI, "hdfs: //%s:%d" , host, ( int )(port)); You should be able to remove HADOOP_UNIX_USER_GROUP_INFO now Some lines are > 80 cols Can remove some of the debug statements you added, eg: + System .out.println( "GETCURRENTUSER CALLED" ); Have you checked that this doesn't introduce additional leaks, eg by looking at memory usage?
          Hide
          Devaraj Das added a comment -

          This patch addresses the minor comments from Eli. I also simplified the patch to do operations based on just the username (removed the groups thing). It doesn't make sense anymore since the group resolution happens on the server.
          I don't think this requires new tests. An existing test covers the api i changed.
          Also, this patch is for 20.

          Show
          Devaraj Das added a comment - This patch addresses the minor comments from Eli. I also simplified the patch to do operations based on just the username (removed the groups thing). It doesn't make sense anymore since the group resolution happens on the server. I don't think this requires new tests. An existing test covers the api i changed. Also, this patch is for 20.
          Hide
          Suresh Srinivas added a comment -

          Congrats on getting HDFS-1000

          1. Patch is only applicable for branch 0.20. Can you please also attach a patch for trunk?
          2. Patch does not apply to apache 0.20 branch.
          3. hdfs.c
            • HADOOP_USER_GROUP_INFO is not used
            • remove groups in the comment "conect with NULL as user name/groups".
            • properly indent goto done;

          I tested the patch - previous tests no longer fail.

          Show
          Suresh Srinivas added a comment - Congrats on getting HDFS-1000 Patch is only applicable for branch 0.20. Can you please also attach a patch for trunk? Patch does not apply to apache 0.20 branch. hdfs.c HADOOP_USER_GROUP_INFO is not used remove groups in the comment "conect with NULL as user name/groups". properly indent goto done; I tested the patch - previous tests no longer fail.
          Hide
          Devaraj Das added a comment -

          Addresses the concerns Suresh had. The patch for trunk is coming soon.

          Show
          Devaraj Das added a comment - Addresses the concerns Suresh had. The patch for trunk is coming soon.
          Hide
          Devaraj Das added a comment -

          Attaching a patch that adds javadoc in the FileSystem class.

          Show
          Devaraj Das added a comment - Attaching a patch that adds javadoc in the FileSystem class.
          Hide
          Devaraj Das added a comment -

          Patch for trunk. I ran the libhdfs unit test manually and it worked fine with the changes in this patch (I had to copy the bin directory from the Common repository in order to run the test).

          Show
          Devaraj Das added a comment - Patch for trunk. I ran the libhdfs unit test manually and it worked fine with the changes in this patch (I had to copy the bin directory from the Common repository in order to run the test).
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12446548/hdfs-1000-trunk.1.patch
          against trunk revision 951555.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/400/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/400/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/400/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/400/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446548/hdfs-1000-trunk.1.patch against trunk revision 951555. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/400/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/400/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/400/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/400/console This message is automatically generated.
          Hide
          Devaraj Das added a comment -

          The test failures are unrelated. In fact the patch doesn't touch any Java source file..

          Show
          Devaraj Das added a comment - The test failures are unrelated. In fact the patch doesn't touch any Java source file..
          Hide
          Eli Collins added a comment -

          Closing out HDFS-940 as a dupe of this one.

          Show
          Eli Collins added a comment - Closing out HDFS-940 as a dupe of this one.
          Hide
          Devaraj Das added a comment -

          If this patch is slated for 0.21, we should commit the other jiras - HADOOP-6769 & HADOOP-6813 to 0.21 as well..

          Show
          Devaraj Das added a comment - If this patch is slated for 0.21, we should commit the other jiras - HADOOP-6769 & HADOOP-6813 to 0.21 as well..
          Hide
          Tom White added a comment -

          > we should commit the other jiras - HADOOP-6769 & HADOOP-6813 to 0.21

          Done

          Show
          Tom White added a comment - > we should commit the other jiras - HADOOP-6769 & HADOOP-6813 to 0.21 Done
          Hide
          Suresh Srinivas added a comment -

          +1 patch looks good.

          Show
          Suresh Srinivas added a comment - +1 patch looks good.
          Hide
          Devaraj Das added a comment -

          I just committed this.

          Show
          Devaraj Das added a comment - I just committed this.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #314 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/314/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #314 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/314/ )

            People

            • Assignee:
              Devaraj Das
              Reporter:
              Devaraj Das
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development