meaningful errno values in libhdfs

• Key: HADOOP-3549
• Type: Improvement
• Status: Closed
• Resolution: Fixed
• Priority: Major
• Assignee: Ben Slusky
• Reporter: Ben Slusky
• Affects Version/s: 0.17.1
• Fix Version/s: 0.19.0

Description

A comment near the top of hdfs.h says that "All APIs set errno to meaningful values." Unfortunately this is a big fat lie – bigger and fatter still now that we have permissions in HDFS. If Hadoop throws an AccessControlException, then libhdfs sets errno to EINTERNAL, even though this is not an "internal" error in any sense of the word.

With the attached patches, allow libhdfs to derive a useful errno value when an exception is thrown. I've implemented handling for AccessControlException and a few others whose semantics seem obvious.

Tangential question: The signatures for invokeMethod() and constructNewObjectOfClass() in hdfsJniHelper.h disagreed with their comments. Was the "exc" argument of those functions removed in the past, or never implemented? I (re)instated it in my patch.

Comments

[Hadoop QA]

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12383928/libhdfs-02-meaningful-errno.patch
against trunk revision 667040.

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.

-1 patch. The patch command could not apply the patch.

Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2651/console

This message is automatically generated.

[Ben Slusky]

Ok, so layered diffs are bad. Good to know. This is a combined diff against trunk. I removed the translation of DiskOutOfSpaceException to ENOSPC, since DiskOutOfSpaceException is apparently not thrown in the client.

Testing this automatically is a tricky business. AFAICT all hdfs tests are run as the superuser, so there's no way to trigger EACCES. I had written up a test to trigger ENOSPC, only to find that the exception that gets thrown when you fill up the disk is ambiguous (not to mention different from DFS to local FS).

[Hadoop QA]

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12383981/libhdfs-03-combined.patch
against trunk revision 669646.

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2701/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2701/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2701/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2701/console

This message is automatically generated.

[Hadoop QA]

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12383981/libhdfs-03-combined.patch
against trunk revision 677872.

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2903/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2903/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2903/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2903/console

This message is automatically generated.

[Ben Slusky]

If libhdfs had a permissions API, then we could make a test that would trigger EACCES.

[Owen O'Malley]

I'm sorry this patch sat so long to review, but I'd rather that the macro CHECK_EXCEPTION_IN_METHOD_INVOC declared the temporary jthr rather than just assuming it exists in the function that includes it. It would have also been better to fix the macro to take the arguments env and exc, although I know it is already using env implicitly.

[Ben Slusky]

How about this patch, which eliminates that ugly macro altogether?

[Owen O'Malley]

I just committed this. Thanks, Ben!

[Hudson]

Integrated in Hadoop-trunk #581 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/581/)