Hadoop Common
  1. Hadoop Common
  2. HADOOP-7045

TestDU fails on systems with local file systems with extended attributes

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.0
    • Component/s: fs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The test reports that the file takes an extra 4k on disk:

      Testcase: testDU took 5.74 sec
              FAILED
      expected:<32768> but was:<36864>
      junit.framework.AssertionFailedError: expected:<32768> but was:<36864>
              at org.apache.hadoop.fs.TestDU.testDU(TestDU.java:79)
      

      This is because du reports 32k for the file and 4k because the file system it lives on uses extended attributes.

      common-branch-0.20 $ dd if=/dev/zero of=data bs=4096 count=8
      8+0 records in
      8+0 records out
      32768 bytes (33 kB) copied, 9.6e-05 seconds, 341 MB/s
      common-branch-0.20 $ du data
      36	data
      common-branch-0.20 $ du --apparent-size data
      32	data
      

      We should modify the test to allow for some extra on-disk slack. The on-disk usage could also be smaller if the file data is all zeros or compression is enabled. The test currently handles the former by writing random data, we're punting on the latter.

        Issue Links

          Activity

          Hide
          Eli Collins added a comment -

          Patch attached. Modifies the test to pass if the disk usage is greater than the written size, but no greater than a given amount of slack (4k). The test now passes on local ext3 file systems with extended attributes.

          Show
          Eli Collins added a comment - Patch attached. Modifies the test to pass if the disk usage is greater than the written size, but no greater than a given amount of slack (4k). The test now passes on local ext3 file systems with extended attributes.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12459966/hadoop-7045-1.patch
          against trunk revision 1036694.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/113//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/113//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/113//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12459966/hadoop-7045-1.patch against trunk revision 1036694. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/113//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/113//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/113//console This message is automatically generated.
          Hide
          Konstantin Boudnik added a comment -

          +1 patch looks good!

          Show
          Konstantin Boudnik added a comment - +1 patch looks good!
          Hide
          Eli Collins added a comment -

          Thanks Cos! I've committed this.

          Show
          Eli Collins added a comment - Thanks Cos! I've committed this.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #435 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/435/)
          HADOOP-7045. TestDU fails on systems with local file systems with extended attributes. Contributed by Eli Collins

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #435 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/435/ ) HADOOP-7045 . TestDU fails on systems with local file systems with extended attributes. Contributed by Eli Collins
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #520 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/520/)

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #520 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/520/ )

            People

            • Assignee:
              Eli Collins
              Reporter:
              Eli Collins
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development