Hadoop Common
  1. Hadoop Common
  2. HADOOP-7045

TestDU fails on systems with local file systems with extended attributes

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.0
    • Component/s: fs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The test reports that the file takes an extra 4k on disk:

      Testcase: testDU took 5.74 sec
              FAILED
      expected:<32768> but was:<36864>
      junit.framework.AssertionFailedError: expected:<32768> but was:<36864>
              at org.apache.hadoop.fs.TestDU.testDU(TestDU.java:79)
      

      This is because du reports 32k for the file and 4k because the file system it lives on uses extended attributes.

      common-branch-0.20 $ dd if=/dev/zero of=data bs=4096 count=8
      8+0 records in
      8+0 records out
      32768 bytes (33 kB) copied, 9.6e-05 seconds, 341 MB/s
      common-branch-0.20 $ du data
      36	data
      common-branch-0.20 $ du --apparent-size data
      32	data
      

      We should modify the test to allow for some extra on-disk slack. The on-disk usage could also be smaller if the file data is all zeros or compression is enabled. The test currently handles the former by writing random data, we're punting on the latter.

        Issue Links

          Activity

          Eli Collins created issue -
          Eli Collins made changes -
          Field Original Value New Value
          Attachment hadoop-7045-1.patch [ 12459966 ]
          Eli Collins made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Eli Collins made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Resolution Fixed [ 1 ]
          Eli Collins made changes -
          Link This issue is related to HADOOP-7473 [ HADOOP-7473 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Eli Collins
              Reporter:
              Eli Collins
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development