Hadoop Common
  1. Hadoop Common
  2. HADOOP-7473

TestDU is too sensitive to underlying filesystem

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: 0.20.204.0
    • Component/s: None
    • Labels:
      None

      Description

      DU currently uses "du -ks <path>" to determine the size. The returned number can be substantially more than the apparent size of the file, which causes the TestDU to fail.

      1. h-7473.patch
        2 kB
        Owen O'Malley

        Issue Links

          Activity

          Hide
          Owen O'Malley added a comment -

          We could fix this by changing the command to use: "du -ks --apparent-size <file>", which causes it to return the part of the allocation that is actually used. That however, isn't available on non-gnu systems.

          I'd propose instead that we just loosen the TestDU test to require that the size be between:

          size <= returned size < size + 10%

          Show
          Owen O'Malley added a comment - We could fix this by changing the command to use: "du -ks --apparent-size <file>", which causes it to return the part of the allocation that is actually used. That however, isn't available on non-gnu systems. I'd propose instead that we just loosen the TestDU test to require that the size be between: size <= returned size < size + 10%
          Hide
          Eli Collins added a comment -

          Have you seen HADOOP-7045?

          Show
          Eli Collins added a comment - Have you seen HADOOP-7045 ?
          Hide
          Owen O'Malley added a comment -

          I'd missed that. This is clearly a duplicate, although I'd lean toward my solution of using 10% slack instead of a hard coded 4k.

          Show
          Owen O'Malley added a comment - I'd missed that. This is clearly a duplicate, although I'd lean toward my solution of using 10% slack instead of a hard coded 4k.
          Hide
          Eli Collins added a comment -

          10% slack is fine with me, I used 4k because it addresses the specific case of ext attrs.

          Show
          Eli Collins added a comment - 10% slack is fine with me, I used 4k because it addresses the specific case of ext attrs.
          Hide
          Owen O'Malley added a comment -

          Hadoop 0.20.204.0 was released today.

          Show
          Owen O'Malley added a comment - Hadoop 0.20.204.0 was released today.

            People

            • Assignee:
              Unassigned
              Reporter:
              Owen O'Malley
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development