Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-9884 Hadoop calling du -sk is expensive
  3. HADOOP-17155

DF implementation of CachingGetSpaceUsed makes DFS Used size not correct

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Minor
    • Resolution: Not A Problem
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      When   we calculate DN's storage used, we add each Volume's used size together and each volume's size comes from it's BP's size. 

      When we use DF instead of DU, we know that DF check disk space usage (not disk size of a directory). so when check BP dir path,  What you're actually checking is the corresponding disk directory space. 

       

      When we use this with federation, under each volume  may have more than one BP, each BP return it's corresponding disk directory space. 

       

      If we have two BP under one volume, we will make DN's storage info's Used size double than real size.

        Attachments

        1. HADOOP-17155.1.patch
          2 kB
          angerszhu

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              angerszhuuu angerszhu
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: