Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-9884 Hadoop calling du -sk is expensive
  3. HADOOP-17155

DF implementation of CachingGetSpaceUsed makes DFS Used size not correct

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Minor
    • Resolution: Not A Problem
    • None
    • None
    • None
    • None

    Description

      When   we calculate DN's storage used, we add each Volume's used size together and each volume's size comes from it's BP's size. 

      When we use DF instead of DU, we know that DF check disk space usage (not disk size of a directory). so when check BP dir path,  What you're actually checking is the corresponding disk directory space. 

       

      When we use this with federation, under each volume  may have more than one BP, each BP return it's corresponding disk directory space. 

       

      If we have two BP under one volume, we will make DN's storage info's Used size double than real size.

      Attachments

        1. HADOOP-17155.1.patch
          2 kB
          angerszhu

        Activity

          People

            Unassigned Unassigned
            angerszhuuu angerszhu
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: