Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10285 Storage Policy Satisfier in HDFS
  3. HDFS-12225

[SPS]: Optimize extended attributes for tracking SPS movements

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: HDFS-10285, 3.2.0
    • Component/s: datanode, namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      We have discussed to optimize number extended attributes and asked to report separate JIRA while implementing HDFS-11150

      This is the JIRA to track that work

      For the context, comment copied from HDFS-11150

      Yuanbo Liu wrote : I've tried that before. There is an issue here if we only mark the directory. When recovering from FsImage, the InodeMap isn't built up, so we don't know the sub-inode of a given inode, in the end, We cannot add these inodes to movement queue in FSDirectory#addToInodeMap, any thoughts?

      Uma Maheswara Rao G wrote: I got what you are saying. Ok for simplicity we can add for all Inodes now. For this to handle 100%, we may need intermittent processing, like first we should add them to some intermittentList while loading fsImage, once fully loaded and when starting active services, we should process that list and do required stuff. But it would add some additional complexity may be. Let's do with all file inodes now and we can revisit later if it is really creating issues. How about you raise a JIRA for it and think to optimize separately?

      Andrew Wang wrote in HDFS-10285 merge time review comment : HDFS-10899 also the cursor of the iterator in the EZ root xattr to track progress and handle restarts. I wonder if we can do something similar here to avoid having an xattr-per-file being moved.

        Attachments

        1. HDFS-12225-HDFS-10285-01.patch
          51 kB
          Surendra Singh Lilhore
        2. HDFS-12225-HDFS-10285-02.patch
          52 kB
          Surendra Singh Lilhore
        3. HDFS-12225-HDFS-10285-03.patch
          59 kB
          Surendra Singh Lilhore
        4. HDFS-12225-HDFS-10285-04.patch
          60 kB
          Surendra Singh Lilhore
        5. HDFS-12225-HDFS-10285-05.patch
          60 kB
          Surendra Singh Lilhore
        6. HDFS-12225-HDFS-10285-06.patch
          62 kB
          Surendra Singh Lilhore
        7. HDFS-12225-HDFS-10285-07.patch
          64 kB
          Surendra Singh Lilhore
        8. HDFS-12225-HDFS-10285-08.patch
          64 kB
          Surendra Singh Lilhore
        9. HDFS-12225-HDFS-10285-09.patch
          67 kB
          Surendra Singh Lilhore

          Issue Links

            Activity

              People

              • Assignee:
                surendrasingh Surendra Singh Lilhore
                Reporter:
                umamaheswararao Uma Maheswara Rao G
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: