Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-487

HDFS should expose a fileid to uniquely identify a file

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      HDFS should expose a id that uniquely identifies a file. This helps in developing applications that work correctly even when files are moved from one directory to another. A typical use-case is to make the Pluggable Block Placement Policy (HDFS-385) use fileid instead of filename.

      1. fileid1.txt
        33 kB
        dhruba borthakur

        Issue Links

          Activity

          Hide
          dhruba borthakur added a comment -

          A few preliminary requirements as I see it:

          1. The fileid should be unique per path for the lifetime of a filesystem.
          2. FileStatus should contain the fileid, typical aplications like "hadoop dfs -ls" and "fsck" should be able to display the fileid
          3. There is no need for an API to map a fileid to a pathname.
          4. Regular files as well as directories have valid fileids.

          Show
          dhruba borthakur added a comment - A few preliminary requirements as I see it: 1. The fileid should be unique per path for the lifetime of a filesystem. 2. FileStatus should contain the fileid, typical aplications like "hadoop dfs -ls" and "fsck" should be able to display the fileid 3. There is no need for an API to map a fileid to a pathname. 4. Regular files as well as directories have valid fileids.
          Hide
          Hong Tang added a comment -

          How about using UUID? They have advantages:

          • can be calculated independently by any process.
          • guaranteed to be unique globally, even across two different HDFS instances, which would make it easier if we want to build a federated file system.
          Show
          Hong Tang added a comment - How about using UUID ? They have advantages: can be calculated independently by any process. guaranteed to be unique globally, even across two different HDFS instances, which would make it easier if we want to build a federated file system.
          Hide
          dhruba borthakur added a comment -

          This patch implements a 64 bit fileid per file. It uses the already existing generation stamp counter to populate this field.

          One major question that is unanswered by this patch: what to do with files that were created by older versions of hadoop. I think a good answer will be assign them unique fileids.

          Show
          dhruba borthakur added a comment - This patch implements a 64 bit fileid per file. It uses the already existing generation stamp counter to populate this field. One major question that is unanswered by this patch: what to do with files that were created by older versions of hadoop. I think a good answer will be assign them unique fileids.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > can be calculated independently by any process.

          This may not be trivial. What are the parameters for calculating a UUID?

          Show
          Tsz Wo Nicholas Sze added a comment - > can be calculated independently by any process. This may not be trivial. What are the parameters for calculating a UUID?
          Hide
          dhruba borthakur added a comment -

          Using a globally unique UUID means that it has to be at least 128 bits whereas a id that is unique within a cluster needs 64 bits. This has some impact on the amount of memory needed by the NN. What do other people think about sing 64 bit ids vs 128 bit ids?

          Show
          dhruba borthakur added a comment - Using a globally unique UUID means that it has to be at least 128 bits whereas a id that is unique within a cluster needs 64 bits. This has some impact on the amount of memory needed by the NN. What do other people think about sing 64 bit ids vs 128 bit ids?
          Hide
          Hong Tang added a comment -

          The wikipedia page tells you 5 ways of doing it.

          Show
          Hong Tang added a comment - The wikipedia page tells you 5 ways of doing it.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > ... What are the parameters for calculating a UUID?

          I mean what parameters should we choose. Obviously, path is needed. Anything else?

          > ... This has some impact on the amount of memory needed by the NN. ...

          Ideally, if we can compute the UUID in runtime, no additional memory is needed.

          Show
          Tsz Wo Nicholas Sze added a comment - > ... What are the parameters for calculating a UUID? I mean what parameters should we choose. Obviously, path is needed. Anything else? > ... This has some impact on the amount of memory needed by the NN. ... Ideally, if we can compute the UUID in runtime, no additional memory is needed.
          Hide
          Raghu Angadi added a comment -

          Given that there is mapping from id to the file, I am not sure if it really works like an id. Can you explain the use case bit more? (I haven't read the patch for pluggable block placement)

          When I first read the title of the jira, thought this might be about file handles...

          Show
          Raghu Angadi added a comment - Given that there is mapping from id to the file, I am not sure if it really works like an id. Can you explain the use case bit more? (I haven't read the patch for pluggable block placement) When I first read the title of the jira, thought this might be about file handles...
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > ... path is needed ...
          Oops, should be "path is not needed".

          Show
          Tsz Wo Nicholas Sze added a comment - > ... path is needed ... Oops, should be "path is not needed".
          Hide
          Raghu Angadi added a comment -

          > Given that there is mapping [...]
          should be 'there is NO mapping [...]'

          Show
          Raghu Angadi added a comment - > Given that there is mapping [...] should be 'there is NO mapping [...] '
          Hide
          Sanjay Radia added a comment -

          It would be useful to articulate the various use cases for the file id plus the mapping data structures that would be needed.

          Dhruba has given one use case.
          Another one is that if we propagate the fileId to the DNs than one can reconstruct files from blocks (though without their file names.)
          Hard links is another use case though I am not a fan of hard links.
          Others?
          Would fileIds simplify things inside the NN?

          Mappings:
          Currently, inside a NN we have mappings from filename->inode,, inode->blocks.

          To make the file id useful we would have to add a mapping from fileId->inode. This mapping would use up memory which is scarce inside a NN, but perhaps it could replace
          other data structures.
          Wouldn't one need the fileId to inode->blocks for puggable block placement policies?

          Show
          Sanjay Radia added a comment - It would be useful to articulate the various use cases for the file id plus the mapping data structures that would be needed. Dhruba has given one use case. Another one is that if we propagate the fileId to the DNs than one can reconstruct files from blocks (though without their file names.) Hard links is another use case though I am not a fan of hard links. Others? Would fileIds simplify things inside the NN? Mappings: Currently, inside a NN we have mappings from filename->inode,, inode->blocks. To make the file id useful we would have to add a mapping from fileId->inode. This mapping would use up memory which is scarce inside a NN, but perhaps it could replace other data structures. Wouldn't one need the fileId to inode->blocks for puggable block placement policies?
          Hide
          Sanjay Radia added a comment -

          Are you proposing a UID so that fileIds can be global across NN volumes?
          Isn't an ID unique to the NN volume sufficient for our needs?

          Show
          Sanjay Radia added a comment - Are you proposing a UID so that fileIds can be global across NN volumes? Isn't an ID unique to the NN volume sufficient for our needs?
          Hide
          dhruba borthakur added a comment -

          I am of the opinion that it would be sufficient to go with a 64 bit fileid that is unique to the namespace.

          Another use case: "distcp -update" looks at modification time and length of a file to determine whether it should be copied or not. This is not ideal, especially if somebody deletes and recreates a file with different contents within the time-precision of the clock used by HDFS. Also, it will not work when HDFS supports "truncates". (The modtime of a file can be set by an application). Having a unique fileid makes distcp work accurately.

          Show
          dhruba borthakur added a comment - I am of the opinion that it would be sufficient to go with a 64 bit fileid that is unique to the namespace. Another use case: "distcp -update" looks at modification time and length of a file to determine whether it should be copied or not. This is not ideal, especially if somebody deletes and recreates a file with different contents within the time-precision of the clock used by HDFS. Also, it will not work when HDFS supports "truncates". (The modtime of a file can be set by an application). Having a unique fileid makes distcp work accurately.
          Hide
          Raghu Angadi added a comment -

          So truncating a file would change the fileid?

          btw, 'distcp -update' is expected guarantee just that 'copy-if-modified' and nothing else.. just like any rsync or other sync that rely modification files and file lengths (by default). If user wants to handle other cases, then they should ask distcp to check file checksums.

          I am still not clear about block placement use case.. may be it can use id of the first block (it comes for free).

          Even for hardlinks I don't see how fileids help...

          Show
          Raghu Angadi added a comment - So truncating a file would change the fileid? btw, 'distcp -update' is expected guarantee just that 'copy-if-modified' and nothing else.. just like any rsync or other sync that rely modification files and file lengths (by default). If user wants to handle other cases, then they should ask distcp to check file checksums. I am still not clear about block placement use case.. may be it can use id of the first block (it comes for free). Even for hardlinks I don't see how fileids help...
          Hide
          Hong Tang added a comment -

          UUID is not a requirement based on the usage cases everybody presented here. But it would probably make the system a bit more future proof. The few cases I can think of (not necessarily independent) are (1) cross co-lo synchronization; (2) federated hdfs systems; (3) separation of block management.

          Show
          Hong Tang added a comment - UUID is not a requirement based on the usage cases everybody presented here. But it would probably make the system a bit more future proof. The few cases I can think of (not necessarily independent) are (1) cross co-lo synchronization; (2) federated hdfs systems; (3) separation of block management.
          Hide
          dhruba borthakur added a comment -

          > So truncating a file would change the fileid?

          Truncating a file does not change the fileid. There isn't an operation that can change the fileid of an existing file. The filid is associated with a file at file creation time. If you delete a file and then recreate a file with the same pathname, the new file will get a new fileid. The reason I mention truncate is to exemplify the fact that the heuristic used in "distcp -update" option might not work very well when hdfs supports truncates. "distcp -update" could use the fileid to reduce the probability of not detecting modified files.

          > I am still not clear about block placement use case.. may be it can use id of the first block (it comes for free).

          A blockid of a block is a concatenation of a 64 bit blockid and a 64 bit generation stamp. An error while writing to a block causes the generation stamp of that block to be modified. So, the blockid of the first block of a file does not remain fixed for the lifetime of that file. That means, it cannot be used as an unique identifier for a file.

          > (3) separation of block management.

          UUIDs probably make it somewhat futureproof, but we can also upgrade the unique-within-filesystem-fileid to a globally-unique-fileid when the use case arises. Such an upgrade will be easy to do. (The tradeoff is using more memory in the NN)

          Show
          dhruba borthakur added a comment - > So truncating a file would change the fileid? Truncating a file does not change the fileid. There isn't an operation that can change the fileid of an existing file. The filid is associated with a file at file creation time. If you delete a file and then recreate a file with the same pathname, the new file will get a new fileid. The reason I mention truncate is to exemplify the fact that the heuristic used in "distcp -update" option might not work very well when hdfs supports truncates. "distcp -update" could use the fileid to reduce the probability of not detecting modified files. > I am still not clear about block placement use case.. may be it can use id of the first block (it comes for free). A blockid of a block is a concatenation of a 64 bit blockid and a 64 bit generation stamp. An error while writing to a block causes the generation stamp of that block to be modified. So, the blockid of the first block of a file does not remain fixed for the lifetime of that file. That means, it cannot be used as an unique identifier for a file. > (3) separation of block management. UUIDs probably make it somewhat futureproof, but we can also upgrade the unique-within-filesystem-fileid to a globally-unique-fileid when the use case arises. Such an upgrade will be easy to do. (The tradeoff is using more memory in the NN)
          Hide
          Raghu Angadi added a comment -

          Dhruba, can you describe how fileid helps with pluggable block placement. Many use cases are mentioned here but I can't find description of how.

          'distcp -update' is not expected to be fool-proof (just like rsync). Even with fileid, should distcp store fileids in previous update (or source and destination files are expected to have same fileid?)?

          Show
          Raghu Angadi added a comment - Dhruba, can you describe how fileid helps with pluggable block placement. Many use cases are mentioned here but I can't find description of how. 'distcp -update' is not expected to be fool-proof (just like rsync). Even with fileid, should distcp store fileids in previous update (or source and destination files are expected to have same fileid?)?
          Hide
          dhruba borthakur added a comment -

          The API for pluggable block placement (HDFS-385) provides the pathname of the file to the block placement policy. The block placement policy can use the filename to determine what kind of placement algorithm to use for blocks in that file. This works well in the current NN design. However, if in future, we separate out the Block Manager from the NN, the Block Manager might not know the pathname for which the block belongs to. In that case, the Block manager will not be able to provide the filename when invoking the pluggable-block-placement-policy API. So, in some sense, using a fileid (instead of a filename) is future-proofing the API.

          Again to emphasize, HDFS-385 does not really need fileids, although it is good to have. The API designed in HDFS-385 shoudl be marked as "experimental", and we can change it if/when the Block Manager is separated out from the NN. Which option do you prefer?

          Show
          dhruba borthakur added a comment - The API for pluggable block placement ( HDFS-385 ) provides the pathname of the file to the block placement policy. The block placement policy can use the filename to determine what kind of placement algorithm to use for blocks in that file. This works well in the current NN design. However, if in future, we separate out the Block Manager from the NN, the Block Manager might not know the pathname for which the block belongs to. In that case, the Block manager will not be able to provide the filename when invoking the pluggable-block-placement-policy API. So, in some sense, using a fileid (instead of a filename) is future-proofing the API. Again to emphasize, HDFS-385 does not really need fileids, although it is good to have. The API designed in HDFS-385 shoudl be marked as "experimental", and we can change it if/when the Block Manager is separated out from the NN. Which option do you prefer?
          Hide
          Raghu Angadi added a comment -

          Thanks Dhruba. It looks like the specifics of why a fileid is required (or if it even helps) depend on future features. I think it is better to wait.

          I think should really have strong requirements before adding such important features or contracts in HDFS. Once it is provided it will stay for a long time.. even if there is limited use.

          Show
          Raghu Angadi added a comment - Thanks Dhruba. It looks like the specifics of why a fileid is required (or if it even helps) depend on future features. I think it is better to wait. I think should really have strong requirements before adding such important features or contracts in HDFS. Once it is provided it will stay for a long time.. even if there is limited use.
          Hide
          dhruba borthakur added a comment -

          > Thanks Dhruba. It looks like the specifics of why a fileid is required (or if it even helps) depend on future features. I think it is better to wait.

          Thanks Raghu.

          Sanjay: given Raghu's opinion, do you really think that the HDFS-385 needs to change to use the fileid (and not the filename).

          Show
          dhruba borthakur added a comment - > Thanks Dhruba. It looks like the specifics of why a fileid is required (or if it even helps) depend on future features. I think it is better to wait. Thanks Raghu. Sanjay: given Raghu's opinion, do you really think that the HDFS-385 needs to change to use the fileid (and not the filename).
          Hide
          dhruba borthakur added a comment -

          Hi folks, now that HDFS-385 has been committed I would like to resurrect the discussion about fileids for HDFS files.

          The negative feedback earlier was from Raghu who mentioned that it may be premature to introduce the concept of a fileid. In some sense, this is true but is more of a chicken-and-egg problem. The existence of a fileid simplifies many-a-applications. Can we have some discussion on what the possible dis-advantages of introducing a fileid might be?

          Show
          dhruba borthakur added a comment - Hi folks, now that HDFS-385 has been committed I would like to resurrect the discussion about fileids for HDFS files. The negative feedback earlier was from Raghu who mentioned that it may be premature to introduce the concept of a fileid. In some sense, this is true but is more of a chicken-and-egg problem. The existence of a fileid simplifies many-a-applications. Can we have some discussion on what the possible dis-advantages of introducing a fileid might be?
          Hide
          Tigran Mkrtchyan added a comment -

          I have implemented nfsv4.1/pNFS server and interfaced it with hdfs. The missing path to get it production ready is a unique
          file id and a way to access a file with it as nfs IO operations done with filehandles. Currently I maintain a path to it mapping,
          but this is not en elegant solution and does not survives restarts. A native hadoop way of doing that is required.

          Show
          Tigran Mkrtchyan added a comment - I have implemented nfsv4.1/pNFS server and interfaced it with hdfs. The missing path to get it production ready is a unique file id and a way to access a file with it as nfs IO operations done with filehandles. Currently I maintain a path to it mapping, but this is not en elegant solution and does not survives restarts. A native hadoop way of doing that is required.
          Hide
          dhruba borthakur added a comment -

          Hi Tigran, thanks for doing this. As you rightly pointed out, hdfs will need to support a persistent fileid for a file. If you are interested in implementing this, you could follow the steps outlined in http://wiki.apache.org/hadoop/HowToContribute. For a big feature like this, a good way to start is with a requirements and design document.

          Show
          dhruba borthakur added a comment - Hi Tigran, thanks for doing this. As you rightly pointed out, hdfs will need to support a persistent fileid for a file. If you are interested in implementing this, you could follow the steps outlined in http://wiki.apache.org/hadoop/HowToContribute . For a big feature like this, a good way to start is with a requirements and design document.
          Hide
          Tigran Mkrtchyan added a comment -

          Hi Dhruba, this will take some time as up to now my interaction with hdfs was based on javadoc only. I will check what can I do.

          Show
          Tigran Mkrtchyan added a comment - Hi Dhruba, this will take some time as up to now my interaction with hdfs was based on javadoc only. I will check what can I do.
          Hide
          Aleksandr Levchuk added a comment -

          Is fileid in the foreseeable future?

          Show
          Aleksandr Levchuk added a comment - Is fileid in the foreseeable future?
          Hide
          dhruba borthakur added a comment -

          as far as i know, nobody is working on this one

          Show
          dhruba borthakur added a comment - as far as i know, nobody is working on this one
          Hide
          Brandon Li added a comment -

          This requirement has been addressed by HDFS-4489.

          Show
          Brandon Li added a comment - This requirement has been addressed by HDFS-4489 .

            People

            • Assignee:
              dhruba borthakur
              Reporter:
              dhruba borthakur
            • Votes:
              3 Vote for this issue
              Watchers:
              19 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development