Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-5977

FSImageFormatPBINode does not respect "-renameReserved" upgrade flag

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Later
    • Affects Version/s: 2.4.0
    • Fix Version/s: None
    • Component/s: namenode
    • Labels:
    • Target Version/s:

      Description

      HDFS-5709 added a new upgrade flag "-renameReserved" which can be used to automatically rename reserved paths like "/.reserved" encountered during upgrade. The new protobuf loading code does not have a similar facility, so future reserved paths cannot be automatically renamed via "-renameReserved".

        Activity

        Haohui Mai made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Later [ 7 ]
        Hide
        Haohui Mai added a comment -

        Thanks Andrew Wang and Suresh Srinivas for the info. Let me resolve this jira as later and keep it around. We can reopen this jira if we need to add another reserved path.

        Show
        Haohui Mai added a comment - Thanks Andrew Wang and Suresh Srinivas for the info. Let me resolve this jira as later and keep it around. We can reopen this jira if we need to add another reserved path.
        Hide
        Suresh Srinivas added a comment -

        For future upgrades, I think that this mechanism is no longer required.

        I think this mechanism may be required, as not everything can start with /.reserved. I think we should address this issue. We should also do another code review to ensure all the previous relevant fsimage related mechanism is available in protobuf based fsimage solution as well.

        Show
        Suresh Srinivas added a comment - For future upgrades, I think that this mechanism is no longer required. I think this mechanism may be required, as not everything can start with /.reserved. I think we should address this issue. We should also do another code review to ensure all the previous relevant fsimage related mechanism is available in protobuf based fsimage solution as well.
        Hide
        Andrew Wang added a comment -

        Hi Haohui, I had the same thought and agree in spirit, but if you look at the comment history on HDFS-5709, Suresh Srinivas had some concerns about adding more reserved paths beyond our current two. We agreed to implement a more general solution for this reason.

        I agree that we don't need to implement this for the PB loader until we add another reserved path, but I filed this JIRA so "-renameReserved" isn't forgotten about in the future.

        Show
        Andrew Wang added a comment - Hi Haohui, I had the same thought and agree in spirit, but if you look at the comment history on HDFS-5709 , Suresh Srinivas had some concerns about adding more reserved paths beyond our current two. We agreed to implement a more general solution for this reason. I agree that we don't need to implement this for the PB loader until we add another reserved path, but I filed this JIRA so "-renameReserved" isn't forgotten about in the future.
        Hide
        Haohui Mai added a comment -

        There are two cases here:

        1. If the user is upgrading from a version that uses the old fsimage, NN will use the old loader to load the fsimage, which has handled the flag already.
        2. For future upgrades, I think that this mechanism is no longer required. For example, currently the NN has already reserved .reserved in the namespace. What we need to do here is to regulate ourselves to put special names into .reserved. Making this assumption explicit eliminates the needs of renaming during upgrades, therefore the whole workflow for upgrades can be simplified.
        Show
        Haohui Mai added a comment - There are two cases here: If the user is upgrading from a version that uses the old fsimage, NN will use the old loader to load the fsimage, which has handled the flag already. For future upgrades, I think that this mechanism is no longer required. For example, currently the NN has already reserved .reserved in the namespace. What we need to do here is to regulate ourselves to put special names into .reserved . Making this assumption explicit eliminates the needs of renaming during upgrades, therefore the whole workflow for upgrades can be simplified.
        Andrew Wang made changes -
        Field Original Value New Value
        Target Version/s 2.4.0 [ 12326143 ]
        Andrew Wang created issue -

          People

          • Assignee:
            Unassigned
            Reporter:
            Andrew Wang
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development