Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: HDFS-5698 (FSImage in protobuf)
    • Component/s: namenode
    • Labels:
      None

      Description

      The old code that loads the fsimage tracks the progress during loading. This jira proposes to implement the same functionality in the new code which serializes the fsimage using protobuf..

      1. HDFS-5771.000.patch
        13 kB
        Haohui Mai
      2. HDFS-5771.001.patch
        12 kB
        Haohui Mai
      3. HDFS-5771.002.patch
        11 kB
        Haohui Mai
      4. HDFS-5771.003.patch
        11 kB
        Haohui Mai

        Activity

        Hide
        Chris Nauroth added a comment -

        I committed the patch to the HDFS-5698 feature branch. Thanks again, Haohui.

        Show
        Chris Nauroth added a comment - I committed the patch to the HDFS-5698 feature branch. Thanks again, Haohui.
        Hide
        Chris Nauroth added a comment -

        +1 for the v3 patch. Thanks for incorporating those changes, Haohui.

        Show
        Chris Nauroth added a comment - +1 for the v3 patch. Thanks for incorporating those changes, Haohui.
        Hide
        Haohui Mai added a comment -

        The v3 patch places the currentStep variable correctly.

        Show
        Haohui Mai added a comment - The v3 patch places the currentStep variable correctly.
        Hide
        Haohui Mai added a comment -

        Thanks Chris for the review. The v2 patch makes sure that beginStep() and endStep() are called exactly once for each step.

        It also records the storage path in the step.

        Show
        Haohui Mai added a comment - Thanks Chris for the review. The v2 patch makes sure that beginStep() and endStep() are called exactly once for each step. It also records the storage path in the step.
        Hide
        Chris Nauroth added a comment -

        Hi, Haohui. A couple of notes:

        1. I see there are multiple sections that do beginStep/endStep for StepType#INODES. Considering the way the StartupProgress class works, the effect of this will be that progress jumps to 100% complete the first time endStep gets called. After that, the subsequent calls to beginStep/endStep are no-ops. Are all of the various inode sections serialized sequentially in the new format? If so, then would it be possible to do the beginStep call for StepType#INODES before the first inode section, and then do the endStep after the last inode section?
        2. There is a similar situation with saveInodes and saveSnapshots trying to begin/end the same step.
        Show
        Chris Nauroth added a comment - Hi, Haohui. A couple of notes: I see there are multiple sections that do beginStep / endStep for StepType#INODES . Considering the way the StartupProgress class works, the effect of this will be that progress jumps to 100% complete the first time endStep gets called. After that, the subsequent calls to beginStep / endStep are no-ops. Are all of the various inode sections serialized sequentially in the new format? If so, then would it be possible to do the beginStep call for StepType#INODES before the first inode section, and then do the endStep after the last inode section? There is a similar situation with saveInodes and saveSnapshots trying to begin/end the same step.
        Hide
        Haohui Mai added a comment -

        Rebased

        Show
        Haohui Mai added a comment - Rebased

          People

          • Assignee:
            Haohui Mai
            Reporter:
            Haohui Mai
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development