Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: HA branch (HDFS-1623)
    • Fix Version/s: HA branch (HDFS-1623)
    • Component/s: documentation, ha
    • Labels:
      None

      Description

      We need to document the configuration changes in HDFS-2231 and the new CLI introduced by HADOOP-7774.

      1. HDFS-2733-HDFS-1623.patch
        17 kB
        Aaron T. Myers
      2. HDFSHighAvailability.html
        26 kB
        Aaron T. Myers
      3. HDFSHighAvailability.html
        26 kB
        Aaron T. Myers
      4. HDFS-2733-HDFS-1623.patch
        17 kB
        Aaron T. Myers
      5. HDFS-2733-HDFS-1623.patch
        18 kB
        Aaron T. Myers
      6. HDFSHighAvailability.html
        27 kB
        Aaron T. Myers
      7. HDFSHighAvailability.html
        29 kB
        Aaron T. Myers
      8. HDFS-2733-HDFS-1623.patch
        20 kB
        Aaron T. Myers
      9. HDFS-2733-HDFS-1623.patch
        20 kB
        Aaron T. Myers
      10. HDFSHighAvailability.html
        29 kB
        Aaron T. Myers
      11. HDFSHighAvailability.html
        29 kB
        Aaron T. Myers
      12. HDFS-2733-HDFS-1623.patch
        21 kB
        Aaron T. Myers

        Issue Links

          Activity

          Hide
          Aaron T. Myers added a comment -

          Here's a patch which adds some docs. Note that these are actually under the MR project, to mirror the Federation documentation. This location should be changed once HDFS-2771 is implemented.

          I'm also attaching the generated HTML from this doc, so people can review the generated form more easily.

          Show
          Aaron T. Myers added a comment - Here's a patch which adds some docs. Note that these are actually under the MR project, to mirror the Federation documentation. This location should be changed once HDFS-2771 is implemented. I'm also attaching the generated HTML from this doc, so people can review the generated form more easily.
          Hide
          Aaron T. Myers added a comment -

          If HDFS-2819 gets committed before this does, this patch will need to be updated slightly to reflect the changing of the configuration keys for the fencing options.

          Show
          Aaron T. Myers added a comment - If HDFS-2819 gets committed before this does, this patch will need to be updated slightly to reflect the changing of the configuration keys for the fencing options.
          Hide
          Aaron T. Myers added a comment -

          Slightly updated patch to reflect the config changes in HDFS-2819, which just got committed.

          Show
          Aaron T. Myers added a comment - Slightly updated patch to reflect the config changes in HDFS-2819 , which just got committed.
          Hide
          Eli Collins added a comment -

          Looks good. Comments follow.

          • Might make sense to use machine and host consistently (maybe pick one).
          • In the "option of running two redundant NameNodes" paragraph would mention that we now "support an Active/Passive configuration with a hot standby" since that's the standard HA description for our configuration.
          • Perhaps be explicit that there are two types of state, persistent and ephemeral state. That all persistent state modifications and some emphemeral (eg block allocation) are logged, with the remaining ephemeral state (block reports) are handled by the Datanodes communicating two both Namenodes.
          • Might be worth mentioning that we fence both with the fencers but the NNs also fence operations outside the edits log (eg operations from block reports)
          • In the NFS server section, technically there's no requirement it be an NFS server. Would mention that the availability of the system is limited by the availability of the shared edits dir, therefore in order to remove all single points of failure there needs to be redundancy for the shared edits dir. Specifically, multiple network paths to the storage, and redundancy in the storage itself (disk, network and power).
          • Perhaps mention that today manual failover is supported and automatic failover is coming?
          • Can mention explicitly that the NN/SBN layout allows people to re-use their existing 2NN/SBN hardware configuration.
          • s/any change/change/
          • Wrt syncing metadata directories, perhaps worth noting that you only need to sync the edits directory since the shared edits dir only contains edits (people may be confused why the old image they copied there never gets updated)
          • Should mention that checkHealth currently doesn't do anything
          Show
          Eli Collins added a comment - Looks good. Comments follow. Might make sense to use machine and host consistently (maybe pick one). In the "option of running two redundant NameNodes" paragraph would mention that we now "support an Active/Passive configuration with a hot standby" since that's the standard HA description for our configuration. Perhaps be explicit that there are two types of state, persistent and ephemeral state. That all persistent state modifications and some emphemeral (eg block allocation) are logged, with the remaining ephemeral state (block reports) are handled by the Datanodes communicating two both Namenodes. Might be worth mentioning that we fence both with the fencers but the NNs also fence operations outside the edits log (eg operations from block reports) In the NFS server section, technically there's no requirement it be an NFS server. Would mention that the availability of the system is limited by the availability of the shared edits dir, therefore in order to remove all single points of failure there needs to be redundancy for the shared edits dir. Specifically, multiple network paths to the storage, and redundancy in the storage itself (disk, network and power). Perhaps mention that today manual failover is supported and automatic failover is coming? Can mention explicitly that the NN/SBN layout allows people to re-use their existing 2NN/SBN hardware configuration. s/any change/change/ Wrt syncing metadata directories, perhaps worth noting that you only need to sync the edits directory since the shared edits dir only contains edits (people may be confused why the old image they copied there never gets updated) Should mention that checkHealth currently doesn't do anything
          Hide
          Aaron T. Myers added a comment -

          Thanks a lot for the review, Eli. Here's a patch which addresses all of your feedback, except for the following:

          Perhaps be explicit that there are two types of state, persistent and ephemeral state. That all persistent state modifications and some emphemeral (eg block allocation) are logged, with the remaining ephemeral state (block reports) are handled by the Datanodes communicating two both Namenodes.

          Might be worth mentioning that we fence both with the fencers but the NNs also fence operations outside the edits log (eg operations from block reports)

          Both of these facts seem more developer-oriented to me, and not particularly pertinent to operators. I think adding this material would only serve to confuse operators, and add little value.

          Wrt syncing metadata directories, perhaps worth noting that you only need to sync the edits directory since the shared edits dir only contains edits (people may be confused why the old image they copied there never gets updated)

          I already do say that you should copy over the fsimage/edits to the second NameNode, and only put the recent edits in the shared directory. What more do you think should be said?

          Show
          Aaron T. Myers added a comment - Thanks a lot for the review, Eli. Here's a patch which addresses all of your feedback, except for the following: Perhaps be explicit that there are two types of state, persistent and ephemeral state. That all persistent state modifications and some emphemeral (eg block allocation) are logged, with the remaining ephemeral state (block reports) are handled by the Datanodes communicating two both Namenodes. Might be worth mentioning that we fence both with the fencers but the NNs also fence operations outside the edits log (eg operations from block reports) Both of these facts seem more developer-oriented to me, and not particularly pertinent to operators. I think adding this material would only serve to confuse operators, and add little value. Wrt syncing metadata directories, perhaps worth noting that you only need to sync the edits directory since the shared edits dir only contains edits (people may be confused why the old image they copied there never gets updated) I already do say that you should copy over the fsimage/edits to the second NameNode, and only put the recent edits in the shared directory. What more do you think should be said?
          Hide
          Uma Maheswara Rao G added a comment -

          Aaron, Thanks a lot for the document. It really looks great.

          couple of typos, please take care while committing the patch if there are no more comments.
          1)For exaple:
          2) to transiton to
          3) transitonToActive and transitionToStandby

          Also along with the below lines, will it be good of explaining how Standby Node is different than other available checkpointing capable nodes today (BackupNode, CheckpointNode, Secondary NameNode) ?

          Note that, in an HA cluster, the Standby NameNode also performs checkpoints of the namespace state, and thus it is not necessary to run a Secondary NameNode, CheckpointNode, or BackupNode in an HA cluster. 
          
          Show
          Uma Maheswara Rao G added a comment - Aaron, Thanks a lot for the document. It really looks great. couple of typos, please take care while committing the patch if there are no more comments. 1)For exaple : 2) to transiton to 3) transitonToActive and transitionToStandby Also along with the below lines, will it be good of explaining how Standby Node is different than other available checkpointing capable nodes today (BackupNode, CheckpointNode, Secondary NameNode) ? Note that, in an HA cluster, the Standby NameNode also performs checkpoints of the namespace state, and thus it is not necessary to run a Secondary NameNode, CheckpointNode, or BackupNode in an HA cluster.
          Hide
          Stephen Barlow added a comment -

          • Consider simplifying the first sentence describing each of the 8 configuration parameters, and have it on a separate line from any additional discussion to improve scanability. Short noun phrases are better than imperatives for introducing parameters (e.g., for #3, "Similarly to rpc-address above, set the addresses for both NameNodes' HTTP servers to listen on." becomes "The addresses for both NameNodes' HTTP servers to listen on.") Also regarding #3, 'exaple' -> 'example'

          • Unless config parameters must be set in a particular order, consider switching to an unordered list (even though I just referenced an item by number)

          • Wherever 'once' means 'after', use 'after' (reduces ambiguity)

          Show
          Stephen Barlow added a comment - • Consider simplifying the first sentence describing each of the 8 configuration parameters, and have it on a separate line from any additional discussion to improve scanability. Short noun phrases are better than imperatives for introducing parameters (e.g., for #3, "Similarly to rpc-address above, set the addresses for both NameNodes' HTTP servers to listen on." becomes "The addresses for both NameNodes' HTTP servers to listen on.") Also regarding #3, 'exaple' -> 'example' • Unless config parameters must be set in a particular order, consider switching to an unordered list (even though I just referenced an item by number) • Wherever 'once' means 'after', use 'after' (reduces ambiguity)
          Hide
          Aaron T. Myers added a comment -

          Thanks a lot for the reviews, Uma and Stephen. Here's an updated patch which should address all of your comments except the following:

          Also along with the below lines, will it be good of explaining how Standby Node is different than other available checkpointing capable nodes today (BackupNode, CheckpointNode, Secondary NameNode) ?

          The top of the document specifies that knowledge of the node types in HDFS is a prerequisite. Thus, I punted on this.

          Show
          Aaron T. Myers added a comment - Thanks a lot for the reviews, Uma and Stephen. Here's an updated patch which should address all of your comments except the following: Also along with the below lines, will it be good of explaining how Standby Node is different than other available checkpointing capable nodes today (BackupNode, CheckpointNode, Secondary NameNode) ? The top of the document specifies that knowledge of the node types in HDFS is a prerequisite. Thus, I punted on this.
          Hide
          Aaron T. Myers added a comment -

          New patch fixes a small typo Stephen pointed out to me offline.

          Show
          Aaron T. Myers added a comment - New patch fixes a small typo Stephen pointed out to me offline.
          Hide
          Todd Lipcon added a comment -
          • "In the event of an unplanned event" could be worded better
          • The doc for dfs.federation.nameservices should indicate that, if federation is also being used, that this would be a comma-separated list of nameservice IDs (some of which may be HA and some of which might not be)
          • I think the example nameservice ID should be something like "mycluster" or "mynameservice" rather than "ha-nn-uri" – it doubles as the URI but that's a side effect rather than the important bit, I think.
          • Add a "note" for dfs.namenode.http-address that https-address should be configured on a secure cluster
          • For the "shell" fencing method, should note that the address of the service to be fenced will be inserted as the first argument
          • "This is not yet implemented, and at present will always return success" – if the NN is down it will return non-zero, right? So it still has some basic use.
          Show
          Todd Lipcon added a comment - "In the event of an unplanned event" could be worded better The doc for dfs.federation.nameservices should indicate that, if federation is also being used, that this would be a comma-separated list of nameservice IDs (some of which may be HA and some of which might not be) I think the example nameservice ID should be something like "mycluster" or "mynameservice" rather than "ha-nn-uri" – it doubles as the URI but that's a side effect rather than the important bit, I think. Add a "note" for dfs.namenode.http-address that https-address should be configured on a secure cluster For the "shell" fencing method, should note that the address of the service to be fenced will be inserted as the first argument "This is not yet implemented, and at present will always return success" – if the NN is down it will return non-zero, right? So it still has some basic use.
          Hide
          Aaron T. Myers added a comment -

          Thanks a lot for the review, Todd. Here's an updated patch which should address your feedback.

          Show
          Aaron T. Myers added a comment - Thanks a lot for the review, Todd. Here's an updated patch which should address your feedback.
          Hide
          Todd Lipcon added a comment -

          +1, lgtm.

          Show
          Todd Lipcon added a comment - +1, lgtm.
          Hide
          Aaron T. Myers added a comment -

          Thanks a lot for the reviews, everyone. I've just committed this to the HA branch.

          Show
          Aaron T. Myers added a comment - Thanks a lot for the reviews, everyone. I've just committed this to the HA branch.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-HAbranch-build #71 (See https://builds.apache.org/job/Hadoop-Hdfs-HAbranch-build/71/)
          HDFS-2733. Document HA configuration and CLI. Contributed by Aaron T. Myers.

          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241183
          Files :

          • /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/CHANGES.HDFS-1623.txt
          • /hadoop/common/branches/HDFS-1623/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HDFSHighAvailability.apt.vm
          • /hadoop/common/branches/HDFS-1623/hadoop-project/src/site/site.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-HAbranch-build #71 (See https://builds.apache.org/job/Hadoop-Hdfs-HAbranch-build/71/ ) HDFS-2733 . Document HA configuration and CLI. Contributed by Aaron T. Myers. atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241183 Files : /hadoop/common/branches/ HDFS-1623 /hadoop-hdfs-project/hadoop-hdfs/CHANGES. HDFS-1623 .txt /hadoop/common/branches/ HDFS-1623 /hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HDFSHighAvailability.apt.vm /hadoop/common/branches/ HDFS-1623 /hadoop-project/src/site/site.xml

            People

            • Assignee:
              Aaron T. Myers
              Reporter:
              Eli Collins
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development