Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: HA branch (HDFS-1623)
    • Fix Version/s: HA branch (HDFS-1623)
    • Component/s: ha, test
    • Labels:
      None

      Description

      Right now the MiniDFS builder is somewhat inflexible - it just takes a "numNameNodes" parameter, which is used to specify federated nameservices. In order to add HA support, we need to be able to be more specific when configuring the NNs – eg to test the case where there is one nameservice that is HA and another which is not.

      1. hdfs-2591.txt
        38 kB
        Todd Lipcon
      2. hdfs-2591-v2.txt
        46 kB
        Todd Lipcon
      3. hdfs-2591-v3.txt
        48 kB
        Todd Lipcon

        Activity

        Hide
        tlipcon Todd Lipcon added a comment -

        Attached patch introduces a new MiniDFSNNTopology class which is used to describe the layout of NNs in the cluster. Most unit tests are passing, including those that make federated miniclusters. (ran tests with this applied on top of HDFS-2582, though it may also work without that patch)

        Show
        tlipcon Todd Lipcon added a comment - Attached patch introduces a new MiniDFSNNTopology class which is used to describe the layout of NNs in the cluster. Most unit tests are passing, including those that make federated miniclusters. (ran tests with this applied on top of HDFS-2582 , though it may also work without that patch)
        Hide
        tlipcon Todd Lipcon added a comment -

        (this is still a work in progress - testing it by updating the latest HDFS-1975 patch to run on top of this infrastructure, and identifying a few missing pieces)

        Show
        tlipcon Todd Lipcon added a comment - (this is still a work in progress - testing it by updating the latest HDFS-1975 patch to run on top of this infrastructure, and identifying a few missing pieces)
        Hide
        eli Eli Collins added a comment -

        What you have so far looks good. If we end up creating a class for the map in HDFS-2582 it could probably share the code here since it's mostly the same map essentially in class form.

        Show
        eli Eli Collins added a comment - What you have so far looks good. If we end up creating a class for the map in HDFS-2582 it could probably share the code here since it's mostly the same map essentially in class form.
        Hide
        tlipcon Todd Lipcon added a comment -

        v2 patch fixes a number of issues discovered trying to build HA test cases on top of this. In particular, it has code to set up shared edits directories (to be enabled by HDFS-1971), as well as initialize the SBNN by copying one of the formatted namedirs from the active NN. (otherwise the two NNs would have different nsids/bpids and not be able to failover).

        Show
        tlipcon Todd Lipcon added a comment - v2 patch fixes a number of issues discovered trying to build HA test cases on top of this. In particular, it has code to set up shared edits directories (to be enabled by HDFS-1971 ), as well as initialize the SBNN by copying one of the formatted namedirs from the active NN. (otherwise the two NNs would have different nsids/bpids and not be able to failover).
        Hide
        eli Eli Collins added a comment -

        +1 lgtm

        Show
        eli Eli Collins added a comment - +1 lgtm
        Hide
        tlipcon Todd Lipcon added a comment -

        Small update to fix some unit test failures discovered when testing this patch against the tip of branch:

        • add a toString method to ConfiguredNNAddress (helpful when debugging)
        • change addNameNode to fail when trying to add a NN to a minicluster which wasn't set up as federated (to fix failure of TestDatanodeMultipleRegistrations)
        • fix TestDatanodeMulitipleBlockScanner to use the DN's old configuration when refreshing it

        Fairly trivial delta between the two, so I'll commit unless there's any comment in the next few hours.

        Show
        tlipcon Todd Lipcon added a comment - Small update to fix some unit test failures discovered when testing this patch against the tip of branch: add a toString method to ConfiguredNNAddress (helpful when debugging) change addNameNode to fail when trying to add a NN to a minicluster which wasn't set up as federated (to fix failure of TestDatanodeMultipleRegistrations) fix TestDatanodeMulitipleBlockScanner to use the DN's old configuration when refreshing it Fairly trivial delta between the two, so I'll commit unless there's any comment in the next few hours.
        Hide
        eli Eli Collins added a comment -

        +1 the delta lgtm

        Show
        eli Eli Collins added a comment - +1 the delta lgtm
        Hide
        tlipcon Todd Lipcon added a comment -

        Committed v3 to HA branch

        Show
        tlipcon Todd Lipcon added a comment - Committed v3 to HA branch

          People

          • Assignee:
            tlipcon Todd Lipcon
            Reporter:
            tlipcon Todd Lipcon
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development