Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14118

Support using DNS to resolve nameservices to IP addresses

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.3.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      HDFS clients can use a single domain name to discover servers (namenodes/routers/observers) instead of explicitly listing out all hosts in the config

      Description

      In router based federation (RBF), clients will need to know about routers to talk to the HDFS cluster (obviously), and having routers updating (adding/removing) will have to make config change in every client, which is a painful process.

      DNS can be used here to resolve the single domain name clients knows to a list of routers in the current config. However, DNS won't be able to consider only resolving to the working router based on certain health thresholds.

      There are some ways about how this can be solved. One way is to have a separate script to regularly check the status of the router and update the DNS records if a router fails the health thresholds. In this way, security might be carefully considered for this way. Another way is to have the client do the normal connecting/failover after they get the list of routers, which requires the change of current failover proxy provider.

      See the attached design document for details about the proposed solution.

        Attachments

        1. HDFS design doc_ Single domain name for clients - Google Docs-1.pdf
          76 kB
          Fengnan Li
        2. HDFS design doc_ Single domain name for clients - Google Docs.pdf
          66 kB
          Fengnan Li
        3. HDFS-14118.patch
          3 kB
          Fengnan Li
        4. HDFS-14118.024.patch
          32 kB
          Yongjun Zhang
        5. HDFS-14118.023.patch
          32 kB
          Fengnan Li
        6. HDFS-14118.022.patch
          32 kB
          Fengnan Li
        7. HDFS-14118.021.patch
          32 kB
          Fengnan Li
        8. HDFS-14118.020.patch
          32 kB
          Fengnan Li
        9. HDFS-14118.019.patch
          31 kB
          Fengnan Li
        10. HDFS-14118.018.patch
          30 kB
          Fengnan Li
        11. HDFS-14118.017.patch
          29 kB
          Fengnan Li
        12. HDFS-14118.016.patch
          28 kB
          Fengnan Li
        13. HDFS-14118.015.patch
          28 kB
          Fengnan Li
        14. HDFS-14118.014.patch
          26 kB
          Íñigo Goiri
        15. HDFS-14118.013.patch
          26 kB
          Íñigo Goiri
        16. HDFS-14118.012.patch
          26 kB
          Íñigo Goiri
        17. HDFS-14118.011.patch
          25 kB
          Fengnan Li
        18. HDFS-14118.010.patch
          25 kB
          Fengnan Li
        19. HDFS-14118.009.patch
          25 kB
          Fengnan Li
        20. HDFS-14118.008.patch
          20 kB
          Fengnan Li
        21. HDFS-14118.007.patch
          19 kB
          Fengnan Li
        22. HDFS-14118.006.patch
          19 kB
          Fengnan Li
        23. HDFS-14118.005.patch
          19 kB
          Fengnan Li
        24. HDFS-14118.004.patch
          19 kB
          Fengnan Li
        25. HDFS-14118.003.patch
          20 kB
          Fengnan Li
        26. HDFS-14118.002.patch
          20 kB
          Fengnan Li
        27. HDFS-14118.001.patch
          21 kB
          Fengnan Li
        28. DNS testing log
          7 kB
          Fengnan Li

          Issue Links

            Activity

              People

              • Assignee:
                fengnanli Fengnan Li
                Reporter:
                fengnanli Fengnan Li
              • Votes:
                0 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: