Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-2776

Web interface uses internal hostnames on EC2

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 0.15.1
    • Fix Version/s: None
    • Component/s: contrib/cloud
    • Labels:
      None
    • Environment:

      EC2 ami-a324c1ca

      Description

      The web interface, for example http://$MASTER_HOST:50030/machines.jsp, uses internal hostnames when running on EC2. This makes it impossible to access from outside EC2.

      The slaves file has the public names listed. Resolving a public name inside EC2 returns the private IP (which would reverse to the internal DNS name).

        Issue Links

          Activity

          Hide
          ncarlson Nate Carlson added a comment -

          I get by this by setting up nph-proxy (http://www.jmarshall.com/tools/cgiproxy/) on the master Hadoop node, and use that to browse the hosts. Kind of a hack, but works well for me.

          Show
          ncarlson Nate Carlson added a comment - I get by this by setting up nph-proxy ( http://www.jmarshall.com/tools/cgiproxy/ ) on the master Hadoop node, and use that to browse the hosts. Kind of a hack, but works well for me.
          Hide
          cwensel Chris K Wensel added a comment -

          Also note that the new EC2 scripts add SOCKS support:

          hadoop-ec2 proxy <cluster name>

          By installing FoxyProxy or setting you socks firewall settings, you can browse your whole cluster.

          that said, Nate's suggestion might be less clunky...

          see:
          https://issues.apache.org/jira/browse/HADOOP-2410

          Show
          cwensel Chris K Wensel added a comment - Also note that the new EC2 scripts add SOCKS support: hadoop-ec2 proxy <cluster name> By installing FoxyProxy or setting you socks firewall settings, you can browse your whole cluster. that said, Nate's suggestion might be less clunky... see: https://issues.apache.org/jira/browse/HADOOP-2410
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1609 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1609/)
          HADOOP-2776. Missing interface annotation on JournalSet. Contributed by Brandon Li.

          jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230412
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1609 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1609/ ) HADOOP-2776 . Missing interface annotation on JournalSet. Contributed by Brandon Li. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230412 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1536 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1536/)
          HADOOP-2776. Missing interface annotation on JournalSet. Contributed by Brandon Li.

          jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230412
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1536 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1536/ ) HADOOP-2776 . Missing interface annotation on JournalSet. Contributed by Brandon Li. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230412 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1554 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1554/)
          HADOOP-2776. Missing interface annotation on JournalSet. Contributed by Brandon Li.

          jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230412
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1554 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1554/ ) HADOOP-2776 . Missing interface annotation on JournalSet. Contributed by Brandon Li. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230412 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #923 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/923/)
          HADOOP-2776. Missing interface annotation on JournalSet. Contributed by Brandon Li.

          jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230412
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #923 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/923/ ) HADOOP-2776 . Missing interface annotation on JournalSet. Contributed by Brandon Li. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230412 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #956 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/956/)
          HADOOP-2776. Missing interface annotation on JournalSet. Contributed by Brandon Li.

          jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230412
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #956 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/956/ ) HADOOP-2776 . Missing interface annotation on JournalSet. Contributed by Brandon Li. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230412 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java
          Hide
          nate.sammons Nate Sammons added a comment -

          Any more updates on this? Running Hadoop inside Amazon EC2 is super annoying because most of the links are broken (internal vs external addresses). This bug has been open for FOUR YEARS.

          Show
          nate.sammons Nate Sammons added a comment - Any more updates on this? Running Hadoop inside Amazon EC2 is super annoying because most of the links are broken (internal vs external addresses). This bug has been open for FOUR YEARS.
          Hide
          aw Allen Wittenauer added a comment -

          I'm going to close this as won't fix. I don't think this is anything that we actually can fix here other than providing a complicated hostname mapping system for web interfaces.

          Part of the frustration I'm sure stems from a misunderstanding of what is actually happening:

          The slaves file has the public names listed.

          The slaves file is only used by the shell code to run ssh connections. It has absolutely zero impact on the core of Hadoop.

          Resolving a public name inside EC2 returns the private IP (which would reverse to the internal DNS name).

          Hadoop makes the perfectly valid assumption that the hostname the system tells us is a valid, network-connectable hostname. It is, from the inside of EC2. We have no way to know that you are attempting to connect from a completely different address that is being forwarded from some external entity.

          Proxying connections into a private network space is a perfectly valid solution.

          Show
          aw Allen Wittenauer added a comment - I'm going to close this as won't fix. I don't think this is anything that we actually can fix here other than providing a complicated hostname mapping system for web interfaces. Part of the frustration I'm sure stems from a misunderstanding of what is actually happening: The slaves file has the public names listed. The slaves file is only used by the shell code to run ssh connections. It has absolutely zero impact on the core of Hadoop. Resolving a public name inside EC2 returns the private IP (which would reverse to the internal DNS name). Hadoop makes the perfectly valid assumption that the hostname the system tells us is a valid, network-connectable hostname. It is, from the inside of EC2. We have no way to know that you are attempting to connect from a completely different address that is being forwarded from some external entity. Proxying connections into a private network space is a perfectly valid solution.

            People

            • Assignee:
              Unassigned
              Reporter:
              electrum David Phillips
            • Votes:
              6 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development