Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-7809

Backport HADOOP-5839 to 0.20-security - fixes to ec2 scripts to allow remote job submission

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Won't Do
    • None
    • None
    • contrib/cloud
    • None

    Description

      The fix for HADOOP-5839 was committed to 0.21 more than a year ago. This bug is to backport the change (which is only 14 lines) to branch-0.20-security.
      ===========
      Original description:
      i would very much like the option of submitting jobs from a workstation outside ec2 to a hadoop cluster in ec2. This has been explored here:

      http://www.nabble.com/public-IP-for-datanode-on-EC2-tt19336240.html

      the net result of this is that we can make this work (along with using a socks proxy) with a couple of changes in the ec2 scripts:
      a) use public 'hostname' for fs.default.name setting (instead of the private hostname being used currently)
      b) mark hadoop.rpc.socket.factory.class.default as final variable in the generated hadoop-site.xml (that applies to server side)

      #a has no downside as far as i can tell since public hostnames resolve to internal/private IP addresses within ec2 (so traffic is optimally routed).

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            jsensarma Joydeep Sen Sarma
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment