Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-477

Streaming should execute Unix commands and scripts in well known languages without user specifying the path

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: None
    • Labels:
      None

      Description

      If the executables for -mapper or -reducer are well-known (grep, cat, awk), Streaming should make sure that the executable is found.
      If a script for -mapper or -reducer are in a well-known language (.pl, .py), Streaming should execute it with the correct language processor.

      Reason:
      many jobs get started from machines with a different environment from that on the cluster.
      On another hand, different clusters may have different environments.
      Also, a user may have no access to the cluster machines.
      Because of this, a user may be unable to specify correct paths for standard commands, and correct language processors for scripts.

      Implementation:
      Stream may tailr the commands by prepending the path, or the name of language processor.
      Another solution is to make sure that the commands are executed in a "meaningful" environment (with good $PATH, and other variables Unix users are accustomed to count upon).

      Once again, Streaming is user facing tool – it is not a library or a hackable example that the users are to modify for their needs. So it should work out of the box.

        Attachments

        1. streamingpath.patch
          6 kB
          Dhruba Borthakur

          Activity

            People

            • Assignee:
              dhruba Dhruba Borthakur
              Reporter:
              arkady arkady borkovsky
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: