Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-3570

Including user specified jar files in the client side classpath path in Hadoop 0.17 streaming

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.17.0
    • Fix Version/s: 0.19.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Hadoop 0.17 streaming allows specifying multiple user specific jar files on the command line along with the hadoop-streaming.jar. These user specific jar files are not automatically added on the client side class path when hadoop runs the streaming job

      For example, in the following streaming job <user-app.jar> is not automatically included in client side path before executing the streaming job

      hadoop jar -libjars <user-app.jar> $HADOOP_HOME/hadoop-streaming.jar \
      -input </input-data-path> \
      -output </output data path> \
      -inputformat <Input-format> \
      -outputformat <OutputFormat> \
      -mapper /bin/cat -reducer /bin/cat \
      -additionalconfspec <xxxx> \
      -jobconf mapred.reduce.tasks=3\

      Workaround:
      Current work around is to set these user specific jar files in
      HADOOP_CLASSPATH environment variable on the client side before running above example.

        Attachments

        1. 3570_v1.patch
          2 kB
          Sharad Agarwal
        2. 3570_v2.patch
          3 kB
          Sharad Agarwal
        3. 3570_v3.patch
          5 kB
          Sharad Agarwal

          Activity

            People

            • Assignee:
              sharadag Sharad Agarwal
              Reporter:
              vgogate Suhas
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: