Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1773

streaming doesn't support jobclient.output.filter

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.22.0
    • contrib/streaming
    • None
    • Linux

    • Reviewed
    • Improved console messaging for streaming jobs by using the generic JobClient API itself instead of the existing streaming-specific code.

    Description

      the streaming Jobclient implementation i.e contrib/streaming/src/java/org/apache/hadoop/streaming/StreamJob.java is significantly different than the core hadoop mapred/org/apache/hadoop/mapred/JobClient.java.

      for example unlike StreamJob.java, JobClient.java it gets tasks log when jobclient.output.filter=ALL is specified .

      With hod-logs going away in hadoop 0.20 (due to new scheduler) user has no good way of programmitically getting logs

      We should have intermediate adaptor class to implement Tools for the purpose of submitting jobs via m/r, streaming, pipes so that we don't miss some core functionality.

      GenericJobClient implements Tools and then StreamJob extends GenericJobClient, JobClient extends GenericJobClient

      Alok

      Attachments

        1. patch-1773.txt
          2 kB
          Amareshwari Sriramadasu
        2. patch-1773-1.txt
          2 kB
          Amareshwari Sriramadasu

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            amareshwari Amareshwari Sriramadasu
            aloknsingh Alok Singh
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment