• Type: New Feature New Feature
    • Status: Reopened
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:


      In some cases it would be useful to be able to "signal" a job and its tasks about some external condition, or to broadcast a specific message to all tasks in a job. Currently we can only send a single pseudo-signal, that is to kill a job.

      Example 1: some jobs may be gracefully terminated even if they didn't complete all their work, e.g. Fetcher in Nutch may be running for a very long time if it blocks on relatively few sites left over from the fetchlist. In such case it would be very useful to send it a message requesting that it discards the rest of its input and gracefully completes its map tasks.

      Example 2: available bandwidth for fetching may be different at different times of day, e.g. daytime vs. nighttime, or total external link usage by other applications. Fetcher jobs often run for several hours. It would be good to be able to send a "signal" to the Fetcher to throttle or un-throttle its bandwidth usage depending on external conditions.

      Job implementations could react to these messages either by implementing a method, or by registering a listener, whichever seems more natural.

      I'm not quite sure how to go about implementing it, I guess this would have to be a part of TaskUmbilicalProtocol but my knowledge here is a bit fuzzy ... Comments are welcome.

        Issue Links


          Andrzej Bialecki created issue -
          Andrzej Bialecki made changes -
          Field Original Value New Value
          Link This issue is related to NUTCH-368 [ NUTCH-368 ]
          Doug Cutting made changes -
          Assignee Owen O'Malley [ owen.omalley ]
          Owen O'Malley made changes -
          Project Hadoop Common [ 12310240 ] Hadoop Map/Reduce [ 12310941 ]
          Key HADOOP-490 MAPREDUCE-205
          Affects Version/s 0.6.0 [ 12312025 ]
          Component/s mapred [ 12310690 ]
          Harsh J made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Not A Problem [ 8 ]
          Harsh J made changes -
          Resolution Not A Problem [ 8 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Assignee Owen O'Malley [ owen.omalley ]


            • Assignee:
              Andrzej Bialecki
            • Votes:
              0 Vote for this issue
              4 Start watching this issue


              • Created: