Hive
  1. Hive
  2. HIVE-410

Heartbeating for streaming jobs should not depend on stdout

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.4.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      jobs that require iterative processing may take longer than 10 mins to produce rows. This shouldn't be cause to kill the job. Producing keepalive dummy rows to stdout is bad if the data has to go into a Hive table or other Hive steps.

      If we adopt the solution of using stderr to indicate heartbeats, can that be combined with streaming counters (http://hadoop.apache.org/core/docs/current/streaming.html#How+do+I+update+counters+in+streaming+applications%3F )? Also, will limitations on size of stderr break this?

      1. patch-410-2.txt
        3 kB
        Ashish Thusoo
      2. patch-410.txt
        3 kB
        Ashish Thusoo

        Issue Links

          Activity

          Hide
          Venky Iyer added a comment -

          ping?

          Show
          Venky Iyer added a comment - ping?
          Hide
          Ashish Thusoo added a comment -

          Still investigating this. Will update this week.

          Show
          Ashish Thusoo added a comment - Still investigating this. Will update this week.
          Hide
          Ashish Thusoo added a comment -

          The fix is quite simple. I have not been able to add a test case yet as I need miniMR for that. If this goes in before miniMR stuff, I will add a test case in a separate JIRA.

          Show
          Ashish Thusoo added a comment - The fix is quite simple. I have not been able to add a test case yet as I need miniMR for that. If this goes in before miniMR stuff, I will add a test case in a separate JIRA.
          Hide
          Namit Jain added a comment -

          1. Why did you increase the max memory from 256 to 512 m ?
          2. Instead of hard coding 5 minutes, can you make it a function of map reduce timeout (10 minutes in the above note)

          Show
          Namit Jain added a comment - 1. Why did you increase the max memory from 256 to 512 m ? 2. Instead of hard coding 5 minutes, can you make it a function of map reduce timeout (10 minutes in the above note)
          Hide
          Ashish Thusoo added a comment -

          Added code to parameterize this based on the expiry interval in map reduce.

          I had to bump the memory for junit as our tests intermittently fail with out of memory exception otherwise. Looks like we are operating near the 256m limit.

          Show
          Ashish Thusoo added a comment - Added code to parameterize this based on the expiry interval in map reduce. I had to bump the memory for junit as our tests intermittently fail with out of memory exception otherwise. Looks like we are operating near the 256m limit.
          Hide
          Namit Jain added a comment -

          +1
          looks good - will commit once the tests pass

          Show
          Namit Jain added a comment - +1 looks good - will commit once the tests pass
          Hide
          Namit Jain added a comment -

          Committed. Thanks Ashish

          Show
          Namit Jain added a comment - Committed. Thanks Ashish

            People

            • Assignee:
              Ashish Thusoo
              Reporter:
              Venky Iyer
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development