Hive
  1. Hive
  2. HIVE-410

Heartbeating for streaming jobs should not depend on stdout

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.4.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      jobs that require iterative processing may take longer than 10 mins to produce rows. This shouldn't be cause to kill the job. Producing keepalive dummy rows to stdout is bad if the data has to go into a Hive table or other Hive steps.

      If we adopt the solution of using stderr to indicate heartbeats, can that be combined with streaming counters (http://hadoop.apache.org/core/docs/current/streaming.html#How+do+I+update+counters+in+streaming+applications%3F )? Also, will limitations on size of stderr break this?

      1. patch-410-2.txt
        3 kB
        Ashish Thusoo
      2. patch-410.txt
        3 kB
        Ashish Thusoo

        Issue Links

          Activity

          Venky Iyer created issue -
          Hide
          Venky Iyer added a comment -

          ping?

          Show
          Venky Iyer added a comment - ping?
          Ashish Thusoo made changes -
          Field Original Value New Value
          Assignee Ashish Thusoo [ athusoo ]
          Hide
          Ashish Thusoo added a comment -

          Still investigating this. Will update this week.

          Show
          Ashish Thusoo added a comment - Still investigating this. Will update this week.
          Hide
          Ashish Thusoo added a comment -

          The fix is quite simple. I have not been able to add a test case yet as I need miniMR for that. If this goes in before miniMR stuff, I will add a test case in a separate JIRA.

          Show
          Ashish Thusoo added a comment - The fix is quite simple. I have not been able to add a test case yet as I need miniMR for that. If this goes in before miniMR stuff, I will add a test case in a separate JIRA.
          Ashish Thusoo made changes -
          Attachment patch-410.txt [ 12408590 ]
          Namit Jain made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Namit Jain added a comment -

          1. Why did you increase the max memory from 256 to 512 m ?
          2. Instead of hard coding 5 minutes, can you make it a function of map reduce timeout (10 minutes in the above note)

          Show
          Namit Jain added a comment - 1. Why did you increase the max memory from 256 to 512 m ? 2. Instead of hard coding 5 minutes, can you make it a function of map reduce timeout (10 minutes in the above note)
          Hide
          Ashish Thusoo added a comment -

          Added code to parameterize this based on the expiry interval in map reduce.

          I had to bump the memory for junit as our tests intermittently fail with out of memory exception otherwise. Looks like we are operating near the 256m limit.

          Show
          Ashish Thusoo added a comment - Added code to parameterize this based on the expiry interval in map reduce. I had to bump the memory for junit as our tests intermittently fail with out of memory exception otherwise. Looks like we are operating near the 256m limit.
          Ashish Thusoo made changes -
          Attachment patch-410-2.txt [ 12408627 ]
          Hide
          Namit Jain added a comment -

          +1
          looks good - will commit once the tests pass

          Show
          Namit Jain added a comment - +1 looks good - will commit once the tests pass
          Hide
          Namit Jain added a comment -

          Committed. Thanks Ashish

          Show
          Namit Jain added a comment - Committed. Thanks Ashish
          Namit Jain made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Fix Version/s 0.4.0 [ 12313714 ]
          Resolution Fixed [ 1 ]
          Zheng Shao made changes -
          Link This issue is related to HIVE-690 [ HIVE-690 ]
          Carl Steinbach made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          36d 16h 51m 1 Namit Jain 20/May/09 19:07
          Patch Available Patch Available Resolved Resolved
          4h 20m 1 Namit Jain 20/May/09 23:27
          Resolved Resolved Closed Closed
          940d 1h 40m 1 Carl Steinbach 17/Dec/11 00:07

            People

            • Assignee:
              Ashish Thusoo
              Reporter:
              Venky Iyer
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development