Hive
  1. Hive
  2. HIVE-2678

Programmatically limit CLI status updates

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: CLI, Logging
    • Labels:
      None

      Description

      Provide a way to configure the frequency of Hive logging output, i.e., these:

      2011-12-23 22:31:20,979 Stage-1 map = 16%, reduce = 0%, Cumulative CPU 567.27 sec

      Some jobs update more than once per second, which is way more than necessary (and runs users out of scrollback buffer when using the CLI in screen).

      Default should be to update when map % or reduce % complete has gone up by one, and should be configurable via "SET mapred.update.rate=N;" to indicate that I would like updates every N seconds.

        Activity

        Hide
        tamtam180 added a comment -

        Hi,
        You can use parameter "hive.exec.counters.pull.interval" after version-0.8.
        Default value is 1000[ms].

        Show
        tamtam180 added a comment - Hi, You can use parameter "hive.exec.counters.pull.interval" after version-0.8. Default value is 1000 [ms] .
        Hide
        tamtam180 added a comment -

        This issue seem to had resolved by HIVE-1950.

        Show
        tamtam180 added a comment - This issue seem to had resolved by HIVE-1950 .
        Hide
        Adam Kramer added a comment -

        No, that is not enough. For some queries ms makes sense, but for others % done makes sense. If a query increases in more than 1% done in the interval, the lack of output is maddening.

        Show
        Adam Kramer added a comment - No, that is not enough. For some queries ms makes sense, but for others % done makes sense. If a query increases in more than 1% done in the interval, the lack of output is maddening.

          People

          • Assignee:
            Unassigned
            Reporter:
            Adam Kramer
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:

              Development