Hive
  1. Hive
  2. HIVE-478

Surface "processor time" for queries

    Details

    • Type: Wish Wish
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Logging, Query Processor
    • Labels:

      Description

      We currently list real-time metrics of how long queries take--"finished in: 1min 13sec" appears on the job tracker. However, this is affected by a lot more than just the quality or implementation of the query. For example, number of mappers used varies a lot when you use subqueries versus single-query aggregation, as does the amount of work necessary.

      For implementation comparisons (e.g., "should I use this version of the query or that one"), ti would be great to know the processor time used instead of the real time used...both in terms of "mapper cpu seconds" and "reducer cpu seconds."

        Activity

        Hide
        Adam Kramer added a comment -

        Also, in case it was not obvious, the current system counts time going by while mappers/reducers are "pending." This request would tell me how much time I actually used, e.g., not include time spent waiting for mappers or reducers.

        Show
        Adam Kramer added a comment - Also, in case it was not obvious, the current system counts time going by while mappers/reducers are "pending." This request would tell me how much time I actually used, e.g., not include time spent waiting for mappers or reducers.
        Hide
        Namit Jain added a comment -

        Can you set the configuration parameter hive.task.progress to true. It will dump the total time taken by each operator.
        Please check if this meets your requirements, we can enhance it to add more stuff.

        Show
        Namit Jain added a comment - Can you set the configuration parameter hive.task.progress to true. It will dump the total time taken by each operator. Please check if this meets your requirements, we can enhance it to add more stuff.
        Hide
        Adam Kramer added a comment -

        Sorry for the month-long delay. This is all I need. But it would be great, in general, to report this in the standard way:

        Time taken: 185.89 seconds (23,194,570 CPU_MILLISECONDS)

        ...in the CLI. Otherwise ok to mark issue resolved.

        Show
        Adam Kramer added a comment - Sorry for the month-long delay. This is all I need. But it would be great, in general, to report this in the standard way: Time taken: 185.89 seconds (23,194,570 CPU_MILLISECONDS) ...in the CLI. Otherwise ok to mark issue resolved.

          People

          • Assignee:
            Unassigned
            Reporter:
            Adam Kramer
          • Votes:
            4 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:

              Development