Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-4518

Counter Strike: Operation Operator

Log workAgile BoardRank to TopRank to BottomBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 0.13.0
    • None
    • None
    • Added config setting hive.counters.group.name: counter group name for counters used during query execution.

    Description

      Queries of the form:

      from foo
      insert overwrite table bar partition (p) select ...
      insert overwrite table bar partition (p) select ...
      insert overwrite table bar partition (p) select ...

      Generate a huge amount of counters. The reason is that task.progress is turned on for dynamic partitioning queries.

      The counters not only make queries slower than necessary (up to 50%) you will also eventually run out. That's because we're wrapping them in enum values to comply with hadoop 0.17.

      The real reason we turn task.progress on is that we need CREATED_FILES and FATAL counters to ensure dynamic partitioning queries don't go haywire.

      The counters have counter-intuitive names like C1 through C1000 and don't seem really useful by themselves.

      With hadoop 20+ you don't need to wrap the counters anymore, each operator can simply create and increment counters. That should simplify the code a lot.

      Attachments

        1. HIVE-4518.9.patch
          376 kB
          Jason Dere
        2. HIVE-4518.8.patch
          374 kB
          Jason Dere
        3. HIVE-4518.7.patch
          374 kB
          Jason Dere
        4. HIVE-4518.6.patch.txt
          372 kB
          Navis Ryu
        5. HIVE-4518.5.patch
          369 kB
          Gunther Hagleitner
        6. HIVE-4518.4.patch
          367 kB
          Gunther Hagleitner
        7. HIVE-4518.3.patch
          360 kB
          Gunther Hagleitner
        8. HIVE-4518.2.patch
          360 kB
          Gunther Hagleitner
        9. HIVE-4518.11.patch
          374 kB
          Jason Dere
        10. HIVE-4518.10.patch
          374 kB
          Jason Dere
        11. HIVE-4518.1.patch
          354 kB
          Gunther Hagleitner

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            hagleitn Gunther Hagleitner Assign to me
            hagleitn Gunther Hagleitner
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment