Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-3981

Use Tez counter to collect per output stats

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 0.18.0
    • Component/s: tez
    • Labels:
      None

      Description

      Pig currently goes to hdfs to collect those information. That will take from several hundred milliseconds to several seconds, which adds overhead for namenode and hurt performance especially for iterative queries.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                daijy Daniel Dai
                Reporter:
                daijy Daniel Dai
              • Votes:
                1 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated: