Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-4407

Misleading split info in TezSplitGrouper logs when adjusting small splits

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 0.10.1
    • None
    • None
    • None

    Description

      The log message in [TezSplitGrouper.getGroupedSplits|
      https://github.com/apache/tez/blob/627f33077480afdcefc0611fbde87d6be0010176/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/grouper/TezSplitGrouper.java#L272] is misleading and quite often it may show that the "Desired splits" is the same with "New desired splits" which does not make much sense.

      2022-04-19 01:59:05,064 [INFO] [App Shared Pool - #18] |grouper.TezSplitGrouper|: Desired splits: 4 too large.  Desired splitLength: 7589213 Min splitLength: 268435456 New desired splits: 4 Final desired splits: 4 All splits have localhost: false Total length: 1047311531 Original splits: 18
      

      Due to the above it is difficult/impossible to see what was the initial desired splits without reading the code.

      This was caused by TEZ-3291.

      Attachments

        Issue Links

          Activity

            People

              zabetak Stamatis Zampetakis
              zabetak Stamatis Zampetakis
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 40m
                  40m