Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-6197

Duplicate entries in inputProfiles of minor fragments for specific operators

    Details

      Description

      Minor fragments for the following operators show duplicate entries of the inputProfile (org.apache.drill.exec.ops.OperatorStats instance) when viewed in the Profile UI.
      e.g

      {
          ...
      	"query": "select * from sys.version",
          ...
      	        [ ...
      			{
      				"inputProfile": [{
      					"records": 0,
      					"batches": 0,
      					"schemas": 0
      				}],
      				"operatorId": 0,
      				"operatorType": 13,
      				"setupNanos": 0,
      				"processNanos": 0,
      				"peakLocalMemoryAllocated": 27131904,
      				"waitNanos": 0
      			},
      			{
      				"inputProfile": [{
      					"records": 1,
      					"batches": 1,
      					"schemas": 1
      				}],
      				"operatorId": 0,
      				"operatorType": 13,
      				"setupNanos": 0,
      				"processNanos": 752448,
      				"peakLocalMemoryAllocated": 27131904,
      				"metric": [{
      					"metricId": 0,
      					"longValue": 178
      				}],
      				"waitNanos": 889492
      			}]
      	...
      }
      

      operatorType: 13 is the screen operator, for which there can be only one inputProfile.

      It turns out that by default, all minor fragments' operators are provide a list of inputProfiles by org.apache.drill.exec.ops.FragmentStats.newOperatorStats(OpProfileDef, BufferAllocator). However, for the following 4 operators, the org.apache.drill.exec.physical.impl.BaseRootExec constructors also inject OperatorStats.

      org.apache.drill.exec.proto.beans.CoreOperatorType.SCREEN
      org.apache.drill.exec.proto.beans.CoreOperatorType.SINGLE_SENDER
      org.apache.drill.exec.proto.beans.CoreOperatorType.BROADCAST_SENDER
      org.apache.drill.exec.proto.beans.CoreOperatorType.HASH_PARTITION_SENDER
      

      All updates to the inputProfiles are done by the latter, while the former only reports zero values.

      The workaround is to have org.apache.drill.exec.ops.FragmentStats skip injecting the org.apache.drill.exec.ops.OperatorStats instance for these operators

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                kkhatua Kunal Khatua
                Reporter:
                kkhatua Kunal Khatua
                Reviewer:
                Aman Sinha
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - 24h
                  24h
                  Remaining:
                  Remaining Estimate - 24h
                  24h
                  Logged:
                  Time Spent - Not Specified
                  Not Specified