Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-4059 Pig on Spark
  3. PIG-4809

Implement to collect metric data like getSMMSpillCount() in SparkJobStats

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: spark-branch
    • Component/s: spark
    • Labels:
      None

      Description

      In current code, we don't implement to collect metric data in SparkJobStats like:
      SparkJobStats#getSMMSpillCount:

          @Override
          public long getSMMSpillCount() {
              throw new UnsupportedOperationException();
          }
      

      We just throw UnsupportedOperationException when this function is called.
      We need implement following functions in SparkJobStats if the metric data can be collected in spark mode:

          @Override
          public int getNumberMaps() {
              throw new UnsupportedOperationException();
          }
      
          @Override
          public int getNumberReduces() {
              throw new UnsupportedOperationException();
          }
      
          @Override
          public long getMaxMapTime() {
              throw new UnsupportedOperationException();
          }
      
          @Override
          public long getMinMapTime() {
              throw new UnsupportedOperationException();
          }
      
          @Override
          public long getAvgMapTime() {
              throw new UnsupportedOperationException();
          }
      
          @Override
          public long getMaxReduceTime() {
              throw new UnsupportedOperationException();
          }
      
          @Override
          public long getMinReduceTime() {
              throw new UnsupportedOperationException();
          }
      
          @Override
          public long getAvgREduceTime() {
              throw new UnsupportedOperationException();
          }
      
          @Override
          public long getMapInputRecords() {
              throw new UnsupportedOperationException();
          }
      
          @Override
          public long getMapOutputRecords() {
              throw new UnsupportedOperationException();
          }
      
          @Override
          public long getReduceInputRecords() {
              throw new UnsupportedOperationException();
          }
      
          @Override
          public long getReduceOutputRecords() {
              throw new UnsupportedOperationException();
          }
      
          @Override
          public long getSMMSpillCount() {
              throw new UnsupportedOperationException();
          }
      
          @Override
          public long getProactiveSpillCountObjects() {
              throw new UnsupportedOperationException();
          }
      
          @Override
          public long getProactiveSpillCountRecs() {
              throw new UnsupportedOperationException();
          }
      
        @Override
          public Map<String, Long> getMultiStoreCounters() {
              throw new UnsupportedOperationException();
          }
      
          @Override
          public Map<String, Long> getMultiInputCounters() {
              throw new UnsupportedOperationException();
          }
      

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              kellyzly liyunzhang
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: