Uploaded image for project: 'Crunch'
  1. Crunch
  2. CRUNCH-624

temporary table size is 0, which makes reducer number too small

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 0.15.0
    • Component/s: Core
    • Labels:
      None

      Description

      if the pipeline produce temporary table , the reduce number of the temporary table whose input table is temporary table may become very small in some cases, since temporary table has no content .

      And, I may found the root cause in my caseļ¼š

      PCollectionImpl.java
      public void materializeAt(SourceTarget<S> sourceTarget) {
        this.materializedAt = sourceTarget;
        this.size = materializedAt.getSize(getPipeline().getConfiguration());
      }
      
      @Override
      public long getSize() {
          if (size < 0) {
              this.size = getSizeInternal();
          }
          return size;
      }
      

      PColletionImpl.materializeAt(sourceTarget) this method will be invoked when node splits to create temporary table, source sourceTarget binds with the new temporary table whose size is 0, since its path was just created, the this.size will be 0. After that, when getSize() was invoked by setting reduce number, since the size is 0, it will just return 0, which makes reduce number too small.

      So i think the code of materializeAt() should check sourceTarget's size, like below:

      PCollectionImpl.java
      public void materializeAt(SourceTarget<S> sourceTarget) {
        this.materializedAt = sourceTarget;
        long size = materializedAt.getSize(getPipeline().getConfiguration());
        if (size > 0)
            this.size = size;
      }
      

        Attachments

        1. CRUNCH-624.patch
          1 kB
          Josh Wills

          Activity

            People

            • Assignee:
              jwills Josh Wills
              Reporter:
              celix JingChen
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: