Uploaded image for project: 'Calcite'
  1. Calcite
  2. CALCITE-603

Metadata providers for size, memory, parallelism

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.1.0-incubating
    • Component/s: None
    • Labels:
      None

      Description

      Add a metadata provider for average row size and column size (both in bytes).

      This can be used, among other things, to determine the degree of parallelism and whether there is adequate memory to put multiple operators in the same process. (See email thread.)

      I think this should be a single interface with two methods:

      interface Volume {
        List<Double> averageColumnSize();
        Double averageRowSize();
      }

        Activity

        Show
        julianhyde Julian Hyde added a comment - Please review https://github.com/julianhyde/incubator-calcite/tree/calcite-603 .
        Hide
        julianhyde Julian Hyde added a comment -

        Adding to the list of providers:

          interface Size extends Metadata {
            Double averageRowSize();
            List<Double> averageColumnSizes();
          }
        
          interface Parallelism extends Metadata {
            Boolean isPhaseTransition();
            Integer splitCount();
          }
        
          interface Memory extends Metadata {
            Double memory();
            Double cumulativeMemoryWithinPhase();
            Double cumulativeMemoryWithinPhaseSplit();
          }
        

        See discussion.

        Show
        julianhyde Julian Hyde added a comment - Adding to the list of providers: interface Size extends Metadata { Double averageRowSize(); List< Double > averageColumnSizes(); } interface Parallelism extends Metadata { Boolean isPhaseTransition(); Integer splitCount(); } interface Memory extends Metadata { Double memory(); Double cumulativeMemoryWithinPhase(); Double cumulativeMemoryWithinPhaseSplit(); } See discussion .
        Show
        julianhyde Julian Hyde added a comment - Fixed in http://git-wip-us.apache.org/repos/asf/incubator-calcite/commit/dacde8f2 .
        Hide
        julianhyde Julian Hyde added a comment -

        Closing now that 1.1.0-incubating has been released.

        Show
        julianhyde Julian Hyde added a comment - Closing now that 1.1.0-incubating has been released.

          People

          • Assignee:
            julianhyde Julian Hyde
            Reporter:
            julianhyde Julian Hyde
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development