Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.12.0
    • Component/s: Catalog, Planner/Optimizer
    • Labels:
      None

      Description

      Currently, PartitionedTableRewriter take a look into partition directories for rewriting filter conditions. It get all sub directories of table path because catalog doesn’t provide partition directories. But if there are lots of sub directories on HDFS, such as, more than 10,000 directories, it might be cause overload to NameNode. Thus, CatalogStore need to provide partition directories for specified filter conditions. I designed new method to CatalogStore as follows:

      • method name: getPartitionsWithConditionFilters
      • first parameter: database name
      • second parameter: table name
      • third parameter: where clause (included target column name and partition value)
      • return values: List<org.apache.tajo.catalog.proto.CatalogProtos.TablePartitionProto>
      • description: It scan right partition directories on CatalogStore with where caluse.
        For examples, users set parameters as following:
        • first parameter: default
        • second parameter: table1
        • third parameter: COLUMN_NAME = 'col1' AND PARTITION_VALUE = '3

      In the previous cases, this method will create select clause as follows.

      SELECT DISTINCT A.PATH
      FROM PARTITIONS A, (
        SELECT B.PARTITION_ID
        FROM PARTITION_KEYS B
        WHERE B.PARTITION_ID > 0 
        AND (
          COLUMN_NAME = 'col1' AND PARTITION_VALUE = '3'
        )
      ) B
      WHERE A.PARTITION_ID > 0
      AND A.TID = ${table_id}
      AND A.PARTITION_ID = B.PARTITION_ID
      

      At the first time, I considered to use EvalNode instead of where clause. But I can’t use it because of recursive related problems between tajo-catalog module and tajo-plan module. So, I’ll implement utility class to convert EvalNode to SQL.

        Attachments

        1. TAJO-1493_2.patch
          137 kB
          Jaehwa Jung
        2. TAJO-1493_3.patch
          138 kB
          Jaehwa Jung
        3. TAJO-1493_4.patch
          142 kB
          Jaehwa Jung
        4. TAJO-1493_5.patch
          172 kB
          Jaehwa Jung
        5. TAJO-1493_6.patch
          182 kB
          Jaehwa Jung
        6. TAJO-1493.patch
          133 kB
          Jaehwa Jung

          Issue Links

            Activity

              People

              • Assignee:
                blrunner Jaehwa Jung
                Reporter:
                blrunner Jaehwa Jung
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: