Uploaded image for project: 'Apache Lens (Retired)'
  1. Apache Lens (Retired)
  2. LENS-511

identify facts associated with underlying hive/jdbc tables

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.0
    • Fix Version/s: None
    • Component/s: api, cube
    • Labels:
    • Flags:
      Important

      Description

      Consider following storage table defn inside a fact:

      <x_fact_table cube_name="sample_cube" name="fact2" weight="200.0"
      xmlns="uri:lens:cube:0.1"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xsi:schemaLocation="uri:lens:cube:0.1 cube-0.1.xsd ">
      ...
      <storage_tables>
      <storage_table>
      <update_periods>
      <update_period>HOURLY</update_period>
      <update_period>DAILY</update_period>
      </update_periods>
      <storage_name>local</storage_name>
      <table_desc external="true" field_delimiter=","
      table_location="/tmp/examples/fact2_local">
      <part_cols>
      <column comment="Time column" name="dt" type="STRING"/>
      </part_cols>
      <time_part_cols>dt</time_part_cols>
      </table_desc>
      </storage_table>
      </storage_tables>
      </x_fact_table>

      In an event a new partition is added to the external table location
      "/tmp/examples/fact2_local" then I wish to add a new partition on the
      fact2, however I have no way to find what all facts are built on
      external table location "/tmp/examples/fact2_local". We can possibly do
      it by matching the location, however that doesn't seem to be quite
      nice.. Consider for a JDBC source the table_location is kind of dummy as
      its not really used to query the content from that location. JDBCDriver
      expects a table with the name as storageName_factName in the target
      datastore, thus no indication on which all facts to be updated.

      Problem Statement: Current storage_table definition doesn't give me
      enough detail to find where all partitions needs to be added given a new
      partition is added to an external table.

      I propose that we add a table_name property as part of table_desc and
      provide following API's:
      GET /storages/

      {storageName}/tableNames/
      GET /storages/{storageName}

      /tableNames/

      {tableName}

      /facts

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                praagarw Pranav Kumar Agarwal
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated: