[LENS-511] identify facts associated with underlying hive/jdbc tables

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.0
• Component/s: api, cube
• Labels: Hackathon-July
• Flags: Important
• Assignee: Unassigned
• Reporter: Pranav Kumar Agarwal
• Created: 16/Apr/15 08:59
• Updated: 10/Jul/15 04:52

Description

Consider following storage table defn inside a fact:

<x_fact_table cube_name="sample_cube" name="fact2" weight="200.0"
xmlns="uri:lens:cube:0.1"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="uri:lens:cube:0.1 cube-0.1.xsd ">
...
<storage_tables>
<storage_table>
<update_periods>
<update_period>HOURLY</update_period>
<update_period>DAILY</update_period>
</update_periods>
<storage_name>local</storage_name>
<table_desc external="true" field_delimiter=","
table_location="/tmp/examples/fact2_local">
<part_cols>
<column comment="Time column" name="dt" type="STRING"/>
</part_cols>
<time_part_cols>dt</time_part_cols>
</table_desc>
</storage_table>
</storage_tables>
</x_fact_table>

In an event a new partition is added to the external table location
"/tmp/examples/fact2_local" then I wish to add a new partition on the
fact2, however I have no way to find what all facts are built on
external table location "/tmp/examples/fact2_local". We can possibly do
it by matching the location, however that doesn't seem to be quite
nice.. Consider for a JDBC source the table_location is kind of dummy as
its not really used to query the content from that location. JDBCDriver
expects a table with the name as storageName_factName in the target
datastore, thus no indication on which all facts to be updated.

Problem Statement: Current storage_table definition doesn't give me
enough detail to find where all partitions needs to be added given a new
partition is added to an external table.

I propose that we add a table_name property as part of table_desc and
provide following API's:
GET /storages/

{storageName}/tableNames/
GET /storages/{storageName}

/tableNames/

{tableName}

/facts

Issue Links

• is related to: LENS-340 Using existing storage tables to create facts and dimensions