[HIVE-18051] qfiles: dataset support

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0
• Component/s: Testing Infrastructure
• Assignee: László Bodor
• Reporter: Zoltan Haindrich
• Created: 13/Nov/17 13:06
• Updated: 22/May/18 23:12
• Resolved: 06/Mar/18 14:28

Description

it would be great to have some kind of test dataset support; currently there is the q_test_init.sql which is quite large; and I'm often override it with an invalid string; because I write independent qtests most of the time - and the load of src and other tables are just a waste of time for me ; not to mention that the loading of those tables may also trigger breakpoints - which is a bit annoying.

Most of the tests are "only" using the src table and possibly 2 others; however the main init script contains a bunch of tables - meanwhile there are quite few other tests which could possibly also benefit from a more general feature; for example the creation of bucket_small is present in 20 q files.

the proposal would be to enable the qfiles to be annotated with metadata like datasets:

--! qt:dataset:src,bucket_small

proposal for storing a dataset:

• the loader script would be at: data/datasets/_NAME_/load.hive.sql
• the table data could be stored under that location

a draft about this; and other qfiles related ideas:
https://docs.google.com/document/d/1KtcIx8ggL9LxDintFuJo8NQuvNWkmtvv_ekbWrTLNGc/edit?usp=sharing

Issue Links

• Dependent: HIVE-19330 multi_insert_partitioned.q fails with "src table does not exist" message.
• relates to: HIVE-18862 qfiles: prepare .q files for using datasets

Sub-Tasks

• qfiles: prepare .q files for using datasets: Closed