Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-18051

qfiles: dataset support

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0
    • Component/s: Testing Infrastructure
    • Labels:
      None

      Description

      it would be great to have some kind of test dataset support; currently there is the q_test_init.sql which is quite large; and I'm often override it with an invalid string; because I write independent qtests most of the time - and the load of src and other tables are just a waste of time for me ; not to mention that the loading of those tables may also trigger breakpoints - which is a bit annoying.

      Most of the tests are "only" using the src table and possibly 2 others; however the main init script contains a bunch of tables - meanwhile there are quite few other tests which could possibly also benefit from a more general feature; for example the creation of bucket_small is present in 20 q files.

      the proposal would be to enable the qfiles to be annotated with metadata like datasets:

      --! qt:dataset:src,bucket_small
      

      proposal for storing a dataset:

      • the loader script would be at: data/datasets/_NAME_/load.hive.sql
      • the table data could be stored under that location

      a draft about this; and other qfiles related ideas:
      https://docs.google.com/document/d/1KtcIx8ggL9LxDintFuJo8NQuvNWkmtvv_ekbWrTLNGc/edit?usp=sharing

        Attachments

        1. HIVE-18051.01.patch
          30 kB
          László Bodor
        2. HIVE-18051.02.patch
          30 kB
          László Bodor
        3. HIVE-18051.03.patch
          41 kB
          László Bodor
        4. HIVE-18051.04.patch
          41 kB
          László Bodor
        5. HIVE-18051.05.patch
          41 kB
          László Bodor
        6. HIVE-18051.06.patch
          33 kB
          László Bodor
        7. HIVE-18051.07.patch
          46 kB
          László Bodor
        8. HIVE-18051.08.patch
          45 kB
          László Bodor
        9. HIVE-18051.09.patch
          45 kB
          László Bodor
        10. HIVE-18051.10.patch
          45 kB
          László Bodor
        11. HIVE-18051.11.patch
          45 kB
          László Bodor
        12. HIVE-18051.12.patch
          45 kB
          László Bodor

          Issue Links

            Activity

              People

              • Assignee:
                abstractdog László Bodor
                Reporter:
                kgyrtkirk Zoltan Haindrich
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: