Hive
  1. Hive
  2. HIVE-1079

CREATE VIEW followup: derive dependencies on underlying base table partitions from view definition

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.6.0
    • Fix Version/s: None
    • Component/s: Query Processor
    • Labels:
      None

      Description

      When querying a view, it would be useful to know which underlying base table partitions it depends on in order to know how fresh the result is (or to be able to wait until all of those partitions have been loaded consistently). The task is to come up with a way to perform this analysis automatically (possibly overconservatively), or alternately to let the view creator annotate the view definition with this dependency information, or some combination of the two.

      Note that this would be useful for any complex query which directly accesses base tables (not just view definitions).

        Issue Links

          Activity

          Hide
          John Sichi added a comment -

          Note that the manual approach has been implemented in HIVE-1941; see https://cwiki.apache.org/confluence/display/Hive/PartitionedViews

          for more info on it. The implicit (derived) approach remains to be implemented.

          Show
          John Sichi added a comment - Note that the manual approach has been implemented in HIVE-1941 ; see https://cwiki.apache.org/confluence/display/Hive/PartitionedViews for more info on it. The implicit (derived) approach remains to be implemented.
          Hide
          John Sichi added a comment -

          For the manual approach, a very simple thing we can start with is just letting users add partitions to views explicitly as a way of indicating that the underlying data is ready.

          Show
          John Sichi added a comment - For the manual approach, a very simple thing we can start with is just letting users add partitions to views explicitly as a way of indicating that the underlying data is ready.
          Hide
          John Sichi added a comment -

          Additional user feedback:

          (1) Take the 80/20 approach and focus on the single-table use case for starters; don't get hung up on the complicated join/union cases.

          (2) Propagating metadata about the partitioning key through to the view will be important in UI's which auto-suggest a filter on the partitioning columns. A nuance here is that if the filter is baked into the view definition itself (i.e. the view is partition-specific), then we might not want to propagate it (if the view creator forgot to project it away).

          (3) It would be nice if a system function CURRENT_PARTITION() were provided so that the view definition could bake in a dynamic filter like WHERE ds=CURRENT_PARTITION(). This is an independent but related feature whose semantics remain to be worked out.

          Show
          John Sichi added a comment - Additional user feedback: (1) Take the 80/20 approach and focus on the single-table use case for starters; don't get hung up on the complicated join/union cases. (2) Propagating metadata about the partitioning key through to the view will be important in UI's which auto-suggest a filter on the partitioning columns. A nuance here is that if the filter is baked into the view definition itself (i.e. the view is partition-specific), then we might not want to propagate it (if the view creator forgot to project it away). (3) It would be nice if a system function CURRENT_PARTITION() were provided so that the view definition could bake in a dynamic filter like WHERE ds=CURRENT_PARTITION(). This is an independent but related feature whose semantics remain to be worked out.

            People

            • Assignee:
              Prajakta Kalmegh
              Reporter:
              John Sichi
            • Votes:
              1 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:

                Development