Details
-
Sub-task
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Description
Currently the PDMService only works if the hcat dependency URI partitions match exactly with all the partitions in the notification. This is mostly not the case as there will be lot of sub-partitions in a table, but only the date partition will be specified for dependency check. So need to do a subset match.
Also need to extract out the dependency cache and make an interface, so that a cache which spills to disk can later be substituted. Spillable cache is required for scale when number of pending dependencies will be high.