[SAMZA-2752] DiskSpace monitor reports 0 disk usage even for a stateful job with lots of disk usage

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Lakshmi Manasa Gaduputi
• Created: 27/Jun/22 20:16
• Updated: 27/Jun/22 20:18

Description

Stateful jobs after PR (#1491): report 0 disk usage for their disk spaced used by their rocksdb kv stores. 

Reason:

1. PR changed the init of rocksdb task stores from constructor of ContainerStorageManager to ContainerStorageManager.restoreStores() (which is invoked at run and not in the constructor). see these code changes of the PR for file ContainerStorageManager
2.  However, SamzaContainer creates the disk space monitor during setup (here) (before run() which invokes restoreStores()) and SamzaContainer uses the store paths returned by ContainerStorageManager after its constructor is done.
3. Hence, the task stores (not side input) are not part of the disk space monitoring.
4. logs show "2022-06-22 20:46:41.897 [main] SamzaContainer$ [INFO] Initialized disk space monitor watch paths to: [] // basically disk space monitor is not looking at anything" after this PR but should be showing a non-empty set of paths