Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
ghx-label-5
Description
Every partition currently has an HdfsStorageDescriptor attached. In most cases, the number of unique storage descriptors in a warehouse is pretty low (most partitions use the same escaping, file formats, etc). For example, in the functional test data load, we only have 24 unique SDs across ~10k partitions. Each object takes 32 bytes (with compressed oops) or 40 (without). So, we can get some small memory/object-count savings by interning these.