Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Description
Due to Spark's limitations with broadcasts larger than 2G, in SystemML we use partitioned broadcasts that split a large side input into potentially many broadcast variables. For historic reasons the meta data is still maintained in the individual partitioned blocks. However, for many operations this meta data is accessed on the first partitioned block which leads to potentially unnecessary broadcast fetches.