Affects Version/s: 2.3.0
Fix Version/s: None
During investigation of CBO and DataSourceV2 we found, that
misses attribute/column statistics and that
without forwarding the optional rowCount if present.
However rowCount and attributeStats are used during CBO e.g. in JoinEstimation and AggregateEstimation.
We propose that:
- org.apache.spark.sql.sources.v2.reader.Statistics mirrors org.apache.spark.sql.catalyst.plans.logical.Statistics
- DataSourceV2Relation forwards all the information to be available during CBO