Uploaded image for project: 'Apache Sedona'
  1. Apache Sedona
  2. SEDONA-408

Set a reasonable default size for RasterUDT

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.5.0

    Description

      The defaultSize method of UserDefinedType is used by Spark SQL query optimizer to decide whether to broadcast the DataFrame or not. For RasterUDT, the default value is 100 bytes, which is the default size of BinaryType. This is almost always too small for RasterUDT and will lead to large raster DataFrames being mistakenly broadcasted. We can override this method and set a better default size for RasterUDT. Maybe 512 KB is a reasonable default value.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              kontinuation Kristin Cowalcijk
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m