Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-6288

Pyrolite calls hashCode to cache previously serialized objects

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 1.0.2, 1.1.1, 1.2.1, 1.3.0
    • 1.4.0
    • PySpark
    • None

    Description

      https://github.com/irmen/Pyrolite/blob/v2.0/java/src/net/razorvine/pickle/Pickler.java#L140

      This operation could be quite expensive, compared to serializing the object directly, because hashCode usually needs to access all data stored in the object. Maybe we should disable this feature by default.

      Attachments

        Issue Links

          Is contained by

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-7314 Upgrade Pyrolite to 4.4

          • Major - Major loss of function.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-5186 Vector.equals and Vector.hashCode are very inefficient and fail on SparseVectors with large size

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-7314 Upgrade Pyrolite to 4.4

          • Major - Major loss of function.
          • Resolved

          Task - A task that needs to be done. SPARK-7360 Compare Pyrolite performance affected by useMemo

          • Major - Major loss of function.
          • Closed

          Activity

            People

              mengxr Xiangrui Meng
              mengxr Xiangrui Meng
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: