Uploaded image for project: 'TinkerPop'
  1. TinkerPop
  2. TINKERPOP-2934

Optimize ObjectWritable for displaying content of Java Collection or Map to reduce OOM

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 3.6.2
    • 3.7.0, 3.6.3, 3.5.6
    • None
    • None

    Description

      OLAP query easily triggers OOM, I found a possible reason is https://github.com/apache/tinkerpop/blob/master/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/structure/io/ObjectWritable.java#L72 dump a huge string output. See what I captureed from SparkUI. In fact, that toString() does not make much sense for OLAP processing except for displaying in UI. Furthermore, those too long string is not readable for people. So, ideally the toString() for Map or Collection should be optimized to be java class + size(). Then the OOM will disappear.

      Attachments

        Activity

          People

            Unassigned Unassigned
            redriver Redriver
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: