Uploaded image for project: 'TinkerPop'
  1. TinkerPop
  2. TINKERPOP-911

Allow setting Thread Specific Spark JobGroup/Custom Properties based on hadoop conf

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.2-incubating
    • Fix Version/s: 3.1.0-incubating
    • Component/s: hadoop
    • Labels:
      None

      Description

      When using a Persistant Spark context it can be beneficial to pass in new configuration options for new users/ GraphComputers. Currently the .getOrCreate call will always use the configuration from the initial construction. To work around this we should iterate over all of the properties passed into the graph computer and set them as local context properties on the thread we are operating on.

      See
      https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkContext.scala#L630-L640

      This would let different graph computers set different spark properties for use with things like the Spark Fair Scheduler.

        Attachments

          Activity

            People

            • Assignee:
              okram Marko A. Rodriguez
              Reporter:
              rspitzer Russell Spitzer

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment