Uploaded image for project: 'Phoenix'
  1. Phoenix
  2. PHOENIX-5197

Use extraOptions to set the configuration for Spark Workers

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: connectors-6.0.0
    • Fix Version/s: connectors-6.0.0
    • Component/s: None
    • Labels:
      None

      Description

      We need a standardized way to set configurations for Spark driver and workers when reading and writing data.

      Quoting from my offline discussion with Thomas D'Silva:

      "if we use phoenixTableAsRDD to create the rdd then we can pass in a config
      if we use the standard df.write.format() we can't pass in a conf
      maybe the best was to do this is to put config in the option map when we call read/write .format and then set those in the config"

        Attachments

          Activity

            People

            • Assignee:
              ckulkarni Chinmay Kulkarni
              Reporter:
              ckulkarni Chinmay Kulkarni

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 20m
                20m

                  Issue deployment