Uploaded image for project: 'Phoenix'
  1. Phoenix
  2. PHOENIX-5197

Use extraOptions to set the configuration for Spark Workers

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: connectors-6.0.0
    • Fix Version/s: connectors-6.0.0
    • Component/s: None
    • Labels:
      None

      Description

      We need a standardized way to set configurations for Spark driver and workers when reading and writing data.

      Quoting from my offline discussion with Thomas D'Silva:

      "if we use phoenixTableAsRDD to create the rdd then we can pass in a config
      if we use the standard df.write.format() we can't pass in a conf
      maybe the best was to do this is to put config in the option map when we call read/write .format and then set those in the config"

        Attachments

        1. PHOENIX-5197-v1.patch
          32 kB
          Chinmay Kulkarni
        2. PHOENIX-5197.patch
          19 kB
          Chinmay Kulkarni

          Issue Links

            Activity

              People

              • Assignee:
                ckulkarni Chinmay Kulkarni
                Reporter:
                ckulkarni Chinmay Kulkarni
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m