[IGNITE-3078] IgniteContext needs option to read config from HDFS

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.5.0.final
• Component/s: spark
• Labels: bigdata
• Assignee: Unassigned
• Reporter: Franco
• Created: 02/May/16 00:30
• Updated: 27/May/20 03:58

Description

If you are running Ignite in YARN mode then obviously we aren't going to have Ignite installation directories on each of the Hadoop cluster nodes. Therefore we need to store the configuration in HDFS for it to be accessible everywhere.

What is needed is a new constructor in IgniteContext:

def this(
sc: SparkContext,
hdfsPath: org.apache.hadoop.fs.Path
)

{ this(sc, () ⇒ IgnitionEx.loadHDFSConfiguration(hdfsPath).get1()) }

Issue Links

• links to: GitHub Pull Request #7854