[HBASE-28213] Evaluate using hbase-shaded-client-byo-hadoop for Spark connector

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Done
• Component/s: spark
• Assignee: Istvan Toth
• Reporter: Istvan Toth
• Created: 22/Nov/23 12:11
• Updated: 09/Jan/24 07:54
• Resolved: 09/Jan/24 07:44

Description

Since 3.2 Spark now uses hadoop-client-api and hadoop-client-runtime.
While we don't actually specify what HBase libraries are needed on the Spark client side for the connector, at least the Cloudera docs specify the classes provided by "hbase mapredcp"
which includes the full unshaded Hadoop JAR set.

Investigate whether  hbase-shaded-client-byo-hadoop and the hbase-client-api and hbase-client-runtime is enough for the connector, and if yes, document how to set the Spark classpath.

Alternatively, if hbase-shaded-client-byo-hadoop  is not enough, check if hbase-shaded-mapreduce plus the above two shaded Hadoop client JAR provides everything needed.

Issue Links

• is depended upon by: HBASE-28214 Document Spark classpath requirements for the Spark connector
• is depended upon by: PHOENIX-7120 Investigate changes needed for Spark connectors shading if hbase-shaded-client-byo-hadoop is used
• is related to: SPARK-33618 hadoop-aws doesn't work
• is related to: SPARK-33212 Upgrade to Hadoop 3.2.2 and move to shaded clients for Hadoop 3.x profile