[IMPALA-7712] Impala read from and write to GCS

• Type: New Feature
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 3.3.0
• Fix Version/s: Impala 4.0.0
• Component/s: Backend
• Labels: cloudera connector google-cloud-storage impala
• Epic Color: ghx-label-4
• Assignee: Quanlong Huang
• Reporter: Haaris
• Created: 16/Oct/18 08:08
• Updated: 10/Mar/23 10:11
• Resolved: 19/Mar/21 07:46

Description

Can Impala read from and write to google cloud storage GCS like the way it does with amazon s3

I have tested the use case with S3, but when talking to GCS impala errors out with:
Query: create table gcs_impala2 (title string) location 'gs://mybucket-gcs/some_data/' ERROR: AnalysisException: null CAUSED BY: RuntimeException: java.lang.ClassNotFoundException: Class com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem not found CAUSED BY: ClassNotFoundException: Class com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem not found
 On the same cluster i have Hive talking to GCS using the GCS connector jar provided by google form :
https://cloud.google.com/dataproc/docs/concepts/connectors/install-storage-connector
 
Also, HDFS reads and writes from/to GCS.
 
Made sure java version matches and appropriate values are in classpath.
 
Appreciate your time and effort.
Thanks

Issue Links

• is related to: IMPALA-10561 Add support for spilling to GCS
• is related to: IMPALA-10568 Enable file handle cache for GCS files
• is related to: IMPALA-10562 TestGracefulShutdown::test_shutdown_idle failes in GCE instance
• links to: Code Review