[CARBONDATA-2999] support read schema from S3

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Resolved
• Affects Version/s: 1.5.0
• Assignee: Bo Xu
• Reporter: Bo Xu
• Created: 10/Oct/18 08:47
• Updated: 29/Dec/18 01:48
• Resolved: 29/Dec/18 01:48

Description

support read schema from S3

Code:

      String path = "s3a://sdk/WriterOutput/carbondata5";

        if (args.length > 3) {
            path=args[3];
        }

        Schema schema = CarbonSchemaReader.readSchema(path);
        System.out.println(schema.getFieldsLength());

Exception:

WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain
	at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:117)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3521)
	at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1031)
	at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:994)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:297)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
	at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
	at org.apache.carbondata.core.datastore.filesystem.AbstractDFSCarbonFile.<init>(AbstractDFSCarbonFile.java:74)
	at org.apache.carbondata.core.datastore.filesystem.AbstractDFSCarbonFile.<init>(AbstractDFSCarbonFile.java:66)
	at org.apache.carbondata.core.datastore.filesystem.HDFSCarbonFile.<init>(HDFSCarbonFile.java:41)
	at org.apache.carbondata.core.datastore.filesystem.S3CarbonFile.<init>(S3CarbonFile.java:41)
	at org.apache.carbondata.core.datastore.impl.DefaultFileTypeProvider.getCarbonFile(DefaultFileTypeProvider.java:53)
	at org.apache.carbondata.core.datastore.impl.FileFactory.getCarbonFile(FileFactory.java:99)
	at org.apache.carbondata.sdk.file.CarbonSchemaReader.getCarbonFile(CarbonSchemaReader.java:79)
	at org.apache.carbondata.sdk.file.CarbonSchemaReader.readSchema(CarbonSchemaReader.java:150)
	at org.apache.carbondata.sdk.file.CarbonSchemaReader.readSchema(CarbonSchemaReader.java:109)
	at org.apache.carbondata.examples.sdk.SDKS3SchemaReadExample.main(SDKS3SchemaReadExample.java:51)

Issue Links

• links to: GitHub Pull Request #2931
• links to: GitHub Pull Request #2934