[FLINK-27134] OSS sink can't get access key id

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 1.12.3
• Component/s: FileSystems

Environment

flink version: 1.12.3
patch: flink-11388
fs.oss.sdk.version:3.4.1

add flink-conf.yaml:
fs.oss.credentials.provider: com.aliyun.oss.common.auth.EnvironmentVariableCredentialsProvider
fs.oss.endpoint: cn-shanghai-internal.aliyuncs.com
fs.oss.accessKeyId: ******
fs.oss.accessKeySecret: ******

• External issue URL: https://issues.apache.org/jira/browse/FLINK-11388?jql=project%20%3D%20FLINK%20AND%20component%20%3D%20FileSystems%20AND%20text%20~%20%22oss%22
• External issue ID: 11388

Docs Text

Error message:
org.apache.flink.util.SerializedThrowable: Access key id should not be null or empty.
at org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss.common.auth.EnvironmentVariableCredentialsProvider.getCredentials(EnvironmentVariableCredentialsProvider.java:44) ~[?:?]
at org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss.internal.OSSOperation.createDefaultContext(OSSOperation.java:166) ~[?:?]
at org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:114) ~[?:?]
at org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:102) ~[?:?]
at org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss.internal.OSSMultipartOperation.initiateMultipartUpload(OSSMultipartOperation.java:226) ~[?:?]
at org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss.OSSClient.initiateMultipartUpload(OSSClient.java:727) ~[?:?]
at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemStore.getUploadId(AliyunOSSFileSystemStore.java:640) ~[?:?]
at org.apache.flink.fs.osshadoop.OSSAccessor.startMultipartUpload(OSSAccessor.java:61) ~[?:?]
at org.apache.flink.fs.osshadoop.writer.OSSRecoverableMultipartUpload.<init>(OSSRecoverableMultipartUpload.java:81) ~[?:?]
at org.apache.flink.fs.osshadoop.writer.OSSRecoverableWriter.open(OSSRecoverableWriter.java:73) ~[?:?]
at org.apache.flink.streaming.api.functions.sink.filesystem.OutputStreamBasedPartFileWriter$OutputStreamBasedBucketWriter.openNewInProgressFile(OutputStreamBasedPartFileWriter.java:90) ~[flink-table-blink_2.11-1.12.3.jar:1.12.3]
at org.apache.flink.streaming.api.functions.sink.filesystem.RowWiseBucketWriter.openNewInProgressFile(RowWiseBucketWriter.java:34) ~[flink-table-blink_2.11-1.12.3.jar:1.12.3]
at org.apache.flink.streaming.api.functions.sink.filesystem.Bucket.rollPartFile(Bucket.java:243) ~[flink-dist_2.11-1.12.3.jar:1.12.3]
at org.apache.flink.streaming.api.functions.sink.filesystem.Bucket.write(Bucket.java:220) ~[flink-dist_2.11-1.12.3.jar:1.12.3]
at org.apache.flink.streaming.api.functions.sink.filesystem.Buckets.onElement(Buckets.java:305) ~[flink-dist_2.11-1.12.3.jar:1.12.3]
at org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSinkHelper.onElement(StreamingFileSinkHelper.java:103) ~[flink-dist_2.11-1.12.3.jar:1.12.3]
at org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink.invoke(StreamingFileSink.java:492) ~[flink-dist_2.11-1.12.3.jar:1.12.3]
at org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:54) ~[flink-dist_2.11-1.12.3.jar:1.12.3]
at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:191) ~[flink-dist_2.11-1.12.3.jar:1.12.3]
at org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.processElement(StreamTaskNetworkInput.java:204) ~[flink-dist_2.11-1.12.3.jar:1.12.3]
at org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.emitNext(StreamTaskNetworkInput.java:174) ~[flink-dist_2.11-1.12.3.jar:1.12.3]
at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65) ~[flink-dist_2.11-1.12.3.jar:1.12.3]
at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:396) ~[flink-dist_2.11-1.12.3.jar:1.12.3]
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:191) ~[flink-dist_2.11-1.12.3.jar:1.12.3]
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:617) ~[flink-dist_2.11-1.12.3.jar:1.12.3]
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:581) ~[flink-dist_2.11-1.12.3.jar:1.12.3]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:755) ~[flink-dist_2.11-1.12.3.jar:1.12.3]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:570) ~[flink-dist_2.11-1.12.3.jar:1.12.3]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_221]

• Assignee: Unassigned
• Reporter: Alex Z
• Created: 08/Apr/22 09:21
• Updated: 20/May/22 11:05
• Resolved: 08/Apr/22 10:24

Description

When I cherry-pick the flink-11388 patch to our flink code(version:1.12.3),the OSS sink can't Work.The error message is Access key id should not be null or empty,but I has been added the OSS configuration in flink-conf.yaml .And the configuration propertity also print the correct fs.oss.accessKeyId