[BEAM-12881] Issue with Beam JDBC IO for GCP Dataflow - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Open
Priority: P3
Resolution: Unresolved
Affects Version/s: 2.31.0
Fix Version/s: None
Component/s: cross-language, io-java-jdbc
Labels:
- GCP
- beam
- dataflow
- jdbc_connector
- python3
Environment:
python 3.7.3
openjdk version "11.0.12" 2021-07-20
OpenJDK Runtime Environment (build 11.0.12+7-post-Debian-2deb10u1)

Flags:

Important
Language:
- Python

Description

Hi,

We have a dataflow job in GCP that we are hoping to use PostgreSQL as our sink for streaming data. Currently we are trying to connect to instance using apache_beam.io.jdbc(default expansion service) but we are constantly getting error while trying to create the tempate for dataflow job. The error seems to come at "INFO: Expanding 'load to sql' with URN 'beam:external:java:schemaio:jdbc:write:v1'"

Below is the code and error log.

Connection portion.
load_data = data | "load to sql" >> WriteToJdbc(
                                            table_name='test',
                                            driver_class_name='org.postgresql.Driver',
                                            jdbc_url='jdbc:{}://{}:{}/{}'.format("postgresql", "ip", "5432", "postgres"),
                                            username='test_user',
                                            password='password'
                                        )            

Error:
INFO:root:Dataflow job is running on dev INFO:root:Job Run Id is 3618 INFO:root:Job Name is validation-parser-20210913-17-46 INFO:apache_beam.utils.subprocess_server:Using cached job server jar from https://repo.maven.apache.org/maven2/org/apache/beam/beam-sdks-java-extensions-schemaio-expansion-service/2.31.0/beam-sdks-java-extensions-schemaio-expansion-service-2.31.0.jar INFO:apache_beam.utils.subprocess_server:Starting service with ['java' '-jar' '/home/user/.apache_beam/cache/jars/beam-sdks-java-extensions-schemaio-expansion-service-2.31.0.jar' '39547'] INFO:apache_beam.utils.subprocess_server:b'Starting expansion service at localhost:39547' INFO:apache_beam.utils.subprocess_server:b'Sep 13, 2021 5:46:34 PM org.apache.beam.sdk.expansion.service.ExpansionService loadRegisteredTransforms' INFO:apache_beam.utils.subprocess_server:b'INFO: Registering external transforms: [beam:external:java:schemaio:jdbc:read:v1, beam:external:java:schemaio:jdbc:write:v1, beam:external:java:schemaio:avro:read:v1, beam:external:java:schemaio:avro:write:v1, beam:external:java:generate_sequence:v1]' INFO:apache_beam.utils.subprocess_server:b'\tbeam:external:java:schemaio:jdbc:read:v1: org.apache.beam.sdk.expansion.service.ExpansionService$ExternalTransformRegistrarLoader$$Lambda$11/0x000000084007f840@13b6d03' INFO:apache_beam.utils.subprocess_server:b'\tbeam:external:java:schemaio:jdbc:write:v1: org.apache.beam.sdk.expansion.service.ExpansionService$ExternalTransformRegistrarLoader$$Lambda$11/0x000000084007f840@f5f2bb7' INFO:apache_beam.utils.subprocess_server:b'\tbeam:external:java:schemaio:avro:read:v1: org.apache.beam.sdk.expansion.service.ExpansionService$ExternalTransformRegistrarLoader$$Lambda$11/0x000000084007f840@73035e27' INFO:apache_beam.utils.subprocess_server:b'\tbeam:external:java:schemaio:avro:write:v1: org.apache.beam.sdk.expansion.service.ExpansionService$ExternalTransformRegistrarLoader$$Lambda$11/0x000000084007f840@64c64813' INFO:apache_beam.utils.subprocess_server:b'\tbeam:external:java:generate_sequence:v1: org.apache.beam.sdk.expansion.service.ExpansionService$ExternalTransformRegistrarLoader$$Lambda$11/0x000000084007f840@3ecf72fd' INFO:apache_beam.utils.subprocess_server:b'Sep 13, 2021 5:46:35 PM org.apache.beam.sdk.expansion.service.ExpansionService expand' INFO:apache_beam.utils.subprocess_server:b"INFO: Expanding 'load to sql' with URN 'beam:external:java:schemaio:jdbc:write:v1'" ERROR:root:Failed to launch datapipeline. Error org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalArgumentException: Unexpected type_info: TYPEINFO_NOT_SET at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2050) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache.get(LocalCache.java:3952) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958) at org.apache.beam.runners.core.construction.RehydratedComponents.getPCollection(RehydratedComponents.java:139) at org.apache.beam.sdk.expansion.service.ExpansionService.lambda$expand$0(ExpansionService.java:425) at java.base/java.util.stream.Collectors.lambda$uniqKeysMapAccumulator$1(Collectors.java:178) at java.base/java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169) at java.base/java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet.lambda$entryConsumer$0(Collections.java:1576) at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133) at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) at java.base/java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet$UnmodifiableEntrySetSpliterator.forEachRemaining(Collections.java:1601) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) at org.apache.beam.sdk.expansion.service.ExpansionService.expand(ExpansionService.java:420) at org.apache.beam.sdk.expansion.service.ExpansionService.expand(ExpansionService.java:494) at org.apache.beam.model.expansion.v1.ExpansionServiceGrpc$MethodHandlers.invoke(ExpansionServiceGrpc.java:219) at org.apache.beam.vendor.grpc.v1p36p0.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:182) at org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:331) at org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:797) at org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalArgumentException: Unexpected type_info: TYPEINFO_NOT_SET at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2050) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache.get(LocalCache.java:3952) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958) at org.apache.beam.runners.core.construction.RehydratedComponents.getCoder(RehydratedComponents.java:168) at org.apache.beam.runners.core.construction.PCollectionTranslation.fromProto(PCollectionTranslation.java:51) at org.apache.beam.runners.core.construction.RehydratedComponents$3.load(RehydratedComponents.java:108) at org.apache.beam.runners.core.construction.RehydratedComponents$3.load(RehydratedComponents.java:98) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3528) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2277) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2154) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2044) ... 27 more Caused by: java.lang.IllegalArgumentException: Unexpected type_info: TYPEINFO_NOT_SET at org.apache.beam.sdk.schemas.SchemaTranslation.fieldTypeFromProtoWithoutNullable(SchemaTranslation.java:362) at org.apache.beam.sdk.schemas.SchemaTranslation.fieldTypeFromProto(SchemaTranslation.java:261) at org.apache.beam.sdk.schemas.SchemaTranslation.fieldTypeFromProtoWithoutNullable(SchemaTranslation.java:358) at org.apache.beam.sdk.schemas.SchemaTranslation.fieldTypeFromProto(SchemaTranslation.java:261) at org.apache.beam.sdk.schemas.SchemaTranslation.fieldFromProto(SchemaTranslation.java:255) at org.apache.beam.sdk.schemas.SchemaTranslation.schemaFromProto(SchemaTranslation.java:229) at org.apache.beam.runners.core.construction.CoderTranslators$8.fromComponents(CoderTranslators.java:169) at org.apache.beam.runners.core.construction.CoderTranslators$8.fromComponents(CoderTranslators.java:151) at org.apache.beam.runners.core.construction.CoderTranslation.fromKnownCoder(CoderTranslation.java:170) at org.apache.beam.runners.core.construction.CoderTranslation.fromProto(CoderTranslation.java:145) at org.apache.beam.runners.core.construction.RehydratedComponents$2.load(RehydratedComponents.java:87) at org.apache.beam.runners.core.construction.RehydratedComponents$2.load(RehydratedComponents.java:82) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3528) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2277) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2154) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2044) ... 38 more Traceback (most recent call last): File "muthu_sql.py", line 921, in run password='E9d3X5h' File "/home/user/venv/lib/python3.7/site-packages/apache_beam/pvalue.py", line 136, in __or__ return self.pipeline.apply(ptransform, self) File "/home/user/venv/lib/python3.7/site-packages/apache_beam/pipeline.py", line 641, in apply transform.transform, pvalueish, label or transform.label) File "/home/user/venv/lib/python3.7/site-packages/apache_beam/pipeline.py", line 651, in apply return self.apply(transform, pvalueish) File "/home/user/venv/lib/python3.7/site-packages/apache_beam/pipeline.py", line 694, in apply pvalueish_result = self.runner.apply(transform, pvalueish, self._options) File "/home/user/venv/lib/python3.7/site-packages/apache_beam/runners/dataflow/dataflow_runner.py", line 141, in apply return super(DataflowRunner, self).apply(transform, input, options) File "/home/user/venv/lib/python3.7/site-packages/apache_beam/runners/runner.py", line 185, in apply return m(transform, input, options) File "/home/user/venv/lib/python3.7/site-packages/apache_beam/runners/runner.py", line 215, in apply_PTransform return transform.expand(input) File "/home/user/venv/lib/python3.7/site-packages/apache_beam/transforms/external.py", line 297, in expand raise RuntimeError(response.error) RuntimeError: org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalArgumentException: Unexpected type_info: TYPEINFO_NOT_SET at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2050) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache.get(LocalCache.java:3952) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958) at org.apache.beam.runners.core.construction.RehydratedComponents.getPCollection(RehydratedComponents.java:139) at org.apache.beam.sdk.expansion.service.ExpansionService.lambda$expand$0(ExpansionService.java:425) at java.base/java.util.stream.Collectors.lambda$uniqKeysMapAccumulator$1(Collectors.java:178) at java.base/java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169) at java.base/java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet.lambda$entryConsumer$0(Collections.java:1576) at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133) at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) at java.base/java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet$UnmodifiableEntrySetSpliterator.forEachRemaining(Collections.java:1601) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) at org.apache.beam.sdk.expansion.service.ExpansionService.expand(ExpansionService.java:420) at org.apache.beam.sdk.expansion.service.ExpansionService.expand(ExpansionService.java:494) at org.apache.beam.model.expansion.v1.ExpansionServiceGrpc$MethodHandlers.invoke(ExpansionServiceGrpc.java:219) at org.apache.beam.vendor.grpc.v1p36p0.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:182) at org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:331) at org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:797) at org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalArgumentException: Unexpected type_info: TYPEINFO_NOT_SET at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2050) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache.get(LocalCache.java:3952) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958) at org.apache.beam.runners.core.construction.RehydratedComponents.getCoder(RehydratedComponents.java:168) at org.apache.beam.runners.core.construction.PCollectionTranslation.fromProto(PCollectionTranslation.java:51) at org.apache.beam.runners.core.construction.RehydratedComponents$3.load(RehydratedComponents.java:108) at org.apache.beam.runners.core.construction.RehydratedComponents$3.load(RehydratedComponents.java:98) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3528) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2277) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2154) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2044) ... 27 more Caused by: java.lang.IllegalArgumentException: Unexpected type_info: TYPEINFO_NOT_SET at org.apache.beam.sdk.schemas.SchemaTranslation.fieldTypeFromProtoWithoutNullable(SchemaTranslation.java:362) at org.apache.beam.sdk.schemas.SchemaTranslation.fieldTypeFromProto(SchemaTranslation.java:261) at org.apache.beam.sdk.schemas.SchemaTranslation.fieldTypeFromProtoWithoutNullable(SchemaTranslation.java:358) at org.apache.beam.sdk.schemas.SchemaTranslation.fieldTypeFromProto(SchemaTranslation.java:261) at org.apache.beam.sdk.schemas.SchemaTranslation.fieldFromProto(SchemaTranslation.java:255) at org.apache.beam.sdk.schemas.SchemaTranslation.schemaFromProto(SchemaTranslation.java:229) at org.apache.beam.runners.core.construction.CoderTranslators$8.fromComponents(CoderTranslators.java:169) at org.apache.beam.runners.core.construction.CoderTranslators$8.fromComponents(CoderTranslators.java:151) at org.apache.beam.runners.core.construction.CoderTranslation.fromKnownCoder(CoderTranslation.java:170) at org.apache.beam.runners.core.construction.CoderTranslation.fromProto(CoderTranslation.java:145) at org.apache.beam.runners.core.construction.RehydratedComponents$2.load(RehydratedComponents.java:87) at org.apache.beam.runners.core.construction.RehydratedComponents$2.load(RehydratedComponents.java:82) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3528) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2277) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2154) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2044) ... 38 moreTraceback (most recent call last): File "muthu_sql.py", line 985, in <module> main(args.env, args.run_date) File "muthu_sql.py", line 970, in main run() File "muthu_sql.py", line 921, in run password='E9d3X5h' File "/home/user/venv/lib/python3.7/site-packages/apache_beam/pvalue.py", line 136, in __or__ return self.pipeline.apply(ptransform, self) File "/home/user/venv/lib/python3.7/site-packages/apache_beam/pipeline.py", line 641, in apply transform.transform, pvalueish, label or transform.label) File "/home/user/venv/lib/python3.7/site-packages/apache_beam/pipeline.py", line 651, in apply return self.apply(transform, pvalueish) File "/home/user/venv/lib/python3.7/site-packages/apache_beam/pipeline.py", line 694, in apply pvalueish_result = self.runner.apply(transform, pvalueish, self._options) File "/home/user/venv/lib/python3.7/site-packages/apache_beam/runners/dataflow/dataflow_runner.py", line 141, in apply return super(DataflowRunner, self).apply(transform, input, options) File "/home/user/venv/lib/python3.7/site-packages/apache_beam/runners/runner.py", line 185, in apply return m(transform, input, options) File "/home/user/venv/lib/python3.7/site-packages/apache_beam/runners/runner.py", line 215, in apply_PTransform return transform.expand(input) File "/home/user/venv/lib/python3.7/site-packages/apache_beam/transforms/external.py", line 297, in expand raise RuntimeError(response.error) RuntimeError: org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalArgumentException: Unexpected type_info: TYPEINFO_NOT_SET at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2050) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache.get(LocalCache.java:3952) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958) at org.apache.beam.runners.core.construction.RehydratedComponents.getPCollection(RehydratedComponents.java:139) at org.apache.beam.sdk.expansion.service.ExpansionService.lambda$expand$0(ExpansionService.java:425) at java.base/java.util.stream.Collectors.lambda$uniqKeysMapAccumulator$1(Collectors.java:178) at java.base/java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169) at java.base/java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet.lambda$entryConsumer$0(Collections.java:1576) at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133) at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) at java.base/java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet$UnmodifiableEntrySetSpliterator.forEachRemaining(Collections.java:1601) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) at org.apache.beam.sdk.expansion.service.ExpansionService.expand(ExpansionService.java:420) at org.apache.beam.sdk.expansion.service.ExpansionService.expand(ExpansionService.java:494) at org.apache.beam.model.expansion.v1.ExpansionServiceGrpc$MethodHandlers.invoke(ExpansionServiceGrpc.java:219) at org.apache.beam.vendor.grpc.v1p36p0.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:182) at org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:331) at org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:797) at org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalArgumentException: Unexpected type_info: TYPEINFO_NOT_SET at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2050) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache.get(LocalCache.java:3952) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958) at org.apache.beam.runners.core.construction.RehydratedComponents.getCoder(RehydratedComponents.java:168) at org.apache.beam.runners.core.construction.PCollectionTranslation.fromProto(PCollectionTranslation.java:51) at org.apache.beam.runners.core.construction.RehydratedComponents$3.load(RehydratedComponents.java:108) at org.apache.beam.runners.core.construction.RehydratedComponents$3.load(RehydratedComponents.java:98) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3528) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2277) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2154) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2044) ... 27 more Caused by: java.lang.IllegalArgumentException: Unexpected type_info: TYPEINFO_NOT_SET at org.apache.beam.sdk.schemas.SchemaTranslation.fieldTypeFromProtoWithoutNullable(SchemaTranslation.java:362) at org.apache.beam.sdk.schemas.SchemaTranslation.fieldTypeFromProto(SchemaTranslation.java:261) at org.apache.beam.sdk.schemas.SchemaTranslation.fieldTypeFromProtoWithoutNullable(SchemaTranslation.java:358) at org.apache.beam.sdk.schemas.SchemaTranslation.fieldTypeFromProto(SchemaTranslation.java:261) at org.apache.beam.sdk.schemas.SchemaTranslation.fieldFromProto(SchemaTranslation.java:255) at org.apache.beam.sdk.schemas.SchemaTranslation.schemaFromProto(SchemaTranslation.java:229) at org.apache.beam.runners.core.construction.CoderTranslators$8.fromComponents(CoderTranslators.java:169) at org.apache.beam.runners.core.construction.CoderTranslators$8.fromComponents(CoderTranslators.java:151) at org.apache.beam.runners.core.construction.CoderTranslation.fromKnownCoder(CoderTranslation.java:170) at org.apache.beam.runners.core.construction.CoderTranslation.fromProto(CoderTranslation.java:145) at org.apache.beam.runners.core.construction.RehydratedComponents$2.load(RehydratedComponents.java:87) at org.apache.beam.runners.core.construction.RehydratedComponents$2.load(RehydratedComponents.java:82) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3528) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2277) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2154) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2044) ... 38 more(venv) user@cloudshell:~/sql_test (edp-dev-eds)$

Attachments

Activity

People

Assignee:: Unassigned

Reporter:: Gaurav Pathak

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Dates

Created:: 14/Sep/21 14:28

Updated:: 04/Jun/22 21:53