[BEAM-10858] beam.io.jdbc not working on Google Cloud Dataflow

• Type: Bug
• Status: Open
• Priority: P3
• Resolution: Unresolved
• Affects Version/s: 2.23.0
• Component/s: cross-language, extensions-java-gcp, runner-py-interactive
• Labels: Clarified GCP MySQL cross-platform dataflow java jdbc python
• Environment: Apache Beam 2.25.0.dev0 Jupyter Notebook running on Google Cloud Dataflow
• Assignee: Unassigned
• Reporter: Ehab El Badrawi
• Created: 08/Sep/20 13:25
• Updated: 30/Jan/21 17:13

Description

Using the JDBC python wrapper to connect to an external server is failing on Cloud Dataflow.

The error produced is;

RuntimeError: /root/apache-beam-custom/packages/beam/sdks/java/extensions/schemaio-expansion-service/build/libs/beam-sdks-java-extensions-schemaio-expansion-service-2.25.0-SNAPSHOT.jar not found. Please build the server with cd /root/apache-beam-custom/packages/beam; ./gradlew :sdks:java:extensions:schemaio-expansion-service:shadowJar

You can recreate the issue by running the below

from apache_beam.io import jdbc
jdbc.ReadFromJdbc(
 table_name='[tablename]',
 driver_class_name='com.mysql.cj.jdbc.Driver',
 jdbc_url='jdbc:mysql://[host]:3306/[databasename]',
 username='[username]',
 password='[password]')