Details

      Description

      [this is my first issue request here, please apologize if something is missing]

      JDBCInputFormat of flink 1.1-SNAPSHOT fails with an NPE in Row.productArity:

      java.io.IOException: Couldn't access resultSet
      at org.apache.flink.api.java.io.jdbc.JDBCInputFormat.nextRecord(JDBCInputFormat.java:288)
      at org.apache.flink.api.java.io.jdbc.JDBCInputFormat.nextRecord(JDBCInputFormat.java:98)
      at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:162)
      at org.apache.flink.runtime.taskmanager.Task.run(Task.java:588)
      at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.NullPointerException
      at org.apache.flink.api.table.Row.productArity(Row.scala:28)
      at org.apache.flink.api.java.io.jdbc.JDBCInputFormat.nextRecord(JDBCInputFormat.java:279)
      ... 4 more

      The code reproduce this can be found in this gist: https://gist.github.com/zeitgeist/b91a60460661618ca4585e082895c616

      The reason for the NPE, I believe, is the way through which Flink creates Row instances through Kryo. As Row expects the number of fields to allocate as a parameter, which Kryo does not provide, the ‘fields’ member of Row ends up being null. As I’m neither a reflection nor a Kryo expert, I rather leave a true analysis to more knowledgable programmers.

      Part of the aforementioned example is a not very elegant workaround though a custom type and a cast (function jdbcNoIssue + custom Row type MyRow) to serve as a further hint towards my theory.

        Issue Links

          Activity

          Hide
          twalthr Timo Walther added a comment -

          Actually, RowTypeInfo was intended for the Table API only. If it is used outside of the Table API there should be additional tests.

          Show
          twalthr Timo Walther added a comment - Actually, RowTypeInfo was intended for the Table API only. If it is used outside of the Table API there should be additional tests.
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user twalthr opened a pull request:

          https://github.com/apache/flink/pull/2619

          FLINK-4108 [scala] Consider ResultTypeQueryable for input formats

          Thanks for contributing to Apache Flink. Before you open your pull request, please take the following check list into consideration.
          If your changes take all of the items into account, feel free to open your pull request. For more information and/or questions please refer to the [How To Contribute guide](http://flink.apache.org/how-to-contribute.html).
          In addition to going through the list, please provide a meaningful description of your changes.

          • [x] General
          • The pull request references the related JIRA issue ("[FLINK-XXX] Jira title text")
          • The pull request addresses only one issue
          • Each commit in the PR has a meaningful commit message (including the JIRA id)
          • [x] Documentation
          • Documentation has been added for new functionality
          • Old documentation affected by the pull request has been updated
          • JavaDoc for public methods has been added
          • [x] Tests & Build
          • Functionality added by the pull request is covered by tests
          • `mvn clean verify` has been executed successfully locally or a Travis build has passed

          This PR fixes issues with input formats (i.e. the JDBC input format) in the Scala API. Now the API also considers explicit information defines with ResultTypeQueryable.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/twalthr/flink FLINK-4108

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flink/pull/2619.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #2619


          commit 24b0cac3b58c1ea52b3abc7af3484f6701ab5047
          Author: twalthr <twalthr@apache.org>
          Date: 2016-10-11T09:19:32Z

          FLINK-4108 [scala] Consider ResultTypeQueryable for input formats


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user twalthr opened a pull request: https://github.com/apache/flink/pull/2619 FLINK-4108 [scala] Consider ResultTypeQueryable for input formats Thanks for contributing to Apache Flink. Before you open your pull request, please take the following check list into consideration. If your changes take all of the items into account, feel free to open your pull request. For more information and/or questions please refer to the [How To Contribute guide] ( http://flink.apache.org/how-to-contribute.html ). In addition to going through the list, please provide a meaningful description of your changes. [x] General The pull request references the related JIRA issue (" [FLINK-XXX] Jira title text") The pull request addresses only one issue Each commit in the PR has a meaningful commit message (including the JIRA id) [x] Documentation Documentation has been added for new functionality Old documentation affected by the pull request has been updated JavaDoc for public methods has been added [x] Tests & Build Functionality added by the pull request is covered by tests `mvn clean verify` has been executed successfully locally or a Travis build has passed This PR fixes issues with input formats (i.e. the JDBC input format) in the Scala API. Now the API also considers explicit information defines with ResultTypeQueryable. You can merge this pull request into a Git repository by running: $ git pull https://github.com/twalthr/flink FLINK-4108 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/2619.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2619 commit 24b0cac3b58c1ea52b3abc7af3484f6701ab5047 Author: twalthr <twalthr@apache.org> Date: 2016-10-11T09:19:32Z FLINK-4108 [scala] Consider ResultTypeQueryable for input formats
          Hide
          githubbot ASF GitHub Bot added a comment -
          Show
          githubbot ASF GitHub Bot added a comment - Github user albertoRamon commented on the issue: https://github.com/apache/flink/pull/2619 Works OK ¡¡ [Capture Output] ( http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/jdbc-JDBCInputFormat-tp9393p9489.html ) Thanks, Alb
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user fhueske commented on a diff in the pull request:

          https://github.com/apache/flink/pull/2619#discussion_r83489370

          — Diff: flink-scala/src/main/scala/org/apache/flink/api/scala/ExecutionEnvironment.scala —
          @@ -110,7 +110,7 @@ class ExecutionEnvironment(javaEnv: JavaEnv) {
          */
          @PublicEvolving
          def getRestartStrategy: RestartStrategyConfiguration = {

          • javaEnv.getRestartStrategy()
            + javaEnv.getRestartStrategy
              • End diff –

          We call methods on Java objects with parentheses.

          Show
          githubbot ASF GitHub Bot added a comment - Github user fhueske commented on a diff in the pull request: https://github.com/apache/flink/pull/2619#discussion_r83489370 — Diff: flink-scala/src/main/scala/org/apache/flink/api/scala/ExecutionEnvironment.scala — @@ -110,7 +110,7 @@ class ExecutionEnvironment(javaEnv: JavaEnv) { */ @PublicEvolving def getRestartStrategy: RestartStrategyConfiguration = { javaEnv.getRestartStrategy() + javaEnv.getRestartStrategy End diff – We call methods on Java objects with parentheses.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user fhueske commented on the issue:

          https://github.com/apache/flink/pull/2619

          Reverting the change and merging

          Show
          githubbot ASF GitHub Bot added a comment - Github user fhueske commented on the issue: https://github.com/apache/flink/pull/2619 Reverting the change and merging
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user fhueske commented on a diff in the pull request:

          https://github.com/apache/flink/pull/2619#discussion_r83502181

          — Diff: flink-scala/src/main/scala/org/apache/flink/api/scala/ExecutionEnvironment.scala —
          @@ -110,7 +110,7 @@ class ExecutionEnvironment(javaEnv: JavaEnv) {
          */
          @PublicEvolving
          def getRestartStrategy: RestartStrategyConfiguration = {

          • javaEnv.getRestartStrategy()
            + javaEnv.getRestartStrategy
              • End diff –

          Apparently this is not the case in this class. Similar method are also called without parentheses. So lets keep this change to have consistent code in this class.

          Show
          githubbot ASF GitHub Bot added a comment - Github user fhueske commented on a diff in the pull request: https://github.com/apache/flink/pull/2619#discussion_r83502181 — Diff: flink-scala/src/main/scala/org/apache/flink/api/scala/ExecutionEnvironment.scala — @@ -110,7 +110,7 @@ class ExecutionEnvironment(javaEnv: JavaEnv) { */ @PublicEvolving def getRestartStrategy: RestartStrategyConfiguration = { javaEnv.getRestartStrategy() + javaEnv.getRestartStrategy End diff – Apparently this is not the case in this class. Similar method are also called without parentheses. So lets keep this change to have consistent code in this class.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/flink/pull/2619

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/2619
          Hide
          fhueske Fabian Hueske added a comment -

          Fixed for 1.1.4 with 9f7269808f3694815bba1e4dbf050db2a2dfe15f
          Fixed for 1.2.0 with dc99deb43fb876400615b0b77cf3471768482646

          Show
          fhueske Fabian Hueske added a comment - Fixed for 1.1.4 with 9f7269808f3694815bba1e4dbf050db2a2dfe15f Fixed for 1.2.0 with dc99deb43fb876400615b0b77cf3471768482646

            People

            • Assignee:
              twalthr Timo Walther
              Reporter:
              ms@funkpopes.org Martin Scholl
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development