Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-5576

extend deserialization functions of KvStateRequestSerializer to detect unconsumed bytes

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.2.0, 1.3.0
    • Component/s: Queryable State
    • Labels:
      None

      Description

      KvStateRequestSerializer#deserializeValue and KvStateRequestSerializer#deserializeList both deserialize a given byte array. This is used by clients and unit tests and it is fair to assume that these byte arrays represent a complete value since we do not offer a method to continue reading from the middle of the array anyway. Therefore, we can treat unconsumed bytes as errors, e.g. from a wrong serializer being used, and throw a IOException with an appropriate failure message.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user NicoK opened a pull request:

          https://github.com/apache/flink/pull/3174

          FLINK-5576 extend deserialization functions of KvStateRequestSerializer to detect unconsumed bytes

          `KvStateRequestSerializer#deserializeValue()` deserializes a given byte array. This is used by clients and unit tests and it is fair to assume that these byte arrays represent a complete value since we do not offer a method to continue reading from the middle of the array anyway. Therefore, we can treat unconsumed bytes as errors, e.g. from a wrong serializer being used, and throw an `IOException` with an appropriate failure message.

          Similarly, this adds a better failure message to exceptions in `KvStateRequestSerializer#deserializeList()` by wrapping the original `IOException` into a new one with an appropriate error message just as in #3172.

          The new unit tests require #3171 to be accepted first on which this PR is also based.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/NicoK/flink flink-5576

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flink/pull/3174.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3174


          commit e29d5ef3099d5f5ffc817f3e5cdbb79c523b34c7
          Author: Nico Kruber <nico@data-artisans.com>
          Date: 2017-01-18T17:52:57Z

          FLINK-5561 fix DataInputDeserializer#available() 1 smaller than correct

          commit e589058e92d1f2a57d242987327777baa1fe64d4
          Author: Nico Kruber <nico@data-artisans.com>
          Date: 2017-01-19T16:06:00Z

          FLINK-5576 let KvStateRequestSerializer#deserializeValue() detect unconsumed bytes

          KvStateRequestSerializer#deserializeValue deserializes a given byte array. This is used by clients and unit tests and it is fair to assume that these byte arrays represent a complete value since we do not offer a method to continue reading from the middle of the array anyway. Therefore, we can treat unconsumed bytes as errors, e.g. from a wrong serializer being used, and throw a IOException with an appropriate failure message.

          commit 12d2108333b23e01b7b978a08e868a5f888506f9
          Author: Nico Kruber <nico@data-artisans.com>
          Date: 2017-01-19T16:10:54Z

          FLINK-5576 add a better failure message to exceptions in KvStateRequestSerializer#deserializeList()

          As in FLINK-5559, wrap the original IOException into a new one with an
          appropriate error message to better diagnose it.

          commit 9770e7f3269adfa5b80dd2125bcdbf8cebc4dab8
          Author: Nico Kruber <nico@data-artisans.com>
          Date: 2017-01-19T16:12:02Z

          FLINK-5576 add more unit tests for KvStateRequestSerializer#deserializeList()

          These tests ensure that some special cases not properly tested before are
          handled correctly in future.


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/3174 FLINK-5576 extend deserialization functions of KvStateRequestSerializer to detect unconsumed bytes `KvStateRequestSerializer#deserializeValue()` deserializes a given byte array. This is used by clients and unit tests and it is fair to assume that these byte arrays represent a complete value since we do not offer a method to continue reading from the middle of the array anyway. Therefore, we can treat unconsumed bytes as errors, e.g. from a wrong serializer being used, and throw an `IOException` with an appropriate failure message. Similarly, this adds a better failure message to exceptions in `KvStateRequestSerializer#deserializeList()` by wrapping the original `IOException` into a new one with an appropriate error message just as in #3172. The new unit tests require #3171 to be accepted first on which this PR is also based. You can merge this pull request into a Git repository by running: $ git pull https://github.com/NicoK/flink flink-5576 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3174.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3174 commit e29d5ef3099d5f5ffc817f3e5cdbb79c523b34c7 Author: Nico Kruber <nico@data-artisans.com> Date: 2017-01-18T17:52:57Z FLINK-5561 fix DataInputDeserializer#available() 1 smaller than correct commit e589058e92d1f2a57d242987327777baa1fe64d4 Author: Nico Kruber <nico@data-artisans.com> Date: 2017-01-19T16:06:00Z FLINK-5576 let KvStateRequestSerializer#deserializeValue() detect unconsumed bytes KvStateRequestSerializer#deserializeValue deserializes a given byte array. This is used by clients and unit tests and it is fair to assume that these byte arrays represent a complete value since we do not offer a method to continue reading from the middle of the array anyway. Therefore, we can treat unconsumed bytes as errors, e.g. from a wrong serializer being used, and throw a IOException with an appropriate failure message. commit 12d2108333b23e01b7b978a08e868a5f888506f9 Author: Nico Kruber <nico@data-artisans.com> Date: 2017-01-19T16:10:54Z FLINK-5576 add a better failure message to exceptions in KvStateRequestSerializer#deserializeList() As in FLINK-5559 , wrap the original IOException into a new one with an appropriate error message to better diagnose it. commit 9770e7f3269adfa5b80dd2125bcdbf8cebc4dab8 Author: Nico Kruber <nico@data-artisans.com> Date: 2017-01-19T16:12:02Z FLINK-5576 add more unit tests for KvStateRequestSerializer#deserializeList() These tests ensure that some special cases not properly tested before are handled correctly in future.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/flink/pull/3174

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3174
          Hide
          uce Ufuk Celebi added a comment -

          Fixed in 7a0e3d6 (release-1.2), 21742b2 (master).

          Show
          uce Ufuk Celebi added a comment - Fixed in 7a0e3d6 (release-1.2), 21742b2 (master).

            People

            • Assignee:
              NicoK Nico Kruber
              Reporter:
              NicoK Nico Kruber
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development