Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-5397

Fail to deserialize savepoints in v1.1 when there exist missing fields in class serialization descriptors

    Details

      Description

      To restore from the savepoints in previous versions, Flink now keeps all classes whose serialization is changed and put them in a separated package ("migration").

      When deserializing the old savepoints, flink will look up correct descriptors (ObjectStreamClass) for these classes, without using those ones written in serialized data. The implementation however is problematic when there exist missing field descriptors in the serialized data.

      When serializing an object, Java will only write the descriptors of those non-null fields. But when we look up class descriptors with given classes, all fields will be put into the descriptors. As a result, we will deserialize the savepoints with incorrect descriptors, leading to serialization exceptions.

      A simple resolution is to update the name of read descriptors using Reflections, without using different descriptors.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user StefanRRichter opened a pull request:

          https://github.com/apache/flink/pull/3050

          FLINK-5397 Do not replace ObjectStreamClass on deserialization of m…

          This PR fixes FLINK-5397 by not replacing the ObjectStreamClass in ``readClassDescriptor()`` during deserialization of classes that should be replaced from the migration package. Instead, we replace the class directly in ``resolveClass(...)``.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/StefanRRichter/flink migration-keep-objectstreamclass

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flink/pull/3050.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3050


          commit 01834068b50a9cf4186ce83fa9c57d5b8f2bfc8c
          Author: Stefan Richter <s.richter@data-artisans.com>
          Date: 2016-12-28T11:50:00Z

          FLINK-5397 Do not replace ObjectStreamClass on deserialization of migration package classes, override resolveClass(...) instead


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user StefanRRichter opened a pull request: https://github.com/apache/flink/pull/3050 FLINK-5397 Do not replace ObjectStreamClass on deserialization of m… This PR fixes FLINK-5397 by not replacing the ObjectStreamClass in ``readClassDescriptor()`` during deserialization of classes that should be replaced from the migration package. Instead, we replace the class directly in ``resolveClass(...)``. You can merge this pull request into a Git repository by running: $ git pull https://github.com/StefanRRichter/flink migration-keep-objectstreamclass Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3050.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3050 commit 01834068b50a9cf4186ce83fa9c57d5b8f2bfc8c Author: Stefan Richter <s.richter@data-artisans.com> Date: 2016-12-28T11:50:00Z FLINK-5397 Do not replace ObjectStreamClass on deserialization of migration package classes, override resolveClass(...) instead
          Hide
          srichter Stefan Richter added a comment -

          Thanks for pointing out this problem. I opened a PR #3050 with a fix that does not require overriding strings by reflection. The idea is to simply replace the class only during resolveClass(...).

          Show
          srichter Stefan Richter added a comment - Thanks for pointing out this problem. I opened a PR #3050 with a fix that does not require overriding strings by reflection. The idea is to simply replace the class only during resolveClass(...) .
          Hide
          xiaogang.shi Xiaogang Shi added a comment -

          The idea does work and is better. Very thanks for the quick fix

          Show
          xiaogang.shi Xiaogang Shi added a comment - The idea does work and is better. Very thanks for the quick fix
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user StephanEwen commented on the issue:

          https://github.com/apache/flink/pull/3050

          Looks good, merging this...

          Show
          githubbot ASF GitHub Bot added a comment - Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/3050 Looks good, merging this...
          Hide
          StephanEwen Stephan Ewen added a comment -

          Fixed in 1.2 via 3554c96d118a411906a22b1f1087de073617a4c7

          Show
          StephanEwen Stephan Ewen added a comment - Fixed in 1.2 via 3554c96d118a411906a22b1f1087de073617a4c7
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/flink/pull/3050

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3050
          Hide
          StephanEwen Stephan Ewen added a comment -

          Fixed in

          • 1.2.0 via 3554c96d118a411906a22b1f1087de073617a4c7
          • 1.3.0 via 09614cc82486b2682ba08876b47019cc604574ed
          Show
          StephanEwen Stephan Ewen added a comment - Fixed in 1.2.0 via 3554c96d118a411906a22b1f1087de073617a4c7 1.3.0 via 09614cc82486b2682ba08876b47019cc604574ed

            People

            • Assignee:
              srichter Stefan Richter
              Reporter:
              xiaogang.shi Xiaogang Shi
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development