I am upgrading my Spark version from 2.4.5 to 3.0.1 and I cannot load anymore the PipelineModel objects that use a "DecisionTreeClassifier" stage.
In my code I load several PipelineModel, all the PipelineModel with stages ["CountVectorizer_[uid]", "LinearSVC_[uid]"] are loading fine whereas the models with stages
["CountVectorizer_[uid]","DecisionTreeClassifier_[uid]"] are throwing the following exception:
Here is the code I am using and the full stacktrace:
These pipeline models where saved using Spark 2.4.3, I can load them fine using Spark 2.4.5.
I tried to investigate further and load each stage separately. Loading the CountVectorizerModel with
yields a CountVectorizerModel, but my code fails when trying to load the DecisionTreeClassificationModel:
And here is the content of the "data" of my Decision Tree Classifier: