Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
1.99.6
-
None
-
None
Description
This one will take a bit longer to explain, so please beer with me
The current flow of configuration class is this one:
- Connector developer defines class and marks inputs that are required from users with @Input annotation. This annotation can have several parameters (sensitive, size, ...).
- As we can't use those classes everywhere (mainly we can't send them to clients over the wire nor serialize them to repository), we have ConfigUtil class that serializes them to model objects.
- Now ConnectorManager takes the model objects and persist them in memory for the lifetime of Sqoop 2 server process (after restart we simply run the ConfigUtils again, so nothing is lost).
- The RepositoryManager takes the model objects and persist them in database. This is desirable because this way we can detect if the structures actually changed (e.g. connector code has been updated) and we can call upgrade routines.
On REST layer, we have following behavior:
- For /connector/ we always call ConnectorManager to get the in memory structures.
- For /job and /links - e.g. to get actual values for those model objects - we are using RepositoryManager instead. As the RepositoryManager is currently not using ConnectorManager it uses the model objects that are serialized in the database instead.
This behavior is a bit limiting because now if we want to improve the model classes and add new information (Validations, Sensitive keys for maps) we absolutely have to serialize that information in database. And that means that we have to do database upgrade, that means that if this information is changed we have to do data upgrade, ... Even though that this is not required as those kind of things can be kept in memory as we don't need to do any action when they change.
Hence I would like to propose changing current behavior and making RepositoryManager use ConnectroManager to read jobs and links objects. This way we can have more in memory state and what state is persisted in database.
Please note that I'm not suggesting to drop the model classes from repository - we still need them to detect if metadata (the @Input annotation) has changed and run upgrade if required. I'm only suggesting to not used this serialized metadata if we are reading job and links objects.
Attachments
Attachments
Issue Links
- links to