Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
2.1.0
-
None
-
None
-
None
Description
I'm using Atlas 2.1.0, trying to integrate with https://github.com/AbsaOSS/spline-spark-agent 0.5.5 to process lineage.
Our Atlas server is remote, so I naturally thought that creating a v2-client and passing URL + creds would work.
I'm however having problems because Atlas expects there to be an atlas-application.config even in this narrow use case (I want Atlas models, and to send stuff through the REST endpoint).
Chief of the problems is that Spline depends on Spark 2.4, which in turn depends on Hadoop 2.7.
Hadoop 2.7 then depends on an Apache Commons version that's a major version different to Atlas 2.0+.
This causes issues reading the atlas-application.properties file (trying to parse a list while finding a bool).
org.apache.atlas.AtlasException: Failed to load application properties
at org.apache.atlas.ApplicationProperties.get(ApplicationProperties.java:147)
...
Caused by: org.apache.commons.configuration.ConversionException: 'atlas.graph.index.search.solr.wait-searcher' doesn't map to a List object: true, a java.lang.Boolean
at org.apache.commons.configuration.AbstractConfiguration.getList(AbstractConfiguration.java:1144)
Someone on the user mailing list came across the issue above, but received no responses, so I'm opening this here instead. I'm a commiter on another Apache project, so I know it's not best-practise; but I think I have a valid use-case.
There is also ATLAS-3886 which is related, in that the user got the same error there.
In my case however, I'm hoping for a way to 'just use the REST API component' without having to reinvent the wheel by starting from scratch.
I have tried various combinations, such as:
- Going back to Atlas 2.0.0
- Using Spark 3.0.x (it still builds with an old Apache Commons)
- Removing the Hadoop transitive dependency, but my application ends up not working
The problem is also not on Spline's side, it's a case of Atlas being too ahead of the curve by using a more recent Apache Commons library.
Thanks