Uploaded image for project: 'TinkerPop'
  1. TinkerPop
  2. TINKERPOP-1315

HadoopConfiguration will not allow an ArrayList to be serialized in vertexProgram configuration unless setProperty is overriden



    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Won't Fix
    • Affects Version/s: 3.2.1
    • Fix Version/s: None
    • Component/s: hadoop
    • Labels:


      I have been implementing a "PrecisionBulkLoader" class that takes a ScriptTraversal with bindings that can execute against the target graph to getOrCreate vertices/edges with more precision - this follows from my realization that currently IncrementalBulkLoader will overwrite the first edge of the same label in the target graph that is between the two vertex endpoints - this is an issue for self-loops and multi-edges:


      I finally got it to work with the script bindings being propagated to workers, but in order to do so without just taking the last value of the Array I had to override the setProperty method in org.apache.tinkerpop.gremlin.hadoop.structure.HadoopConfiguration - before I did that, when ConfigurationUtils.copy(conf1, conf2) was called with a HadoopConfiguration on either end (conf1 or conf2), any multi-valued / list properties get clobbered and only the last value would be there after storeState/loadState goes through the first cycle in BulkLoaderVertexProgram. This is something that was bugging me for a while with multiple hosts configured for TitanGraph in the config and the HadoopConf only opening a connection against the last host in the list - this change to HadoopConfiguration causes it to read standardtitangraph[cassandrathrift:[host1, host2, ...]] in the spark executor logs instead like you might expect, and allows the bindings for the ScriptTraversal to survive storeState/loadState and be applied to the traversal.

      I suppose I was wondering if this is dangerous or bad somehow? I know that in a few places I saw the values of the configuration being explicitly toString()'d...




            • Assignee:
              okram Marko A. Rodriguez
              dylanht Dylan Bethune-Waddell
            • Votes:
              0 Vote for this issue
              2 Start watching this issue


              • Created: