Uploaded image for project: 'Spot'
  1. Spot
  2. SPOT-128

[ML] Make every pipeline to return results to main program to invoke a single save to HDFS method

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Trivial
    • Resolution: Fixed
    • None
    • 1.0

    Description

      Currently, every pipeline invokes dataFrame.write.mode.parquet to save results in HDFS. To all the *SuspiciousConnectsAnalysis.scala free of that method call, we can return a couple of data frames to SuspiciousConnects.scala and from there save results.
      With that, we are going to be able to write unit tests for *SuspiciousConnectsAnalysis.run method.

      Attachments

        Activity

          People

            rabarona Ricardo Barona
            rabarona Ricardo Barona
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: