Uploaded image for project: 'Metron (Retired)'
  1. Metron (Retired)
  2. METRON-955

Make the default sync policy for HDFS Writer be based on the batch size

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Done
    • Major
    • Resolution: Done
    • None
    • 0.4.0
    • None

    Description

      Right now, we do a sync per record written in HDFS. This has performance penalties as it requires a Namenode contact. We get around this by letting the user define the sync policy in Flux, but that's less convenient and more sensible defaults could be created. We should use the batch size as a sensible default for the sync policy (i.e. sync every batch).

      Attachments

        Issue Links

          links to

          Web Link GitHub Pull Request #589

          Activity

            People

              cestella Casey Stella
              cestella Casey Stella
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: