Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-4069

ListXXX processors can miss files those created while the processor is listing and filesystem does not provide timestamp milliseconds precision

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.0.0
    • 1.4.0
    • Extensions
    • None

    Description

      For some filesystems such as Mac OS X HFS (Hierarchical File System) or EXT3 are known that only support timestamp in seconds precision. Also some FTP server is reported that it can only provides timestamp precision in minutes.

      This can cause files to NOT be listed as ListXXX processors logic expects timestamps in milliseconds.

      Specifically, if generate several files in one second, not all files will be listened.

      Steps to reproduce:
      1. start processor ListFile
      2. generate 10000 zero size files with following command:

      for i in {1..10000}; do touch ./test_$i; done
      

      3. see processor stats: out 3952 (0 bytes)

      Current AbstractListProcessor logic adopts LISTING_LAG_NANOS (100ms) and postponing the files those have the latest timestamp within a listing iteration to next iteration, however with those filesystem without milliseconds precision, these logics do not work as expected.

      This issue is originally reported at nifi-dev ML. http://apache-nifi-developer-list.39713.n7.nabble.com/processors-ListFile-ListSFTP-do-not-store-milliseconds-in-timestamp-td16037.html

      Attachments

        1. ListFilesWithoutMilliseconds.png
          67 kB
          Koji Kawamura

        Issue Links

          Activity

            People

              ijokarumawak Koji Kawamura
              ijokarumawak Koji Kawamura
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: