Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
Description
ListFile is used in many different contexts. We often see users with a specific use case, though, which is to run ListFile on a Primary Node in a cluster, in order to obtain a file listing of an NFS-mounted share. This works well in most cases, but whenever problems do arise, it is very difficult to understand what the problem is. It would be very helpful to have information such as:
- Is there a problem accessing a specific file on the NFS mount?
- Is there a problem obtaining a listing from the NFS mount?
- Is progress being made at all?
- How long is a listing taking right now?
- How long does a listing typically take?
- Is this problem related to NiFi or to the operating system / infrastructure?
It would be helpful to track information about each disk access that is occurring, as well as the overall listing progress and issue warnings if we see clear problems arise. We can do this by timing how long each disk access takes, what file was being accessed, and what operating was being performed. If we capture this data in a rolling window, we can assess the data to determine if the listing is now taking longer than it did previously and alert to this fact. Or alert if performing a specific disk operation is taking a long time.
Gathering this information will likely be fairly heap intensive, so it is best to make the functionality optional.
Attachments
Issue Links
- links to
