Uploaded image for project: 'Falcon'
  1. Falcon
  2. FALCON-870

Retention support delete files/dirs based on modify time or access time

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: feed-lifecycle, retention
    • Labels:
      None

      Description

      In our company, we have a requirement that delete files/dirs based on it's create time or access time, so we hope retention supports delete files based on modify time or access time.

      1. FALCON-870.patch
        8 kB
        yunjiong zhao
      2. FALCON-870-2.patch
        15 kB
        yunjiong zhao
      3. FALCON-870-3.patch
        17 kB
        yunjiong zhao

        Activity

        Hide
        zhaoyunjiong yunjiong zhao added a comment -

        Attached patch is prototype for proposal function.

        Show
        zhaoyunjiong yunjiong zhao added a comment - Attached patch is prototype for proposal function.
        Hide
        svenkat Venkatesh Seetharam added a comment -

        Thanks yunjiong zhao for taking this up. The approach looks good with couple of nits:

        • patch needs to be rebased, these have moved to storage implementations
        • the retention type is only applicable for file system storage, no? If so, this needs to be validated as well
        • Adding an extension point for retention type might be useful to add extensions to this in future such as number of instances as against time
          A simple strategy or template implementation would help

        Thanks!

        Show
        svenkat Venkatesh Seetharam added a comment - Thanks yunjiong zhao for taking this up. The approach looks good with couple of nits: patch needs to be rebased, these have moved to storage implementations the retention type is only applicable for file system storage, no? If so, this needs to be validated as well Adding an extension point for retention type might be useful to add extensions to this in future such as number of instances as against time A simple strategy or template implementation would help Thanks!
        Hide
        zhaoyunjiong yunjiong zhao added a comment -

        Thanks Venkatesh Seetharam for your time and sorry for late response.
        Update patch rebased on the current master.

        the retention type is only applicable for file system storage, no? If so, this needs to be validated as well

        Yes, only for file system storage.
        Where is the best place for validation in your opinion?

        Show
        zhaoyunjiong yunjiong zhao added a comment - Thanks Venkatesh Seetharam for your time and sorry for late response. Update patch rebased on the current master. the retention type is only applicable for file system storage, no? If so, this needs to be validated as well Yes, only for file system storage. Where is the best place for validation in your opinion?
        Hide
        sriksun Srikanth Sundarrajan added a comment -

        yunjiong zhao, Validations for this needs to go into FeedEntityParser

        Show
        sriksun Srikanth Sundarrajan added a comment - yunjiong zhao , Validations for this needs to go into FeedEntityParser
        Hide
        zhaoyunjiong yunjiong zhao added a comment -

        Thanks Srikanth Sundarrajan.
        Update patch to add validate logic.

        Show
        zhaoyunjiong yunjiong zhao added a comment - Thanks Srikanth Sundarrajan. Update patch to add validate logic.
        Hide
        shaik.idris Shaik Idris Ali added a comment -

        Changing the status to patch-available, as it is attached.

        Show
        shaik.idris Shaik Idris Ali added a comment - Changing the status to patch-available, as it is attached.
        Hide
        sriksun Srikanth Sundarrajan added a comment -
        +    private long retentionLimitToMills(String retentionLimit) throws NumberFormatException {
        +        Frequency frequency = Frequency.fromString(retentionLimit);
        +        long timePoint = Long.parseLong(frequency.getFrequency());
        +        if (frequency.getTimeUnit() == Frequency.TimeUnit.minutes) {
        +            timePoint *= 60;
        +        } else if (frequency.getTimeUnit() == Frequency.TimeUnit.hours) {
        +            timePoint *= 60 * 60;
        +        } else if (frequency.getTimeUnit() == Frequency.TimeUnit.days) {
        +            timePoint *= 24 * 60 * 60;
        +        } else if (frequency.getTimeUnit() == Frequency.TimeUnit.months) {
        +            timePoint *= 31 * 24 * 60 * 60;
        +        }
        +        return timePoint * 1000;
        +    }
        

        It might be worthwhile to consider using Calendar. The current implementation might surprise the users.

        Also, noticed that the current patch seems to be selectively deleting files in a instance directory based on access/modify time. While one might expect an instance to be either present or absent, deleting contents partially might cause unexpected behavior from a consumer stand point. Also what rules are to be enforced vis-a-vis feed validity period on process submission or instance operations, that also seems to be missing in the patch.

        Show
        sriksun Srikanth Sundarrajan added a comment - + private long retentionLimitToMills(String retentionLimit) throws NumberFormatException { + Frequency frequency = Frequency.fromString(retentionLimit); + long timePoint = Long.parseLong(frequency.getFrequency()); + if (frequency.getTimeUnit() == Frequency.TimeUnit.minutes) { + timePoint *= 60; + } else if (frequency.getTimeUnit() == Frequency.TimeUnit.hours) { + timePoint *= 60 * 60; + } else if (frequency.getTimeUnit() == Frequency.TimeUnit.days) { + timePoint *= 24 * 60 * 60; + } else if (frequency.getTimeUnit() == Frequency.TimeUnit.months) { + timePoint *= 31 * 24 * 60 * 60; + } + return timePoint * 1000; + } It might be worthwhile to consider using Calendar. The current implementation might surprise the users. Also, noticed that the current patch seems to be selectively deleting files in a instance directory based on access/modify time. While one might expect an instance to be either present or absent, deleting contents partially might cause unexpected behavior from a consumer stand point. Also what rules are to be enforced vis-a-vis feed validity period on process submission or instance operations, that also seems to be missing in the patch.
        Hide
        ajayyadava Ajay Yadava added a comment -

        Cancelling the patch as more work needs to be done on this as per Srikanth's comments. yunjiong zhao , can you please provide an updated patch? Let us know if you need some clarifications / help.

        Show
        ajayyadava Ajay Yadava added a comment - Cancelling the patch as more work needs to be done on this as per Srikanth's comments. yunjiong zhao , can you please provide an updated patch? Let us know if you need some clarifications / help.
        Hide
        me.venkatr Venkatesan Ramachandran added a comment -

        Ajay Yadava should this be another plugin using Lifecycle?

        Show
        me.venkatr Venkatesan Ramachandran added a comment - Ajay Yadava should this be another plugin using Lifecycle?
        Hide
        me.venkatr Venkatesan Ramachandran added a comment -

        yunjiong zhao If you are not looking into this issue, can I take this one?

        Show
        me.venkatr Venkatesan Ramachandran added a comment - yunjiong zhao If you are not looking into this issue, can I take this one?
        Hide
        ajayyadava Ajay Yadava added a comment -

        Yes, this will be another policy using lifecycle.

        Show
        ajayyadava Ajay Yadava added a comment - Yes, this will be another policy using lifecycle.

          People

          • Assignee:
            me.venkatr Venkatesan Ramachandran
            Reporter:
            zhaoyunjiong yunjiong zhao
          • Votes:
            1 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:

              Development