Commons IO
  1. Commons IO
  2. IO-279

Tailer erroneously considers file as new

    Details

    • Type: Bug Bug
    • Status: Reopened
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 2.0.1, 2.4
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Tailer sometimes erroneously considers the tailed file as new, forcing a repositioning at the start of the file: I'm still unable to reproduce this in a test case, because it only happens to me with huge log files during Apache Tomcat startup.

      This is the piece of code causing the problem:

      // See if the file needs to be read again
      if (length > position) {
      
          // The file has more content than it did last time
          last = System.currentTimeMillis();
          position = readLines(reader);
      
      } else if (FileUtils.isFileNewer(file, last)) {
      
          /* This can happen if the file is truncated or overwritten
              * with the exact same length of information. In cases like
              * this, the file position needs to be reset
              */
          position = 0;
          reader.seek(position); // cannot be null here
      
          // Now we can read new lines
          last = System.currentTimeMillis();
          position = readLines(reader);
      }
      

      What probably happens is that the new file content is about to be written on disk, the date is already updated but content is still not flushed, so actual length is untouched and there you go.

      In other words, I think there should be some better method to verify the condition above, rather than relying only on dates: keeping and comparing the hash code of the latest line may be a solution, but may hurt performances ... other ideas?

      1. modify-test-fixed.patch
        2 kB
        Herman Meerlo
      2. modify-test.patch
        2 kB
        Herman Meerlo
      3. IO-279.patch
        1 kB
        Niall Pemberton
      4. fix-tailer.patch
        1 kB
        Herman Meerlo
      5. disable_resetting.patch
        3 kB
        Karol Grzegorczyk

        Activity

        Hide
        Chris Baron added a comment -

        There are at least two additional causes that I've identified:

        (1) "last" time stamp does not include time spent reading or listener handling.

        last = System.currentTimeMillis();
        position = readLines(reader);

        readLines(...) continues to read and handle lines from the log until it reaches the EOF.

        An erroneous truncation can be detected ff (a) content is added to the file between the recording of the "last" timestamp and (b) before readLine encounters EOF and (c) no content is added during the delay time.

        The fix is to reverse the two lines such that the timestamp is recorded after the call to readLines(...).

        (2) On very highly loaded system content could be written between the point the file length is saved and the timestamp is compared.

        The fix is to compare the file date to the "last" timestamp prior to checking its length and to use that boolean result in the nested else if.

        Show
        Chris Baron added a comment - There are at least two additional causes that I've identified: (1) "last" time stamp does not include time spent reading or listener handling. last = System.currentTimeMillis(); position = readLines(reader); readLines(...) continues to read and handle lines from the log until it reaches the EOF. An erroneous truncation can be detected ff (a) content is added to the file between the recording of the "last" timestamp and (b) before readLine encounters EOF and (c) no content is added during the delay time. The fix is to reverse the two lines such that the timestamp is recorded after the call to readLines(...). (2) On very highly loaded system content could be written between the point the file length is saved and the timestamp is compared. The fix is to compare the file date to the "last" timestamp prior to checking its length and to use that boolean result in the nested else if.
        Hide
        Mark Baker added a comment -

        I see this bug as well, I am using this class to tail log files during a lengthly build process and occasionally the entire log file will be regurgitated

        Show
        Mark Baker added a comment - I see this bug as well, I am using this class to tail log files during a lengthly build process and occasionally the entire log file will be regurgitated
        Hide
        Sergio Bossa added a comment -

        Mark, that should be fixed in my fork: https://github.com/sbtourist/tayler

        Show
        Sergio Bossa added a comment - Mark, that should be fixed in my fork: https://github.com/sbtourist/tayler
        Hide
        Sebb added a comment -

        There's a general problem here, in that it's not possible to obtain both the file position and the current timestamp (System or File) as part of a single transaction.

        However, the critical case is where the File timestamp is greater than the System timestamp, so it does not matter if the File timestamp is measured too early or the System timestamp is measured too late.

        Show
        Sebb added a comment - There's a general problem here, in that it's not possible to obtain both the file position and the current timestamp (System or File) as part of a single transaction. However, the critical case is where the File timestamp is greater than the System timestamp, so it does not matter if the File timestamp is measured too early or the System timestamp is measured too late.
        Hide
        Niall Pemberton added a comment -

        Firstly I don't know why System.currentTimeMillis() is used. What matters is if the files lastModified time compared to its previous lastModified value.

        I agree with Chris that the lastModified time should be stored after the file is read.

        Show
        Niall Pemberton added a comment - Firstly I don't know why System.currentTimeMillis() is used. What matters is if the files lastModified time compared to its previous lastModified value. I agree with Chris that the lastModified time should be stored after the file is read.
        Hide
        Niall Pemberton added a comment -

        Ooo, my bad - this is already fixed. Still same as my patch except using file.lastModified() rather than System.currentTimeMillis()

        Show
        Niall Pemberton added a comment - Ooo, my bad - this is already fixed. Still same as my patch except using file.lastModified() rather than System.currentTimeMillis()
        Hide
        Sebb added a comment -

        What matters is if the files lastModified time compared to its previous lastModified value.

        Yes, but if that is measured after calling readLines, this might trigger case (2) above.

        Show
        Sebb added a comment - What matters is if the files lastModified time compared to its previous lastModified value. Yes, but if that is measured after calling readLines, this might trigger case (2) above.
        Hide
        Grzegorz Molenda added a comment -

        I am tailing with the fixed Tailer (commons-io 2.4.0) a log4j log file and I still see the issue. Despite the fact that the log file was neither rotated nor new data was added, the position is being reset to 0, causing the Tailer re-reading the monitored file from the begining.

        Since log4j's asynchronous logger is used to log into the monitored file, it might happen, that the modifiedDate is set before the content is actually flushed to the file.

        I assume reseting position was added to cover the case, when the monitored file is overriden. I think it is imposiilble for the Tailer to determine this. The current implementation covers only the case, when the file length is equal to the last read position. If the file legth after being overriden is higher than the last read position, then the Tailer will assume data was normally appended and process the file from the last read position.

        Assuming the data is only appended to the file, I'd just get rid of the reseting position feature from Tailer to resolve the issue finally.

        Show
        Grzegorz Molenda added a comment - I am tailing with the fixed Tailer (commons-io 2.4.0) a log4j log file and I still see the issue. Despite the fact that the log file was neither rotated nor new data was added, the position is being reset to 0, causing the Tailer re-reading the monitored file from the begining. Since log4j's asynchronous logger is used to log into the monitored file, it might happen, that the modifiedDate is set before the content is actually flushed to the file. I assume reseting position was added to cover the case, when the monitored file is overriden. I think it is imposiilble for the Tailer to determine this. The current implementation covers only the case, when the file length is equal to the last read position. If the file legth after being overriden is higher than the last read position, then the Tailer will assume data was normally appended and process the file from the last read position. Assuming the data is only appended to the file, I'd just get rid of the reseting position feature from Tailer to resolve the issue finally.
        Hide
        Sebb added a comment -

        It seems odd that the OS should update the file modification date before the file has actually been modified.
        I would expect the flush to write the data to the file and then update the date.

        But perhaps it does behave that way.

        Could you provide a patch that works with your use case?

        Show
        Sebb added a comment - It seems odd that the OS should update the file modification date before the file has actually been modified. I would expect the flush to write the data to the file and then update the date. But perhaps it does behave that way. Could you provide a patch that works with your use case?
        Hide
        Richard Hawkes added a comment -

        Guys, I have downloaded 2.4 which (I think) you are saying has fixed it. However, I notice that the fileRotated is still getting called erroneously. I have done a fair bit of research into this, and it would seem that the file.length() method is not always 100% up to date, which leads to position occasionally being greater than file.length() !! Quite often it seems to be a few miliseconds behind the actual position. I suppose with that much data bouncing around the network.

        I have added a check after the readLines(reader) to see if position is greater than file.length() if it is, it waits a second. That seems to mop up this issue, although I know it's one ALMIGHTY hack!

        Show
        Richard Hawkes added a comment - Guys, I have downloaded 2.4 which (I think) you are saying has fixed it. However, I notice that the fileRotated is still getting called erroneously. I have done a fair bit of research into this, and it would seem that the file.length() method is not always 100% up to date, which leads to position occasionally being greater than file.length() !! Quite often it seems to be a few miliseconds behind the actual position. I suppose with that much data bouncing around the network. I have added a check after the readLines(reader) to see if position is greater than file.length() if it is, it waits a second. That seems to mop up this issue, although I know it's one ALMIGHTY hack!
        Hide
        Herman Meerlo added a comment -

        Just a mere 'touch <file>' triggers a complete reload of the file. I can not imagine that that is wanted behaviour.

        Show
        Herman Meerlo added a comment - Just a mere 'touch <file>' triggers a complete reload of the file. I can not imagine that that is wanted behaviour.
        Hide
        Richard Hawkes added a comment -

        Herman, I don't think anyone's looking at this. I would say that the tailer is flawed and should not be used. It's no better than reading the file via standard Java methods. I had really hoped to leverage this, but such is the way with open source

        Show
        Richard Hawkes added a comment - Herman, I don't think anyone's looking at this. I would say that the tailer is flawed and should not be used. It's no better than reading the file via standard Java methods. I had really hoped to leverage this, but such is the way with open source
        Hide
        Sebb added a comment -

        The issue has been marked resolved; if you have a patch please re-open and provide the patch, preferably with a test case that demonstrates the problem.

        Show
        Sebb added a comment - The issue has been marked resolved; if you have a patch please re-open and provide the patch, preferably with a test case that demonstrates the problem.
        Hide
        Herman Meerlo added a comment -

        Well, this is the world turned upside down. I can only reopen the issue if I have a patch for it. That doesn't make sense. The problem still exists and therefore the ticket should be reopened. I have patched it locally for myself but I doubt that my patch is ok for everyone because I completely removed the last else if statement. For me it makes no sense to check if the file is newer. The only use case would be that the file had been overwritten with exactly the same amount of data. Truncation is not an issue because that would mean that the length and position must have been 0 anyway. For me it is way more likely that the file's modified time has been updated than that the content has been overwritten with the same amount of bytes.

        Show
        Herman Meerlo added a comment - Well, this is the world turned upside down. I can only reopen the issue if I have a patch for it. That doesn't make sense. The problem still exists and therefore the ticket should be reopened. I have patched it locally for myself but I doubt that my patch is ok for everyone because I completely removed the last else if statement. For me it makes no sense to check if the file is newer. The only use case would be that the file had been overwritten with exactly the same amount of data. Truncation is not an issue because that would mean that the length and position must have been 0 anyway. For me it is way more likely that the file's modified time has been updated than that the content has been overwritten with the same amount of bytes.
        Hide
        Richard Hawkes added a comment -

        Strikes me that this should simply be re-opened. Issue is recreatable, but as yet no fix is known.

        Show
        Richard Hawkes added a comment - Strikes me that this should simply be re-opened. Issue is recreatable, but as yet no fix is known.
        Hide
        Sebb added a comment -

        I can only reopen the issue if I have a patch for it.

        That's not what I meant. The issue had been marked resolved, so developers were unlikely to look at it.
        But without a proposed patch (even if incomplete) or a test case, there's not a lot developers can do.

        Show
        Sebb added a comment - I can only reopen the issue if I have a patch for it. That's not what I meant. The issue had been marked resolved, so developers were unlikely to look at it. But without a proposed patch (even if incomplete) or a test case, there's not a lot developers can do.
        Hide
        Herman Meerlo added a comment -

        Hi, here is the requested test case patch. It tests both cases: only the lastmodified updated and content overwritten with exactly the same amount of bytes.

        Show
        Herman Meerlo added a comment - Hi, here is the requested test case patch. It tests both cases: only the lastmodified updated and content overwritten with exactly the same amount of bytes.
        Hide
        Sebb added a comment -

        The test seems wrong to me.
        Only one line is written to the file, yet the check says:

        assertEquals("1 line count", 2, lines.size());
        

        Also, I'm not sure that changing the file modification date should be ignored.
        How can one tell the difference between a file that has been touched from one that has been re-written to the same length?

        Potentially it may even be the same data - that would be an unusual use-case, but not impossible.
        For example, a rotating logger that records events but does not include a timestamp. The same event sequence could recur.

        A further problem with the test case in the patch is that it does not check the data, only the line count.

        Show
        Sebb added a comment - The test seems wrong to me. Only one line is written to the file, yet the check says: assertEquals( "1 line count" , 2, lines.size()); Also, I'm not sure that changing the file modification date should be ignored. How can one tell the difference between a file that has been touched from one that has been re-written to the same length? Potentially it may even be the same data - that would be an unusual use-case, but not impossible. For example, a rotating logger that records events but does not include a timestamp. The same event sequence could recur. A further problem with the test case in the patch is that it does not check the data, only the line count.
        Hide
        Sebb added a comment -

        Having said that, if there is still a problem whereby the code does not follow the file properly, please provide full details.

        Show
        Sebb added a comment - Having said that, if there is still a problem whereby the code does not follow the file properly, please provide full details.
        Hide
        Herman Meerlo added a comment -

        I'm sorry, I had the test correct but modified it before making the patch. I will correct it and upload it in a few minutes.

        Let's be clear, I'm not suggesting to ignore the file modification date as a solution. For me that would be the perfect solution and I think the most common use case as well. The likelihood of the file being touched seems way higher to me than the likelihood that the exact same sequence of bytes are written, especially when the files get larger. And as can be seen from other comments above there are more people reporting this problem. It is probably not an option to make Java 7 a requirement so we can use the WatchService?

        Show
        Herman Meerlo added a comment - I'm sorry, I had the test correct but modified it before making the patch. I will correct it and upload it in a few minutes. Let's be clear, I'm not suggesting to ignore the file modification date as a solution. For me that would be the perfect solution and I think the most common use case as well. The likelihood of the file being touched seems way higher to me than the likelihood that the exact same sequence of bytes are written, especially when the files get larger. And as can be seen from other comments above there are more people reporting this problem. It is probably not an option to make Java 7 a requirement so we can use the WatchService?
        Hide
        Herman Meerlo added a comment -

        Fixed testcase, mea culpa.

        Show
        Herman Meerlo added a comment - Fixed testcase, mea culpa.
        Hide
        Sebb added a comment -

        My point was that discriminating between 'touch' and an updated file is tricky and not always possible.
        I don't consider it a fault that the a touched file is seen as new (cf. backup).

        We really need a test case that shows the exact same error.

        Also it would be helpful to know if the failures occurred on Unix or Windows, and whether reOpen is true or false.

        Show
        Sebb added a comment - My point was that discriminating between 'touch' and an updated file is tricky and not always possible. I don't consider it a fault that the a touched file is seen as new (cf. backup). We really need a test case that shows the exact same error. Also it would be helpful to know if the failures occurred on Unix or Windows, and whether reOpen is true or false.
        Hide
        Herman Meerlo added a comment -

        I totally agree, it is very hard to discriminate between the different use cases. It might only be possible with Java 7. What do you mean with (cf. backup) by the way?

        My case occurs on Linux (Debian) where I wrote a tool to tail GlassFish log files and out put them to Kafka. Every now and then it spits out the entire log file again, which makes the Tailer useless for me. I have a suspicion that the problem might be related to the fact that the 'last' is set to System.currentTimeMillis() instead of to file.lastModified(). Maybe there is a granularity difference between the two, where the FS rounds the last modified upwards? If I stat the file then it always has a 1 sec precision. That would explain it I guess. I will patch it here and run a test today.

        Show
        Herman Meerlo added a comment - I totally agree, it is very hard to discriminate between the different use cases. It might only be possible with Java 7. What do you mean with (cf. backup) by the way? My case occurs on Linux (Debian) where I wrote a tool to tail GlassFish log files and out put them to Kafka. Every now and then it spits out the entire log file again, which makes the Tailer useless for me. I have a suspicion that the problem might be related to the fact that the 'last' is set to System.currentTimeMillis() instead of to file.lastModified(). Maybe there is a granularity difference between the two, where the FS rounds the last modified upwards? If I stat the file then it always has a 1 sec precision. That would explain it I guess. I will patch it here and run a test today.
        Hide
        Sebb added a comment -

        OK thanks.

        (cf. backup)

        I meant that backup treats touched files as new, so Tailer should too.

        Show
        Sebb added a comment - OK thanks. (cf. backup) I meant that backup treats touched files as new, so Tailer should too.
        Hide
        Herman Meerlo added a comment -

        Ok, I've tested the patch for a few days now. The problem has not reoccured anymore whereas before it used to happen multiple times per day. I have attached the patch.

        Show
        Herman Meerlo added a comment - Ok, I've tested the patch for a few days now. The problem has not reoccured anymore whereas before it used to happen multiple times per day. I have attached the patch.
        Hide
        Sebb added a comment -

        Thanks. Can you confirm exactly what you changed? Did you replace all 3 instances of System.currentTimeMillis() or only some of them?

        Also the new test case testModifiedTime fails for me both with the current code and when the code is patched by replacing all 3 System.currentTimeMillis() with file.lastModifiedTime(). Is that what you expect? Or should the test succeed?

        Show
        Sebb added a comment - Thanks. Can you confirm exactly what you changed? Did you replace all 3 instances of System.currentTimeMillis() or only some of them? Also the new test case testModifiedTime fails for me both with the current code and when the code is patched by replacing all 3 System.currentTimeMillis() with file.lastModifiedTime(). Is that what you expect? Or should the test succeed?
        Hide
        Herman Meerlo added a comment -

        Yes I replaced all 3 instances of System.currentTimeMillis()

        The test will indeed still fail, it doesn't solve the specific case of differentiating between the touch of a file and overwriting the contents of the file with the exact same amount of bytes. It solves this specific bug as the title says 'Tailer erroneously considers file as new'. So I guess it is better to create a new ticket and attach the testcase to that ticket, because that is a different bug (which is very hard to solve as has already been said by most of us).

        Show
        Herman Meerlo added a comment - Yes I replaced all 3 instances of System.currentTimeMillis() The test will indeed still fail, it doesn't solve the specific case of differentiating between the touch of a file and overwriting the contents of the file with the exact same amount of bytes. It solves this specific bug as the title says 'Tailer erroneously considers file as new'. So I guess it is better to create a new ticket and attach the testcase to that ticket, because that is a different bug (which is very hard to solve as has already been said by most of us).
        Hide
        Sebb added a comment -

        OK, thanks, I'll apply the same fix to Tailer.

        Show
        Sebb added a comment - OK, thanks, I'll apply the same fix to Tailer.
        Hide
        Sebb added a comment -

        URL: http://svn.apache.org/r1476097
        Log:
        IO-279 Tailer erroneously considers file as new.
        Fix to use file.lastModified() rather than System.currentTimeMillis()

        Modified:
        commons/proper/io/trunk/src/changes/changes.xml
        commons/proper/io/trunk/src/main/java/org/apache/commons/io/input/Tailer.java

        Show
        Sebb added a comment - URL: http://svn.apache.org/r1476097 Log: IO-279 Tailer erroneously considers file as new. Fix to use file.lastModified() rather than System.currentTimeMillis() Modified: commons/proper/io/trunk/src/changes/changes.xml commons/proper/io/trunk/src/main/java/org/apache/commons/io/input/Tailer.java
        Hide
        Otis Gospodnetic added a comment -

        My case occurs on Linux (Debian) where I wrote a tool to tail GlassFish log files and out put them to Kafka. Every now and then it spits out the entire log file again, which makes the Tailer useless for me.

        What about tracking the current position/line in the file, at least approximately.
        Then, after detecting apparent new/rotated file one could check things like size of the file or some such and compare it with the offset to answer the question such as "Does this apparently new file that I'm about to start tailing from its beginning actually already have the offset I was at before? If so, maybe this is the same file and somebody just touched it. In that case, let me just jump to that offset".

        Doable?

        Show
        Otis Gospodnetic added a comment - My case occurs on Linux (Debian) where I wrote a tool to tail GlassFish log files and out put them to Kafka. Every now and then it spits out the entire log file again, which makes the Tailer useless for me. What about tracking the current position/line in the file, at least approximately. Then, after detecting apparent new/rotated file one could check things like size of the file or some such and compare it with the offset to answer the question such as "Does this apparently new file that I'm about to start tailing from its beginning actually already have the offset I was at before? If so, maybe this is the same file and somebody just touched it. In that case, let me just jump to that offset". Doable?
        Hide
        Sebb added a comment -

        Note that the particular problem you quoted has been solved.

        We already keep track of the location in the file within the code, and we compare file sizes and times.

        The problem is trying to distinguish a file that has been touched from a file that has been rewritten or truncated to exactly the same size.

        Show
        Sebb added a comment - Note that the particular problem you quoted has been solved. We already keep track of the location in the file within the code, and we compare file sizes and times. The problem is trying to distinguish a file that has been touched from a file that has been rewritten or truncated to exactly the same size.
        Hide
        Otis Gospodnetic added a comment - - edited

        Thanks Sebb. I see. So things like logrotate can confuse the tailer if they truncate files instead of moving them?

        Show
        Otis Gospodnetic added a comment - - edited Thanks Sebb. I see. So things like logrotate can confuse the tailer if they truncate files instead of moving them?
        Hide
        Casey Doerschuk added a comment -

        Just wanted to confirm to anyone who cares, commons-io 2.4 tag with the April patch attached to this JIRA, still has the issue in the scenerio we are using it. We have a daemon network listener process written in C++ that opens a log file, appends new data, closes the file, repeatedly, for which we are trying to use the Tailer classes to pump the log through Kafka, similar to Herman in the above thread. Using commons-io 2.4 prebuilt jar we were getting the intermittent reatart on almost all hosts more than once a day. Using the patched jar it happened less, but still happens. I am trying the forked version in github published by Sergio. I will respond with my findings.

        Show
        Casey Doerschuk added a comment - Just wanted to confirm to anyone who cares, commons-io 2.4 tag with the April patch attached to this JIRA, still has the issue in the scenerio we are using it. We have a daemon network listener process written in C++ that opens a log file, appends new data, closes the file, repeatedly, for which we are trying to use the Tailer classes to pump the log through Kafka, similar to Herman in the above thread. Using commons-io 2.4 prebuilt jar we were getting the intermittent reatart on almost all hosts more than once a day. Using the patched jar it happened less, but still happens. I am trying the forked version in github published by Sergio. I will respond with my findings.
        Hide
        Karol Grzegorczyk added a comment - - edited

        In many cases it can be assumed that a file can not be overwritten with the exact same length of data (always will be smaller after reset). In our project we are using a slightly patched version of commons-io library with a flag added to the Tailer class that enables/disables resetting file position when a file update is encountered but a file length is not changed. If we are sure that a file can not be overwritten with the exact size then we disable the flag to prevent this issue. I've attached the patch we are using (disable_resetting.patch). It is based on the version 2.4. Maybe it would be worth to apply this patch to the trunk?

        Show
        Karol Grzegorczyk added a comment - - edited In many cases it can be assumed that a file can not be overwritten with the exact same length of data (always will be smaller after reset). In our project we are using a slightly patched version of commons-io library with a flag added to the Tailer class that enables/disables resetting file position when a file update is encountered but a file length is not changed. If we are sure that a file can not be overwritten with the exact size then we disable the flag to prevent this issue. I've attached the patch we are using ( disable_resetting.patch ). It is based on the version 2.4. Maybe it would be worth to apply this patch to the trunk?
        Hide
        Robert Olofsson added a comment -

        I stumbled across this issue while tailing a file on a remote server via Samba.

        The clock on the server was running a few seconds ahead of my local machine which caused the file to be seen as newer even though it wasn't.

        I solved this by simply replacing the line:

        last = System.currentTimeMillis();

        With:

        last = file.lastModified();

        That way it doesn't matter if the clocks are not in perfect sync.

        Show
        Robert Olofsson added a comment - I stumbled across this issue while tailing a file on a remote server via Samba. The clock on the server was running a few seconds ahead of my local machine which caused the file to be seen as newer even though it wasn't. I solved this by simply replacing the line: last = System.currentTimeMillis(); With: last = file.lastModified(); That way it doesn't matter if the clocks are not in perfect sync.
        Hide
        Sebb added a comment -

        last = file.lastModified();

        That change has already been made in trunk and will be in 2.5

        Show
        Sebb added a comment - last = file.lastModified(); That change has already been made in trunk and will be in 2.5
        Hide
        liuhy added a comment -

        I have downloaded from the trunk, but the question remains.Repeat output for three times, it seems that problems unresolved.

        Show
        liuhy added a comment - I have downloaded from the trunk, but the question remains.Repeat output for three times, it seems that problems unresolved.
        Hide
        Shawn M. Crawford added a comment -

        Hi,
        I was curious if there has been any progress on this issue?

        Thanks

        Show
        Shawn M. Crawford added a comment - Hi, I was curious if there has been any progress on this issue? Thanks

          People

          • Assignee:
            Unassigned
            Reporter:
            Sergio Bossa
          • Votes:
            4 Vote for this issue
            Watchers:
            17 Start watching this issue

            Dates

            • Created:
              Updated:

              Development