Directory ApacheDS
  1. Directory ApacheDS
  2. DIRSERVER-1790

When we restart the server, some user index are lost

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-M9
    • Fix Version/s: 2.0.0-M10
    • Component/s: None
    • Labels:
      None

      Description

      Indexes are not anymore used when the server is restarted. The index was available when the server was started the first time, then after a restart, even if the index exists on disk, it's not anymore used by the server, causing a full scan to be done.

        Activity

        Emmanuel Lecharny made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        Emmanuel Lecharny added a comment -

        Closed all the resolved issues

        Show
        Emmanuel Lecharny added a comment - Closed all the resolved issues
        Emmanuel Lecharny made changes -
        Field Original Value New Value
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Emmanuel Lecharny added a comment -

        Fixed with http://svn.apache.org/viewvc?rev=1435201&view=rev

        We added a TransactionManager to be sure that the base remains consistent.

        Show
        Emmanuel Lecharny added a comment - Fixed with http://svn.apache.org/viewvc?rev=1435201&view=rev We added a TransactionManager to be sure that the base remains consistent.
        Hide
        Emmanuel Lecharny added a comment -

        Ok, I think the pb occurs when the server is shutdown 'brutally'.

        We have a shutdown hook in the server, which flush all the partitions on disk when the server is shutdown using a kill -TERM or a ^C. It's called, and in this case, all the *.db files are correctly created and filled. Otherwise, doing a kill -9 will let the index in a intermediate state, where the *.lg files contain all the updates, but the *.db files all have a 0 size.

        A workaround for the moment is to be sure to correctly stop the server using a kill -TERM on linux, or when stopping the server.

        Still, we should periodically flush the journal files (.lg) so that the data files (.db) are updated, and more important, detect that we are in a transient state when the server is started, and update the data files, applying the journal on them. This would solve the pb we will face in case of a server crash. This is definitively needed.

        Show
        Emmanuel Lecharny added a comment - Ok, I think the pb occurs when the server is shutdown 'brutally'. We have a shutdown hook in the server, which flush all the partitions on disk when the server is shutdown using a kill -TERM or a ^C. It's called, and in this case, all the *.db files are correctly created and filled. Otherwise, doing a kill -9 will let the index in a intermediate state, where the *.lg files contain all the updates, but the *.db files all have a 0 size. A workaround for the moment is to be sure to correctly stop the server using a kill -TERM on linux, or when stopping the server. Still, we should periodically flush the journal files ( .lg) so that the data files ( .db) are updated, and more important, detect that we are in a transient state when the server is started, and update the data files, applying the journal on them. This would solve the pb we will face in case of a server crash. This is definitively needed.
        Hide
        Emmanuel Lecharny added a comment -

        After some investigation, here are some elemets :

        • first, the embedded tests work. If we inject some data which are indexed, and shutdown the DirectoryService, and restart it, the index is still available and working.
        • in standalone mode, although, when we shutdown the server and restart it, the index are not anymore usable.

        That means two things :

        • we may not save the index correctly when we brutally shutdown the server. However, in my case, I waited the 15s needed for all the data to be sync() (and and saw the data being synced)
        • we may have some difference duing the init phase for indexes when we are using the server embedded or standalone.

        I'm investigating the second use case atm.

        Show
        Emmanuel Lecharny added a comment - After some investigation, here are some elemets : first, the embedded tests work. If we inject some data which are indexed, and shutdown the DirectoryService, and restart it, the index is still available and working. in standalone mode, although, when we shutdown the server and restart it, the index are not anymore usable. That means two things : we may not save the index correctly when we brutally shutdown the server. However, in my case, I waited the 15s needed for all the data to be sync() (and and saw the data being synced) we may have some difference duing the init phase for indexes when we are using the server embedded or standalone. I'm investigating the second use case atm.
        Hide
        Emmanuel Lecharny added a comment -

        Ok, I confirm that the index is not correctly initialized. It seems to be available, but we don't read it at startup, so we don't have any clue about the number of elements it contains, so we default to 0, which leads the search engine to do a full scan.

        I have to fix that now.

        Show
        Emmanuel Lecharny added a comment - Ok, I confirm that the index is not correctly initialized. It seems to be available, but we don't read it at startup, so we don't have any clue about the number of elements it contains, so we default to 0, which leads the search engine to do a full scan. I have to fix that now.
        Emmanuel Lecharny created issue -

          People

          • Assignee:
            Emmanuel Lecharny
            Reporter:
            Emmanuel Lecharny
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development