Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-6659

Remove IndexWriterConfig.get/setMaxThreadStates

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.3, 6.0
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      Ever since LUCENE-5644, IndexWriter will aggressively reuse its internal thread states across threads, whenever one is free.

      I think this means we can safely remove the sneaky maxThreadStates limit (default 8) that we have today: IW will only ever allocate as many thread states as there are actual concurrent threads running through it.

      1. LUCENE-6659.patch
        32 kB
        Michael McCandless

        Activity

        Hide
        mikemccand Michael McCandless added a comment -

        Patch, I think it's ready ... I'm beasting tests now.

        Show
        mikemccand Michael McCandless added a comment - Patch, I think it's ready ... I'm beasting tests now.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 1690299 from Michael McCandless in branch 'dev/trunk'
        [ https://svn.apache.org/r1690299 ]

        LUCENE-6659: remove IWC's max thread states limit

        Show
        jira-bot ASF subversion and git services added a comment - Commit 1690299 from Michael McCandless in branch 'dev/trunk' [ https://svn.apache.org/r1690299 ] LUCENE-6659 : remove IWC's max thread states limit
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 1690301 from Michael McCandless in branch 'dev/branches/branch_5x'
        [ https://svn.apache.org/r1690301 ]

        LUCENE-6659: remove IWC's max thread states limit

        Show
        jira-bot ASF subversion and git services added a comment - Commit 1690301 from Michael McCandless in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1690301 ] LUCENE-6659 : remove IWC's max thread states limit
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 1690826 from Michael McCandless in branch 'dev/trunk'
        [ https://svn.apache.org/r1690826 ]

        LUCENE-6659: never allocate a new ThreadState when DW is aborting

        Show
        jira-bot ASF subversion and git services added a comment - Commit 1690826 from Michael McCandless in branch 'dev/trunk' [ https://svn.apache.org/r1690826 ] LUCENE-6659 : never allocate a new ThreadState when DW is aborting
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 1690827 from Michael McCandless in branch 'dev/branches/branch_5x'
        [ https://svn.apache.org/r1690827 ]

        LUCENE-6659: never allocate a new ThreadState when DW is aborting

        Show
        jira-bot ASF subversion and git services added a comment - Commit 1690827 from Michael McCandless in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1690827 ] LUCENE-6659 : never allocate a new ThreadState when DW is aborting
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 1691662 from Michael McCandless in branch 'dev/trunk'
        [ https://svn.apache.org/r1691662 ]

        LUCENE-6659: woops

        Show
        jira-bot ASF subversion and git services added a comment - Commit 1691662 from Michael McCandless in branch 'dev/trunk' [ https://svn.apache.org/r1691662 ] LUCENE-6659 : woops
        Hide
        shalinmangar Shalin Shekhar Mangar added a comment -

        Bulk close for 5.3.0 release

        Show
        shalinmangar Shalin Shekhar Mangar added a comment - Bulk close for 5.3.0 release
        Hide
        mintern Brandon Mintern added a comment -

        We just upgraded from 5.2.1 to 5.3.1, and none of the intermediate change logs seem to mention this change.

        We were previously using IndexWriterConfig#setMaxThreadStates to allow us to configure the number of threads that can write to the index at the same time. Looking at the source, it used to construct a new DocumentsWriterPerThreadPool instance and setting it.

        Unfortunately, setIndexerThreadPool has package visibility, and besides, that constructor has been removed from DocumentsWriterPerThreadPool.

        Were we mistaken in our understanding of the purpose of this method? Going forward, how can we be limit the number of threads that can write to an index?

        Show
        mintern Brandon Mintern added a comment - We just upgraded from 5.2.1 to 5.3.1, and none of the intermediate change logs seem to mention this change. We were previously using IndexWriterConfig#setMaxThreadStates to allow us to configure the number of threads that can write to the index at the same time. Looking at the source, it used to construct a new DocumentsWriterPerThreadPool instance and setting it. Unfortunately, setIndexerThreadPool has package visibility, and besides, that constructor has been removed from DocumentsWriterPerThreadPool. Were we mistaken in our understanding of the purpose of this method? Going forward, how can we be limit the number of threads that can write to an index?
        Hide
        mikemccand Michael McCandless added a comment -

        Going forward, how can we be limit the number of threads that can write to an index?

        You should fix your application to limit the threads that are allowed to be inside IndexWriter concurrently ... e.g. use Semaphore.

        Show
        mikemccand Michael McCandless added a comment - Going forward, how can we be limit the number of threads that can write to an index? You should fix your application to limit the threads that are allowed to be inside IndexWriter concurrently ... e.g. use Semaphore.
        Hide
        mintern Brandon Mintern added a comment -

        OK, thank you.

        Show
        mintern Brandon Mintern added a comment - OK, thank you.

          People

          • Assignee:
            mikemccand Michael McCandless
            Reporter:
            mikemccand Michael McCandless
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development