Lucene - Core
  1. Lucene - Core
  2. LUCENE-6659

Remove IndexWriterConfig.get/setMaxThreadStates

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.3, 6.0
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      Ever since LUCENE-5644, IndexWriter will aggressively reuse its internal thread states across threads, whenever one is free.

      I think this means we can safely remove the sneaky maxThreadStates limit (default 8) that we have today: IW will only ever allocate as many thread states as there are actual concurrent threads running through it.

      1. LUCENE-6659.patch
        32 kB
        Michael McCandless

        Activity

        Hide
        Michael McCandless added a comment -

        Patch, I think it's ready ... I'm beasting tests now.

        Show
        Michael McCandless added a comment - Patch, I think it's ready ... I'm beasting tests now.
        Hide
        ASF subversion and git services added a comment -

        Commit 1690299 from Michael McCandless in branch 'dev/trunk'
        [ https://svn.apache.org/r1690299 ]

        LUCENE-6659: remove IWC's max thread states limit

        Show
        ASF subversion and git services added a comment - Commit 1690299 from Michael McCandless in branch 'dev/trunk' [ https://svn.apache.org/r1690299 ] LUCENE-6659 : remove IWC's max thread states limit
        Hide
        ASF subversion and git services added a comment -

        Commit 1690301 from Michael McCandless in branch 'dev/branches/branch_5x'
        [ https://svn.apache.org/r1690301 ]

        LUCENE-6659: remove IWC's max thread states limit

        Show
        ASF subversion and git services added a comment - Commit 1690301 from Michael McCandless in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1690301 ] LUCENE-6659 : remove IWC's max thread states limit
        Hide
        ASF subversion and git services added a comment -

        Commit 1690826 from Michael McCandless in branch 'dev/trunk'
        [ https://svn.apache.org/r1690826 ]

        LUCENE-6659: never allocate a new ThreadState when DW is aborting

        Show
        ASF subversion and git services added a comment - Commit 1690826 from Michael McCandless in branch 'dev/trunk' [ https://svn.apache.org/r1690826 ] LUCENE-6659 : never allocate a new ThreadState when DW is aborting
        Hide
        ASF subversion and git services added a comment -

        Commit 1690827 from Michael McCandless in branch 'dev/branches/branch_5x'
        [ https://svn.apache.org/r1690827 ]

        LUCENE-6659: never allocate a new ThreadState when DW is aborting

        Show
        ASF subversion and git services added a comment - Commit 1690827 from Michael McCandless in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1690827 ] LUCENE-6659 : never allocate a new ThreadState when DW is aborting
        Hide
        ASF subversion and git services added a comment -

        Commit 1691662 from Michael McCandless in branch 'dev/trunk'
        [ https://svn.apache.org/r1691662 ]

        LUCENE-6659: woops

        Show
        ASF subversion and git services added a comment - Commit 1691662 from Michael McCandless in branch 'dev/trunk' [ https://svn.apache.org/r1691662 ] LUCENE-6659 : woops
        Hide
        Shalin Shekhar Mangar added a comment -

        Bulk close for 5.3.0 release

        Show
        Shalin Shekhar Mangar added a comment - Bulk close for 5.3.0 release
        Hide
        Brandon Mintern added a comment -

        We just upgraded from 5.2.1 to 5.3.1, and none of the intermediate change logs seem to mention this change.

        We were previously using IndexWriterConfig#setMaxThreadStates to allow us to configure the number of threads that can write to the index at the same time. Looking at the source, it used to construct a new DocumentsWriterPerThreadPool instance and setting it.

        Unfortunately, setIndexerThreadPool has package visibility, and besides, that constructor has been removed from DocumentsWriterPerThreadPool.

        Were we mistaken in our understanding of the purpose of this method? Going forward, how can we be limit the number of threads that can write to an index?

        Show
        Brandon Mintern added a comment - We just upgraded from 5.2.1 to 5.3.1, and none of the intermediate change logs seem to mention this change. We were previously using IndexWriterConfig#setMaxThreadStates to allow us to configure the number of threads that can write to the index at the same time. Looking at the source, it used to construct a new DocumentsWriterPerThreadPool instance and setting it. Unfortunately, setIndexerThreadPool has package visibility, and besides, that constructor has been removed from DocumentsWriterPerThreadPool. Were we mistaken in our understanding of the purpose of this method? Going forward, how can we be limit the number of threads that can write to an index?
        Hide
        Michael McCandless added a comment -

        Going forward, how can we be limit the number of threads that can write to an index?

        You should fix your application to limit the threads that are allowed to be inside IndexWriter concurrently ... e.g. use Semaphore.

        Show
        Michael McCandless added a comment - Going forward, how can we be limit the number of threads that can write to an index? You should fix your application to limit the threads that are allowed to be inside IndexWriter concurrently ... e.g. use Semaphore.
        Hide
        Brandon Mintern added a comment -

        OK, thank you.

        Show
        Brandon Mintern added a comment - OK, thank you.

          People

          • Assignee:
            Michael McCandless
            Reporter:
            Michael McCandless
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development