• Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: jcs-, jcs-
    • Fix Version/s: jcs-
    • Component/s: TCP Lateral Cache
    • Labels:
    • Environment:



      We encountered a problem with an infinite loop in
      which we were able to fix by ensuring that access to
      it was synchronized
      inside I thought I should
      report exactly what we
      found and what we did to solve the problem so that
      perhaps similar changes
      might be made for the next release of jcs.

      We were using the LateralTCPCache in jcs. and
      were performing
      serveral thousand gets and puts per minute from
      multiple threads all using
      the same JCS object. Very quickly, one of these
      threads would end up in an
      infinite loop in ObjectOutputStream.lookup()
      (offending loop in bold)

      int lookup(Object obj) {
      if (size == 0)

      { return -1; }

      int index = hash(obj) % spine.length;
      for (int i = spine[index]; i >= 0; i = next[i]) {
      if (objs[i] == obj)

      { return i; }

      return -1;

      Once one thread ended up in this loop, all other
      threads would end up
      blocking, waiting to obtain a lock on this.getLock in
      LateralTCPSender.sendAndReceive() (found by performing
      a thread dump after
      the application had stopped responding). But since
      the thread in the
      inifinite loop already had a lock on this.getLock,
      our application was
      effectively dead.

      Now ObjectOutputStream is not thread-safe. Indeed, if
      multiple threads
      attempt to write objects to the same
      ObjectOutputStream simultaneously, it
      is likely that the internal object cache will become
      corrupted and
      subsequent writes will end up locked in the
      aforementioned infinite loop.

      Looking at more carefully, we
      noticed that the send(
      LateralElementDescriptor led ) method did perform
      non-thread-safe writes to
      the shared ObjectOutputStream. Wrapping access to
      it with a
      synchronized(this.getLock) block (exactly as was done
      in the sendAndReceive
      method appears to have fixed out problem. I've
      included the
      version of this method with the changes we made in


      • Sends commands to the lateral cache listener.
      • <p>
      • @param led
      • @throws IOException
        public void send( LateralElementDescriptor led )
        throws IOException
        if ( log.isInfoEnabled() )
        Unknown macro: { if ( sendCnt % 100 == 0 ) { "Send Count (port " + port + ") = " + sendCnt ); } }

      if ( log.isDebugEnabled() )

      { log.debug( "sending LateralElementDescriptor" ); }

      if ( led == null )

      { return; }

      if ( address == null )

      { throw new IOException( "No remote host is set for LateralTCPSender." ); }

      if ( oos != null )
      { try
      oos.writeObject( led );
      if ( ++counter >= RESET_FREQUENCY
      counter = 0;
      // Failing to reset the object
      output stream every
      now and
      // then creates a serious
      memory leak.
      if ( log.isDebugEnabled() )

      { log.debug( "Doing oos.reset()" ); }

      catch ( IOException e )

      { oos = null; log.error( "Detected problem with connection: " + e ); throw e; }

      } }

      John (jklame)


        asmuts Aaron Smuts added a comment -

        I put in the synchronized block and ran some tets. I think it should fix the problem.

        asmuts Aaron Smuts added a comment - I put in the synchronized block and ran some tets. I think it should fix the problem.


          • Assignee:
            asmuts Aaron Smuts
            asmuts Aaron Smuts
          • Votes:
            0 Vote for this issue
            0 Start watching this issue


            • Created: