Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-716

bootstrapping does not work properly using multiple DataFileDirectory

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Normal
    • Resolution: Fixed
    • Fix Version/s: 0.5
    • Component/s: None
    • Labels:
      None
    • Environment:
    • Severity:
      Normal

      Description

      I was adding a new machine A which has 2 DataFileDirectories into the ring. The A will throw exception while bootstrapping.
      DEBUG [MESSAGING-SERVICE-POOL:4] 2010-01-19 11:43:32,837 ContentStreamState.java (line 88) Removing stream context /home/store0/data/pic/raw_data-tmp-1-Data.db:209833142
      WARN [MESSAGING-SERVICE-POOL:4] 2010-01-19 11:43:32,837 TcpConnection.java (line 484) Problem reading from socket connected to : java.nio.channels.SocketChannel[connected local
      =/10.81.37.65:7000 remote=/10.81.42.26:10418]
      WARN [MESSAGING-SERVICE-POOL:4] 2010-01-19 11:43:32,837 TcpConnection.java (line 485) Exception was generated at : 01/19/2010 11:43:32 on thread MESSAGING-SERVICE-POOL:4
      java.io.IOException: rename failed of /home/store0/data/pic/raw_data-1-Filter.db
      java.io.IOError: java.io.IOException: rename failed of /home/store0/data/pic/raw_data-1-Filter.db
      at org.apache.cassandra.io.SSTableWriter.rename(SSTableWriter.java:154)
      at org.apache.cassandra.io.SSTableWriter.renameAndOpen(SSTableWriter.java:162)
      at org.apache.cassandra.io.Streaming$StreamCompletionHandler.onStreamCompletion(Streaming.java:284)
      at org.apache.cassandra.net.io.ContentStreamState.handleStreamCompletion(ContentStreamState.java:108)
      at org.apache.cassandra.net.io.ContentStreamState.read(ContentStreamState.java:90)
      at org.apache.cassandra.net.io.TcpReader.read(TcpReader.java:96)
      at org.apache.cassandra.net.TcpConnection$ReadWorkItem.run(TcpConnection.java:445)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
      at java.lang.Thread.run(Thread.java:619)
      Caused by: java.io.IOException: rename failed of /home/store0/data/pic/raw_data-1-Filter.db
      at org.apache.cassandra.utils.FBUtilities.renameWithConfirm(FBUtilities.java:306)
      at org.apache.cassandra.io.SSTableWriter.rename(SSTableWriter.java:150)
      ... 9 more

      I traced the exception and maybe found the reason.
      StreamInitiateVerbHandler::doVerb() will create 3 temporary files(index, filter, data) for each ssTable. The name for each file is generated by getNewFileNameFromOldContextAndNames(). This method will generate a file name and a path for each ssTable, but the path is generated with DatabaseDescriptor.getDataFileLocationForTable() which will return different path for ech call when we configure multi-DataFileDirectory.
      eg: the ssTable raw_data-1 may have 3 temporary files :
      /home/store0/data/pic/raw_data-tmp-1-Index.db
      /home/store1/data/pic/raw_data-tmp-1-Filter.db
      /home/store0/data/pic/raw_data-tmp-1-Data.db

      After receiving all data, StreamCompletionHandler::onStreamCompletion() will rename all temporary files and this method think all ssTable files will have the same path as data.db file.
      if (streamContext.getTargetFile().contains("-Data.db"))
      {
      ......
      try

      { SSTableReader sstable = SSTableWriter.renameAndOpen(streamContext.getTargetFile()); ...... }

      ......
      }
      Then the renameAndOpen() will throw that exception.

        Attachments

          Activity

            People

            • Assignee:
              gdusbabek Gary Dusbabek
              Reporter:
              david.pan david.pan
              Authors:
              Gary Dusbabek
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 24h
                24h
                Remaining:
                Remaining Estimate - 24h
                24h
                Logged:
                Time Spent - Not Specified
                Not Specified