Cassandra
  1. Cassandra
  2. CASSANDRA-5185

symlinks to data directories are broken in 1.2.0

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Fix Version/s: 1.2.2
    • Component/s: Core
    • Labels:
      None

      Description

      symlinks to data directories is broken in 1.2.0

      cd ~
      tar xzf apache-cassandra-1.2.0-bin.tar.gz
      cd apache-cassandra-1.2.0/conf
      vim cassandra.yaml
      #set data/commitlog/savecache dirs to
      #~/apache-cassandra-1.2.0/var/...
      cd ../bin
      # start once to make folders
      ./cassandra -f
      #cntrl-c
      cd ..
      mkdir var/lib/cassandra2
      mv var/lib/cassandra/data/system var/lib/cassandra2/system
      cd var/lib/cassandra/data
      ln -s ../../cassandra2/system .
      cd ~/apache-cassandra-1.2.0/bin/
      cassandra -f
      #get lots of assertion errors see attached log file
      
       INFO 21:59:44,883 Enqueuing flush of Memtable-local@1578022692(52/52 serialized/live bytes, 2 ops)
       INFO 21:59:44,890 Writing Memtable-local@1578022692(52/52 serialized/live bytes, 2 ops)
      ERROR 21:59:44,892 Exception in thread Thread[FlushWriter:1,5,main]
      java.lang.AssertionError
      	at org.apache.cassandra.io.sstable.Descriptor.<init>(Descriptor.java:190)
      	at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:593)
      	at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:588)
      	at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:428)
      	at org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:417)
      	at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
      	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      	at java.lang.Thread.run(Thread.java:662)
      

      I traced it back some, I think it is coming from:

      org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:428)
      which calls: cfs.directories.getLocationForDisk(dataDirectory)
      
          public File getLocationForDisk(File dataDirectory)
          {
              for (File dir : sstableDirectories)
              {
                  if (FileUtils.getCanonicalPath(dir).startsWith(FileUtils.getCanonicalPath(dataDirectory)))
                      return dir;
              }
              return null;
          }
      

      My guess is that the FileUtils.getCanonicalPath calls aren't matching because of the symlinks. So null is being returned there.

      1. 5185.txt
        0.6 kB
        Yuki Morishita
      2. 5185-errors.log
        8 kB
        Jeremiah Jordan

        Activity

        Jeremiah Jordan created issue -
        Jeremiah Jordan made changes -
        Field Original Value New Value
        Description symlinks to data directories is broken in 1.2.0

        cd ~
        tar xzf apache-cassandra-1.2.0-bin.tar.gz
        cd apache-cassandra-1.2.0/conf
        vim cassandra.yaml
        #set data/commitlog/savecache dirs to
        #~/apache-cassandra-1.2.0/var/...
        cd ../bin
        # start once to make folders
        ./cassandra -f
        #cntrl-c
        cd ..
        mkdir var/lib/cassandra2
        mv var/lib/cassandra/data/system var/lib/cassandra2/system
        cd var/lib/cassandra/data
        ln -s ../../cassandra2/system .
        cd ~/apache-cassandra-1.2.0/bin/
        cassandra -f
        # get lots of assertion errors see attached log file
        symlinks to data directories is broken in 1.2.0

        cd ~
        tar xzf apache-cassandra-1.2.0-bin.tar.gz
        cd apache-cassandra-1.2.0/conf
        vim cassandra.yaml
        #set data/commitlog/savecache dirs to
        #~/apache-cassandra-1.2.0/var/...
        cd ../bin
        # start once to make folders
        ./cassandra -f
        #cntrl-c
        cd ..
        mkdir var/lib/cassandra2
        mv var/lib/cassandra/data/system var/lib/cassandra2/system
        cd var/lib/cassandra/data
        ln -s ../../cassandra2/system .
        cd ~/apache-cassandra-1.2.0/bin/
        cassandra -f
        #get lots of assertion errors see attached log file
        Jeremiah Jordan made changes -
        Description symlinks to data directories is broken in 1.2.0

        cd ~
        tar xzf apache-cassandra-1.2.0-bin.tar.gz
        cd apache-cassandra-1.2.0/conf
        vim cassandra.yaml
        #set data/commitlog/savecache dirs to
        #~/apache-cassandra-1.2.0/var/...
        cd ../bin
        # start once to make folders
        ./cassandra -f
        #cntrl-c
        cd ..
        mkdir var/lib/cassandra2
        mv var/lib/cassandra/data/system var/lib/cassandra2/system
        cd var/lib/cassandra/data
        ln -s ../../cassandra2/system .
        cd ~/apache-cassandra-1.2.0/bin/
        cassandra -f
        #get lots of assertion errors see attached log file
        symlinks to data directories is broken in 1.2.0
        {noformat}
        cd ~
        tar xzf apache-cassandra-1.2.0-bin.tar.gz
        cd apache-cassandra-1.2.0/conf
        vim cassandra.yaml
        #set data/commitlog/savecache dirs to
        #~/apache-cassandra-1.2.0/var/...
        cd ../bin
        # start once to make folders
        ./cassandra -f
        #cntrl-c
        cd ..
        mkdir var/lib/cassandra2
        mv var/lib/cassandra/data/system var/lib/cassandra2/system
        cd var/lib/cassandra/data
        ln -s ../../cassandra2/system .
        cd ~/apache-cassandra-1.2.0/bin/
        cassandra -f
        #get lots of assertion errors see attached log file
        {noformat}
        Jeremiah Jordan made changes -
        Attachment 5185-errors.log [ 12566374 ]
        Jeremiah Jordan made changes -
        Description symlinks to data directories is broken in 1.2.0
        {noformat}
        cd ~
        tar xzf apache-cassandra-1.2.0-bin.tar.gz
        cd apache-cassandra-1.2.0/conf
        vim cassandra.yaml
        #set data/commitlog/savecache dirs to
        #~/apache-cassandra-1.2.0/var/...
        cd ../bin
        # start once to make folders
        ./cassandra -f
        #cntrl-c
        cd ..
        mkdir var/lib/cassandra2
        mv var/lib/cassandra/data/system var/lib/cassandra2/system
        cd var/lib/cassandra/data
        ln -s ../../cassandra2/system .
        cd ~/apache-cassandra-1.2.0/bin/
        cassandra -f
        #get lots of assertion errors see attached log file
        {noformat}
        symlinks to data directories is broken in 1.2.0
        {noformat}
        cd ~
        tar xzf apache-cassandra-1.2.0-bin.tar.gz
        cd apache-cassandra-1.2.0/conf
        vim cassandra.yaml
        #set data/commitlog/savecache dirs to
        #~/apache-cassandra-1.2.0/var/...
        cd ../bin
        # start once to make folders
        ./cassandra -f
        #cntrl-c
        cd ..
        mkdir var/lib/cassandra2
        mv var/lib/cassandra/data/system var/lib/cassandra2/system
        cd var/lib/cassandra/data
        ln -s ../../cassandra2/system .
        cd ~/apache-cassandra-1.2.0/bin/
        cassandra -f
        #get lots of assertion errors see attached log file
        {noformat}

        {noformat}
         INFO 21:59:44,883 Enqueuing flush of Memtable-local@1578022692(52/52 serialized/live bytes, 2 ops)
         INFO 21:59:44,890 Writing Memtable-local@1578022692(52/52 serialized/live bytes, 2 ops)
        ERROR 21:59:44,892 Exception in thread Thread[FlushWriter:1,5,main]
        java.lang.AssertionError
        at org.apache.cassandra.io.sstable.Descriptor.<init>(Descriptor.java:190)
        at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:593)
        at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:588)
        at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:428)
        at org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:417)
        at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
        {noformat}
        Jeremiah Jordan made changes -
        Description symlinks to data directories is broken in 1.2.0
        {noformat}
        cd ~
        tar xzf apache-cassandra-1.2.0-bin.tar.gz
        cd apache-cassandra-1.2.0/conf
        vim cassandra.yaml
        #set data/commitlog/savecache dirs to
        #~/apache-cassandra-1.2.0/var/...
        cd ../bin
        # start once to make folders
        ./cassandra -f
        #cntrl-c
        cd ..
        mkdir var/lib/cassandra2
        mv var/lib/cassandra/data/system var/lib/cassandra2/system
        cd var/lib/cassandra/data
        ln -s ../../cassandra2/system .
        cd ~/apache-cassandra-1.2.0/bin/
        cassandra -f
        #get lots of assertion errors see attached log file
        {noformat}

        {noformat}
         INFO 21:59:44,883 Enqueuing flush of Memtable-local@1578022692(52/52 serialized/live bytes, 2 ops)
         INFO 21:59:44,890 Writing Memtable-local@1578022692(52/52 serialized/live bytes, 2 ops)
        ERROR 21:59:44,892 Exception in thread Thread[FlushWriter:1,5,main]
        java.lang.AssertionError
        at org.apache.cassandra.io.sstable.Descriptor.<init>(Descriptor.java:190)
        at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:593)
        at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:588)
        at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:428)
        at org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:417)
        at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
        {noformat}
        symlinks to data directories is broken in 1.2.0
        {noformat}
        cd ~
        tar xzf apache-cassandra-1.2.0-bin.tar.gz
        cd apache-cassandra-1.2.0/conf
        vim cassandra.yaml
        #set data/commitlog/savecache dirs to
        #~/apache-cassandra-1.2.0/var/...
        cd ../bin
        # start once to make folders
        ./cassandra -f
        #cntrl-c
        cd ..
        mkdir var/lib/cassandra2
        mv var/lib/cassandra/data/system var/lib/cassandra2/system
        cd var/lib/cassandra/data
        ln -s ../../cassandra2/system .
        cd ~/apache-cassandra-1.2.0/bin/
        cassandra -f
        #get lots of assertion errors see attached log file
        {noformat}

        {noformat}
         INFO 21:59:44,883 Enqueuing flush of Memtable-local@1578022692(52/52 serialized/live bytes, 2 ops)
         INFO 21:59:44,890 Writing Memtable-local@1578022692(52/52 serialized/live bytes, 2 ops)
        ERROR 21:59:44,892 Exception in thread Thread[FlushWriter:1,5,main]
        java.lang.AssertionError
        at org.apache.cassandra.io.sstable.Descriptor.<init>(Descriptor.java:190)
        at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:593)
        at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:588)
        at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:428)
        at org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:417)
        at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
        {noformat}

        I traced it back some, I think it is coming from:
        {noformat}
        org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:428)
        which calls: cfs.directories.getLocationForDisk(dataDirectory)

            public File getLocationForDisk(File dataDirectory)
            {
                for (File dir : sstableDirectories)
                {
                    if (FileUtils.getCanonicalPath(dir).startsWith(FileUtils.getCanonicalPath(dataDirectory)))
                        return dir;
                }
                return null;
            }
        {noformat}

        My guess is that the FileUtils.getCanonicalPath calls aren't matching because of the symlinks. So null is being returned there.
        Jeremiah Jordan made changes -
        Summary symlinks to data directories is broken in 1.2.0 symlinks to data directories are broken in 1.2.0
        Yuki Morishita made changes -
        Attachment 5185.txt [ 12566394 ]
        Yuki Morishita made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Fix Version/s 1.2.1 [ 12322953 ]
        Yuki Morishita made changes -
        Assignee Yuki Morishita [ yukim ]
        Sylvain Lebresne made changes -
        Fix Version/s 1.2.2 [ 12323924 ]
        Fix Version/s 1.2.1 [ 12322953 ]
        Yuki Morishita made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Reviewer jbellis
        Resolution Fixed [ 1 ]
        Gavin made changes -
        Workflow no-reopen-closed, patch-avail [ 12747569 ] patch-available, re-open possible [ 12751862 ]
        Gavin made changes -
        Workflow patch-available, re-open possible [ 12751862 ] reopen-resolved, no closed status, patch-avail, testing [ 12757984 ]

          People

          • Assignee:
            Yuki Morishita
            Reporter:
            Jeremiah Jordan
            Reviewer:
            Jonathan Ellis
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development