Cassandra
  1. Cassandra
  2. CASSANDRA-4230

Deleting a CF always produces an error and that CF remains in an unknown state

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Fix Version/s: 1.1.1
    • Component/s: Core
    • Labels:
      None
    • Environment:

      Debian Linux Squeeze with the cassandra debian package from Apache.

      Description

      From the CLI perspective:

      [default@Disco] drop column family client;
      null
      org.apache.thrift.transport.TTransportException
      at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
      at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
      at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
      at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
      at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
      at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
      at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
      at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
      at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
      at org.apache.cassandra.thrift.Cassandra$Client.recv_system_drop_column_family(Cassandra.java:1222)
      at org.apache.cassandra.thrift.Cassandra$Client.system_drop_column_family(Cassandra.java:1209)
      at org.apache.cassandra.cli.CliClient.executeDelColumnFamily(CliClient.java:1301)
      at org.apache.cassandra.cli.CliClient.executeCLIStatement(CliClient.java:234)
      at org.apache.cassandra.cli.CliMain.processStatementInteractive(CliMain.java:219)
      at org.apache.cassandra.cli.CliMain.main(CliMain.java:346)

      Log:

      INFO [MigrationStage:1] 2012-05-09 11:25:35,686 ColumnFamilyStore.java (line 634) Enqueuing flush of Memtable-schema_columnfamilies@225225949(978/1222 serialized/live bytes, 21 ops)
      INFO [FlushWriter:3] 2012-05-09 11:25:35,687 Memtable.java (line 266) Writing Memtable-schema_columnfamilies@225225949(978/1222 serialized/live bytes, 21 ops)
      INFO [FlushWriter:3] 2012-05-09 11:25:35,748 Memtable.java (line 307) Completed flushing /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-34-Data.db (1041 bytes)
      INFO [MigrationStage:1] 2012-05-09 11:25:35,749 ColumnFamilyStore.java (line 634) Enqueuing flush of Memtable-schema_columns@213209572(586/732 serialized/live bytes, 12 ops)
      INFO [FlushWriter:3] 2012-05-09 11:25:35,750 Memtable.java (line 266) Writing Memtable-schema_columns@213209572(586/732 serialized/live bytes, 12 ops)
      INFO [FlushWriter:3] 2012-05-09 11:25:35,812 Memtable.java (line 307) Completed flushing /var/lib/cassandra/data/system/schema_columns/system-schema_columns-hc-28-Data.db (649 bytes)
      INFO [CompactionExecutor:20] 2012-05-09 11:25:35,814 CompactionTask.java (line 114) Compacting [SSTableReader(path='/var/lib/cassandra/data/system/schema_columns/system-schema_columns-hc-27-Data.db'), SSTableReader
      (path='/var/lib/cassandra/data/system/schema_columns/system-schema_columns-hc-25-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/schema_columns/system-schema_columns-hc-26-Data.db'), SSTableReader(path
      ='/var/lib/cassandra/data/system/schema_columns/system-schema_columns-hc-28-Data.db')]
      INFO [MigrationStage:1] 2012-05-09 11:25:35,918 ColumnFamilyStore.java (line 634) Enqueuing flush of Memtable-Client@864320066(372/465 serialized/live bytes, 6 ops)
      INFO [FlushWriter:3] 2012-05-09 11:25:35,919 Memtable.java (line 266) Writing Memtable-Client@864320066(372/465 serialized/live bytes, 6 ops)
      INFO [CompactionExecutor:20] 2012-05-09 11:25:35,945 CompactionTask.java (line 225) Compacted to [/var/lib/cassandra/data/system/schema_columns/system-schema_columns-hc-29-Data.db,]. 22,486 to 20,621 (~91% of orig
      inal) bytes for 2 keys at 0.150120MB/s. Time: 131ms.
      INFO [FlushWriter:3] 2012-05-09 11:25:36,013 Memtable.java (line 307) Completed flushing /var/lib/cassandra/data/Disco/Client/Disco-Client-hc-5-Data.db (407 bytes)
      ERROR [MigrationStage:1] 2012-05-09 11:25:36,043 CLibrary.java (line 158) Unable to create hard link
      com.sun.jna.LastErrorException: errno was 17
      at org.apache.cassandra.utils.CLibrary.link(Native Method)
      at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:150)
      at org.apache.cassandra.db.Directories.snapshotLeveledManifest(Directories.java:343)
      at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1450)
      at org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1483)
      at org.apache.cassandra.db.DefsTable.dropColumnFamily(DefsTable.java:512)
      at org.apache.cassandra.db.DefsTable.mergeColumnFamilies(DefsTable.java:403)
      at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:270)
      at org.apache.cassandra.service.MigrationManager$1.call(MigrationManager.java:214)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:662)

      ERROR [Thrift:3] 2012-05-09 11:25:36,048 CustomTThreadPoolServer.java (line 204) Error occurred during processing of message.
      java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.IOError: java.io.IOException: Unable to create hard link from /var/lib/cassandra/data/Disco/Client/Client.json to /var/lib/cassandra/data/Disco/Client/snapshots/1336559135918-Client/Client.json (errno 17)
      at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:372)
      at org.apache.cassandra.service.MigrationManager.announce(MigrationManager.java:191)
      at org.apache.cassandra.service.MigrationManager.announceColumnFamilyDrop(MigrationManager.java:182)
      at org.apache.cassandra.thrift.CassandraServer.system_drop_column_family(CassandraServer.java:948)
      at org.apache.cassandra.thrift.Cassandra$Processor$system_drop_column_family.getResult(Cassandra.java:3348)
      at org.apache.cassandra.thrift.Cassandra$Processor$system_drop_column_family.getResult(Cassandra.java:3336)
      at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
      at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
      at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:662)
      Caused by: java.util.concurrent.ExecutionException: java.io.IOError: java.io.IOException: Unable to create hard link from /var/lib/cassandra/data/Disco/Client/Client.json to /var/lib/cassandra/data/Disco/Client/snapshots/1336559135918-Client/Client.json (errno 17)
      at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
      at java.util.concurrent.FutureTask.get(FutureTask.java:83)
      at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:368)
      ... 11 more
      Caused by: java.io.IOError: java.io.IOException: Unable to create hard link from /var/lib/cassandra/data/Disco/Client/Client.json to /var/lib/cassandra/data/Disco/Client/snapshots/1336559135918-Client/Client.json (errno 17)
      at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1454)
      at org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1483)
      at org.apache.cassandra.db.DefsTable.dropColumnFamily(DefsTable.java:512)
      at org.apache.cassandra.db.DefsTable.mergeColumnFamilies(DefsTable.java:403)
      at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:270)
      at org.apache.cassandra.service.MigrationManager$1.call(MigrationManager.java:214)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      ... 3 more
      Caused by: java.io.IOException: Unable to create hard link from /var/lib/cassandra/data/Disco/Client/Client.json to /var/lib/cassandra/data/Disco/Client/snapshots/1336559135918-Client/Client.json (errno 17)
      at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:163)
      at org.apache.cassandra.db.Directories.snapshotLeveledManifest(Directories.java:343)
      at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1450)
      ... 10 more
      ERROR [MigrationStage:1] 2012-05-09 11:25:36,051 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[MigrationStage:1,5,main]
      java.io.IOError: java.io.IOException: Unable to create hard link from /var/lib/cassandra/data/Disco/Client/Client.json to /var/lib/cassandra/data/Disco/Client/snapshots/1336559135918-Client/Client.json (errno 17)
      at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1454)
      at org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1483)
      at org.apache.cassandra.db.DefsTable.dropColumnFamily(DefsTable.java:512)
      at org.apache.cassandra.db.DefsTable.mergeColumnFamilies(DefsTable.java:403)
      at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:270)
      at org.apache.cassandra.service.MigrationManager$1.call(MigrationManager.java:214)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      at java.util.concurrent.FutureTask.run(FutureTask.java:138)

      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:662)
      Caused by: java.io.IOException: Unable to create hard link from /var/lib/cassandra/data/Disco/Client/Client.json to /var/lib/cassandra/data/Disco/Client/snapshots/1336559135918-Client/Client.json (errno 17)
      at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:163)
      at org.apache.cassandra.db.Directories.snapshotLeveledManifest(Directories.java:343)
      at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1450)
      ... 10 more
      INFO [CompactionExecutor:22] 2012-05-09 11:25:36,052 CompactionTask.java (line 114) Compacting [SSTableReader(path='/var/lib/cassandra/data/Disco/Client/Disco-Client-hc-5-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Disco/Client/Disco-Client-hc-4-Data.db')]
      INFO [CompactionExecutor:22] 2012-05-09 11:25:36,187 CompactionTask.java (line 225) Compacted to [/var/lib/cassandra/data/Disco/Client/Disco-Client-hc-6-Data.db,]. 728 to 458 (~62% of original) bytes for 8 keys at 0.003235MB/s. Time: 135ms.

      Schema:

      CREATE COLUMN FAMILY Client WITH
      key_validation_class = UUIDType AND
      comparator = UTF8Type AND
      column_metadata = [

      { column_name: key, validation_class: BytesType } { column_name: name, validation_class: UTF8Type } { column_name: userid, validation_class: UUIDType, index_type: KEYS }

      ] AND
      compression_options =

      { sstable_compression:SnappyCompressor, chunk_length_kb:64 }

      AND
      compaction_strategy = LeveledCompactionStrategy AND
      compaction_strategy_options =

      { sstable_size_in_mb: 10 }

      AND
      gc_grace = 432000;

      State of data dir after deletion attempt:

      1. ls -lah /var/lib/cassandra/data/Disco/Client/
        total 76K
        drwxr-xr-x 3 cassandra cassandra 4.0K May 9 11:25 .
        drwxr-xr-x 17 cassandra cassandra 4.0K May 3 12:34 ..
        rw-rr- 2 cassandra cassandra 420 May 9 11:25 Client-old.json
        rw-rr- 1 cassandra cassandra 418 May 7 18:04 Client.Client_userid_idx-old.json
        rw-rr- 1 cassandra cassandra 418 May 7 18:04 Client.Client_userid_idx.json
        rw-rr- 1 cassandra cassandra 418 May 9 11:25 Client.json
        rw-rr- 1 cassandra cassandra 46 May 9 11:25 Disco-Client-hc-6-CompressionInfo.db
        rw-rr- 1 cassandra cassandra 458 May 9 11:25 Disco-Client-hc-6-Data.db
        rw-rr- 1 cassandra cassandra 976 May 9 11:25 Disco-Client-hc-6-Filter.db
        rw-rr- 1 cassandra cassandra 208 May 9 11:25 Disco-Client-hc-6-Index.db
        rw-rr- 1 cassandra cassandra 4.3K May 9 11:25 Disco-Client-hc-6-Statistics.db
        rw-rr- 4 cassandra cassandra 46 May 7 18:04 Disco-Client.Client_userid_idx-hc-2-CompressionInfo.db
        rw-rr- 4 cassandra cassandra 92 May 7 18:04 Disco-Client.Client_userid_idx-hc-2-Data.db
        rw-rr- 4 cassandra cassandra 496 May 7 18:04 Disco-Client.Client_userid_idx-hc-2-Filter.db
        rw-rr- 4 cassandra cassandra 26 May 7 18:04 Disco-Client.Client_userid_idx-hc-2-Index.db
        rw-rr- 4 cassandra cassandra 4.3K May 7 18:04 Disco-Client.Client_userid_idx-hc-2-Statistics.db
        drwxr-xr-x 6 cassandra cassandra 4.0K May 9 11:25 snapshots
      1. cassandraDEBUG.log
        616 kB
        Cristian Manuel Vertiz Fernandez
      2. CASSANDRA-4230.patch
        3 kB
        Pavel Yaskevich
      3. 4230-v2.txt
        0.8 kB
        Jonathan Ellis

        Activity

        André Cruz created issue -
        Hide
        Jonathan Ellis added a comment -

        probably related to CASSANDRA-4219

        Show
        Jonathan Ellis added a comment - probably related to CASSANDRA-4219
        Jonathan Ellis made changes -
        Field Original Value New Value
        Assignee Pavel Yaskevich [ xedin ]
        Hide
        Pavel Yaskevich added a comment -

        From the exception message I see that it's related to the JNA hardlinks while it does a snapshot:

        ERROR [MigrationStage:1] 2012-05-09 11:25:36,043 CLibrary.java (line 158) Unable to create hard link
        com.sun.jna.LastErrorException: errno was 17
        ...
        Caused by: java.util.concurrent.ExecutionException: java.io.IOError: java.io.IOException: Unable to create hard link from /var/lib/cassandra/data/Disco/Client/Client.json to /var/lib/cassandra/data/Disco/Client/snapshots/1336559135918-Client/Client.json (errno 17)
        

        and errno of 17 means "File exists" so this is something with snapshot or the system rather then migrations.

        Show
        Pavel Yaskevich added a comment - From the exception message I see that it's related to the JNA hardlinks while it does a snapshot: ERROR [MigrationStage:1] 2012-05-09 11:25:36,043 CLibrary.java (line 158) Unable to create hard link com.sun.jna.LastErrorException: errno was 17 ... Caused by: java.util.concurrent.ExecutionException: java.io.IOError: java.io.IOException: Unable to create hard link from /var/lib/cassandra/data/Disco/Client/Client.json to /var/lib/cassandra/data/Disco/Client/snapshots/1336559135918-Client/Client.json (errno 17) and errno of 17 means "File exists" so this is something with snapshot or the system rather then migrations.
        Hide
        Brandon Williams added a comment -

        Since snapshot dirs include a timestamp, perhaps we should just move along if it already exists.

        Show
        Brandon Williams added a comment - Since snapshot dirs include a timestamp, perhaps we should just move along if it already exists.
        Hide
        Pavel Yaskevich added a comment -

        Yeah, we probably should do just that.

        Show
        Pavel Yaskevich added a comment - Yeah, we probably should do just that.
        Hide
        Jonathan Ellis added a comment -

        But why would it attempt the same snapshot twice? Sounds like there's a real bug here.

        Show
        Jonathan Ellis added a comment - But why would it attempt the same snapshot twice? Sounds like there's a real bug here.
        Hide
        Brandon Williams added a comment -

        snapshot on compaction is one way it can happen.

        Show
        Brandon Williams added a comment - snapshot on compaction is one way it can happen.
        Hide
        Pavel Yaskevich added a comment -

        Yeah, it's only called in snapshotWithoutFlush so if you try to drop CF in the middle of compaction snapshoting files it would be the only way to lead to such behavior. I think what we need to do here is to stop compaction and run drop after that...

        Show
        Pavel Yaskevich added a comment - Yeah, it's only called in snapshotWithoutFlush so if you try to drop CF in the middle of compaction snapshoting files it would be the only way to lead to such behavior. I think what we need to do here is to stop compaction and run drop after that...
        Hide
        Jonathan Ellis added a comment -

        snapshot on compaction is one way it can happen.

        I suppose there's a really small chance of that happening if you happen to time it just right, but that wouldn't explain it being easily reproducible (since the compaction and drop snapshots generate their snapshot name independently).

        Also, snapshot-on-compaction is off by default, I doubt André has it enabled. André, can you confirm?

        Show
        Jonathan Ellis added a comment - snapshot on compaction is one way it can happen. I suppose there's a really small chance of that happening if you happen to time it just right, but that wouldn't explain it being easily reproducible (since the compaction and drop snapshots generate their snapshot name independently). Also, snapshot-on-compaction is off by default, I doubt André has it enabled. André, can you confirm?
        Hide
        André Cruz added a comment -

        I suppose you mean this:

        $ cat /etc/cassandra/cassandra.yaml |grep snapshot |grep compac
        # Whether or not to take a snapshot before each compaction.  Be
        snapshot_before_compaction: false
        

        I did not enable it.

        Show
        André Cruz added a comment - I suppose you mean this: $ cat /etc/cassandra/cassandra.yaml |grep snapshot |grep compac # Whether or not to take a snapshot before each compaction. Be snapshot_before_compaction: false I did not enable it.
        Hide
        Pavel Yaskevich added a comment -

        I have figured out that this problem is caused only when LeveledCompaction is used for Secondary Index, it seems like when index's leveled manifest is snapshoted it uses the wrong name.

        Show
        Pavel Yaskevich added a comment - I have figured out that this problem is caused only when LeveledCompaction is used for Secondary Index, it seems like when index's leveled manifest is snapshoted it uses the wrong name.
        Pavel Yaskevich made changes -
        Attachment CASSANDRA-4230.patch [ 12526679 ]
        Pavel Yaskevich made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Reviewer jbellis
        Fix Version/s 1.1.1 [ 12319857 ]
        Hide
        Jonathan Ellis added a comment -

        Do we need this in 1.0 too?

        Show
        Jonathan Ellis added a comment - Do we need this in 1.0 too?
        Hide
        Pavel Yaskevich added a comment -

        No, this bug came together with directory-per-CF which was added into 1.1.

        Show
        Pavel Yaskevich added a comment - No, this bug came together with directory-per-CF which was added into 1.1.
        Hide
        Jonathan Ellis added a comment -

        simpler v2 attached

        Show
        Jonathan Ellis added a comment - simpler v2 attached
        Jonathan Ellis made changes -
        Attachment 4230-v2.txt [ 12526778 ]
        Hide
        Pavel Yaskevich added a comment -

        Heh, I overlooked that possibility, +1.

        Show
        Pavel Yaskevich added a comment - Heh, I overlooked that possibility, +1.
        Hide
        Pavel Yaskevich added a comment -

        Andre, can you please test and confirm that everything works as expected before we commit it?

        Show
        Pavel Yaskevich added a comment - Andre, can you please test and confirm that everything works as expected before we commit it?
        Hide
        André Cruz added a comment -

        It works, thanks!

        Show
        André Cruz added a comment - It works, thanks!
        Hide
        Jonathan Ellis added a comment -

        committed

        Show
        Jonathan Ellis added a comment - committed
        Jonathan Ellis made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Gavin made changes -
        Workflow no-reopen-closed, patch-avail [ 12666553 ] patch-available, re-open possible [ 12749668 ]
        Gavin made changes -
        Workflow patch-available, re-open possible [ 12749668 ] reopen-resolved, no closed status, patch-avail, testing [ 12754358 ]
        Hide
        Cristian Manuel Vertiz Fernandez added a comment -

        Note: Issue reproduced on cassandra 2.0.12 (datastax for windows)
        Log: refer to cassandraDEBUG.log attached file.

        Show
        Cristian Manuel Vertiz Fernandez added a comment - Note: Issue reproduced on cassandra 2.0.12 (datastax for windows) Log: refer to cassandraDEBUG.log attached file.
        Cristian Manuel Vertiz Fernandez made changes -
        Attachment cassandraDEBUG.log [ 12704165 ]
        Hide
        Joshua McKenzie added a comment -

        Cristian Manuel Vertiz Fernandez: doesn't necessarily look like the same issue to me. Your attached logs don't show any errors symlinking for snapshots during compaction though they do contain the same thrift error message:

        DEBUG [Thrift:13] 2015-03-12 09:52:49,291 CustomTThreadPoolServer.java (line 214) Thrift transport error occurred during processing of message.
         org.apache.thrift.transport.TTransportException
        	at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
        

        What behavior produced the errors you're seeing?

        Show
        Joshua McKenzie added a comment - Cristian Manuel Vertiz Fernandez : doesn't necessarily look like the same issue to me. Your attached logs don't show any errors symlinking for snapshots during compaction though they do contain the same thrift error message: DEBUG [Thrift:13] 2015-03-12 09:52:49,291 CustomTThreadPoolServer.java (line 214) Thrift transport error occurred during processing of message. org.apache.thrift.transport.TTransportException at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) What behavior produced the errors you're seeing?
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        4d 4h 18m 1 Pavel Yaskevich 13/May/12 15:47
        Patch Available Patch Available Resolved Resolved
        2d 2h 23m 1 Jonathan Ellis 15/May/12 18:10

          People

          • Assignee:
            Pavel Yaskevich
            Reporter:
            André Cruz
            Reviewer:
            Jonathan Ellis
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development