Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-8225

Production-capable COPY FROM

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Normal
    • Resolution: Fixed
    • Fix Version/s: 2.1.5
    • Component/s: Legacy/Tools
    • Labels:

      Description

      Via Robin Schumacher,

      I pulled down a sourceforge data generator and created a moc file of 500,000 rows that had an incrementing sequence number, date, and SSN. I then used our COPY command and MySQL's LOAD DATA INFILE to load the file on my Mac. Results were:

      mysql> load data infile '/Users/robin/dev/datagen3.txt'  into table p_test  fields terminated by ',';
      Query OK, 500000 rows affected (2.18 sec)
      

      C* 2.1.0 (pre-CASSANDRA-7405)

      cqlsh:dev> copy p_test from '/Users/robin/dev/datagen3.txt' with delimiter=',';
      500000 rows imported in 16 minutes and 45.485 seconds.
      

      Cassandra 2.1.1:

      cqlsh:dev> copy p_test from '/Users/robin/dev/datagen3.txt' with delimiter=',';
      Processed 500000 rows; Write: 4037.46 rows/s
      500000 rows imported in 2 minutes and 3.058 seconds.
      

      [jbellis] 7405 gets us almost an order of magnitude improvement. Unfortunately we're still almost 2 orders slower than mysql.

      I don't think we can continue to tell people, "use sstableloader instead." The number of users sophisticated enough to use the sstable writers is small and (relatively) decreasing as our user base expands.

        Attachments

        1. 8225-2.1.txt
          23 kB
          Tom Hobbs
        2. 8225-2.1-v2.txt
          22 kB
          Tom Hobbs
        3. 8225-2.1-v3.txt
          22 kB
          Tom Hobbs

          Issue Links

            Activity

              People

              • Assignee:
                thobbs Tom Hobbs
                Reporter:
                jbellis Jonathan Ellis
                Authors:
                Tom Hobbs
                Reviewers:
                Aleksey Yeschenko
              • Votes:
                3 Vote for this issue
                Watchers:
                14 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: