Issue 30215 - Support 1048576 rows (was: Further thoughts on row limits)
Summary: Support 1048576 rows (was: Further thoughts on row limits)
Status: CLOSED FIXED
Alias: None
Product: Calc
Classification: Application
Component: code (show other issues)
Version: OOo 1.1.1
Hardware: All All
: P3 Trivial with 173 votes (vote)
Target Milestone: ---
Assignee: oc
QA Contact: issues@sc
URL:
Keywords:
: 54683 65773 74511 96784 (view as issue list)
Depends on: 106522 106249 109369
Blocks:
  Show dependency tree
 
Reported: 2004-06-14 22:05 UTC by bobharvey
Modified: 2017-05-20 10:33 UTC (History)
22 users (show)

See Also:
Issue Type: FEATURE
Latest Confirmation in: ---
Developer Difficulty: ---


Attachments
this is the patch in go-oo (1.03 KB, patch)
2009-03-12 14:32 UTC, kyoshida
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this issue.
Description bobharvey 2004-06-14 22:05:51 UTC
I have watched with interest the painful and exacting progress of Issue 1967, the implimentation 
of a larger row limit to keep step with other products.  Only now has it occured to me, now it's all 
over, to ask why we need a row limit at all?

Now, maybe it would have been brighter to say this 
some time ago, but better late than never.

How about, in some future version, abandoning the 
idea of a row limit - and a column limit.  Or at least making them unfeasibly large?  64 bit processors 
are becoming common, a quad precision number in 64 bits is improbably big.  Is there any reason why 
row and column limits can't be made far bigger than any competing product?
Comment 1 frank 2004-06-15 11:08:23 UTC
Hi Bettina,

one for you.

Frank
Comment 2 dridgway 2005-12-12 22:36:18 UTC
*** Issue 54683 has been marked as a duplicate of this issue. ***
Comment 3 bobharvey 2006-03-27 23:10:09 UTC
See the following:
http://blogs.msdn.com/excel/archive/2005/09/26/474258.aspx
Comment 4 terrye 2006-08-23 13:48:39 UTC
This came up on an OOoForum debate, where a newbie was asking why charting only
still supports 32K rows rather than 64K.  I pointed out that Excel had the same
limitation.  I also made the following observation whch Bob asked me to
replicate here: 

"Why on earth when we upped the various limits in 2.0 match Excel feature by
feature, rather than just go the whole hog and up the limits as sensible. I
occasionally hit a 64K row limit importing CSVs and doing query result sets.
Upping this limit to 256K for examole would have been a brilliant step past Excel."

The 32K/256 (and then 64K/256) limits are buried in the history of how Excel
stored its cell data sparsely with a 4 byte descriptor prefix (type/row/column).
 I am sure that you use nothing like this is Calc, so why on earth impose such
arbitrary and annoying constraints.  As I say it sould have useful product
differentiation. 
Comment 5 Robert Pollak 2007-01-20 11:16:25 UTC
OOo is lagging behind Excel 2007, which allows about 1 million (2^20) rows.
(I am afraid this could be Excel's "unique selling point" in my company.)
Comment 6 kpalagin 2007-02-14 14:43:36 UTC
*** Issue 74511 has been marked as a duplicate of this issue. ***
Comment 7 njglin 2007-02-16 17:44:23 UTC
*** Issue 30215 has been confirmed by votes. ***
Comment 8 a5gpn 2007-02-20 08:45:18 UTC
Not just rows, columns too.  Don't just think vertically, think laterally.
Comment 9 kpalagin 2007-03-25 16:23:55 UTC
*** Issue 31612 has been marked as a duplicate of this issue. ***
Comment 10 cappez 2007-04-12 16:20:29 UTC
More columns and rows are essential for the research project I'm involved in
(due to the large amount of data). Because it is carried out in developing
countries it would be great if Calc could be used - currently we are dependent
on SPSS for managing the data and this is very expensive software.

I really hope this will be added in a future version.
Comment 11 mestech 2007-07-30 20:59:38 UTC
I just voted on this issue as it is important to me.

In science we are getting more detailed data and this means increased data
points to be processed.  Now I have to break my data points into multiple work
sheets because of the row limits.

It would be nice to be able to load my data set into a single work sheet.  I
think OpenOffice needs to exceed Excel in this ability.

Comment 12 ooo 2007-07-31 13:42:08 UTC
*** Issue 65773 has been marked as a duplicate of this issue. ***
Comment 13 bettina.haberer 2007-09-26 15:56:54 UTC
Hi Niklas, these RFEs are in your ownership.
Comment 14 nsaa 2007-10-07 08:55:56 UTC
Excel 2007: 1048576 rows by 16384 columns

References
http://blogs.msdn.com/excel/archive/2005/09/23/473185.aspx
"the Excel 12 grid will be 1,048,576 rows by 16,384 columns. "

http://msdn2.microsoft.com/en-us/library/aa730921.aspx#Office2007excelPerf_BigGridIncreasedLimitsExcel
"The Excel 2007 "Big Grid" increases the maximum number of rows per worksheet
from 65,536 to over 1 million, and the number of columns from 256 (IV) to 16,384
(XFD)."
Comment 15 cboese 2007-10-10 15:56:40 UTC
I completely agree with the comments above, as OOo has real benefits when
compared to Excel but the drag a lagging row limit creates is striking.
Why would M$ invest in such a feature, if everybody would just use a database? ;-)
(Is this a point at all?!)

I would prefer a database myself (which could be related to the fact that I am
capable of writing SQL) and I don't tell users what they should use without a
coercive (technical) reason, which is not given in the concrete discussion.

chris
Comment 16 nsaa 2007-11-07 16:12:31 UTC
Should not this issue get a priority P3 since it will be impossible to open up
Excel 2007 files if this issue isn't fixed?
Comment 17 dca42 2007-11-16 21:17:06 UTC
PLEASE raise the limit on the number of rows!  The first poster got it right. 
The only reason this problem "doesn't affect many people" is because everyone
has conformed to the crippled tradition of Excel.  Being just as broken as Excel
used to be doesn't seem to be a great selling point to me.  1M rows is somewhat
less broken, but the right answer is no limit beyond VM space.  I am perfectly
comfortable writing SQL, and prefer a database for some tasks like persistent
storage.  But for a quick analysis of my latest 60M-row dataset I don't want to
always have to design a schema, import the data, transform it and then do the
analysis.
Comment 18 hotbbq 2007-12-05 12:24:39 UTC
Arbitrary row/column limits would be a boon for the project I'm working on at
the office.  Currently we are forced to use Matlab for doing analysis on large
sets of numbers, which works, but adds significant cost (training new users,
licenses, etc).
Comment 19 norbert2 2007-12-05 13:51:28 UTC
hotbbq: use scilab.org instead of matlab!
Comment 20 rebresb 2007-12-06 09:09:15 UTC
The large data sets I work with are manipulated with PERL scripts and stored in
a database or large flat files.  Reviews of large subsets are facilitated using
a spreadsheet, and often this requires more than 256 columns.  This allows easy
perusal and provides facilities for further analyses and visualization (graphs)
that would otherwise require sophisticated custom script solutions.  We recently
purchased WordPerfect Office X3 to get Quattro Pro because all other
spreadsheets we then had access to (Excel, OOo, Gnumeric (without recompile))
had stuck with the archaic 256-column limit.  I could deal with extracting
smaller subsets, transforming rows/columns, etc., but why should it be that
hard?  Users often employ spreadsheets as formatting tools, and reviewing a
large layout of rows & columns shouldn't be such a technical challenge today. 
As mentioned, with the Excel 2007 boost to near Quattro Pro limits, everyone
else will now need to follow.  Who will take the lead from Quattro Pro?
Comment 21 craketech 2008-01-03 10:52:27 UTC
I had made some posts on the forum before I found this issue. I think the row
limit needs to be increased. I have wanted to move to open office calc for a
while but the row limit has prevented this. I work with large amounts of product
data and currently use Excel 2007 to manipulate the data in a spreadsheet. 

As a comment above mentioned the priority of this should be raised since Open
Office is fast getting left behind by Excel and will require the new 1 million
row limit if it is to be compatible with Excel 2007 files.

I think this could be a key feature of Calc if it had no (or a very large) limit
on rows and columns. 
Comment 22 mcecconi 2008-01-15 09:45:16 UTC
I work in a bank as an EDP responsible. In this bank, all the PC's have OOo 
installed and in 5/6 (of 150 workstations) there is installed also MSOffice 
(because we receive from Istitutional Offices MS-Office files, usally with 
macros).

I often have problems in OOCalc for the reason of this issue (max numbers of 
columns and rows).

I think OOcalc should have AT LEAST the number of columns and rows of the last 
version of MSOffice for this reasons:

1) often the analyze of data requires the import (usally CSV files) which are 
longer than 65536 rows;
2) often this CSV have to be exported to MSOffice;
3) I also receive xls files with more tham 65536 rows.

So I think it should be very nice if we can have this request to be developed.

Greeting.
Comment 23 mestech 2008-02-15 15:50:23 UTC
This issue is related to the increase in the number of columns to
compete/surpass other applications.

http://www.openoffice.org/issues/show_bug.cgi?id=86049

Should be looked at together as far as I am concerned.

Comment 24 nsaa 2008-02-25 23:31:32 UTC
Upgrade the Priority
(http://www.openoffice.org/scdocs/ddIssues_EnterModify.html#priority) to 3. The
number of columns/rows should at least match Excel 2007, and  should be fixed as
sone as possible. If Calc can't read Excel 2007 files it will be devasting. If
it's possible to make the number of columns and rows dynamic (only memory or
other hardware limits) this will be great ...
Comment 25 nsaa 2008-03-21 16:31:02 UTC
Target milestone for this should be set to OOo 3.0, since Calc should support
Microsoft's format OOXML. 
Comment 26 riesling 2008-03-22 18:48:44 UTC
OpenOffice 2.1 Base uses Calc as a "helper application" for the purpose of 
importing data in the form of tab-delimited text.  However, when I attempted to 
use Calc to load data regarding voters in Broward County, Florida (which has 
nearly 1 million voters), Calc failed with an error message indicating that it 
could only load 65,536 records (its design limit) and thus the rest were 
truncated.  This makes OpenOffice Base unusable with respect to this or any 
other non-miniature database.  Request that Calc be revised to eliminate the 
65,536-record limitation, or that some other means be developed whereby Base 
can import and export tab-delimited text containing 1 million or more records.
Comment 27 feinbd 2008-04-30 16:37:25 UTC
i created an account just so i could vote.  I'm surprised, free software is
usually ahead of the curve, but not this time.  i have logs that i load into a
database, and if there is a problem with the load, i need to be able to open the
file and examine the offending record which is often grater than row 64k.
Comment 28 kslays 2008-05-11 20:22:36 UTC
Please, please add support for additional columns.  I occasionally have data
(not suitable for a database) that requires more than 1024, so am unable to use OOo.

Also, Excel has 16,384 (2^14) columns and 1,048,576 (2^20) rows.  
http://msdn2.microsoft.com/en-us/library/aa730921.aspx
OOo needs at least as many as Excel, not just for compatibility's sake, but also
in order to be useful to those of us who need that many columns in a spreadsheet
format.

Could someone please explain the technical limitations of these two
improvements?  Why is an indefinite number (hardware resource limited) of
columns and rows impossible?
Comment 29 nsaa 2008-05-13 11:17:56 UTC
See http://sc.openoffice.org/row-limit.html for an explanation
Comment 31 kslays 2008-05-17 18:02:40 UTC
nsaa, thank you for that link to hack the number of columns!
Just changing MAXCOLCOUNT_DEFINE in sc/inc/address.hxx is good enough for me as
I export to statistical software after doing some data integrity checking anyway.

Comment 32 frank.loehmann 2008-05-22 09:31:45 UTC
This issue is important and listed on the quarterly review for Calc:
http://wiki.services.openoffice.org/wiki/2008_Q2_Review_of_Spreadsheet_Project
Therefore adjusting target to 3.x.
Comment 33 mestech 2008-05-23 16:10:40 UTC
Seeing that the target is now 3.0, I will be looking at testing 3.0 sooner than
later.

This is good news.

I have enough data that requires many more rows than in OOo or previous to 2007
excel.

I may also be able to get other to use OOo sooner because of this.

Comment 34 niklas.nebel 2008-05-23 17:06:14 UTC
mestech, please don't start any false rumors.
Target for this is 3.x, not 3.0.
3.0 will have 1024 x 65536, like the beta version.
Comment 35 mestech 2008-06-03 18:05:59 UTC
Man, I was so excited that I missed the 'x' in my hopes.  :)

I guess I will just have to wait and hope that people don't start going to the
latest MS Office.  

I was hoping that with the change seeming so easy with changing the 
MAXCOLCOUNT_DEFINE working for some, I had my hopes up.

Any increase is better.
Comment 36 kami911 2008-06-03 18:43:10 UTC
I think we should follow MSO here so we have to increase the available
rows/columns. Kohei what is your opinion about it? How hard to implement it?
Comment 37 kyoshida 2008-06-04 00:43:59 UTC
Physically increasing the row and column sizes are as simple as changing the
magic numbers in one file.  The concern for increased sheet size is always with
the performance.  A larger sheet means more data to fill, which in turn means
sorting a maxed out data gets slower, copy & paste gets slower because of the
more expensive re-calculation operation etc.  So, we need to make sure that Calc
is up for handling that large size of data before increasing the maximum size.

You need to buy a larger air conditioning unit when you increase the size of
your house.  The same principle. :-)
Comment 38 kyoshida 2008-06-04 00:46:25 UTC
Also, the reason why the maximum sizes are static is because currently the
sheets are implemented using static size arrays.  We would need to switch to
using dynamically sized arrays to make the sheet size configurable.  It's
possible to do it in theory, but I have no idea how much work would be involved
to make that happen in practice.  (maybe one of those days...)
Comment 39 ooo 2008-06-04 09:31:20 UTC
Performance and internal implementation of arrays aren't the only issues. The
current drawing layer implementation doesn't allow coordinates that far down,
and already at its current limits it positions objects with an error offset.
Going for a million rows or so definitely needs a new drawing layer
implementation, which is on its way anyway, but will take its time.
Comment 40 kami911 2008-06-04 11:22:51 UTC
I did a quick test, and A1:XFD1048576 seems works well. After these coords the
calculation is perfect. I also saw problem with XFD1048576 cell only. I will
build another version with these maximized coords, and come back the findings soon.
* It seems increasing row nomber is not affect the performance so much (but here
we need more testing) Incerasing cols degragate the performance more.
IHMO increasing number of rows is more important than increasing cols.
Comment 41 mikeascott 2008-07-03 08:24:34 UTC
I see two key comments in previous notes here.
"why [do] we need a row limit at all?" and "The concern for increased sheet size
is always with the performance."

But I assume that even with the current hard limits, Calc will keep track of the
highest row/col/sheet used (if it doesn't, it should). In which case a permitted
unlimited size becomes a non-issue, at least as regards performance because Calc
should only be calculating for non-empty cells. Then if a user needs a large
spreadsheet and is prepared to put up with correspondingly slow performance,
that should be his choice; better that than making some large tasks totally
impossible for some users.

Fixed limits is usually a result of hard-coded static arrays. Surely we have
better techniques these days?

There's a lot to be said for the old 0, 1, infinity rule!

Comment 42 bobharvey 2008-08-04 12:50:00 UTC
I made the original suggestion partly because I believed it to be possible.  
Back in about 1985 I was using a cluster of VAX computers for software cross-
development, and to increase usefulness obtained a spreadsheet from one of the 
DECUS user-supported tapes.  That had unlimited numbers of rows and columns, 
and used a sparse matrix technique of double-linked lists to store the data.

no doubt there would be problems saving in older microsoft and ooo formats, but 
a warning and a choice of saving multiple sheets would cover that.

The unlimited spreadsheet is an idea whose time has come.  As, indeed, has a 
simple syntax for relative cell references to allow things like "Starting at 
H2048, 12,000 rows and 8,000 columns
Comment 43 antonvazir 2008-10-02 16:53:31 UTC
A few examples when over 65K would be needed

- Quick self analysis (and charting) of a big amount of electronically produced
data (does everyone have to be a programmer to be able to do it? No.)
- Phone bill detail (CDR) analysis (for a company, the live case) - 
- CDR analysis for small VOIP provider - who don't yet have money for a billing
(Very live case even for a period of 1 year after startup for some cell-phone
companies in developing world - personaly have seen that case)
- Internet traffic bill details analysis (for home user) for correctness
comparision (yes, details are BIG) and applies to any users who is able to get
an internet traffic details from it's ISP. 
- As For a programmer - quick charting the big amount of different
electronically produced data - to not have to learn GNUPLOT or similar soft.

5 minutes thought. And sure who deal with finances and other types of businesses
- have their examples too.
Comment 44 lcon 2008-10-24 12:06:01 UTC
Another need for more than 65k rows is in the university sector.  I'm an academic (science field) working 
with large datasets all the time and the row limit is a constant problem for me and my colleagues and 
students.

If Calc had unlimited rows I could persuade most of my students to start using it (as opposed to shelling 
out for MSO) and grow the OOo user base.

Comment 45 cpmm 2008-11-21 16:47:34 UTC
Please increase row limit to match excel 2007 (at least).  Why can't I open my
list of 70k records as a spreadsheet?  Its 2008 people, and we are still working
with computer limits from what 1997? Earlier?   Please.
Comment 46 dridgway 2008-12-30 03:29:08 UTC
*** Issue 96784 has been marked as a duplicate of this issue. ***
Comment 47 az77 2009-01-12 02:25:38 UTC
Personally I'd like  dynamic row and column sizes, perhaps defaulting to 256
columns and 32k rows, if that would save any space.

In the transition from the old Openoffice 1 format to Openoffice 2 format, I
noticed that smaller spreadsheets generally doubled in size.
If this was due to doubling the number of rows, it would be advantageous to keep
the default at 32k rows for the generally more common smaller files.
(Often I use very small spreadsheets instead of text files for organizing
information.)

At the same time, unlimited dynamic sizes would accomodate the very large
spreadsheets that I and many others sometimes need.
Comment 48 jaapaap 2009-02-11 22:24:34 UTC
More columns would make Calc a good replacement for SPSS in my case. My (market
research) datasets are usually 5-15k columns wide. My colleagues are all raving
about Excel 2007 being able to handle 16k columns; I hope Calc can follow suit soon.
Comment 49 norbert2 2009-02-12 07:59:08 UTC
Since the second OOo3 release is out now and this issue is nearly five years old
now I think it is time to specify an explicit target.
Comment 50 dca42 2009-02-12 14:56:34 UTC
Yes, please!  As many people have noted, spreadsheets can be useful far beyond
traditional accounting.  At least in principle, since all the computational
power needed is available.  But an arbitrary row (or column) limit simply
prevents these applications from working.
Comment 51 lukefury 2009-02-17 08:41:43 UTC
i wish Oo3.1 abandon row limits and column limits which seems an  outdated feature
Comment 52 rthsw 2009-03-09 16:42:23 UTC
Usage in Reality?
-searching Items in your ERP with wrong fields
-searching postings in your ERP for wrong Values
-analysing Item-movements (a quick BI)
-analysing your customers, ledger Entrys

All this types of data are mostly much more than 65K of lines. And all of them
hold the secret, that you don't know exactly what you are searching for. "There
has to be something other than the others".

Shurely, you could use BI and Databases and Reportsystems and Analyzingtools,
but mostly you can speed up your job just by STRG+A,STRG+C,STRG+V in your
Spreadsheet, do one or to re-sorts, or do an Autofilter, or do something very
quickly, very easy, and: only this one time. Next Time you need another quick
way to do your Job.

And if you are just a accounter or controler, you dont like to invest much money
for a job you cant describe before it is present, and you dont like to learn a
database without any suggestion about the need.

PLs pls pls kill this rowlimit, or make it at least configurable (
[_] 65 K combatible to all Excel
[_] 1 Mio compatible to Excel 2007
[_] 2 Mio combatible to yourself and your Family with OO3
[_] Unlimited combatible to nothing and dame slow, but we warned you

i think it would help the mass of the people more to do their Job fast with a
simple, well-known tool, even if this tool slows down a bit, and the OoTeam can
word on speed up Calc where it needs to be speed up (the forum will tell!!),
instead of can't do the Job, but you can't do it very fast...
Comment 53 lszyba1 2009-03-12 14:24:55 UTC
Hello,

It seems as this is an issue for a lot of people. A lot of duplicate bugs have
been raised, and there seem to be a "hack" to make this happen. I propose that
this bug be narrowed down to:

Choice 1: 2 million rows (2^21)
Choice 2: 1024 x 65536 ~ 67 million (Almost no limit. This should satisfy any
csv import). 

Beyond that a performance analysis would need to be done to see if its feasible. 

I vote for choice 2. Is that technically possible? Can an OpenOffice Developer
confirm this is doable and a time frame this would take him/her? If we are
talking about one variable change can somebody knowledgeable on a topic submit a
patch to this ticket. 

I appreciate your contributions. 
Thank you.
Lucas

 
Comment 54 kyoshida 2009-03-12 14:32:23 UTC
Created attachment 60899 [details]
this is the patch in go-oo
Comment 55 kyoshida 2009-03-12 14:38:59 UTC
As is already mentioned, increasing the row limit itself is very simple (as the
patch shows).  In go-oo the row limit has already been increased partly as an
experiment to identify the real-world problems with this change, and also to
properly prioritize those problems.

BTW I have already seen some performance problems when importing a large xlsx
file.  I anticipate other problems as we go along.
Comment 56 dca42 2009-03-12 21:43:44 UTC
The last time I wanted to work with large files, it was 8M rows.  So the bigger
the better.  I wanted to do some statistical analysis as a prelude to data
mining.  Performance in terms of user responsiveness is not so big an issue; I
expect it to take awhile to load, run or do lots of other things when there are
that many rows.  Maybe some quantification of performance problems would be in
order.
Comment 57 rthsw 2009-03-15 19:33:42 UTC
Solution! Solution! Solution?
The Diff-file seems like the Fileversion of the known hack. pls don't laugh
about me, but: how do i use the Diff-File? do i need to recompile the sources? I
am a user of OO (poorly, i know...), not a programmer. is there a how-to for
using this (or other) diff-files?
Comment 58 rthsw 2009-03-17 15:02:01 UTC
Solution only for developer.
Hi there, i've been contacted and told, that
-You realy need to recompile the sources to use the new rowlimit, and this isn't
funny if you don't eat Bits for Breakfast
-there will be soon a 3.1 beta version with this patch integrated, downloadable
as executables for Guys like me (i only eat bytes ,not bits for breakfast).
Comment 59 rthsw 2009-04-09 09:25:09 UTC
***Not inttegrated in Featurelist, releasenote or the newest OOo-dev 3.2.0 300m44***

Bad news: it seems like this usefull feature will still not be avaible to the Oo
Comunity. The next version will be released in less than 50 Days, and the
"unlimted" Rows are still not listed in any featurelist. So there will be only
one solution if you need an easy-to-use spreadsheet with lot of rows: learn to
use and learn to love ribbons. It seems as Open Source wount give us this
feature in the next future...
Comment 60 ombzzz 2009-04-15 03:22:27 UTC
Hi, i'm a system analyst working at a +1000 employee company and i confirm the
need for +64K rows. It is common practice in this company to use spreadsheets
with customer billing information that comprises +1000000 rows.

And i cannot just tell the users "you have to use a database not a spreadsheet
for that". They will simply answer: "Office 2007 handle the data without
problems, OpenOffice is *limited* , so we cannot use OO in this company".

Please, rethink the priority of this issue. Thank you very much.
Comment 61 murdock1965 2009-05-18 22:51:03 UTC
Besides having a personal use for a spreadsheet that allows more than 64K rows, 
it seems only obvious that as computers become more powerful, software should 
reflect the capabilities of current hardware.  

I mean no animosity, but staying at a 64K limit for even older computers is 
akin to using a 16bit operating system for a 64bit processor.  Sure, the 16bit 
OS will work, but it is also a fine waste of resources.  If not for any other 
reason, when we are shelling out cash for more powerful hardware, it is only 
natural that software should accommodate it.
Comment 62 kyoshida 2009-05-18 23:11:33 UTC
Ok, just to let everyone know, I've been working on increasing the row size
limit to match that of Excel 2007.  Along the way, I've discovered tons of
serious performance issues, many of which have been resolved, but some are still
outstanding.  No one is denying the importance of row limit increase, but not
many people seem to realize the scope of work that will be involved to increase
the row limit without substantial performance problems.

Just to give you an example, right after I increased the row limit to 1048576,
setting a page style started to take 70 seconds, saving a small-ish ods document
went into an infinite loop, switching active sheet started to take 10 minutes,
and on and on.  These are the type of the performance problems that I mentioned
would happen.

What I'm trying to say is, you don't need to try to convince us of the need.  We
are convinced.  It's just that it will take efforts to make it happen without
making Calc a totally useless piece of junk.

Just FYI...
Comment 63 murdock1965 2009-05-19 10:00:07 UTC
Hello Kohei,

Thank you for your effort and the feedback.  I was reading through all the 
comments and only yours had any information about how this is issue is 
developing.

Maybe there are things that can influence this issues current standing?

How do you go about giving this enhancement more priority or having main 
developers give it more attention?

Thanks,
Elliot
Comment 64 kyoshida 2009-06-10 21:07:13 UTC
>How do you go about giving this enhancement more priority or having main 
developers give it more attention?

Unfortunately I'm not the right person to answer that question, since I'm just a
lowly external contributor with no power in the project.  However, I can try to
create a cws and commit my changes to it, and lobby for its inclusion.  It's not
entirely finished, and I was hoping to have the code mature a bit.  But since I
have recently been accused (wrongly I might add) of holding back my code from
the upstream inclusion, I will start committing my incomplete changes, and hope
for the best.
Comment 65 kyoshida 2009-06-10 21:10:02 UTC
I'll commit my changes to kohei03 cws once I'm able to create it there.  I have
trouble creating new CWS'es at the moment.
Comment 66 kyoshida 2009-06-10 21:11:02 UTC
.
Comment 67 westlawn 2009-09-03 16:31:21 UTC
 I have been using open office at home for some time. I didn't hesitate to 
bring it into work when I upgraded the computers however...
  I work in a cemetery the calc sheet is used for recording names, burial 
dates, and burial plots. This is an ongoing sheet that is being added to daily. 
It currently holds 40,000 plus rows and is due to expand. I find the program to 
be slow to load, taking nearly a two full minutes, adding to it is slow, as it 
hesitates for a minute before suddenly bringing up what was typed, and resaving 
after adding to it is a long process. I thought about splitting the sheet but 
that would make looking up the names upon request a little difficult as some 
families have members in the A lists and some in the W lists. 
  If this program has row limits on top of the slow lagging response time then 
it will be nessicary for me to convert back to microsoft to meet the needs at 
work. 
  The lag and the limits will o doubt limit open office to home use only.
Comment 68 westlawn 2009-09-03 16:31:30 UTC
 I have been using open office at home for some time. I didn't hesitate to 
bring it into work when I upgraded the computers however...
  I work in a cemetery the calc sheet is used for recording names, burial 
dates, and burial plots. This is an ongoing sheet that is being added to daily. 
It currently holds 40,000 plus rows and is due to expand. I find the program to 
be slow to load, taking nearly a two full minutes, adding to it is slow, as it 
hesitates for a minute before suddenly bringing up what was typed, and resaving 
after adding to it is a long process. I thought about splitting the sheet but 
that would make looking up the names upon request a little difficult as some 
families have members in the A lists and some in the W lists. 
  If this program has row limits on top of the slow lagging response time then 
it will be nessicary for me to convert back to microsoft to meet the needs at 
work. 
  The lag and the limits will no doubt limit open office to home use only.
Comment 69 westlawn 2009-09-03 16:31:32 UTC
 I have been using open office at home for some time. I didn't hesitate to 
bring it into work when I upgraded the computers however...
  I work in a cemetery the calc sheet is used for recording names, burial 
dates, and burial plots. This is an ongoing sheet that is being added to daily. 
It currently holds 40,000 plus rows and is due to expand. I find the program to 
be slow to load, taking nearly a two full minutes, adding to it is slow, as it 
hesitates for a minute before suddenly bringing up what was typed, and resaving 
after adding to it is a long process. I thought about splitting the sheet but 
that would make looking up the names upon request a little difficult as some 
families have members in the A lists and some in the W lists. 
  If this program has row limits on top of the slow lagging response time then 
it will be nessicary for me to convert back to microsoft to meet the needs at 
work. 
  The lag and the limits will no doubt limit open office to home use only.
Comment 70 jodyg 2009-10-15 14:38:51 UTC
My company are currently using Excel 2000/2003 and the row limits are becoming 
more of any issue when working with third parties.
They are keen to upgrade to Excel 2007 for obvious reasons, but I'm try to hold 
them off in the hope that OpenOffice/StarOffice will increase the row limits in 
Calc, do we have any idea of when this is likely to be?
Comment 71 kyoshida 2009-10-15 14:48:32 UTC
My tentative target currently is 3.3.  There is still an unresolved issue of the
resolution of the positioning of drawing objects (e.g. notes at higher rows get
placed slightly off, as much as one cell height off).  Whether or not that will
be a show stopper for this will need to be decided.
Comment 72 norbert2 2009-10-16 08:50:18 UTC
@kohei: Are you working on this issue? Will your patch be integrated into OOo or
only into go-oo?
Comment 73 norbert2 2009-10-16 08:51:54 UTC
What about performance problems? Are they already solved?
Comment 74 kyoshida 2009-10-16 21:06:23 UTC
@norbert2: My plan is to upstream my patches from go-oo to the upstream version,
and I have created a cws for this (koheirowlimitperf).  I haven't done much
there so far because of other priorities, but I'm trying to get this done in
time for 3.3 upstream.

> What about performance problems? Are they already solved?

I've taken care of the major ones we've discovered, so that editing regular
sized documents (i.e. documents with rows less than the old limit of 64K) will
not feel sluggish.  But I doubt I've taken care of all potential performance
problems.  This will be an on-going process, and having my work integrated will
not be the end of it.
Comment 75 steelhoof 2009-10-23 01:57:22 UTC
Here is a reality check.  I just pulled up a list from the IRS. (October 22,
2009)  here is where the 64k limit snipped the list... 

** Beet Street Fort Collins   CO     
  

The list does go to the letter "Z"  

I cannot let OO lose on common folk that cannot parse a file to make it work. 
Can YOU ?  
Comment 76 bobharvey 2009-10-24 09:10:07 UTC
I have been reading the comments, and there is something remarkable about it
all, that might make it worth forking this matter.

The real-life examples given are all about data lists, not calculation.  The
world has, for want of a better application, used spreadsheets to handle what
should be in databases.  None of these, presumably, require hyperbolic trig
functions or regression (for example)

The WP-spreadhseet-graphic-database paradigm for integrated suites appears to be
fundamentally unaltered since the 1980s and 1-2-3.  If huge tranches of
computing users are bending the purpose of the spreadsheet to do something they
cannot otherwise do, then perhaps it is time to invent something else.  We no
longer use steam engines on railways, so why are we still using a mathematical
tool to sort and display data?

It is, at the very least, a damning indictment of database applications.  But it
is also an opportunity to come up with a new type of application that does what
people actually want to do.

Why can't Mr Steelhoof's clients open the list in Base?  what does it /not/ do
that they can do in a spreadsheet?  And, therefore, Why does Base not do that?
Comment 77 riesling 2009-10-24 11:31:35 UTC
bobharvey: as I noted in my earlier comment of Sat Mar 22 2008, "OpenOffice 2.1
Base uses Calc as a "helper application" for the purpose of importing data in
the form of tab-delimited text."  That is why Mr Steelhoof's clients can't open
the list in Base.  The Issue 30215 developers are working very hard to fix this.
 Let's give them our full support.
Comment 78 mestech 2009-10-26 20:27:32 UTC
bobharvey, you are correct to some aspects.

Many users that have issues don't post or complain, they just go back to MSO.

At present, we are just moving to the latest version of MSO (at least those that
can stand the ribbon) so many have not seen the row limits.

I have run into row limits when importing data from trials that I want to
process and do calculations on.  Calc has limits but I have found ways of
working around these with different tools.

As for comparison to WP/Quatro Pro/Lotus, it is interesting.  I don't see many
installations of these anymore.  I do see the odd one though.
Comment 79 tmugan 2009-12-09 06:20:47 UTC
This limitation is being used as a means to justify our corporate migration to
Office 2007.  We are increasingly dealing with larger datasets and the extra
rows and columns of Excel 2007 is an attractive feature for our users.

Comment 80 mestech 2009-12-10 16:44:26 UTC
Tmugan's comment is a reason why this should be a very high priority.  Also for
future, OOo should have a larger limit than Excel.  I see that they are looking
for 3.3, when is that due as we are still at 3.1.1?

Comment 81 kyoshida 2009-12-10 17:16:06 UTC
My current plan is to put performance enhancements relevant to the row limit
increase into 3.3 first, then resolve the following two outstanding issues.

1) formula performance enhancement.
2) drawing objects positioning issue.

We may or may not resolve these by the release of 3.3; we'll try but no
guarantee.  But at least we'll lay the foundation for future row limit increase
post-3.3 in case we don't make it for 3.3.

This may be bad news for those waiting, but please do understand that this
involves a large amount of work, so we need to do this in multiple phases.
Comment 82 shawnvdm 2010-02-12 17:32:07 UTC
Actually look at Gnumeric 1.9.14 you can right click on the tab and select
resize, then you get a slider to change to whatever row or columns you need.
This to me ( a non programmer ) seems to be a useful feature (implementation) of
all this talked about request. Those that need a large number can change to it
an accept the cost in performance and hardware requirements, those that don't
need it don't need to select it.

Being opensource software (assuming both gpl) could you not use methodologies
they implemented to help solve the problem than try reinvent then here?
Comment 83 danielsl 2010-05-28 15:00:20 UTC
I often have to analyze more than 100k records, the limit should be at least 1 
million (without any restrictions on grafics, pivots etc.). To be more future-
proof it would be wise to have either no limit or a very high one (e.g. 
billion, 2^30)
Comment 84 ooo 2010-05-28 17:48:18 UTC
After a few fixes I activated 1M (1048576) rows in CWS koheirowlimitperf.

You can observe the progress and possible integration date of CWS
koheirowlimitperf at
http://tools.services.openoffice.org/EIS2/cws.ShowCWS?Path=DEV300%2Fkoheirowlimitperf
Comment 85 ooo 2010-06-02 15:22:15 UTC
Reassigning to QA for verification.
Comment 86 oc 2010-06-16 08:26:08 UTC
verified in internal build cws_koheirowlimitperf
Comment 87 nate1481 2010-07-26 11:57:41 UTC
Could a default limit not be specified, and the size of a sheet set in the
sheets properties? This would avoid over large/long files but allow users to
customise to their needs.

I originally came across this issue 12-18 months ago when we were expanding
significantly and were seriously considering using open office rather then MS
office for all the new recruits.

The deal breaker was the row limit, we work with catalogue data files the come
in as csv, txt and xls going over 32k a daily occurrence is regular going over
64k is a regular one, while we had no alternative with excel limited we worked
roudn it, but his cause various issues and human error from spiting and merging
(and it just being time consuming)

With the 1m limit on xlsx files we we decided we couldn't justify a shift to
open office. The arbitrary limit was a deal breaker. we are a software as a
service company so the open source option would have been much preferable as it
would enable the possibility of integration with out own programs.
Comment 88 frank0051 2010-07-27 04:51:16 UTC
So any word on if this is officially going into 3.3; by all means I would assume 
so based on the milestone traget, but I'm not really sure how to check.

Thanks
Comment 89 ooo 2010-07-27 12:00:55 UTC
@frank0051:
See #desc85 the link I posted there, this was integrated to DEV300_m84 before
the OOO330 branch-off, and yes the target says OOo3.3
Comment 90 pmad 2010-12-16 02:19:04 UTC
The open source software should be leader in flexibility. The long tradition of
use open source software in science is important too. I am scientist and I am
very disappointed that I can not use my preferred application for analysis of
huge sets of data while my colleagues with no problem use the commercially
available spreadsheet.   
Comment 91 Robert Pollak 2010-12-16 08:49:48 UTC
pmad, why don't you just use the OpenOffice.org 3.3.0 Release Candidate 7?
It contains the fix, as can be seen in
 http://www.openoffice.org/dev_docs/features/3.3/index.html#One_Million_Rows_in_a_Spreadsheet .

(Or switch over to LibreOffice 3.3 Release Candidate 1, FWIW.)