Details
-
Bug
-
Status: Resolved
-
Critical
-
Resolution: Fixed
-
1.18.0
-
Showsample-dataset.zip
Description
Per our discussion in Slack/Dev-list Here are all details and sample data-set to recreate problematic query behavior:
- We are using Drill 1.18.0-SNAPSHOT built on March 6
- We are joining on two small Parquet datasets residing on S3 using the following query:
SELECT CASE WHEN tbl1.`timestamp` IS NULL THEN tbl2.`timestamp` ELSE tbl1.`timestamp` END AS ts, * FROM `s3-store.state.`/164` AS tbl1 FULL OUTER JOIN `s3-store.result`.`/164` AS tbl2 ON tbl1.`timestamp`*10 = tbl2.`timestamp` ORDER BY ts ASC LIMIT 500 OFFSET 0 ROWS
- We are running drill in a single node setup on a 16 core, 64GB ram machine. Drill heap size is set to 16GB, while max direct memory is set to 32GB.
- As the dataset consist of really small files, Drill has been tweaked to parallelize on small item count by tweaking the following variables:
planner.slice_target = 25 planner.width.max_per_node = 16 (to match the core count)
- Without the above parallelization, query speeds on parquet files are super slow (tens of seconds)
- While queries do work, we are seeing non-proportional direct memory/heap utilization. (up 20GB of direct memory used, a min of 12GB heap required)
- We're still encountering the occasional OOM of memory error (we're also seeing heap exhaustion, but I guess that's another indication to same problem. Reducing the node parallelization width to say, 8, reduces memory contention, though it still reaches 8 gb of direct memory
User Error Occurred: One or more nodes ran out of memory while executing the query. (null) org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: One or more nodes ran out of memory while executing the query.null[Error Id: 67b61fc9-320f-47a1-8718-813843a10ecc ] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:657) at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:338) at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.drill.exec.exception.OutOfMemoryException: null at org.apache.drill.exec.vector.complex.AbstractContainerVector.allocateNew(AbstractContainerVector.java:59) at org.apache.drill.exec.test.generated.PartitionerGen5$OutgoingRecordBatch.allocateOutgoingRecordBatch(PartitionerTemplate.java:380) at org.apache.drill.exec.test.generated.PartitionerGen5$OutgoingRecordBatch.initializeBatch(PartitionerTemplate.java:400) at org.apache.drill.exec.test.generated.PartitionerGen5.setup(PartitionerTemplate.java:126) at org.apache.drill.exec.physical.impl.partitionsender.PartitionSenderRootExec.createClassInstances(PartitionSenderRootExec.java:263) at org.apache.drill.exec.physical.impl.partitionsender.PartitionSenderRootExec.createPartitioner(PartitionSenderRootExec.java:218) at org.apache.drill.exec.physical.impl.partitionsender.PartitionSenderRootExec.innerNext(PartitionSenderRootExec.java:188) at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:93) at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:323) at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:310) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:310) ... 4 common frames omitted
I've attached a (real!) sample data-set to match the query above. That same dataset recreates the aforementioned memory behavior
Help, please.
Idan
Attachments
Attachments
- sample-dataset.zip
- 1.19 MB
- Idan Sheinberg
Issue Links
- is related to
-
DRILL-7686 Excessive memory use in partition sender
- Open
-
DRILL-7687 Inaccurate memory estimates in hash join
- Open
- links to
Activity
When run on an out-of-the-box Drill deployment, default memory, parallelism and slice target, we get this error:
RESOURCE ERROR: Not enough memory for internal partitioning and fallback mechanism for HashJoin to use unbounded memory is disabled. Either enable fallback config drill.exec.hashjoin.fallback.enabled using Alter session/system command or increase memory limit for Drillbit
This seems an odd message for 2MB of data.
A workaround is to alter parallelism:
ALTER SYSTEM SET `planner.width.max_per_node`=1
Note: ALTER SYSTEM because I'm using the Web UI which does not support session options.
The result shows that the rows are both quite wide and contain complex types:
1585209600001964 164 1585209600001964 {"ready":true} {"array":[{"assetId":"1909","supply":{"levels":{"array":[{"index":1,"price":252.7,"quantity":103},{"index":2,"price":254.5,"quantity":7},{"index":3,"price":254.7,"quantity":100},{"index":4,"price":256.1,"quantity":555},{"index":5,"price":265.0,"quantity":107}]},"cleared":false,"timestamp":1585209595003064},"demand":{"levels":{"array":[{"index":1,"price":260.0,"quantity":211},{"index":2,"price":254.7,"quantity":1},{"index":3,"price":252.7,"quantity":20},{"index":4,"price":252.5,"quantity":555},{"index":5,"price":252.0,"quantity":100}]},"cleared":false,"timestamp":1585209595003230},"managed":{"orders":{"array":[]},"stage":{"id":"cddc1aac-b9cd-483c-8549-9d53d0b18a5f","types":{"array":[]}},"cleared":false,"timestamp":1585209225712652},"external":{"orders":{"array":[]},"stage":{"id":"73e939ca-e167-4e98-99e6-36b628085986","types":{"array":[{"association":{"type":"Asset","assetId":"1909"},"type":"ASSET_ORDERS"},{"association":{"type":"Heuristic","assetId":"1909","heuristicId":"1076"},"type":"HEURISTIC_ORDERS"}]}},"cleared":false,"timestamp":1585209546760487},"assetHoldings":{"quantity":0,"cleared":false,"timestamp":1585209225711681},"last":{},"status":{"phase":"PRE_OPENING_AUCTION","state":"TRADED","cleared":false,"timestamp":1585209546765981},"auction":{},"base":{"basePrice":259.9,"maxVisibleLevels":5,"priceTicks":{"array":[{"from":0.01,"to":10.0,"size":0.001},{"from":10.0,"to":100.0,"size":0.01},{"from":100.0,"to":2500.0,"size":0.1},{"from":2500.0,"size":1.0}]},"instrumentType":"SECURITY","maxQuantity":670000.0,"minQuantity":7.0,"preOpeningCeilingPrice":350.8,"preOpeningFloorPrice":169.0,"cleared":false,"timestamp":1585209225711206},"fairValueSupply":{},"fairValueDemand":{"price":253.030096,"cleared":false,"timestamp":1585209600000642},"assetViolations":{"violations":{"array":[]},"stage":{"id":"572b7ecf-e7e1-4298-80b4-c1e97570fe51","types":{"array":[]}},"cleared":false,"timestamp":1585209226969761},"platformViolations":{"violations":{"array":[]},"stage":{"id":"cc9346dc-c089-4b24-92c4-cd0113414740","types":{"array":[]}},"cleared":false,"timestamp":1585209228726569},"systemState":{"value":"ACTIVE","cleared":false,"timestamp":1585209233708361}}]} null null null {"array":[]} {} {"types":{"array":[]}}
Wide rows are likely to hit memory issues for any operator that overly-aggressively allocates memory to each column. (Operators don't know how many rows they will process.)
The query profile shows that, when running with parallelization of 1, we get 11K rows and use 13 MB of memory in one scan, 128K rows and 13MB in the other scan. The partition sender uses 80MB of memory. The join uses 300MB of memory. The other operators use minimal memory. Interestingly, query time is 9 sec. in this mode vs. the 38 sec. in the earlier test.
It is not clear, however, if the profile shows all memory; perhaps there is additional unaccounted memory in the Parquet column readers.
On a Drill server, increase parallelism to 2. Query time decreases to 3 seconds.
ALTER SYSTEM SET `planner.width.max_per_node`=2
Note that, according to the query profile, we don't actually create more fragments.
The hash partition senders now consume 145MB each, for two threads, for nearly 300MB total: far larger than the actual data size.
At a parallelism of 3 hit the following error:
org.apache.drill.common.exceptions.UserRemoteException: RESOURCE ERROR: Not enough memory for internal partitioning and fallback mechanism for HashJoin to use unbounded memory is disabled. Either enable fallback config drill.exec.hashjoin.fallback.enabled using Alter session/system command or increase memory limit for Drillbit Fragment: 3:1 ... org.apache.drill.exec.physical.impl.join.HashJoinBatch.disableSpilling(HashJoinBatch.java:997) at org.apache.drill.exec.physical.impl.join.HashJoinBatch.partitionNumTuning(HashJoinBatch.java:968) at org.apache.drill.exec.physical.impl.join.HashJoinBatch.executeBuildPhase(HashJoinBatch.java:1057) at org.apache.drill.exec.physical.impl.join.HashJoinBatch.innerNext(HashJoinBatch.java:560) ...
The hash join does some memory calcs and vastly overestimates the memory required, resulting in the following warning in the log file:
When using the minimum number of partitions 2 we require 713 MB memory but only have 682 MB available. Forcing legacy behavior of using unbounded memory in order to prevent regressions.
The hash join then checks an option and emits the message shown above.
So, we have a number of problems:
1. The hash join vastly overestimated memory needs.
2. The problem only occurs as we increase parallelism, despite the fact that the planner did not actually creates extra fragments.
Let's do what the error message suggests:
ALTER SESSION SET `drill.exec.hashjoin.fallback.enabled`=true
The query now passes in the debug environment with parallelism of 3. In the test case, was able to push parallelism to 8 (on a 8-core, 16 thread machine). Query run time decreases to 30 sec.
On a running Drillbit, parallelism of 3 reduces run time to 2.6 seconds. Parallelism of 4 gives 2.5 seconds. Increasing to parallelism of 8 provides no additional reduction of run time.
The next step is to force more parallelism as in the ticket description:
ALTER SESSION SET `planner.slice_target` = 25
The meaning of this option is not especially clear: it is the number of (estimated) rows per operator at which the planner will parallelize that operator to the limit we set earlier. The default is 100K (which seems overly large since Drill batches are on the order of several thousand nodes. Perhaps another thing to fix.) Looking at the plan for this query, the planner has guessed between 20K and 1M row, depending on operator.
The user reduced the number to 25, which pretty much forces even the smallest operator to be parallelized.
Let's clear the max per node so we use the default, then change the slice target to 1000. We now get 6 scan threads (on an 8-core CPU), run time of 3.2 seconds. So, with data on a local SSD, there is no benefit to additional parallelism. (Perhaps S3 sees a different result.) However, memory use has increased, 520 MB * 6 = 3 GB for the hash partition sender. (Remember, we only have 2 MB of data.) So, this is clearly another problem, and suggests we'll have problems as we increase parallelism.
Dropping the slice target to 25 produces the same result as a target of 1000 rows.
Increase max per node (parallelism) to 8. Run time increases to 4 seconds. Memory use has increased to 550 MB per each of the 8 partition senders.
At a parallelism of 10 we see the expected out-of-memory error:
RESOURCE ERROR: One or more nodes ran out of memory while executing the query.
The Web UI does not show the stack trace. When run in the debugger:
org.apache.drill.common.exceptions.UserRemoteException: RESOURCE ERROR: One or more nodes ran out of memory while executing the query. null Fragment: 3:1 ... Caused by: org.apache.drill.exec.exception.OutOfMemoryException: at org.apache.drill.exec.vector.complex.AbstractContainerVector.allocateNew(AbstractContainerVector.java:59) at org.apache.drill.exec.test.generated.PartitionerGen64$OutgoingRecordBatch.allocateOutgoingRecordBatch(PartitionerTemplate.java:380) at org.apache.drill.exec.test.generated.PartitionerGen64$OutgoingRecordBatch.initializeBatch(PartitionerTemplate.java:400) at org.apache.drill.exec.test.generated.PartitionerGen64.setup(PartitionerTemplate.java:126) at org.apache.drill.exec.physical.impl.partitionsender.PartitionSenderRootExec.createClassInstances(PartitionSenderRootExec.java:263) at org.apache.drill.exec.physical.impl.partitionsender.PartitionSenderRootExec.createPartitioner(PartitionSenderRootExec.java:218) at org.apache.drill.exec.physical.impl.partitionsender.PartitionSenderRootExec.innerNext(PartitionSenderRootExec.java:188) at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:93) ...
Which is indeed what we expected: the hash partition sender is using vastly too much memory for the simple query.
Digging deeper, the partition sender allocates memory for:
- For each of two data sources,
- In each of 10 sender threads,
- One batch per receiver: so 10 receivers when parallelism is 10,
- 1023 records per batch,
- ~34 fields per record (some in arrays, so multiply by 5 at each level),
- 8 - 50 bytes per field
For one table, this gives approximately:
- ~1500 bytes/record
- ~1.5 MB/batch (1,476,608 actual value from debugger)
- 15 MB/sender
- 150 MB total
The other table has six columns, but contains nested maps with a total of 247 fields, nested multiple levels deep. The result is that each batch is 66,297,984 bytes in size (giving an estimated record size of 64K.) Expand this out using the above math and we end up with the total of 677,561,600 bytes (645MB from instrumented code) per sender.
The question is, how do we tell the code, which has no visibility to actual data, that the row count size estimate is too large? Drill has no schema or stats to predict actual array carnality, so the code tends to assume 5 elements per array.
This is a classic problem with this kind of partitioning: the amount of memory increases with the square of the the parallelization count – if we hold the send buffer size constant.
Added a system/session option to allow the buffer size to shrink linearly with the increase in slice count, over some limit: exec.partition.mem_throttle:
- The default is 0, which leaves the current logic unchanged.
- If set to a positive value, then when the slice count exceeds that amount, the buffer size per sender is reduced.
- The reduction factor is 1 / (slice count - threshold).
So, if we set the threshold at 2, and run 10 slices, each slice will get 1024 / 8 = 256 records.
This option controls memory, but at obvious cost of increasing overhead. One could argue that this is a good thing. As the number of senders increases, the number of records going to each sender decreases, which increases the time that batches must accumulate before they are sent.
With this change, the test program can now run at a parallelization of 10. However, run time doubles, from 38 to 76 seconds.
sheinbergon from the investigations, we see that the issue is caused by a combination of:
- The very wide row in one of the two data sets (247 columns)
- The need to allocate 1K of records for each of the 16 receiving threads (which will likely total more than 1 GB per sending thread per exchange, and your query has two exchanges)
- The amount of memory available to run the query
Some suggestions:
- Select only the columns you need to avoid the need to shuffle unused columns.
- ALTER SESSION SET `drill.exec.hashjoin.fallback.enabled`=true to resolve the hash join memory panic (seen in my tests, not reported by you yet.)
- Slice target is OK, it simply causes parallelization to occur since your data has less than 100K records.
- Avoid being overly aggressive with parallelization. Find out where you get benefit and leave it there to avoid Drill exhausting memory just for unneeded send buffers.
- Consider using the new exec.partition.mem_throttle option to reduce buffer size per sender, but see the caveats above.
In my testing, I got far better performance using standard options than when forcing parallelization to increase and using the exec.partition.mem_throttle option. But, S3 may perform differently than the local SSD file system.
paul-rogers commented on pull request #2047: DRILL-7675: Work around for partitions sender memory use
URL: https://github.com/apache/drill/pull/2047
DRILL-7675(https://issues.apache.org/jira/browse/DRILL-7675): Work around for partitions sender memory use
-
- Description
DRILL-7675 describes a combination of factors which exposed a flaw in the partition sender:
- The partition sender holds one buffer for each of the receivers, resulting in n^2 buffers total in the system; all on a single machine for a one-node Drill.
- Every buffer holds 1024 rows.
- The size of each row depends on the row shape. In
DRILL-7675, one table has 250+ columns, some nested within repeated maps. Since each needs a vector of 1024 values (or 5 * 1024 or even 5 * 5 * 1024), the total memory size is large.
The result is that Drill attempts to allocate many GB of buffers. But, the actual data set is only 2 MB in size.
DRILL-7686 describes the needed longer-term redesign. This PR includes a workaround: the ability to reduce the number of rows per send buffer as the receiver count increases. See Documentation below.
By enabling the new option, the query will now run in the configuration that the user describes in DRILL-7675. The cost, however, is slower performance, which is exactly what the user was trying to prevent by enabling excessive parallelism. The best workaround in this case (at least with local files) is to go with default parallelism.
Also includes a number of cleanup and diagnostic fixes found during the investigation.
-
- Documentation
Adds a new system/session option to allow the buffer size to shrink linearly with the increase in slice count, over some limit: `exec.partition.mem_throttle`:
- The default is 0, which leaves the current logic unchanged.
- If set to a positive value, then when the slice count exceeds that amount, the buffer size per sender is reduced.
- The reduction factor is 1 / (slice count - threshold), with a minimum batch size of 256 records.
So, if we set the threshold at 2, and run 10 slices, each slice will get 1024 / 8 = 256 records.
This option controls memory, but at obvious cost of increasing overhead. One could argue that this is a good thing. As the number of senders increases, the number of records going to each sender decreases, which increases the time that batches must accumulate before they are sent.
If the option is enabled, and buffer size reduction kicks in, you'll find an info-level log message which details the reduction:
```
exec.partition.mem_throttle is set to 2: 10 receivers, reduced send buffer size from 1024 to 256 rows
```
-
- Testing
Created an ad-hoc test using the query from DRILL-7675. Ran this test with a variety of options, including with the new option enabled and disabled. See DRILL-7675 for a full description of the analysis.
Ran the query from DRILL-7675 in the Drill server using the Web console with the new option on and off (along with other variations.) Verified that, with the option off, performance is the same before and after the change. (3 sec on my machine.) Verified that, with the option on, the query completes even with excessive parallelism (though the query does run slower in that case.)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Created DRILL-7686 to describe the required longer-term redesign. Issued a PR for the work-around system/session option described above.
arina-ielchiieva commented on issue #2047: DRILL-7675: Work around for partitions sender memory use
URL: https://github.com/apache/drill/pull/2047#issuecomment-609407652
@paul-rogers changes look good to me. You have added really nice documentation section in the PR but I think it would be much better if you include this information in the PR itself. Especially the part describing how setting the option works. You may include some information in the option description, some in the Javadoc above this option.
```
new OptionDescription("Linearly reduces partition sender buffer row count after this number of receivers. Default is 0 (disabled)."));
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
paul-rogers commented on issue #2047: DRILL-7675: Work around for partitions sender memory use
URL: https://github.com/apache/drill/pull/2047#issuecomment-609455569
@arina-ielchiieva, thanks for the review. How do you recommend we modify the documentation?
My thought was this: I put a one-line summary in the option description. Details on how the option works are in the file where the option is used. My thought was that the detailed user description should go into the docs somewhere.
We should decide how we want to handle option documentation. Most require more background explanation than fits in a simple doc string. (For this one, we have to explain how the partition sender works in order to know why you even need the option.) To me, the docs seem the place for these explanations. This leaves the doc string as a reminder and summary.
An improvement might be:
1. Put the doc strings in a resource file.
2. Add a link to the docs where we provide the full description.
Checking the docs, I didn't immediately find a reference section for the options, so looks like that is an opportunity for improvement. We'd want two entries for each option:
1. A theory-of-operation section that explains the background along with relevant SQL commands, start-up options and system/session options.
2. A tabular reference section that contains a link to the theory of operation section and the relevant details (default value, one-line doc string, etc.)
By having the option descriptions in a resource file, we could generate the reference section above.
All of these are beyond the scope of this PR, however.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
arina-ielchiieva commented on pull request #2047: DRILL-7675: Work around for partitions sender memory use
URL: https://github.com/apache/drill/pull/2047#discussion_r403903697
##########
File path: exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java
##########
@@ -189,6 +188,12 @@ private ExecConstants() {
public static final BooleanValidator HASHAGG_FALLBACK_ENABLED_VALIDATOR = new BooleanValidator(HASHAGG_FALLBACK_ENABLED_KEY,
new OptionDescription("Hash Aggregates ignore memory limits when enabled (true). When disabled (false), Hash Aggregates fail when memory is set too low."));
+ // Partitioner options
+ public static final String PARTITIONER_MEMORY_REDUCTION_THRESHOLD_KEY = "exec.partition.mem_throttle";
+ public static final LongValidator PARTITIONER_MEMORY_REDUCTION_THRESHOLD_VALIDATOR =
+ new RangeLongValidator(PARTITIONER_MEMORY_REDUCTION_THRESHOLD_KEY, 0, Integer.MAX_VALUE,
+ new OptionDescription("Linearly reduces partition sender buffer row count after this number of receivers. Default is 0 (disabled)."));
Review comment:
Please also add Drill version when option was introduced: `(Drill 1.18+)`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
arina-ielchiieva commented on issue #2047: DRILL-7675: Work around for partitions sender memory use
URL: https://github.com/apache/drill/pull/2047#issuecomment-609650653
@paul-rogers as you know before Drill project had a document writer who was adding documentation to the site (https://drill.apache.org/docs/). Now it's up to the Drill contributor to ensure if his work is documented or not. `OptionDescription` information is used on Drill Web UI to describe options, so of course it is pretty useful if would have nice description. You can also add your description to the Drill site (create the PR, got it reviewed and then regenerate the site, there is an instruction how it should be done - https://github.com/apache/drill/tree/gh-pages-master/).
Why I have asked you to add java doc? I don't know how and when you are going to document new option. But you wrote really nice description and I did not want it to be lost, having it at least in java doc would help people in future who might be reverse-engineering this code or even users who would need to understand how this option works.
Frankly saying, having this documentation section in PR template, does not make any difference since who do we expect to transfer documentation to the Drill web site? Developer who created the PR? I did not see much documentation PRs expect of @vvysotskyi who added info about Drill Metastore. cc @cgivre
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
paul-rogers commented on issue #2047: DRILL-7675: Work around for partitions sender memory use
URL: https://github.com/apache/drill/pull/2047#issuecomment-610200359
@arina-ielchiieva you made good points. I copied the PR description into the file where the option is used. This will at least explain the option for developers.
I've been impressed with the documentation which @vvysotskyi has provided. It is first class. My thought is to do a bunch of updates as a sparate project when we get closer to Drill 1.18. There is this option. There is the confusion over the various `typeof()` methods. There will be new JSON and CSV options. I'm hoping there will be better error handling in plugin configs (if I get that part done in time.) Gathering the growing list of provided schema options (I'm working on the ability to include partition dirs as first-class columns.)
I've been leaving ever more detailed notes in the PRs so I can refer to them when doing the doc updates. There is even a JIRA ticket for how we might restructure the docs just a bit; because at present it can be hard to find a place for an extended description of some topics.
Will this address your concerns?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
arina-ielchiieva commented on issue #2047: DRILL-7675: Work around for partitions sender memory use
URL: https://github.com/apache/drill/pull/2047#issuecomment-610263997
@paul-rogers thanks for adding comment. +1, LGTM.
Please squash the commits.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
paul-rogers commented on issue #2047: DRILL-7675: Work around for partitions sender memory use
URL: https://github.com/apache/drill/pull/2047#issuecomment-610549374
@arina-ielchiieva, thanks for the review. Squashed commits.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
paul-rogers commented on issue #2047: DRILL-7675: Work around for partitions sender memory use
URL: https://github.com/apache/drill/pull/2047#issuecomment-612289545
Build failed in code unrelated to this PR. Tests passed on my dev. box. Is this a known issue?
```
2020-04-07T19:15:03.3970943Z [ERROR] Errors:
2020-04-07T19:15:03.4027624Z [ERROR] TestAggregateFunctions.stddevEmptyNonexistentNullableInput:544 » Rpc org.apach...
2020-04-07T19:15:03.4045069Z [ERROR] TestAggregateFunctions.testAggGroupByWithNullDecimal:477 » IllegalState org.ap...
2020-04-07T19:15:03.4050818Z [ERROR] TestAggregateFunctions.testAvgDecimal:391 » IllegalState org.apache.drill.comm...
2020-04-07T19:15:03.4068993Z [ERROR] TestAggregateFunctions.testCollectListStreamAgg:1075 » IllegalState org.apache...
2020-04-07T19:15:03.4075691Z [ERROR] TestAggregateFunctions.testCollectToListVarcharHashAgg:1128 » IllegalState org...
2020-04-07T19:15:03.4081969Z [ERROR] TestAggregateFunctions.testCountDistinctOnBoolColumn:105 » org.apache.drill.c...
2020-04-07T19:15:03.4086351Z [ERROR] TestAggregateFunctions.testCountStarRequired:945 » Rpc org.apache.drill.common...
2020-04-07T19:15:03.4091663Z [ERROR] TestAggregateFunctions.testGBExprWithDrillFunc:194 » org.apache.drill.common....
2020-04-07T19:15:03.4098295Z [ERROR] TestAggregateFunctions.testGroupByWithoutAggregate:1050->ClusterTest.run:122 » User
2020-04-07T19:15:03.4101846Z [ERROR] TestAggregateFunctions.testHashAggSingleValueFunction:658->getBaselineRecords:687 » Rpc
2020-04-07T19:15:03.4106742Z [ERROR] TestAggregateFunctions.testInjectVariablesHashAgg:1239 » IllegalState org.apac...
2020-04-07T19:15:03.4110807Z [ERROR] TestAggregateFunctions.testMergeSchemaFunctionStreamAgg:1200 » IllegalState or...
2020-04-07T19:15:03.4122628Z [ERROR] TestAggregateFunctions.testSchemaFunctionStreamAgg:1152 » IllegalState org.apa...
2020-04-07T19:15:03.4130655Z [ERROR] TestAggregateFunctions.testSingleValueFunction:601 » Rpc org.apache.drill.comm...
2020-04-07T19:15:03.4162870Z [ERROR] TestAggregateFunctions.testSingleValueWithComplexInput:722 » Rpc org.apache.dr...
2020-04-07T19:15:03.4167823Z [ERROR] TestAggregateFunctions.testSumAvgDecimalLimit0:444 » IllegalState org.apache.d...
2020-04-07T19:15:03.4173300Z [ERROR] TestAggregateFunctions.testSumDecimal:371 » IllegalState org.apache.drill.comm...
2020-04-07T19:15:03.4177814Z [ERROR] TestAggregateFunctions.testVarPopDecimal:312 » IllegalState org.apache.drill.c...
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
asfgit commented on pull request #2047: DRILL-7675: Work around for partitions sender memory use
URL: https://github.com/apache/drill/pull/2047
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Merged into master with commit id ae25994ef40dfa3724a47fc804b83ca3c5128851.
sheinbergon, thanks for the detailed instructions and the sample data. With a bit of fiddling, I have the test case working on my dev. box, using the local file system. I see the query returns 500 records. On my box (vintage Intel i7, SSD) with a single fragment, run via a test case, the query takes 38 seconds. I'll now start poking around to understand where the time and memory usage occurs.