Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-6235 Address various 2G limits
  3. SPARK-6190

create LargeByteBuffer abstraction for eliminating 2GB limit on blocks

    Details

    • Type: Sub-task
    • Status: In Progress
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Spark Core
    • Labels:
      None

      Description

      A key component in eliminating the 2GB limit on blocks is creating a proper abstraction for storing more than 2GB. Currently spark is limited by a reliance on nio ByteBuffer and netty ByteBuf, both of which are limited at 2GB. This task will introduce the new abstraction and the relevant implementation and utilities, without effecting the existing implementation at all.

        Issue Links

          Activity

          Hide
          irashid Imran Rashid added a comment -

          design doc

          Show
          irashid Imran Rashid added a comment - design doc
          Hide
          irashid Imran Rashid added a comment -

          Hi Reynold Xin, I've attached a design doc here. I outlined a few proposals for dealing w/ ManagedBuffer, I think option #1 (creating a LargeManagedBuffer and removing LengthFieldBasedFrameDecoder) is the most straightforward. I'm going to begin prototyping the decoder for larger messages. Would appreciate any feedback on the design.

          Show
          irashid Imran Rashid added a comment - Hi Reynold Xin , I've attached a design doc here. I outlined a few proposals for dealing w/ ManagedBuffer, I think option #1 (creating a LargeManagedBuffer and removing LengthFieldBasedFrameDecoder) is the most straightforward. I'm going to begin prototyping the decoder for larger messages. Would appreciate any feedback on the design.
          Hide
          irashid Imran Rashid added a comment -

          actually, there isn't any need to change the msg format – the length is already encoded as a long, and the receiving end is free to chunk the entire message into smaller ByteBuf to stick in the LargeByteBuf however it likes.

          Show
          irashid Imran Rashid added a comment - actually, there isn't any need to change the msg format – the length is already encoded as a long, and the receiving end is free to chunk the entire message into smaller ByteBuf to stick in the LargeByteBuf however it likes.
          Hide
          irashid Imran Rashid added a comment -

          Another observation as I've dug into the implementation a little more. LargeByteBuf (the version which wraps netty's ByteBuf) isn't necessary on the sending side. The large messages are in ChunkFetchSuccess, which is either sending a file region, or something that is already available in a nio ByteBuffer. We do still need a LargeByteBuf to make it easy for the receiving end to get these messages, though. (Example decoder: https://github.com/apache/spark/blob/5e83a55daa30a19840214f77681248e112635bf6/network/common/src/main/java/org/apache/spark/network/protocol/FixedChunkLargeFrameDecoder.java) But that simplifies the api it needs to expose – the encode / decoding can still be done on ByteBuf, and we just expose the bulk of the data in the LargeByteBuf via conversion to nio LargeByteBuffer and expose as a InputStream.

          Here is a WIP branch, that demonstrates the basics of transferring large blocks, though its got a fair amount of cleanup necessary before you look too closely. I'm pretty sure some of the functionality here is not needed (eg. LargeByteBuf#getInt, since the decoding can really just use one of the ByteBuf s).

          https://github.com/apache/spark/compare/master...squito:SPARK-6190_largeBB

          Show
          irashid Imran Rashid added a comment - Another observation as I've dug into the implementation a little more. LargeByteBuf (the version which wraps netty's ByteBuf ) isn't necessary on the sending side. The large messages are in ChunkFetchSuccess , which is either sending a file region, or something that is already available in a nio ByteBuffer . We do still need a LargeByteBuf to make it easy for the receiving end to get these messages, though. (Example decoder: https://github.com/apache/spark/blob/5e83a55daa30a19840214f77681248e112635bf6/network/common/src/main/java/org/apache/spark/network/protocol/FixedChunkLargeFrameDecoder.java ) But that simplifies the api it needs to expose – the encode / decoding can still be done on ByteBuf , and we just expose the bulk of the data in the LargeByteBuf via conversion to nio LargeByteBuffer and expose as a InputStream . Here is a WIP branch, that demonstrates the basics of transferring large blocks, though its got a fair amount of cleanup necessary before you look too closely. I'm pretty sure some of the functionality here is not needed (eg. LargeByteBuf#getInt , since the decoding can really just use one of the ByteBuf s). https://github.com/apache/spark/compare/master...squito:SPARK-6190_largeBB
          Hide
          rxin Reynold Xin added a comment - - edited

          Hi Imran Rashid,

          As I said earlier, I would advise against attacking the network transfer problem at this point. We don't hear that often from users complaining about the 2G limit, and the complain of various issues drop probably by an order of magnitude in the following order:

          • caching 2g
          • fetching 2g non shuffle block
          • fetching 2g shuffle block
          • uploading 2g

          I think it'd make sense to solve the caching 2g limit first. It is important to think about the network part, but I would not try to address it here. It is much more complicated to deal with, e.g. transferring very large data in one shot brings all sorts of complicated resource management problems (e.g. large transfer blocking small ones, memory management, allocation...).

          For caching, I can think of two ways to do this. The first approach, as proposed in this ticket, is to have a large byte buffer abstraction that encapsulates multiple, smallers buffers. The second approach is to assume the block manager can only handle blocks < 2g, and then have the upper layers (e.g. CacheManager) handle the chunking and reconnecting. It is not yet clear to me which one is better. While the first approach provides a better, clearer abstraction, the 2nd approach would be less intrusive and allow us to cache partial blocks. Do you have any thoughts on this?

          Now for the large buffer abstraction here – I'm confused. The proposed design is read-only. How do we even create a buffer?

          Show
          rxin Reynold Xin added a comment - - edited Hi Imran Rashid , As I said earlier, I would advise against attacking the network transfer problem at this point. We don't hear that often from users complaining about the 2G limit, and the complain of various issues drop probably by an order of magnitude in the following order: caching 2g fetching 2g non shuffle block fetching 2g shuffle block uploading 2g I think it'd make sense to solve the caching 2g limit first. It is important to think about the network part, but I would not try to address it here. It is much more complicated to deal with, e.g. transferring very large data in one shot brings all sorts of complicated resource management problems (e.g. large transfer blocking small ones, memory management, allocation...). For caching, I can think of two ways to do this. The first approach, as proposed in this ticket, is to have a large byte buffer abstraction that encapsulates multiple, smallers buffers. The second approach is to assume the block manager can only handle blocks < 2g, and then have the upper layers (e.g. CacheManager) handle the chunking and reconnecting. It is not yet clear to me which one is better. While the first approach provides a better, clearer abstraction, the 2nd approach would be less intrusive and allow us to cache partial blocks. Do you have any thoughts on this? Now for the large buffer abstraction here – I'm confused. The proposed design is read-only. How do we even create a buffer?
          Hide
          irashid Imran Rashid added a comment -

          Hi Reynold Xin,

          I've been adding scatterered notes across the various tickets which I think has led to a lot of the confusion – lemme try to summarize things here.

          I completely agree about the importance of the various cases. Caching large blocks is by far the most important case. However, I think its worth exploring the other cases now for two reasons. (a) I still think they need to be solved eventually for a consistent user experience. Eg., if caching locally works, but reading from a remote cache doesn't, a user will be baffled when on run 1 of their job, everything works fine, but run 2, with the same data & same code, tasks get scheduled slightly different and require a remote fetch, and KABOOM! thats the kind of experience that makes the average user want to throw spark out the window. (This is actually what I thought you were pointing out in your comments on the earlier jira – that we can forget about uploading at this point, but need to make sure remote fetches work.) (b) We should make sure that whatever approach we take at least leaves the door open for solutions to all the problems. At least for myself, I wasn't sure if this approach would work for everything initially, but exploring the options makes me feel like its all possible. (which gets to your question about large blocks vs. multi-blocks.)

          The proposal isn't exactly "read-only", it also supports writing via LargeByteBufferOutputStream. It turns out thats all we need. The BlockManager currently exposes ByteBuffers, but it actually doesn't need to. For example, currently local shuffle fetches only expose a FileInputStream over the data – thats why there isn't a 2GB limit on local shuffles. (it gets wrapped in a FileSegmentManagedBuffer and eventually read here: https://github.com/apache/spark/blob/55c4831d68c8326380086b5540244f984ea9ec27/core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala#L300) It also makes sense that we only need stream access, since RDDs & broadcast vars are immutable – eg. we never say "treat bytes 37-40 as an int, and increment its value".

          Fundamentally, blocks are always created via serialization – more specifically, Serializer#serializeStream. Obviously there isn't any limit when writing to a FileOutputStream, we just need a way to write to an in-memory output stream over 2GB. We can create an Array[Array[Byte]] already with ByteArrayChunkOutputStream https://github.com/apache/spark/blob/55c4831d68c8326380086b5540244f984ea9ec27/core/src/main/scala/org/apache/spark/util/io/ByteArrayChunkOutputStream.scala (currently used to create multiple blocks by TorrentBroadcast). We can use that to write out more than 2GB, eg. creating many chunks of max size 64K.

          Similarly, we need a way to convert the various representations of large blocks back into InputStreams. File-based input streams have no problem (DiskStore only fails b/c the code currently tries to convert to a ByteBuffer, though conceptually this is unnecessary). For in-memory large blocks, represented as Array[Array[Byte]], again we can do the same as TorrentBroadcast. The final case is network transfer. This involves changing the netty frame decoder to handle frames that are > 2GB – then we just use the same input stream for the in-memory case. That was the last piece that I was prototyping, and was mentioning in my latest comments. I have an implementation available here: https://github.com/squito/spark/blob/5e83a55daa30a19840214f77681248e112635bf6/network/common/src/main/java/org/apache/spark/network/protocol/FixedChunkLargeFrameDecoder.java

          Its a good question about whether we should allow large blocks, or instead we should have blocks be limited at 2GB and have another layer put multiple blocks together. I don't know if I have very clear objective arguments for one vs. the other, but I did consider both and felt like this version was much simpler to implement. Especially given the limited api that is actually needed (only stream access), the changes proposed here really aren't that big. It keeps the changes more nicely contained to the layers underneath BlockManager (with mostly cosmetic / naming changes required in outer layers since we'd no longer be returning ByteBuffers). Going down this road certainly doesn't prevent us from later deciding to have blocks be fragmented (then its just a question of naming: are "blocks" the smallest units that we work with in the internals, and there is some new logical unit which wraps blocks? or are "blocks" the logical unit that is exposed, and there is some new smaller unit which is used by the internals? the proposed solution pushes us towards the latter. To some extent it is already creating smaller units by making Array[Array[Byte]], though as proposed these aren't chunks you'd work with directly, and in fact the sending & receiving side can "chunk" in different ways if they like.)

          Your questions about resource management are good, but IMO they are orthogonal to this change. To me, those questions apply just as well to a 1.99GB block as to a 2.01 GB block. We might consider making changes for those purposes whether or not we support blocks > 2GB. I was just discussing this w/ Marcelo, and he pointed out that (if my understanding of the network layers is correct) with the current implementation, if you fetch a 1.99 GB block (shuffle or non-shuffle), the entire block is downloaded before any processing is done, because the netty layer creates one large ByteBuf for the data before making it available. Doesn't make a lot of sense, given that we only need to expose it as an InputStream. The goal of these changes is not to optimize the performance of large blocks, its just to make them work (though of course we need reasonable efficiency). Even with my proposed changes, it probably still make sense for a spark app to be tuned such that blocks are significantly smaller than 2GB. But thats very different than having a hard failure at 2GB – especially in production systems which you expect to run regularly. I think optimizing performance for large blocks (eg. changing to a streaming interface on the receiving side of the network) would be a great addition to Spark, but should be tackled separately.

          To me, the main remaining question is, out of the assorted 2GB limits that come up, at what granularity can we check in fixes? Eg., is it an acceptable user experience to have local caching of 2GB work, but remote fetches of the cache not? As you noted, there are 4 areas we could address:

          (a) caching
          (b) fetching non-shuffle blocks (mostly for reading cached data remotely)
          (c) fetching shuffle blocks
          (d) uploading (aka replicating) blocks

          (a) was already solved by my previous PR (the code is not ready for merge, but I think conceptually it is a good solution, just needs polish.) (b) is what my latest prototyping for sending large blocks over the network was trying to address. Though (b) probably occurs far less than (a) , I feel a little uneasy about solving (a) w/out including (b) b/c of the confusing user-experience it would lead to, as I mentioned earlier. Remote fetches from the cache are not that unlikely when there is load. I'm OK with delaying (c) & (d), though I think they will actually be relatively straightforward given a solution to (b). (The earlier PR had a solution to (d), but now that I see how to transfer over 2gb directly I think that solution was too complicated; I definitely do not want to merge that.)

          whew, that was a long spiel, but hopefully writing this all down collects everything together. thanks for reading this far

          Show
          irashid Imran Rashid added a comment - Hi Reynold Xin , I've been adding scatterered notes across the various tickets which I think has led to a lot of the confusion – lemme try to summarize things here. I completely agree about the importance of the various cases. Caching large blocks is by far the most important case. However, I think its worth exploring the other cases now for two reasons. (a) I still think they need to be solved eventually for a consistent user experience. Eg., if caching locally works, but reading from a remote cache doesn't, a user will be baffled when on run 1 of their job, everything works fine, but run 2, with the same data & same code, tasks get scheduled slightly different and require a remote fetch, and KABOOM! thats the kind of experience that makes the average user want to throw spark out the window. (This is actually what I thought you were pointing out in your comments on the earlier jira – that we can forget about uploading at this point, but need to make sure remote fetches work.) (b) We should make sure that whatever approach we take at least leaves the door open for solutions to all the problems. At least for myself, I wasn't sure if this approach would work for everything initially, but exploring the options makes me feel like its all possible. (which gets to your question about large blocks vs. multi-blocks.) The proposal isn't exactly "read-only", it also supports writing via LargeByteBufferOutputStream . It turns out thats all we need. The BlockManager currently exposes ByteBuffers , but it actually doesn't need to. For example, currently local shuffle fetches only expose a FileInputStream over the data – thats why there isn't a 2GB limit on local shuffles. (it gets wrapped in a FileSegmentManagedBuffer and eventually read here: https://github.com/apache/spark/blob/55c4831d68c8326380086b5540244f984ea9ec27/core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala#L300 ) It also makes sense that we only need stream access, since RDDs & broadcast vars are immutable – eg. we never say "treat bytes 37-40 as an int, and increment its value". Fundamentally, blocks are always created via serialization – more specifically, Serializer#serializeStream . Obviously there isn't any limit when writing to a FileOutputStream , we just need a way to write to an in-memory output stream over 2GB. We can create an Array[Array[Byte]] already with ByteArrayChunkOutputStream https://github.com/apache/spark/blob/55c4831d68c8326380086b5540244f984ea9ec27/core/src/main/scala/org/apache/spark/util/io/ByteArrayChunkOutputStream.scala (currently used to create multiple blocks by TorrentBroadcast). We can use that to write out more than 2GB, eg. creating many chunks of max size 64K. Similarly, we need a way to convert the various representations of large blocks back into InputStreams . File-based input streams have no problem ( DiskStore only fails b/c the code currently tries to convert to a ByteBuffer , though conceptually this is unnecessary). For in-memory large blocks, represented as Array[Array[Byte]] , again we can do the same as TorrentBroadcast . The final case is network transfer. This involves changing the netty frame decoder to handle frames that are > 2GB – then we just use the same input stream for the in-memory case. That was the last piece that I was prototyping, and was mentioning in my latest comments. I have an implementation available here: https://github.com/squito/spark/blob/5e83a55daa30a19840214f77681248e112635bf6/network/common/src/main/java/org/apache/spark/network/protocol/FixedChunkLargeFrameDecoder.java Its a good question about whether we should allow large blocks, or instead we should have blocks be limited at 2GB and have another layer put multiple blocks together. I don't know if I have very clear objective arguments for one vs. the other, but I did consider both and felt like this version was much simpler to implement. Especially given the limited api that is actually needed (only stream access), the changes proposed here really aren't that big. It keeps the changes more nicely contained to the layers underneath BlockManager (with mostly cosmetic / naming changes required in outer layers since we'd no longer be returning ByteBuffers). Going down this road certainly doesn't prevent us from later deciding to have blocks be fragmented (then its just a question of naming: are "blocks" the smallest units that we work with in the internals, and there is some new logical unit which wraps blocks? or are "blocks" the logical unit that is exposed, and there is some new smaller unit which is used by the internals? the proposed solution pushes us towards the latter. To some extent it is already creating smaller units by making Array[Array[Byte]] , though as proposed these aren't chunks you'd work with directly, and in fact the sending & receiving side can "chunk" in different ways if they like.) Your questions about resource management are good, but IMO they are orthogonal to this change. To me, those questions apply just as well to a 1.99GB block as to a 2.01 GB block. We might consider making changes for those purposes whether or not we support blocks > 2GB. I was just discussing this w/ Marcelo, and he pointed out that (if my understanding of the network layers is correct) with the current implementation, if you fetch a 1.99 GB block (shuffle or non-shuffle), the entire block is downloaded before any processing is done, because the netty layer creates one large ByteBuf for the data before making it available. Doesn't make a lot of sense, given that we only need to expose it as an InputStream . The goal of these changes is not to optimize the performance of large blocks, its just to make them work (though of course we need reasonable efficiency). Even with my proposed changes, it probably still make sense for a spark app to be tuned such that blocks are significantly smaller than 2GB. But thats very different than having a hard failure at 2GB – especially in production systems which you expect to run regularly. I think optimizing performance for large blocks (eg. changing to a streaming interface on the receiving side of the network) would be a great addition to Spark, but should be tackled separately. To me, the main remaining question is, out of the assorted 2GB limits that come up, at what granularity can we check in fixes? Eg., is it an acceptable user experience to have local caching of 2GB work, but remote fetches of the cache not? As you noted, there are 4 areas we could address: (a) caching (b) fetching non-shuffle blocks (mostly for reading cached data remotely) (c) fetching shuffle blocks (d) uploading (aka replicating) blocks (a) was already solved by my previous PR (the code is not ready for merge, but I think conceptually it is a good solution, just needs polish.) (b) is what my latest prototyping for sending large blocks over the network was trying to address. Though (b) probably occurs far less than (a) , I feel a little uneasy about solving (a) w/out including (b) b/c of the confusing user-experience it would lead to, as I mentioned earlier. Remote fetches from the cache are not that unlikely when there is load. I'm OK with delaying (c) & (d), though I think they will actually be relatively straightforward given a solution to (b). (The earlier PR had a solution to (d), but now that I see how to transfer over 2gb directly I think that solution was too complicated; I definitely do not want to merge that.) whew, that was a long spiel, but hopefully writing this all down collects everything together. thanks for reading this far
          Hide
          rxin Reynold Xin added a comment -

          If I can guarantee at the block manager level, all large blocks are chunked into smaller ones less than 2G, then there is no reason to support +2GB blocks at the block manager level.

          This affects the very core of Spark. It is important to think about how this will affect the long term Spark evolution (including explicit memory management, operating directly against records in the form of raw bytes, etc), rather than just rushing in, patching individual problems and leading to a codebase that has tons of random abstractions.

          On a separate topic, based on your design doc, LargeByteBuffer is still read only. There is no interface for LargeByteBufferOutputStream to even write to LargeByteBuffer. Can you include that?

          Show
          rxin Reynold Xin added a comment - If I can guarantee at the block manager level, all large blocks are chunked into smaller ones less than 2G, then there is no reason to support +2GB blocks at the block manager level. This affects the very core of Spark. It is important to think about how this will affect the long term Spark evolution (including explicit memory management, operating directly against records in the form of raw bytes, etc), rather than just rushing in, patching individual problems and leading to a codebase that has tons of random abstractions. On a separate topic, based on your design doc, LargeByteBuffer is still read only. There is no interface for LargeByteBufferOutputStream to even write to LargeByteBuffer. Can you include that?
          Hide
          irashid Imran Rashid added a comment -

          I've updated the design doc to include some code.

          No, we shouldn't be making changes willy nilly, but we've also got to commit to fixing major bugs like this. Part of the reason the proposal for LargeByteBuffer focused on what was missing was to emphasize the limited set of functionality. Its much easier to expose more later than to strip things out.

          I've looked into limiting all blocks to < 2GB. Pretty quickly you see that each block must belong to a "BlockGroup", which doesn't have a 2GB limit. If you try to cache a partition over 2GB, then it would need to make create multiple blocks. Later on when you try fetch the data for that partition, you have no idea which blocks to fetch – you only know the BlockGroup. Similarly, for dropping blocks, its pretty useless to only drop some of the blocks in a BlockGroup, since you'd need to recreate the entire BlockGroup in any case. So we'd have to support put, drop, and get for BlockGroups (though get could be streaming-ish, eg. returning an iterator over blocks)

          Nonetheless, it might (a) make the changes to the network layer simpler and (b) open the door for future optimizations, such as auto-splitting large partitions. I don't think that introducing LargeByteBuffer would even make it any harder to eventually move to blocks < 2GB, but I'm open to either approach.

          Show
          irashid Imran Rashid added a comment - I've updated the design doc to include some code. No, we shouldn't be making changes willy nilly, but we've also got to commit to fixing major bugs like this. Part of the reason the proposal for LargeByteBuffer focused on what was missing was to emphasize the limited set of functionality. Its much easier to expose more later than to strip things out. I've looked into limiting all blocks to < 2GB. Pretty quickly you see that each block must belong to a " BlockGroup ", which doesn't have a 2GB limit. If you try to cache a partition over 2GB, then it would need to make create multiple blocks. Later on when you try fetch the data for that partition, you have no idea which blocks to fetch – you only know the BlockGroup. Similarly, for dropping blocks, its pretty useless to only drop some of the blocks in a BlockGroup, since you'd need to recreate the entire BlockGroup in any case. So we'd have to support put , drop , and get for BlockGroups (though get could be streaming-ish, eg. returning an iterator over blocks) Nonetheless, it might (a) make the changes to the network layer simpler and (b) open the door for future optimizations, such as auto-splitting large partitions. I don't think that introducing LargeByteBuffer would even make it any harder to eventually move to blocks < 2GB, but I'm open to either approach.
          Hide
          apachespark Apache Spark added a comment -

          User 'squito' has created a pull request for this issue:
          https://github.com/apache/spark/pull/5400

          Show
          apachespark Apache Spark added a comment - User 'squito' has created a pull request for this issue: https://github.com/apache/spark/pull/5400
          Hide
          jahubba Jason Hubbard added a comment -

          We often run into the 2gb limit using the jdbcRdd. Because we don't want too many connections to the db we have to allow >2gb. We currently use a different tool to write larger tables to a file, but this goes against why we started using spark as a unified tool. I'm not sure how where jdbcRdd gets stored if it uses the block manager, internally it could chunk the blocks after getting the rows from the db but this is a significant issue for us right now.

          Show
          jahubba Jason Hubbard added a comment - We often run into the 2gb limit using the jdbcRdd. Because we don't want too many connections to the db we have to allow >2gb. We currently use a different tool to write larger tables to a file, but this goes against why we started using spark as a unified tool. I'm not sure how where jdbcRdd gets stored if it uses the block manager, internally it could chunk the blocks after getting the rows from the db but this is a significant issue for us right now.
          Hide
          lisendong hotdog added a comment -

          is there any progress?

          Show
          lisendong hotdog added a comment - is there any progress?
          Hide
          bdolbeare Brian added a comment -

          This is a terrible problem which I'm hoping someone will solve soon! We have jobs that take many hours to get to the point where they fail due to this bug. The only workaround we've found is to try to jiggle some of the spark settings like spark.sql.shuffle.partitions or manually repartition our dataframes/RDDs; however, since our datasets are not static we find that production jobs which have been running fine for months just suddenly die with this unexpected failure. Additionally, this bug makes it very difficult to have confidence that any of our new spark jobs will work - you basically have to try to run the job and if it fails start bumping up the partitions. This leads to a lot of stress when trying to meet project deadlines as jobs are failing because of a 2GB memory limit (it's really hard to explain to management that our jobs have memory problems even though our servers have 256 to 500 GB of ram each and we have close to 100 servers).

          Show
          bdolbeare Brian added a comment - This is a terrible problem which I'm hoping someone will solve soon! We have jobs that take many hours to get to the point where they fail due to this bug. The only workaround we've found is to try to jiggle some of the spark settings like spark.sql.shuffle.partitions or manually repartition our dataframes/RDDs; however, since our datasets are not static we find that production jobs which have been running fine for months just suddenly die with this unexpected failure. Additionally, this bug makes it very difficult to have confidence that any of our new spark jobs will work - you basically have to try to run the job and if it fails start bumping up the partitions. This leads to a lot of stress when trying to meet project deadlines as jobs are failing because of a 2GB memory limit (it's really hard to explain to management that our jobs have memory problems even though our servers have 256 to 500 GB of ram each and we have close to 100 servers).

            People

            • Assignee:
              irashid Imran Rashid
              Reporter:
              irashid Imran Rashid
            • Votes:
              17 Vote for this issue
              Watchers:
              46 Start watching this issue

              Dates

              • Created:
                Updated:

                Development