function readOnly(count){ }
Starting November 20, the site will be set to read-only. On December 4, 2023,
forum discussions will move to the Trailblazer Community.
+ Start a Discussion
J BengelJ Bengel 

System.LimitException: Apex heap size too large

It's complicated. But I can't believe that this hasn't been solved before, so I'm hoping to find one of the Apex Jedi who has seen and conquered it.

We converted a legacy system to a costom app in SF but only convetred the data that was "active" (for a couple of reasons, that woudl take to long to enumerate here). The rest was archive data that we would never need to reference in day to day operations, and in fact if not for this one reporting requirement, we wouldn't have to reference it regularly at all. But we're talking about a federal agency here and apparnelty they need to get the entire data set for all time every quarter. So we send them 2 CSV's every quarter, one that is a reasonable size, and one that is enormous. And as you have probably guessed ('cause you're smart like that) ti's the second one that's causing the weeping and gnashing of teeth.

Since we only converted the data taht we were likely to need, the older archive data is stored as static resources. It took two CSV's to store the big table, but that was the only way to get around the 5Mb size limit on static resources. So the idea was to select all of the records in the object that contains the relevant data for the extract create the CSV records as an array, tehn read the static resource and convert that to an array and .addAll() the second array to the first one. Then the result is flattened and written out to the final CSV for delivery. In the smaller of the extracts this works brilliantly.

But you see what's coming, right?

The "live" data for the second extraction presents no problem. It's bulky, but it still gets in under the limit fairly comfortably. The first of the two static resources gets tacked on without a problem too -- 30,000 or so rows worth. But when the second static resource gets into the act, we hit the wall with a heap size limit exception.

What I'm trying to do, in the abstract, is something like: 
create file1
cat file2 >> file1
cat file3 >> file1
 
And it's the last bit that's failing.

The "create file1" is amassing the results of a query using a typical
for(record:selection) loop happlily hummig along adding the relevant parts of each record to an array called "enrollments" in the execute method of a batch apex job.

When all that's done, the finish method starts with this:
List<String> allLines = NCRAN_Utilities.serializeStaticCSV('USDOLStaticParticipants1');
        enrollments.addAll(allLines);

        allLines.clear();   // clear the previous contents

        allLines = NCRAN_Utilities.serializeStaticCSV('USDOLStaticParticipants2');
        enrollments.addAll(allLines);
NCRAN_Utilities.serializeStaticCSV('USDOLStaticParticipants1'); is a recursive method that parses out the static resource named in the argument (USDOLStaticParticipants1) into a list of lines, which don't need to be broken down into fields because they're already correctly formatted. This is the "cat file2 >> file1". The logs tell me that this works out fine too, and we don't run into trouble until we get to the "cat file 3 >> file 1" part, seen here as 
NCRAN_Utilities.serializeStaticCSV('USDOLStaticParticipants2');

I suspect that the proble doesn't arise until we try to paste the results of the second call to the main array, because the error doesn't occur in the utility class.

What would be ideal would be for the agency in quesiton to use their own resources to store all of this history rather than getting ever-larger uploads from us (or at least give us a web service to send to rather than forcing us to party like it's 1991). Since that's... unlikely to happen, teh next best thing would be a way that the static resources themselves could simply be grafted onto the main file (literally cat file2 >> file1). But I have seen nothing in any of my research so far to suggest that this is an available solution without going outside of Apex. Absent either of those possibilities, I have to find some way of appeasing Apex so that I can join these three bloated arrays topgehter into one and commit that to an attachment.

Any ideas are welcome. (Though ideas that work are preferred.)

​​​​​​​Thanks!
 
J BengelJ Bengel
I thought I had it beat. I figured out a heap-neutral method of transferrign the allLines dat ato the enrollments array without duplicating allLines within enrollments by "popping each line off the top of allLines, and appending tit ot the end of enroolments. It worked great in the sandbox wher eI discovere dthe problem. Unfortunately, taht sandbox is nowhere close to "life-sized", and when I moved my code over to a developer pro box and tested at scale, it slammed into the same wall -- harder.

But since this may help someone solve a similar problem down the road, it basically works like this:
List<String> stuff = new List<String>{'red','orange','yellow','green','blue','indigo','violet'};
List<String> junk = new List<String>();
System.debug(' Initial--> stuff:'+ stuff);
System.debug(' Initial--> junk:'+ junk);

Integer ilimit = stuff.size();

for(Integer i = 0; i < ilimit; i++){
    junk.add(stuff.remove(0));
    System.debug(' Pass '+i+'--> stuff:'+ stuff+' stuff.size(): '+stuff.size());
    System.debug(' Pass '+i+'--> junk:'+ junk+' junk.size(): '+junk.size());
    
}

14:34:13:003 USER_DEBUG [3]|DEBUG| Initial--> stuff:(red, orange, yellow, green, blue, indigo, violet)
14:34:13:003 USER_DEBUG [4]|DEBUG| Initial--> junk:()
14:34:13:003 USER_DEBUG [10]|DEBUG| Pass 0--> stuff:(orange, yellow, green, blue, indigo, violet) stuff.size(): 6
14:34:13:003 USER_DEBUG [11]|DEBUG| Pass 0--> junk:(red) junk.size(): 1
14:34:13:003 USER_DEBUG [10]|DEBUG| Pass 1--> stuff:(yellow, green, blue, indigo, violet) stuff.size(): 5
14:34:13:003 USER_DEBUG [11]|DEBUG| Pass 1--> junk:(red, orange) junk.size(): 2
14:34:13:003 USER_DEBUG [10]|DEBUG| Pass 2--> stuff:(green, blue, indigo, violet) stuff.size(): 4
14:34:13:003 USER_DEBUG [11]|DEBUG| Pass 2--> junk:(red, orange, yellow) junk.size(): 3
14:34:13:004 USER_DEBUG [10]|DEBUG| Pass 3--> stuff:(blue, indigo, violet) stuff.size(): 3
14:34:13:004 USER_DEBUG [11]|DEBUG| Pass 3--> junk:(red, orange, yellow, green) junk.size(): 4
14:34:13:004 USER_DEBUG [10]|DEBUG| Pass 4--> stuff:(indigo, violet) stuff.size(): 2
14:34:13:004 USER_DEBUG [11]|DEBUG| Pass 4--> junk:(red, orange, yellow, green, blue) junk.size(): 5
14:34:13:004 USER_DEBUG [10]|DEBUG| Pass 5--> stuff:(violet) stuff.size(): 1
14:34:13:004 USER_DEBUG [11]|DEBUG| Pass 5--> junk:(red, orange, yellow, green, blue, indigo) junk.size(): 6
14:34:13:004 USER_DEBUG [10]|DEBUG| Pass 6--> stuff:() stuff.size(): 0
14:34:13:004 USER_DEBUG [11]|DEBUG| Pass 6--> junk:(red, orange, yellow, green, blue, indigo, violet) junk.size(): 7
You could also do this in a while stuff.size() > 0 loop, but using a for loop limited by the array size means not making the call to .size() on every pass. Not sure how much that matters, especially in asynchronous Apex, but it's worth considering.