function readOnly(count){ }
Starting November 20, the site will be set to read-only. On December 4, 2023,
forum discussions will move to the Trailblazer Community.
+ Start a Discussion
salesforcecrapsalesforcecrap 

Batch Job raising System.LimitException: Too many DML rows: 10001

Hi hackers,

Ok, I've read the docs, I've pared down my queries and DML, and placed some work in a Batch Apex class.

 

This Batch Apex class' start() method returns a QueryLocator based off of


[SELECT Id, myAttribute__c from Contact]

 

It's execute() method then calls three functions, say fA, fB and fC.

 

fA() executes one dml update on a list of Contacts, say 6,000 of them.

 

fB() executes one dml update on a list of Contacts, say 2,000 of them.

 

fC() executes one dml update on a list of Contacts, about 4,000 of them.

 

When I execute the batch, I can see in the Monitor interface that it queues up 328 batches, which makes sense since 200 per batch x 328 is about 65,600 Contacts, which is about the number of Contact records in our SF domain.

 

What am I not understanding about Batch Apex that is related to my job raising the LimitException?

 

 

 

dmchengdmcheng

Are you saying that for each record you are processing in the execute method, you need to update 12,000 contact reords?  You are exceeding the governor limits for DML processing in an execution:

http://www.salesforce.com/us/developer/docs/apexcode/Content/apex_gov_limits.htm

 

And even if you set the batch execute scope to 1, you will still exceed it.  What kind of business process is requiring you to updated so many contacts based on a single contact record?

salesforcecrapsalesforcecrap

Thanks for the response, dmcheng. So just to clarify, if I have a Batch Apex job that executes a DML update, that update cannot affect more than 10,000 records -- just like plain ol' Apex?

 

My Batch Job generates a "subscription" attribute on Contact. It performs a little logic, makes a few SOQL queries, and ultimately generates a List of Contact objects called, say, myContacts. I then execute

 

update myContacts;

 

I've seen a few tips online about breaking that List into pieces... but I don't see how that can really help within the context of a Batch job.

 

Any tips? Thanks for your time.

trictric

Governor limits are in effect.You cannot update more than 10,000 records through DML statements.Moreover,execute statement is calling 3 fucntions but it is considered a single transaction that is why it is giving you an error.

salesforcecrapsalesforcecrap

Thanks.

 

I guess I need to educate myself on implementation strategies for updating more than 10000 records. (Any tips/links are welcome...)

 

Starz26Starz26

simply break each function (dml update) in the batch job into their own batch job.

 

The 1st batch does fA

 

the second batch does fB

 

etc...

 

 

You may also look at limiting the batch size (scope)

 

by calling the execute batch(batch,5)

 

you can then perform the DML based on 5 contacts at a time. adjust as necessary....

 

if you really need to update more than 10,000 based of of a single contact, we will need a bit more specific info to help.

dmchengdmcheng

If you need to chain your batch process, refer to this Cookbook entry for the way to do it via email.

http://developer.force.com/cookbook/recipe/serialize-batch-apex

salesforcecrapsalesforcecrap

@Starz26: Thanks for the feedback. That is the first solution I thought of. However, now I've lost the benefit of having those three operations as part of one transaction. Or... is it one transaction per dml statement, and not necessarily per execute() call?

 

Here is what I "need to do" (and thanks for offering your insights!):

 

I have a custom Contact attribute called subscriptions__c. That attribute is a simple String. It's value is determined by the Contract and ContractContactRole associated with the Contact. I do need to take the CCR "role" value and place it in the Contact's subscriptions__c attribute. This is also dependent on the contract start date and end date.

 

So, my strategy is to create a Batch Apex class, whose execution I schedule via a Schedulable. My Batch Apex Class does the following:

 

start(): returns a QueryLocator via [SELECT Id, Subscriptions__c from Contact]

 

execute(): calls three methods, resetSubscriptions(), populateFooSubscriptions(), populateBarSubscriptions()

 

Here's what resetSubscriptions does:

 

private void resetSubscriptions() {

    List<Contact> contacts = [SELECT Id, subscriptions__c FROM Contact WHERE subscriptions__c != ''];
    for (Contact c : contacts) {
      c.subscriptions__c = '';
    }
    update contacts;
  }

 I'm taking this approach of just setting all subscriptions__c values to '' because that alleviates the need to calculate Contract expirations and so on. Instead, my intent is to reset them all to '' and then regenerate them all via populateFooSubscriptions() and populateBarSubscriptions().

 

The function resetRubscriptions() would typically affect about 6000 Contact records.

 

populateFooSubscriptions() looks like this (explanation after the code):

 

private void updateFooSubscriptions() {
    // Update Foo Subscribers with active contracts.
    // An active contract has a Role value of "Active Seat - Foo" (where Foo is the edition name)
    // and the contract's startDate and endDate surround the current date.
    List<ContractContactRole> activeCCRs = [SELECT Id, Role, Contract.startDate, Contract.endDate, Contact.subscriptions__c FROM ContractContactRole WHERE Role LIKE 'Foo%'];
    Map<ID, Contact> subscribers = new Map<ID, Contact>();
    for (ContractContactRole ccr : activeCCRs) {
      String editionName = extractEditionNameFromContractContactRole(ccr.Role);
      if (ccr.Contract.startDate <= Date.today() && ccr.Contract.endDate >= Date.today()) {
        if (ccr.Contact == null) {
          continue;
        } else if (subscribers.containsKey(ccr.Contact.Id)) {
          subscribers.get(ccr.Contact.Id).subscriptions__c = appendEditionName(subscribers.get(ccr.Contact.Id).subscriptions__c, editionName);
        } else if (ccr.Contact.subscriptions__c == null || !ccr.Contact.subscriptions__c.contains(editionName)) {
          ccr.Contact.subscriptions__c = appendEditionName(ccr.Contact.subscriptions__c, editionName);
          subscribers.put(ccr.Contact.Id, ccr.Contact);
        }
      }
    }
    List<Contact> uniqueSubscribers = subscribers.values();
    update uniqueSubscribers;
  }

 This function does the following:

- creates a List of ContractContactRole objects

- for every ContractContactRole, checks to see if it should populate the associated Contact's subscriptions__c attribute

- ensures that we've only got distinct Contact objects in the collection (via a Map)

- executes a DML update on those Contacts

 

This function affects about 500 Contact records.

 

Lastly, updateBarSubscriptions() is similar to updateFooSubscriptions() above, but it does the following:

- creates a List of Contract objects via [SELECT Id,startDate,endDate,product__c,subscription_type__c,enterprise_email_extensions__c, Account.Id, Account.parentId FROM Contract WHERE subscription_type__c LIKE '%Bar%' AND startDate <= today AND endDate >= today ORDER BY startDate];
- iterates over that list, and for every Contract, generates a list of Contacts via [SELECT Id, email, subscriptions__c FROM Contact WHERE accountId IN :accountIds]

- it then assigns a value to those Contact objects' subscriptions__c attribute, and adds the Contact to a Map that is outside the scope of the iteration drama

- after iterating over the Contracts, executes a DML update on the Contact objects.

 

This function tends to affect about 6000 Contact objects.

 

So that's 6000 Contact records that I modify via 'reset', and about 6500 that I modify via the other two functions.

 

My question is, is there a way I can "break this up" to avoid hitting the governor limit yet still be part of the same execute() call? Furthermore, what's the point of Batch Apex if we still have to adhere to this limit? I previously had this drama in a standard Apex class and it ran in under 60 seconds. With Batch Apex, which generates 328 batches, it takes over an hour.

 

If i have to split this up, I might as well not use Batch Apex!

 

 

Starz26Starz26

I just did a quick look and from what I see these are 3 seperate transactions and should be placed in thier own batch.

 

I am not sure what yiou mean by this : "now I've lost the benefit of having those three operations as part of one transaction." as it does not appear that one transaction has anything to do with the other...I will look it over more in a bit to try and get a solid understanding.

 

The 10,000 DML is per transaction.

 

**Just in case: For all of these, do not call apex classes them from the batch, actually put all this code in the batch and use the main query in the start function. Just covering that base to ensure that you are not using the code as written below in it's own class**

 

I would simply put #1 in its own batch. You may need to reqork how your query is structured as I believe that you could potentially end up with the "Non specific query error"

 

For #2, again, seperate batch but I would limit the scope when calling the batch to say 10 records or something low. The reason for this is to ensure that you are keeping the dmls li

mited. If you limit the batch size, your for loop will only return the results for the 10 contracts and thus limit the data found per transaction

 

For #3, again seperate transaction, limiting the scope limits the number of DML's per transaction.

 

The key part here is that batch apex is designed to run against ONE query and perform manipulation of the data returned from / based on that query.

Simply putting additional, unrelated queries inside the execute block is not best practice. If you are performing additional Query / dml not based of of the initial query called in the batch then you probably should be doing that in its own batch class.

 

Also, unless one batch depends on another batch (handle that by scheduling or chaining) I do not see any benifit in doing it all in one class..

Starz26Starz26

 

Here is an example of #2

 

global class BatchSubs implements Database.Batchable<sObject> {
 

 
 global BatchSubs() {

  } 
 
  global Database.QueryLocator start(Database.BatchableContext BC){
 


    String query = 'SELECT Id, Role, Contract.startDate, Contract.endDate, Contact.subscriptions__c FROM ContractContactRole WHERE Role LIKE ' + 'Foo%' + ' AND Contract.StartDate <= :date.Today() AND Contract.endDate >= :date.today() AND contact != null';

    return Database.getQueryLocator(query);  
 
  }

  global void execute(Database.BatchableContext BC, List<sObject> scope){   
 
   List<ContractContactRole> activeCCRs = (List<ContractContactRole>)scope;
 
    Map<ID, Contact> subscribers = new Map<ID, Contact>();
    for (ContractContactRole ccr : activeCCRs) {
      String editionName = extractEditionNameFromContractContactRole(ccr.Role);
 
        if (subscribers.containsKey(ccr.Contact.Id)) {
            subscribers.get(ccr.Contact.Id).subscriptions__c = appendEditionName(subscribers.get(ccr.Contact.Id).subscriptions__c, editionName);
        } else if (ccr.Contact.subscriptions__c == null || !ccr.Contact.subscriptions__c.contains(editionName)) {
            ccr.Contact.subscriptions__c = appendEditionName(ccr.Contact.subscriptions__c, editionName);
            subscribers.put(ccr.Contact.Id, ccr.Contact);
        }
      
    }
    List<Contact> uniqueSubscribers = subscribers.values();
    update uniqueSubscribers;
 
  
 

 
    
  }

  global void finish(Database.BatchableContext BC){
 

  }

}

 Then call it like this:

 

BatchSubs batch = New batchSubs();

database.executeBatch(batch,50);

salesforcecrapsalesforcecrap

Thanks Starz26. I need to educate myself on the purpose and meaning of "scope" as well as the purpose and effect of the batch size parameter when calling executeBatch().

 

I thought that the "scope" query should represent "the things we plan on manipulating with DML." Since my DML just updates Contact records, I thought that my scope should be "all contact records" or [SELECT Id from Contact].

 

As for the batch size parameter, I'm not sure how that will help me, can you explain that if you have a moment? The reason I don't understand how this will help is because the default batch size is 200. So shouldn't that mean that I would never hit the 10,000 limit anyway (this is an original assumption I made that is obviously wrong). As such, I don't see how setting that to 50 helps my issue, but I would love to know more.

 

Oh, and you asked about why I felt these three actions should be part of one transaction. Well, the task is to "clear out existing subscrptions__c values, update the Foo subscriptions and then update the Bar subscriptions. Since we don't have control over how Batch Apex jobs are executed, I figured it best to have these tasks part of one execute method, that way I know that clearing, Foo and Bar get executed in that order. I also kind of want all three operations to succeed in concert... so shouldn't I have all three in one execute() method? (I may be misunderstanding the scope of transactions per DML vs. transactions per execute().)

 

Starz26Starz26

Scope

 

When using a query locator in a batch process, scope is the records returned as a result of the querylocator.

 

Lets say you want to update all accounts owned by one of your 20,000 users in say Ohio.

 

Scope could be: [Select ID From Account Where OwnerID IN :setofids];

or Scope could be: [Select ID State__c, From User];

 

The choice is all about how you intend to use it.

 

Batch Size

 

Batch size limits the records you are working with for each transaction. Remember: Limits are reset with eact transaction and each "batch" executed is considered a seperate transaction.

 

So, In the above scope examples:

 

Using account as the scope. Say it returns 30,000 accounts. If you needed to do updates to all those accounts in the execute block, simply executing the batch would give to many query rows.

 

If you limit the batch size to say 100, it will execute the batch 300 time (100 per for total of 30,000). each time it will only be working with 100 records and thus dml will be performed on only 100 records.

 

As for your case, I believe the batch was 200 records, but the DML that you were building was not on those 200 records, rather it was on other records related to thos 200 which ended up being more that 10000 total

 

Per docs:

 

  • A maximum of 50 million records can be returned in theDatabase.QueryLocatorobject. If more than 50 million records are returned, the batch job is immediately terminated and marked as Failed.
  • The maximum value for the optional scope parameter is 2,000. If set to a higher value, Salesforce chunks the records returned by the QueryLocator into smaller batches of up to 2,000 records.

Also:

 

The

Database.executeBatchmethod takes an optional parameter scope. This parameter specifies the number of records that should be passed into theexecutemethod. This value must be greater than 0. There is no upper limit, however, if you use a very high number, you may run into other limits. Use this when you have many operations for each record being passed in and are running into governor limits. By limiting the number of records, you are thereby limiting the operations per transaction.

 

I read the above as it is not 200 per execute. it could be the whole query locator.

 

So reading into this I take it as, each chunk of data returned by the query locator creates a transaction.

 

If acting on 30,000 records returned 30,000 are passed to the execute method..

 

(someone correct me if I am wrong please)

 

Hope this helps.

Starz26Starz26

Oh,

 

In your original example for Contact:

 

Limiting the batch size will reduce the amount of dml per transaction:

 

database.executeBatch(myBatch)

 

Your contact batch should be limited to 2,000

 

Key Point: Limiting the batch size reduces the number of records returned by the query thus limiting the transaction size. when the batch cycles through the queryMore() to get the next chunck, it is a new transaction and thus limits reset

Starz26Starz26

And to your final point:

 

Executing the batch processes in order:

 

See the cookbook receipe post on the first page. it will allow you to do that.

 

Or maybe scheduling your batchs far enough apart will work as well