function readOnly(count){ }
Starting November 20, the site will be set to read-only. On December 4, 2023,
forum discussions will move to the Trailblazer Community.
+ Start a Discussion
sonali  vermasonali verma 

How to execute Multiple batch apex jobs in sequence

Hi
I want to understand a specific concept.
I have created​​ 2 batch apex classes say account and contact and now I need to run contact class only after account class fully completes(without any errors of course).

I schedule account class & then contact class, so when I click setup --> jobs --> scheduled jobs,   I can see both these jobs

As per documentation order of execution of jobs is not guaranteed.

so how do I make sure that contact batch job executes only successful execution of account batch job.

please be specific in replies.

sonali​​​
Best Answer chosen by sonali verma
BroncoBoyBroncoBoy
Your code looks correct, I understand the changes you made. 

The Database.executeBatch method breaks the entire job into chunks of 200 records. Each of these chunks of 200 records is passed to your 'execute' method, this represents a single transaction.  For each transaction you can rollback changes if needed.  Rolling back the entire batch if one transaction fails is not possible as far as I know. For example, if the first transaction of 200 records succeeds but the second transaction fails, the database updates made in the first transaction are not rolled back and are committed to the database. (Taken from this article: http://salesforce.stackexchange.com/questions/8122/rollback-whole-batch-on-error )

Here's a great article on chaining batch jobs.  The article outlines the some necessary considerations for writing and chaining batch apex jobs:
http://andyinthecloud.com/2012/10/14/winter13-chaining-batch-jobs-with-great-power/

The way to rollback a single trasaction example is here:

Example:
global void execute(Database.BatchableContext BC, List < sObject > scope)
{
    List < Account > acctsToUpdate = new List < Account > ();
    //add processed records to a list to update 200 records per transaction
    for (sObject s: scope)
    {
        //loop through and process records code here
    }
    //Here's where the DML update occurs, and will roll back the single transaction of 200 records if one record fails
    if (acctsToUpdate.size() > 0 && acctsToUpdate != null)
    {
        // Set the flag in the update call to false to allow all the possible
        // records to update regardless of errors. We will store any records
        // with errors for reprocessing
        Integer zcmCounter = 0;
        Database.SaveResult[] lsr = Database.update(acctsToUpdate, true); //setting true here specifies whether the operation allows partial success.
        //If you specify true for this parameter and a record fails, the remainder of the DML operation will fail to update. 

        for (Database.SaveResult sr: lsr)
        {
            if (sr.isSuccess())
            {
                //logic here
            }
            else if (!sr.isSuccess())
            {
                //this will log the failed records if you need it
                errorRecords = errorRecords + zcmToUpdate.get(zcmCounter).Id;
            }
        }
    }
}

Does this answer all of your questions?

Thanks,
Bronco

All Answers

BroncoBoyBroncoBoy
You can implement a concept called batch chaining.

A batch consists of a start method (executed once), and execute method (executed multiple times), and a finish method (executed once).  Since the execute method only executes once AND happens after the batches are processed within the execute method, you can simply call the contact batch within the finish method.

Example:

global database.querylocator start(Database.BatchableContext BC)
{
  //start method logic here
}

global void execute(Database.BatchableContext BC, List<sObject> scope)
 {
    //start method logic here
}

global void finish(Database.BatchableContext BC)
 {
       //call next batch
       ContactBatch myContactBatch = new ContactBatch();
      Id batchProcessId = Database.executeBatch(myContactBatch);
     //finish method logic here
}

Does that answer your question?

FYI there are caveats to this approach, see here:  http://salesforce.stackexchange.com/questions/8109/cascading-batch-jobs
BroncoBoyBroncoBoy

Sorry I meant to say:  "Since the finish method only executes once AND happens after the batches are processed within the execute method, you can simply call the contact batch within the finish method."
sonali  vermasonali verma
Hi
OK but how to know the FINISH method of first batch job is fully complete, likewise if there is a failure of first batch then how do we recover from first batch, then how does second batch start.

Request you to shed light on this cause I am not clear on this part.

sonali​
sonali  vermasonali verma
Hi Bronco
also let me know how do we test the second batch job​
BroncoBoyBroncoBoy
Sonali are you processing some additional records using the FINISH method of your first batch job?
Can you post the code from the FINISH method of your first batch job?  I thnk it will help me answer your question.
There is code I can provide that will show if any records in your batch failed and you can use that to determine whether to call the next batch or not.

For example:
global void finish(Database.BatchableContext BC) 
{    
AsyncApexJob a = [SELECT Id, Status, NumberOfErrors, JobItemsProcessed,  TotalJobItems  FROM AsyncApexJob  WHERE Id = : BC.getJobId()];
    //If NumberOfErrors is "0" - if the execute method processed all records without errors - then call next contact batch.
    if (a.NumberOfErrors == 0)
    {     
        ContactBatch myContactBatch = new ContactBatch();   
        Id batchProcessId = Database.executeBatch(myContactBatch);    //finish method logic here
    }
    else
    {
        //process other code
    }
}

Perhaps it would be best to provide the code in your EXECUTE method as well.
-Bronco
sonali  vermasonali verma
Hi Bronco
I am still working on the code and my manager told me that I need to handle the above situation.
I will send the first batch once done.

I made a small modification to the code u sent. let me know if its good enough
I am also checking if count of batch jobs > 5. if its true then I will wait for some time else I will process the next batch class.

AsyncApexJob a = [SELECT Id, Status, NumberOfErrors, JobItemsProcessed,  TotalJobItems  FROM AsyncApexJob  WHERE Id = : BC.getJobId()];

Integer jobs = [Select count() From AsyncApexJob Where JobType = 'BatchApex' and ( Status = 'Queued' or Status = 'Processing' or Status = 'Preparing' )];   

    if ((a.NumberOfErrors == 0) && (jobs < 5))
    {

         //create new instance of second batch
   }
  else
   {
         //wait for 1 hour
  }



secondly , how do we test the second batch class. Is it the same way we test for first batch or is there a difference any approach.

thanks
sonali​​
​​​​​                 
sonali  vermasonali verma
Hi Bronco
In addition to​ my previous post, I want to know one more thing.

In batch , every t​ransaction is discrete.
i.e values of all instance variables and governor limits are reset for every transaction.
Now what if there's a partial DML operation occurred in batch one class?

Is there a way I can rollback, how do I check for partial DML updates and how to handle this scenario.

I request you to please shed light on this.

sonali​​
​​
BroncoBoyBroncoBoy
Your code looks correct, I understand the changes you made. 

The Database.executeBatch method breaks the entire job into chunks of 200 records. Each of these chunks of 200 records is passed to your 'execute' method, this represents a single transaction.  For each transaction you can rollback changes if needed.  Rolling back the entire batch if one transaction fails is not possible as far as I know. For example, if the first transaction of 200 records succeeds but the second transaction fails, the database updates made in the first transaction are not rolled back and are committed to the database. (Taken from this article: http://salesforce.stackexchange.com/questions/8122/rollback-whole-batch-on-error )

Here's a great article on chaining batch jobs.  The article outlines the some necessary considerations for writing and chaining batch apex jobs:
http://andyinthecloud.com/2012/10/14/winter13-chaining-batch-jobs-with-great-power/

The way to rollback a single trasaction example is here:

Example:
global void execute(Database.BatchableContext BC, List < sObject > scope)
{
    List < Account > acctsToUpdate = new List < Account > ();
    //add processed records to a list to update 200 records per transaction
    for (sObject s: scope)
    {
        //loop through and process records code here
    }
    //Here's where the DML update occurs, and will roll back the single transaction of 200 records if one record fails
    if (acctsToUpdate.size() > 0 && acctsToUpdate != null)
    {
        // Set the flag in the update call to false to allow all the possible
        // records to update regardless of errors. We will store any records
        // with errors for reprocessing
        Integer zcmCounter = 0;
        Database.SaveResult[] lsr = Database.update(acctsToUpdate, true); //setting true here specifies whether the operation allows partial success.
        //If you specify true for this parameter and a record fails, the remainder of the DML operation will fail to update. 

        for (Database.SaveResult sr: lsr)
        {
            if (sr.isSuccess())
            {
                //logic here
            }
            else if (!sr.isSuccess())
            {
                //this will log the failed records if you need it
                errorRecords = errorRecords + zcmToUpdate.get(zcmCounter).Id;
            }
        }
    }
}

Does this answer all of your questions?

Thanks,
Bronco
This was selected as the best answer
sonali  vermasonali verma
Hi Bronco.
I understood the above approach​, but I have 2 clarifications
a) Assume  Database.SaveResult[] = Database.update(acctsToUpdate, false);
    so DML occurs and errors can be traced back from debug logs.
    If there are 40 million records in total and say 5 Lakh records are error records.
    Now does it mean I have to restart the entire batch job after fixing the issues ?
    If so, what about the records already comitted? there can be duplicate records , how this situation can be handled.​

b) why​ Database.SaveResult[]  doesn't work with Database.Upsert?

sonali verma​
    ​
BroncoBoyBroncoBoy
a)  If you specifcy false parameter in Database.SaveResult[] sr = Database.update(acctsToUpdate, false),  and 5 records fail, the records that don't have an error will succeed in being committed to the database - only the 5 Lakn records will fail to be committed.  In that scenario 39,999,995 records will succeed in being committed to the database.  Restarting the entire batch job after fixing the issue could be on possible solution to correcting the 5 records that weren't processed - thoough it seems excessive to re-do all of the records.  Perhaps you can specify the records that need reprocessing.  Regarding the records already committed - the duplicate records - there aren't any native batch methods or architectures that I've seen to reprocess those records, identify & remove or merge duplicates.  You would have to code that in a separate custom class/batch process.

In my opinon, batch apex isn't an ideal solution to process 40 million, then records find the records that failed, reprocess them, then remove duplicates.  Perhaps someone has found a way to do that; it may be possible...I just haven't seen it.  At my company, that type of task would be delegated to an external transactional web service written in Java or another server side language.  That's just my opinion.  If you're committed to exploring only batch apex as the solution to all of this, there are classes such as the DuplicateResult Class that may be of use: https://developer.salesforce.com/docs/atlas.en-us.apexcode.meta/apexcode/apex_class_Datacloud_DuplicateResult.htm#apex_class_Datacloud_DuplicateResult. You could explore that as an option.

Keep in mind, this is just my opinion.  Others may know of a batch apex solution/architecture that would work, I do not.  I would recommend posting to the developer board your scenario:  processing 40 million records, then find the records that failed, reprocess the failures, then remove duplicates.  

b)  Try Database.UpsertResult[] ur = Database.upsert(acctsToUpdate, false);

Thanks,
-Bronco