function readOnly(count){ }
Starting November 20, the site will be set to read-only. On December 4, 2023,
forum discussions will move to the Trailblazer Community.
+ Start a Discussion

Batch processing records imported through Data Loader

I'm planning on writing a process using Batch Apex to process new Account records. I see two ways to get data into the batch:

1) Pass the data in via a list (limit 1000 elements)

2) Use a QueryLocator


Also, I see that there's a limit of 5 concurrent batches.


I don't think the QueryLocator option applied in this situation because I can't write a static query that will capture all - and only - the accounts that were just imported. If there's more than 1000 records in the import, I'd have to pass them in using a second call to execute(), which would mean that I'd have another instance of the batch running. If I have over 5,000 records imported, I'd exceed the 5 batches...


It's *possible* to use a formula field or roll-up that I can use to identify the records I need to process, but that's not the cleanest solution. I can't imagine that the batch processing could only handle sets smaller than 5,000 records.


Can someone explain to me what I'm missing?



Can you just embed the logic within an Apex Trigger "on Account before/after insert"? For each batch of inserts that the data loader inserts through the API, the trigger will be invoked. Would that work?
Message Edited by aalbert on 11-02-2009 12:27 PM

Hey, Andrew. I have a trigger doing the work right now, but I was thinking about moving it to the Batch Apex approach because it doesn't need to be done synchronously.


I just noticed, however, that the 1,000 element limit on List size scales with the size of the trigger batch, so the list limit might be as high as 5,000.


Another use case for this logic is that I'll need to batch process all existing accounts (one time) when this app is installed -- and there could be a lot of existing records.


I suppose I could move the logic that's currently in the trigger to another class (non-batchable) then call that method from both the trigger and the batch apex class...


That makes sense - to offload the logic into an asynchronous process.


In your batch apex, can you specify a SOQL query with the TODAY syntax to identify newly created records:

Select Id, Name from Account where CreatedDate = TODAY()


The key here is to make sure it runs the same day the batch ran.