Apex Code Development You need to sign in to do that
Don't have an account?
kay dev Best approach for a callout and processing thousands of records on a schedule
I am looking for the best design to make a daily scheduled callout to process 10s of thousands of records. My plan is to create a schedulable apex class that runs a batch apex. In the start of the batch apex I make the callout and I get back the records from the API. In the execute method, I go through the records and upsert the records into Salesforce. Is this a sound approach? My only concern is that the call out to the api and the processing of the batch (start and execute methods) run in the same context so if the callout takes 3-4 seconds there is the risk of this becoming a long running transaction. My question is first am i correct in having this concern and if yes, how can I make the callout its own async transaction and then once the callout response is recieved then to process the records through batch apex.
I'm not sure about the feasibility of returning tens of thousands of records from the start method -- it may work and I'd love to hear if you find no issues there.
If it *is* a problem, is there any way of chunking the API callout results differently? If you could get a list of the number of results ahead of time, create dummy SObjects for each "chunk" of say, 100-200 records in the start method. Then each execute would callout to request those 100-200 records and process them.