function readOnly(count){ }
Sign Up ›
Starting November 20, the site will be set to read-only. On December 4, 2023,
forum discussions will move to the Trailblazer Community.
Learn More

Browse by Topic
ShowAll Questionssorted byDate Posted
+ Start a Discussion
ManidManid 

confusion regarding data loader

i saw in documentation maximum batch size is 200. if we enabled bulk Api its is upto 10000 .
But how we can import more than 50000 - 5M records if it exceed the size? please anyone explain the overall dataloader process
Best Answer chosen by Manid
jigarshahjigarshah
Manid,

I am not sure what size you would be referring to over here.

Just to clarify a couple of things, Data Loader is capable of performing Data Manipulation operations such as Insert, Update, Delete upto 5 million records.
  • As far as the batch size goes, Data Loader segregates the records to be manipulated into intermediate chunks which it terms as batches. Batch size is the processing size and not the overall size of the records to be manipulated using Data Loader. The maximum batch size can be set to 200 however the preferred optimal value is between 50 to 100. For e.g. if you are uploading 50,000 records then the Data Loader processes these as 250 Batches each containing 200 records if the batch size is set to 200. If the batch size is set to 100 then in that case, Data Loader will process these 50K records as 500 Batches containing 100 records each.
 
  • Additionally, if you set the Use Bulk API setting to TRUE, the processing Batch Size can be extended to at max of 10,000 records i.e.each batch can contain a max of 10,000 records to be processed versus the original 200. Bulk API is a standard SOAP API provided by Salesforce to perform manipulations for heavy data volumes. Data Loader performs calls using the Bulk API when the Use Bulk API checkbox is checked in the Data Loader Settings.
 
  • Generally, the Data Loader processes these batches in a sequential mode i.e. one after another. However, when the Bulk API is used, by default, the batches for data manipulations are processed in parallel i.e. simultaneously. This provides the scale of performance with the heavy volume of records along with low upload times. However, you can explicitly enforce processing in serial mode i.e. one after another even with Bulk API by checking the Enable serial mode for Bulk API in Data Loader Settings.
I believe this should clarify your confusion.

All Answers

jigarshahjigarshah
Manid,

I am not sure what size you would be referring to over here.

Just to clarify a couple of things, Data Loader is capable of performing Data Manipulation operations such as Insert, Update, Delete upto 5 million records.
  • As far as the batch size goes, Data Loader segregates the records to be manipulated into intermediate chunks which it terms as batches. Batch size is the processing size and not the overall size of the records to be manipulated using Data Loader. The maximum batch size can be set to 200 however the preferred optimal value is between 50 to 100. For e.g. if you are uploading 50,000 records then the Data Loader processes these as 250 Batches each containing 200 records if the batch size is set to 200. If the batch size is set to 100 then in that case, Data Loader will process these 50K records as 500 Batches containing 100 records each.
 
  • Additionally, if you set the Use Bulk API setting to TRUE, the processing Batch Size can be extended to at max of 10,000 records i.e.each batch can contain a max of 10,000 records to be processed versus the original 200. Bulk API is a standard SOAP API provided by Salesforce to perform manipulations for heavy data volumes. Data Loader performs calls using the Bulk API when the Use Bulk API checkbox is checked in the Data Loader Settings.
 
  • Generally, the Data Loader processes these batches in a sequential mode i.e. one after another. However, when the Bulk API is used, by default, the batches for data manipulations are processed in parallel i.e. simultaneously. This provides the scale of performance with the heavy volume of records along with low upload times. However, you can explicitly enforce processing in serial mode i.e. one after another even with Bulk API by checking the Enable serial mode for Bulk API in Data Loader Settings.
I believe this should clarify your confusion.
This was selected as the best answer
jigarshahjigarshah
Manid,

Does the abvo explanation clarify your doubt? Do you stil need more clarification?
ManidManid
Thanks jigarshah .And i want know some more information regarding dataloader. We have 3rd party resource in there we have accounts and contacts . I want to insert contacts corresponding to their account (i dont have account ids) . It is possible to insert w.r.t accounts with out account ids? or there is any other way to do that?
 
jigarshahjigarshah
Manid,

There a couple of ways to accomplish this.
  • If the number of records to be uploaded < 50,000 then you can leverage the standard Data Import Wizard offered by Salesforce. You will be able to access this by logging into Salesforce through the web browser and navigating Setup > Data Import Wizard. The advantage here is that you can have Accounts and Contacts in a single file with their respective associations and upload them at once.
 
  • In case the number of records to be uploaded > 50,000 then you will need to use Data Loader. Use the following approach to go about uploading Accounts and Contacts within your Salesforce instance in the absence of Salesforce Record Ids
    • Create a custom key field (Number or Text) on Account and Contact objects in Salesforce
    • Associate Account and Contact records in CSV file using this key field.
    • Upload the Accounts CSV file using Data Loader and once these records are inserted, you will recieve the Record Ids within the results which you can download.
    • Use VLOOKUP() on the key field within CSV, to find and associate the appropriate Account's Salesforce Record Id on the respective Contact records and map the Account record Id field to the standard Account field on Contact.
    • Upload the Contacts CSV file within Salesforce.
    • Once complete, run a custom report on Accounts and Contacts and check the associations.
ManidManid
thanks jigarshah
jigarshahjigarshah
Manid,

Please close this thread in case your query has been addressed.
Nikita Sahi 10Nikita Sahi 10

Hi Jigarshah,

The above response is very helpful. Thank you for posting the detailed information. 
i have query for the same. 
when performing delete operation on multiple records i.e 400, the data loader gives an error-- hed.TDTM_Affiliation: execution of AfterDelete caused by: System.QueryException: unexpected token: ',' (hed)

here TDTM indicates Table driven trigger management which is a TDTM is a framework for managing that code. Actually it's a tool to manage your code in Salesforce and control how Apex behaves. it is used in salesforce eda (Education Cloud)  https://powerofus.force.com/s/article/EDA-TDTM-Overview

This error is only reflected when the delete operation is on a file of 400 records and when the file has only 21 records the delete operation is successful. 
The user said that she changed the BATCH SIZE of data loader settings to 1 and all the 400 records were deleted. 

i am not sure what can be the reason of this error? 
Can you please help me to find the solution.  

Regards
Nikita