function readOnly(count){ }
Sign Up ›
Starting November 20, the site will be set to read-only. On December 4, 2023,
forum discussions will move to the Trailblazer Community.
Learn More

Browse by Topic
ShowAll Questionssorted byDate Posted
+ Start a Discussion
OssieOssie 

Best Practice for Accessing Large volumes of Data from External Systems

Hi,

How can i access large volumes of data without the need to batch upload this info to Salesforce?
For example i want to look up a Marketing Code for an Item we sell and so rather uploadng 1million+ item records with there marketing code in SF.com i would like to search an external source and retrieve the info.  Is this only possible through Web services etc? 
KevinPKevinP
Ossie:

I have two ideas / comments for you.
  1. Batch uploads can be easily automated with tools like Jitterbit or Mulesoft. Once the initial upload is done, this is a completely viable solution for many millions of records. I have worked in orgs with a custom object containing 28million rows of data that was hourly updated. First upload took 2 days, but after that the hourly syncs were generally done in 90 seconds or less.
  2. Salesforce Lightning Connect allows you to connect to an oData data source. If your external data is contained in a SqlServer, Oracle database or Postgresql db you can directly connect to your existing data store and reference it's data (read only) in Salesforce.
James LoghryJames Loghry
Have a look into Salesforce's new feature "Lightning Connect".  With Lightning Connect, you can bring in external data that uses the oData protocol.  It's an additional licensing cost, but if you start up a developer org, you can mess around with it for free and try it out, to see if it will work with your system.

If your system doesn't support the oData protocol, there are also 3rd party middleware solutions that will transform your data into oData.  At least some are working on it (not sure if anything has been released yet).  

That being said, another option is a hosted middleware solution.  So you could use a middleware to transform your external system data into the marketing info you need, and the necessary data into Salesforce (like updating the Marketing Code fields for instance). 

A third solution is running a webservice where you can grab the marketing data you need.

There are more, but I would look into these solutions first.
OssieOssie
Thank You Both. Very much appreciated.
Sumit Sarkar 2Sumit Sarkar 2
Where is your marketing data stored?  I just finished a similar POC with invoice data and published my write up on the dev community (https://developer.salesforce.com/page/Building_a_Data_Integration_Proof_of_Concept_Using_Lightning_Components):