function readOnly(count){ }
Starting November 20, the site will be set to read-only. On December 4, 2023,
forum discussions will move to the Trailblazer Community.
+ Start a Discussion
Marco RispoliMarco Rispoli 

access to large amounts of external data

Hello! I need to allow our customer service reps to view data related to our customers when they pull up their account. The issue is that the data related to our customers is external to SF (customer search data, related marketing data, etc.). This data is large (we are talking about 200-300 GB worth of data in total, 100-200 million records ... potentially growing in the future).

So moving this data (which is read-only as far as the customer service rep is concerned) in SF to me seems impractical and expensive (SF storage is not cheap). 

We already have all this data in the desired format, staged outside of SF (in various on-prem Oracle DBs). 

What would be the best way to approach this? What have other people done so far? 
Hook up the Account page to a VF page that calls a service (Java) that queries the data?
Hook up the Account external objects to oData connecting directly to the database? 
Any preference or difference over databases on-prem or cloud and if cloud ... did anyone have any good results with a specific cloud data storage configuration? (in terms of performance/respons times, etc.).

PS: can't use Heroku because the Heroku DBs are not SOX compliant right now. That's what we would have done but ... we can't. Didn't pass SOX validation. 

At NOWTV we implemented external objects for this pattern. You state this is for customer service agents, so I assume you only access a small subset of customers at a given day - if so, external object is the right pattern. 
You will need the following:
- create a WEB facing api fronting the oracle database. Ideally that should be implemented using OData as it would make the SFDC solution code free. You will need to implement a secuirty solution to insure that API is secure. Obvisouly use basic auth to log in and whitelist SFDC's IP ranges for it, make it available over HTTPS, but your security team may require more
- create external objects to consume the api
NOW - you can use the external objects as if they are custom objects in SFDC - create related lists, relationships, implement access control etc

Most attention is needed for the apis - what will be the key? will they work account by acocunt or can you run bulk queries? wil they support free text search? will they support OData?

depending on answers to the quesiotns above, you may require some custom code on salesforce (e.g. if the api wont support Odata you'll need a custom connector, if access is account by account you may require a VF page if you wish to display multple records at the same time)