function readOnly(count){ }
Starting November 20, the site will be set to read-only. On December 4, 2023,
forum discussions will move to the Trailblazer Community.
+ Start a Discussion
Jenny Chen 9Jenny Chen 9 

small to mid size data warehouse recommendation needed (to integrate with Salesforce.com)

Hello, 
we have a lot of nightly batch jobs. However, some jobs hit limits of 50,000, but with our growing org and data, it is hard to find the daily change and process the changed data and still within the limits. 
I am thinking about data warehouse solutions. 
Do you have any good small to mid size data warehouse recommendation? Really appreciate your help. 
 
Jason Lim 13Jason Lim 13
Hi Jenny,

Middleware ( informatica, boomi, etc ) can alleviate the heavy processing in SFDC, however they don't come cheap, do you have budget for one ? 

Or you can consider building a scripting solution with bulk api ( https://developer.salesforce.com/page/Loading_Large_Data_Sets_with_the_Force.com_Bulk_API ),
with open source db like MySql , MariaDB, Postgres, and use the database functions/ procedures for nightly processing.