Rapid Integration recently completed a six month project involving the migration of over twenty million records between two Salesforce.com organizations. The project used a combination of the Salesforce Bulk API and a Microsoft SQL Server database for staging. Using a Cast Iron horizontal scaling design pattern, we extracted the entire data set from Salesforce in 90 minutes. To make this all work, we learned a few new Salesforce API techniques, and we made some Cast Iron tweaks to improve the SOQL query performance.

A recent Cast Iron project involved taking tab-separated files and producing comma-separated files as a result. Seems easy enough? Of course... except three of the five use cases required merging multiple input files before producing the comma-separated output. Since merging files in memory is not a Cast Iron best practice, Rapid Integration used a MySQL database to stage the data. The database query activity allowed us to use SQL to aggregate, sort and dedupe the data so that Cast Iron could do what it does best.