Skip to main content

1 - Data Migrations Checklist

The following is a semi-profanity-ridden attempt at explaining one way to do data migrations while following best practices. It is rather long and laced with colorful language. If you have read it already, or if you want to avoid the profanity, you can consult the following checklist in the beautiful table below.

Note that all elements are considered mandatory.

As a quick note, and a reminder even if you've read the whole version:

DO NOT MODIFY SOURCE DATA FILES, EVER.

If you're doing data migrations, either use a script to modify the source files and save the edited version, or use excel workbooks that open the source file and then save the edited result elsewhere. Yes, even if the source is an excel file.

Why? Because sources change. People forget stuff, files aren't well formatted, shit gets broken, and people are human - meaning that one-time data import is actually going to be done multiple times. Edit the source file, and get to do everything all over again. Use scripts or workbooks to do the transformations ? Point that to the new source file and BAM Bob's your uncle.

Scripts you might want to use:

Or, if you prefer excel, open a blank workbook, Import the source file via the "data" ribbon tab, select "from text/csv" (or whatever matches based on your source type), then save it as both:

  • the construction excel,
  • a NEW csv file after doing your changes in formula columns.

That way when you change the source file you can just open the construction book again and resave.

Action Completed?
DO YOU HAVE A BACKUP  
Is it UTF-8 encoded  
Did you check it is readable and well formatted  
Does it have carriage returns stored as carriage returns, not as spaces  
Is it up to date  
Do you have a mapping for every object and field  
Did you determine an ExternalID for each object  
Did you determine source of truth (whether to overwrite or not) for reach field  
Did the client sign off on the mapping  
Do you have the source data  
Is it in a format your tool can read  
Are dates and date-times well formatted (yyyy-mm-dd || yyyy-mm-ddT00:00:00z) and are times exported in UTC  
Are field lengths respected (emails not longer than 80 chars, Names not longer than 40, etc)  
Do numbers have the right separators  
Do all tables have the required data for loading (Account Name, Contact Last Name, etc etc etc)  
Do all fields that have special characters or carriage returns have leading and trailing qualifiers (")  
Do all records have an external Id  
Did you do a dummy load with only one field mapped to make your sure tool can read the entire file  
Are you doing transformations  
Did you document them all  
Did you automate them so you can run them again with a click  
Did you read the LDV guide if you are loading more than 1M records  
Did you activate validation rules bypass  
Did you check all automations to deactivate any that should be, including email alerts  
Did you warn the client about when you would do the data load  
Did you warn the client about how long the data load would take  
--------- run the migration -----------  
Did you reactivate all automations  
Did you remove validation rule bypass  
Did you tell the client you were done and they could check  
Did you check the quality of the data  
No Comments
Back to top