Data Migration Tool
Building a UI for Shopify data migrations, because the source data is always different but the destination never changes.
Plenty of Shopify migrations can be handled simply with Matrixify. Shout out to those guys I don't think I have a single Shopify developer in my network who doesn't treat it as the first port of call for data migration.
But its platform support isn't 100% comprehensive, and there are a lot of niche and bespoke platforms in the wilderness of the web. In many cases you won't get a full data export out of the source platform at all, and if you do, it won't be in a format Shopify accepts.
I've dealt with this many times: endless SQL queries and CSVs, trying to jam absolute soup into a complete Shopify import. It's not getting easier either as Shopify's features expand, the data requirements for a successful migration get more complex with them.
What if the transformations lived in an app instead of a folder full of spreadsheets?
The Problem
The real risk isn't the initial import. It's what happens when something goes wrong.
Picture being two weeks out from launch. You spot a giant hole in the data during UAT — a feature that wasn't in scope and that your dataset doesn't account for. Now the last thing you want is a million VLOOKUPs, tracing back through client_data_1.csv, client_data_2.csv, client_data_3.csv.
Because no two datasets are ever the same, it's easy to take every migration project by project and bury yourself in CSVs. The problem that creates is traceability: when something breaks, where did it break, and how many transformations did that one change affect?
That's not just a developer inconvenience. From a client service perspective we're responsible for making sure a migration doesn't damage the performance of the live site, and stock, pricing, variants and product attributes are all vital to supporting the customer and the conversion metrics that follow.
The Idea
The source data may be messy, unstructured and incomplete. The destination never changes.
We know exactly how Shopify data is structured and what it will and won't accept and anything that doesn't fit one of those has to become a custom metafield or a metaobject, depending on what you're looking at.
Even then it's knowable. The import expects metafields and metaobjects in defined structures, and you can use the Shopify API to detect which variations are acceptable rather than guessing.
So the messy half is the only half that varies. That's a strong argument for building the tool around the destination and treating each source dataset as configuration.
What I Built
The first pass was a node script for a paricular client with a specific source set. Too narrow for general use, but it proved the concept: a set of rules could be applied to a dataset to transform it into a Shopify-acceptable format.
Next came a UI tool. You upload a source dataset, then use a set of simple rules to map that data into a Shopify-acceptable format.
For primary product fields — title, description, stock — that mapping is incredibly straightforward. Variants, pricing, images and custom metafields need considerably more work, so rather than trying to build one universal mapper I built different modes to handle different types of incoming dataset. For example, variants in rows, variants expanded from columns, and variants in a single column with a delimiter. Each mode has its own mapping rules and UI.
The tool handles these per line in the UI which reflects the columns in the source dataset, and the rules are stored so that they can be reused between runs. The output is a Shopify-acceptable CSV that can be imported directly into the store.
Then we can use multiple general purpose formatting cleanup rules on each column depending on what they need.
need to strip whitespace, remove HTML, convert to title case, etc. The rules are stored so that they can be reused between runs.
We also implemented a 'configuration' feature that allows the user to save a set of rules for a particular dataset type, so that they can be reused in future migrations. This is particularly useful for clients who have multiple stores or who need to perform regular data migrations. It also means that a developer can download the configuration and pass the source data and the config to another developer to check or replicate the migration.
Really useful future runs.
How It Works
- Get your source data into a CSV. Either from a platform export or a SQL query from the db.
- Upload the CSV to the tool.
- Map the source columns to the Shopify destination columns using the UI.
- Apply any necessary formatting & cleanup rules to the columns.
- Save the configuration for future use.
We have added support for every metafield and metaobject type in Shopify, so that any data can be mapped to the correct destination. The tool will also validate the data before generating the final CSV, so that you can be sure that it will import correctly into Shopify. If you have a limitation on your matrixify plan, the tool will also break the output down into batches to fit the current plan level.
What It Changed
The tool has been used on multiple migrations now, and has saved countless hours of work. It has also reduced the risk of errors in the migration process, as the rules can be tested and validated before the final import. The tool has also been used to create a set of standard configurations for common source datasets, which can be shared with other developers to make future migrations even easier.
It has unified the process across the team, so every developer migrates the same way, so no one needs a stash of csv functions in their notes anymore.
- Faster More Accurate Migrations
- Reduced Risk of Errors
- Reusable Configurations for Future Migrations