Building an SEO Migration Tool

After a successful migration to Shopify, the last thing you want is for the numbers to go backwards because of SEO health.

But building a list of 301 redirects is one of the more thankless jobs in a migration, and two things make it worse than it should be. A Shopify build in development sits behind a password page, so getting a clean list of URLs out of a crawling tool is fiddly. And the data migration is still moving underneath you delta updates, re-imports and corrections all cause the URL set to fluctuate.

Meanwhile developers are head-down in data transformation, where 301s and 404 handling can feel like a soft skill next to the real work.

How do we make redirect mapping a development problem rather than a spreadsheet problem?

---

The Problem

Missed, dropped or heavily altered collections and products are enough for Google to tank a site's health and traffic. At that point you're relying on conversion rate to save you, which is not ideal for the client.

Migration tools help. Matrixify can plug the gaps and mitigate most of this on a lot of projects. But lately I'm seeing more migrations it can't cover, because the product catalogue is too large to handle without the enterprise tier and at that size you're hitting Shopify's variant limit anyway.

Even still its a cumbersome process, import data, export it in the shopify format, vlookup the old URLs, switch it to a redirect sheet & upload again. Plus you cant really manage this in batches as then you introduce the issue of 'check the 301's iv already done'

This means that some 301's cant be done or checked until post lauch, which is exactly what we dont want.

---

What I Built

A CLI that runs locally and produces two CSVs — one per platform.

# the live site
vs-cli seo-checks <live-domain> --output_name.csv

# the Shopify development store
vs-cli seo-checks <shopify-domain> --output_name.csv <store_password>

The first pass hits the live domain over node's https and finds the sitemap by trying the usual sitemap path combinations. I later widened that into a fuzzy search, because plenty of platforms don't put their sitemaps where you'd expect, and what you actually want is the full list of them.

Then the same process runs against the Shopify store, with one extra step at the front: submit the store password to obtain a Shopify cookie. Once you've got the cookie, the sitemaps parse exactly like the live site's.

Deliberately, there's no crawling. At this stage every URL in the sitemaps is all you need, and crawling each one would be slow for no additional information. at this stage, we are only concerned with urls in the site map because all the data the devs migrate, products, collections, pages, blogs, blog posts. Should all exist in the sitemap anyway.

Again like most of my tools, this got refactored into a UI with a DB attached to it so the SEO team can create projects and work on the redirects without needing to run the CLI themselves. The UI also allows for a more visual representation of the matches and the ability to manually adjust them if needed.

---

How It Works

Comparing paths, not URLs

Both CSVs strip the domain live & Shopify, so the comparison only ever looks at URL paths. That's not just tidier: the 301 redirects will be applied in the context of the live store, where the live domain is already the root.

Which means this stops being an SEO problem and becomes string matching, something developers are considerably more comfortable with.

Why the first pass was so bad

The naive version segmented URLs by / and ran regexes to find matches. The results were ludicrously poor:

• Far too many mismatches.
• Slight URL differences defeating otherwise obvious matches.
• End paths matching beginning paths.
• URLs sharing a path not routing properly — cat-1/cat-2/cat-3/product-url against cat-1/cat-2/prod-url.

Normalising, then scoring

Passing each URL through a series of normalisation functions produces better, less strict matches and sometimes several candidate matches for one source URL.

That's what led to scoring. Each match is rated on how many normalisation functions were needed to reach it, which gives a rough probability that two URLs are the same page. Fewer transformations means a more confident match.

I'll be honest that the scoring is rough rather than perfect. It doesn't need to be precise it needs to be good enough to sort the matches you can trust from the ones a human should look at.

Telling the tool the shape of the URLs

Scoring every URL against every other URL is a lot of redundant work on a site with thousands of them. So instead of trying every combination, you tell the tool what the URLs look like first:

/item/{id}/{slug}.html -> /products/{slug}

The left side is the source URL, the arrow is the flow to the destination, and anything in curly braces is a variable. Repeating {slug} on the destination is how you tell the tool that the ideal match is the URL formatted with whatever that variable captured. There's type support too, so {id:d} constrains a variable to digits.

Bottlenecking the comparison like this means that you can run the tool on a site with thousands of URLs in a few seconds, rather than minutes or hours. As we have a rough idea in the rule set of the shape of the source url and the destination url, so it can chunk them out conditionally i.e only compare product urls to product urls and so on.

Workflow

• Get the Live Domain and the Shopify Development Store URL & password
• Create a project in the migrations app
• Run the job to get the CSVs for both the live and dev store
• check your redirect candidate matches and adjust if needed
• write rules that make matching the urls more accurate and run the job again to get a new set of rules
• Once you are happy with the matches, export the final CSV and upload it to Shopify.

The migration app also supports submitting headers with the crawls / sitemap fetches so that we can pass DNS.

---

What It Changed

• Unified the process of creating 301 redirects into a single tool, rather than a spreadsheet exercise.
• Made it possible to run the tool on a dev store behind a password page, so that the SEO team can work on redirects before launch.
• Made it possible to run the tool on a dev store that is still being migrated, so that the SEO team can work on redirects before launch.