A Virtually Unnoticed Migration
Originally posted 2020-10-10
Obligatory disclaimer: all opinions are mine and not of my employer
This is the story of how I pulled off an ambitious migration so cleanly that virtually no one noticed it had happened.
Backstory: My first job
I started my first job in 2014, not long after I dropped out of graduate school. At the time, my coding experience consisted of building a website to coordinate college Go tournaments, using Django. My job interview for HubSpot consisted of one interview (the second interviewer didn’t show up), and a chat with the engineering director who offered me 80k on the spot. I told him that I’d had another offer at 95k, he bumped my offer to 90k, and we shook hands. I don’t even think it was noon by the time I walked out of the building. It still blows my mind how quickly my first job hunt went compared to my second job hunt, which lasted three months.
HubSpot’s backend was in a state of transition when I joined. They had an abandoned C#/MSSQL system and an actively developed Python system, with all new customers being started on the Python system. My manager had written the SQL to dump the MSSQL database contents and was prototyping a data migration tool. My job was to continue picking through this carcass and make the right API calls to reincarnate customers’ data in the Python system. An outsourced team would fix up the tool’s output, and another team in the U.S. would Q/A the customer’s new website before activating it. After spending a month onboarding me, my manager handed development of the tool over to me and moved onto onboarding the next new hire.
9 months into my first job, I decided I would kill the legacy Forms system.
At this point, I should mention that HubSpot’s business was selling Inbound Marketing. In short, Inbound Marketing meant: write useful content, get organic hits from Google search results, collect email addresses via a form, and then court customers by email. HubSpot sold the website builder, blogging platform, contact database, and email platform to execute this vision.
That Forms system I was going to migrate? It collected email addresses and was the linchpin of HubSpot’s business value proposition. If I messed up, there would be a lot of pissed off customers. I don’t think I quite appreciated this at the time, and if I had to do it again, I’d have more precautions in place.
I talked to several eng teams to figure out how the Forms and Contacts APIs worked, and dug into the raw data. The legacy form system stored forms as a raw HTML
<form> element, and relied on a minified JS blob (for which I couldn’t find the source) to make an API request to an older version of the Forms API. My manager’s migration prototype copied the raw HTML blob as a custom HTML widget, and linked to the legacy JS blob in the page header. The form continued to function post-migration, but customers had no way to edit their form and the CSS was unmaintained, so all of the fields and buttons looked out of date. On the other hand, the newest Forms API offered customization options and integrations with marketing campaigns.
Eventually, a plan came together.
- Use BeautifulSoup to extract form fields and metadata from the HTML.
- Intercept the legacy Forms API call to see which fields were important and how they were submitted.
- Use the extracted fields to register a new Form, deduplicating identical forms.
- Replace the custom HTML widget with a Form widget.
- Work with our Q/A team to catch bugs.
After a few very carefully monitored migrations and iterations with our Q/A team, I’d found and implemented migration rules for all of the obscure form field types. I wrote unit tests to ensure my hairy BeautifulSoup logic didn’t regress (file upload fields were particularly annoying), and manually tested the form submission process with a test website. After a few more clean runs, I flipped a switch to route all customer migrations through this codepath.
After the migration tool ran bug-free for a month, I turned my attention to all the customers that we’d migrated with the HTML copy-paste solution. Despite having been migrated to the new systems, they still had an API dependency on the older systems. Conveniently, the HTML blob was migrated as-is, meaning that I could just point my parser at the HTML blob. This time, I worked directly on the live customer websites, without a Q/A team safety net, and within a week, my script had worked through all of the post-migration customers. Now, only unmigrated customers had an API dependency on the old systems.
For a month or two, nothing happened.
One day, a PM frantically came over to me, asking me what I’d done to everybody’s forms. My stomach dropped and I thought, “ah, fuck, what did I miss?”. I explained the migration. The PM, fully appreciative of the risks, spent the meeting white-faced, instead of congratulating me as I’d been hoping.
It turned out that there was no major problem with the migration. The only reason that the PM had even noticed was that customers were wondering why the form submission rate analytics had stopped and reappeared under a different form with identical naming. Messaging then went out to migrated customers telling them that this was an expected side effect of migrating data to the new systems.
A week later, I got a shoutout at the product team monthly congratulating me for migrating the old forms to the new system.
With diminishing returns to working on the migration tool, I moved onto a new team and a new manager. Around that time, management acknowledged the unsustainability of managing (or not managing, in my case) so many junior engineers, and decreed that it was promo or out. My new manager assumed I was incompetent because I hadn’t actually learned how to communicate technically (Exhibit A: this story). After a few months of failing to micromanage me, he fired me.
In my current estimation, the technical difficulty, cross-team coordination, risk mitigation strategy, and engineering impact of this forms migration was worthy of a junior-> senior engineer promotion. At the time, I wasn’t very good at explaining technical problems and solutions, and ultimately couldn’t claim credit for my work. I don’t think my first manager, oversubscribed as he was, actually understood what I’d pulled off.
I’m glad I was arrogant enough to think that I could pull this migration off without a hitch. If I hadn’t done it, the Forms v1 API dependency would have surfaced unexpectedly when the old servers were shut off in a year. Somebody else who was much less familiar with the migrations process and the legacy code would have had to take a swing at the migration, without the benefit of an experienced Q/A team to help iron out the process. The customers got a much improved forms management experience, and the codebase had one less layer of tech debt.
I spent a few years being embarassed that I’d ever been fired, and ambiguously explained that I’d “left” HubSpot. I didn’t really understand what I’d failed to do or why I couldn’t get along with my new manager. I think I understand a bit better now, having had a few more managers since then. In any case, I’m actually glad I got fired, since that’s what got me started on machine learning. My only regret is that I didn’t get to go to the legacy server shutdown party.
What would I do differently today? I would ask each team I’d talked to to give a guess (in writing) of how hard and how important they thought this migration would be to the company. I’d also document the rough steps I was planning to take, and ask a tech lead or PM whether they thought the steps I’d taken had suitably mitigated the risk. Those steps would have added a day to the project, made the PM a little less terrified, and have gotten me my promotion.