Startup update 18: Pivoting out of engineering workstreams

2025-09-11

Progress update

This week is late because I spent a solid week on my consulting gig, and only got a day or two of work on my own app.

The consulting gig has been pretty informative in a lot of ways. On the technical side - I proved that I could dive into a completely unknown codebase, getting my dev loop up and running, building out an LLM backend + frontend chat module in a week. On the PM side - scoping out the feature list, estimating delivery times, and actually being dead-on with my estimate was pretty gratifying. Although… “overpromise, precisely deliver” isn’t how that saying usually goes. On the stakeholder side - seeing what sort of auxiliary features nontechnical folks found valuable was eye-opening. As a one-man shop with the ability to push new deploys within seconds, or to vibecode custom debug UIs in an hour, I hadn’t previously appreciated LangSmith/LangFuse’s value proposition. It very quickly becomes obvious when nontechnical folks started to ask, “hey, can I play around with the prompt?” or “how can we peek at users’ conversations to see how well this is working?”.

I also learned about and integrated some of the tracking JS tools my client used. LogRocket is mindblowingly powerful (and extremely invasive, let’s be honest - you can see everything a user does on your website). I intend to shoulder-surf my first hundred users and get the broad strokes of my app correctly implemented and then turn down tracking to a statistical sample. Strangely, I also found that my filter list for uBlock Origin didn’t actually prevent this user tracking. Perhaps LogRocket has the resources to win the cat and mouse game with filter list maintainers.

I am starting to see a trickle of traffic coming in through the USNCO guide I wrote last week, so in between that traffic flow and the user behavior tracking, I hope to get some stronger signal. I’m belatedly realizing that setting up this user feedback loop is kind of like setting up your devloop. There is only so much code you can write in isolation before you have to actually run the thing and see whether it works, and be able to iterate on changes to your code; similarly there is only so much product you can create in isolation before you have to actually ask users whether the thing is useful!

On the feature side, I uploaded another 10 years’ worth of USNCO national exams, fixed broken mobile CSS, and made various other small tweaks to the UI. Next up is Stripe integration and a “review the solution” feature that kicks you back into an AI chat. The AI features are the only semi-paywalled features, so driving additional demand there will hopefully get some subset of users to start signing up.

I’m also starting to think about new marketing/sales channels. Inbound marketing through a blog is one way, but I should be reaching out to teachers, students on Discord, etc. etc. I knew at an intellectual level that distribution was key, but I feel a bit silly that it’s taken this long to internalize this idea.