Startup update 10: Curriculum compilation

Originally posted 2025-06-29

Obligatory disclaimer: all opinions are mine and not of my employer

Progress update

I’ve built an LLM-administrable AP Chemistry curriculum that I think is decent quality! (Fun fact: it costs me 50 cents to compile this curriculum using Gemini 2.5 flash. I may iterate using Flash and then run Gemini Pro or Opus for the final version, which I’ll put in some work to manually fix up Y.)

At this point, remaining work before I publish the AP Chemistry course online are:

Building a higher-level hierarchy of Unit/Topic/Conversation into the backend
Building UI to allow students to browse this hierarchy
Building UI for students to start a course (only 1 available at first)
Building a simple quiz/test grading UI, with “upload/take photo from camera” functionality
Building LLM system prompts that can administer the various types of lessons (essential knowledge infodump vs. problem solving). The knowledge infodump will be less socratic, since a number of you felt that the Socratic thing was inferior to ChatGPT for these types of conversations.
Skimming through the course and actually trying some lessons; may need more iterations of refining the curriculum.

I think this should take just a week? Of the features on this list, I expect the camera functionality to take the longest. (cue https://xkcd.com/1425/) I should also note that I’ve canceled my Cursor subscription in favor of Claude Code. It really is just a lot more reliable at generating correctly-executing code on the first pass.

Sufficiently Magic Compilation

I very quickly found the official AP Chemistry Teacher Handbook which is a human-curated outline of a course that is to be implemented/compiled into a set of concrete lesson plans by thousands of AP Chem teachers around the U.S. And so I basically wrote an LLM script to do what those teachers would do… and ran into a number of problems.

The framing for this transformation is basically, X -[C]-> Y, where X = curriculum, C = curriculum compiler, and Y = output course. C includes subroutines like “distill the key concepts to be covered in lesson”, “generate short quiz problems for each lesson”, “generate longer test problems for the whole unit”, and “generate solution/grading key for a problem (and rephrase problem if it turns out that the problem is incorrectly posed)”

When Y comes out low-quality, there are 3 broad approaches you can take to fix it.

edit the output Y, accepting its flaws and saying “at least the LLM did 75% of the work”
edit the compiler C
edit the input X

Editing Y was a no-go because it would have taken a month to manually fix up the output. So I tried to improve C. It was hard to get a compiler C that worked for all inputs X. In particular, some units of AP Chemistry are more number-crunchy (e.g. acid-base pH calculation under various limiting conditions), whereas others are more theoretical/conceptual (thermodynamics), and yet others are just fluffy (basics - atomic structure, isotopes, octet rule). I wanted a sufficiently magic compiler that could take any curriculum and generate a good course out of it.

Eventually I realized this was a fool’s errand, because fundamentally, I have stronger (and dare I say, better?) opinions about what an AP chemistry course should look like than the AP chemistry curriculum writers. A compiler capable of creating a correct output from an incorrect input would be logically incoherent.

Instead, I realized that the right approach was to actually revise the input X according to my specifications, and stop trying to get C to do anything other than faithfully compile the input X to a set of output problems and lesson plans. In the future, I may even further simplify X to the simplest possible sketch of a course and move more of the elaborating work into the compiler, but I think I have a decent healthy balance for now.

As an example of the revisions I had to make:

BEFORE

Topic 1.2: Mass Spectra of Elements

Suggested Skill: Mathematical Routines (5.D) - Identify information presented graphically to solve a problem.

Learning Objective: 1.2.A - Explain the quantitative relationship between the mass spectrum of an element and the masses of the element’s isotopes.

Essential Knowledge:

1.2.A.1 - The mass spectrum of a sample containing a single element can be used to determine the identity of the isotopes of that element and the relative abundance of each isotope in nature.
1.2.A.2 - The average atomic mass of an element can be estimated from the weighted average of the isotopic masses using the mass of each isotope and its relative abundance.

Exclusion Statement: Interpreting mass spectra of samples containing multiple elements or peaks arising from species other than singly charged monatomic ions will not be assessed on the AP Exam.

AFTER

Topic 1.2: Isotopic Variation

Learning Objective: 1.2.A - Understand what isotopes are and where they come from.

Essential Knowledge:

An isotope is a variation in the neutron number of an element, but with identical number of protons.
Too many or too few neutrons are both unstable. Most elements have only a few stable isotopes.
Variations in average nucleon mass away from exactly 1 amu are due to nuclear binding energy and form the basis for the famous equation E = mc^2.
The reported mass of an element on the periodic table represents a weighted average mass over isotopes. Some elements, like lead, have lower precision in their average amu due to isotopic variation in their sources (nebular vs. radioactive decay).
Techniques like mass spectroscopy, which measure mass/charge ratios of ions, can identify the presence and proportion of isotopes.

Additional Resources:

https://en.wikipedia.org/wiki/Isotope