OpenStax Word Doc Conversion Project: Part 2

Posted on Aug 30, 2026
tl;dr: Textbook conversion project: converting docx files to HTML and packaging as a Common Cartridge.

Where We Left Off

Back in January, as part of our efforts to increase OER adoption at Shepherd, I converted the source docx files for two textbooks provided by OpenStax to (relatively) clean HTML for importing directly into our LMS (Brightspace). This got me 80% of the way to my goal, which was to have redistributable packages I could share with instructors at any school. I packaged the files into Common Cartridge format by hand (mostly, tediously figuring out which order they went into so I could create the appropriate XML) and we were able to use the books in our spring classes.

Why go through all this pain? Asking students to download a 700-page, 70-MB PDF isn’t a great to way to get them to read the text. And those publisher-provided PDFs weren’t tagged at all. OpenStax did provide Common Cartridge files, but they were just references to the content on their own web site. Neither of those was a good starting point for our own remediation efforts, let alone adapting the content. And it’s content worth adopting and adapting—OpenStax is doing yeoman’s work. It was just going to take a bit of work on our part to get those texts across the finish line.

At the end of Part 1, I had a roughly 60-line bash script that turned a directory of docx files into a directory of HTML files. To put a bow on the first phase of this project I needed to add a requested attribution to each page and fix a problem with the way images were referenced in one of the texts. I also wanted to make some additional accessibility improvements and add some support for the packaging itself. In the meantime, OpenStax had made some updates to the original source files and their PDF files were now tagged, even if still imperfect.

But we also had new OER texts to convert for other classes. What started as two books is now four, in three different shapes:

  • Principles of Marketing (OpenStax), one file per section, named like 1-3-.... This is the book that taught me about layout tables, .so media files, and equations stored as pictures.
  • Introductory Business Statistics 2e and Principles of Economics 3e (OpenStax), one file per section again, but at 169 and 419 files respectively. These are the books that made ordering and manifest structure into real problems, because I didn’t want to create the file list by hand again.
  • Business Communication (OER Commons), one file per chapter, with names like BC-01. Not an OpenStax book, and not built like one: OER Commons seems to be built with the assumption that edits and remixes remain on the platform, so I liberated these chapters by pasting by hand from their editor into docx files and manually remediating the heading structure. This is the book that broke most of my script’s assumptions.

I also moved the goalposts. A page can convert cleanly and still be inaccessible, and some of the accessibility problems seemed to be baked into Pandoc’s defaults rather than into the source documents. I wanted to automate as much remediation as possible, even where some human judgment was called for (alt text and table captions). This was ambitious for someone with a day job who isn’t a Real Programmer, but AI (in this case, mostly Claude Opus 5) had improved enough over the past seven or eight months that it could do the heavy lifting. Also, I hoped that something I’d spotted in Pandoc’s changelog meant one of my problems was fixed. (I was wrong.)

Where the project stands now: about 3,500 lines of code and documentation, available on Github under the unimaginative title “TextbookImprover”.

  • convert.sh, 632 lines
  • figures-and-tables.lua, 1,093 lines
  • build-cartridge.py, 1,191 lines
  • plus two small one-shot repair tools, a documented example config, and a README

I’ll take the two halves of that in order: first converting the documents, then packaging them.

An Aside: Part 1’s Mystery, Solved

I ended Part 1 with this problem:

The only issue I ran into with the script above is that the pipefail option doesn’t seem to be supported when running under WSL, even though I double-checked and the shell does seem to be bash.

There were actually two separate bugs here, and neither was what I thought.

The first was in the script itself. When I edited set -euo pipefail, I accidentally left it as:

set -euo          # not "set -euo pipefail"

-o with no argument doesn’t fail. It tells bash to print its list of options to stdout and do nothing else. So not only was pipefail off, errexit and nounset were off too. My script had been running with no error handling at all, which explains a couple “why did that keep going?” moments I’d written off as Pandoc being generous.

The second bug was the actual WSL error:

convert5.sh: 3: set: Illegal option -o pipefail

That is dash’s error message, not bash’s. /bin/sh on Ubuntu is dash, and a shebang line is ignored entirely when you invoke a script as sh script.sh, which is exactly what I had been doing. The shell “did seem to be bash” because the shell I was typing into was bash. The shell running my script was not.

The fix is a re-exec guard, which has to stay POSIX-parseable, since dash has to be able to read it before handing off:

if [ -z "${BASH_VERSION:-}" ]; then
  exec bash "$0" "$@"
fi

There are two lessons here: the interpreter that runs your script isn’t necessarily the one you asked for, and an error message you can’t explain is usually telling you something true about a thing you haven’t looked at yet.

Converting the Documents

Layout Tables, Figures, and Captions

Here’s the pattern that ate the most time. In these documents, an image is frequently positioned by wrapping it in a one-cell table, with the caption in an ordinary paragraph underneath. Pandoc reproduces that faithfully: you get a <table> with one row, one cell, no header, and no caption, followed by a stray paragraph. But a <table> element is a promise to assistive technology that there is a data relationship here—rows mean something, columns mean something. When there isn’t one, that’s a WCAG 1.3.1 (Info and Relationships) failure. The page looks fine but lies to anyone not looking at it.

I could have attacked this with sed, and a year ago I would have. Instead (thanks entirely to Claude) this became a Lua filter, figures-and-tables.lua, operating on Pandoc’s abstract syntax tree. The advantage is that you can ask structural questions instead of textual ones. To tell a layout table from a data table, look inside it and see whether it contains anything other than images:

-- Returns (images, anchor) when the table holds nothing but images.
local function image_only_table(tbl)
  local found = { images = pandoc.Inlines({}), anchor = nil, extra = false }
  for _, row in ipairs(all_rows(tbl)) do
    for _, cell in ipairs(row.cells) do
      scan_blocks(cell.contents, found)
      if found.extra then return nil end
    end
  end
  ...

Image-only tables become <figure> and <figcaption>. The one wrinkle is that the docx cross-reference anchors live on the table, and the text refers to them (“see Figure 3.2”), so the anchor has to move onto the figure or every #fig-00001 link in the book quietly breaks.

Problems with Pandoc’s Own Output

I want to be clear that I love Pandoc and this project would be impossible without it. (I don’t know how jgm finds the time to be a scholar, a teacher, and also an amazing developer!) But three of its defaults seem to actively work against accessibility, and there’s a fourth problem it faithfully passes through from the source. I wouldn’t have found the first three by reading the HTML.

display: block on tables. Pandoc’s default stylesheet contains:

table { overflow-x: auto; display: block; }

This is a reasonable-looking bit of responsive design—it keeps wide tables from blowing out the layout—and it silently destroys the table. Changing a table’s display property removes its role from the browser’s accessibility tree, so rows and columns stop being exposed to screen readers at all. The fix is to restore display: table and move the horizontal scrolling onto a focusable wrapper element, which the filter now adds.

Caption contrast. Pandoc sets no color on captions, so I picked one. A widely used accessible gray is #767676, which is where it is because that’s the lightest gray that passes AA against pure white. Pandoc’s background is #fdfdfd. Rather than assume, I had Claude compute it:

       fg  vs #fdfdfd    AA    AAA
  #767676      4.47     FAIL  FAIL
  #555555      7.33     PASS  PASS

4.47:1 against a 4.5:1 threshold. That’s the kind of failure that never shows up in casual review but would certainly show up in an audit.

implicit_figures printing alt text as body copy. Some of the equations in these books are included as pictures, and the alt text on those pictures appears to be MathSpeak—the spelled-out spoken form of the equation. Pandoc’s implicit_figures extension wraps any paragraph containing a lone image in a <figure> and uses the image’s alt text as the caption. So pages were displaying

StartLayout 1st Row 1st Column upper C u s t o m e r ...

as visible prose, in the middle of the text. And because Pandoc marks those generated captions aria-hidden="true", the gibberish was visible only to sighted readers. Screen reader users got the clean version. Fixed with -f markdown-implicit_figures.

MathSpeak as alt text. Fixing the visible-prose problem doesn’t fix the underlying content. upper C u s t o m e r left as alt text gets read out letter by letter, which is worse than useless. The filter now rejoins spelled-out identifiers—strictly mechanically. It will turn u s t o m e r back into ustomer, and it will not touch structural words like StartFraction, equals, or Superscript, because a plausible-sounding mistranslation of an equation is more dangerous than obvious gibberish. Obvious gibberish gets reported and fixed; a confidently wrong formula gets read to a student. The filter reports every equation it touches, and the real fix is to author them as actual Word equations upstream.

A few smaller things went in at the same time: lang="en" on the html element, promoting the leading H1 to <title> (Pandoc otherwise uses the filename slug, so every page in Brightspace had been titled 1-3-the-marketing-mix), and stripping fixed height attributes from images so they can reflow.

What the Machine Can’t Know

Three things in these documents genuinely can’t be derived from the source: a descriptive caption for a table that has only a bare label, alt text for an image that has none, and which row of a table is the header row when Word never marked one.

There’s a strong temptation to guess: generate something from the surrounding text; use the first row as a header because it usually is; or summarize the image filename. I tried versions of all three and every one of them produced output that was worse than an honest gap, because a wrong caption is indistinguishable from a right one until someone who needs it hits it. So the pattern I settled on is report what needs a human, accept the answer in a sidecar file, but never invent it.

The script writes a report of what’s missing (table-captions-missing.csv, image-alt-missing.csv, table-headers-missing.csv), and reads back the answers from a corresponding sidecar (table-captions.csv, image-alt.csv). Missing header rows get no sidecar, because I found no satisfactory way to fix that downstream. (The fix probably belongs in Word, but I’d like to revisit this issue.) With this approach, the manual work of fixing table captions and alt text only has to happen once for one set of docx files, even if the HTML is regenerated. It’s much better than manually remediating the output after every run, and much cleaner than multiple edits by multiple people on docx files that might break new things.

The reports are rebuilt from scratch every run and deleted when they’re empty: the file existing at all means there’s work outstanding. No reading, no diffing, no remembering what you did last week.

I started with TSV, on the theory that captions routinely contain commas and Lua has no CSV parser in its standard library. I switched to CSV anyway, because everyone editing these files is going to open them in Excel, and that meant writing a real RFC 4180 parser: quoted fields, doubled quotes inside them, embedded newlines, CRLF line endings, and Excel’s UTF-8 BOM. That last one was the tricky part—without handling it, the first key in the file silently becomes \ufeffTable 2.1, matches nothing, and fails without complaint.

Two bugs in that mechanism are worth recording:

Blank meant “not answered” when it should have meant “answered: nothing”. A row with an empty second column means a human looked at this table and decided it needs no caption. Claude’s first attempt at a loader skipped empty values, which made a reviewed-and-blank row indistinguishable from a row that was never added—so those tables came back in the report every single run, forever. Now nil means absent and '' means deliberately blank, and they behave differently.

Labels that were already fine got reported. Most OpenStax tables are labeled something like Table 12.1 Pricing Objectives, which is a perfectly good caption already. Only bare labels—a prefix, a number, and nothing else—need a human:

local function is_bare_label(label)
  local prefix = opens_with(label, TABLE_PREFIXES)
  if prefix == nil then return false end
  local rest = label:sub(#prefix + 1):match('^%s*(.-)%s*$')
  if rest:find('%s') then return false end
  return rest:find('%d') ~= nil
end

Alt text keys broke when file extensions changed. The alt sidecar was keyed on the image path, but step 2 of the pipeline renames files by detected content type—so a key faithfully recorded as .../rId57.so stopped matching the moment that file became .../rId57.jpg. It’s keyed on the stem now. This is the same lesson as the media resolution rewrite below, learned twice in two places.

The [decorative] Problem

An empty cell in the alt sidecar means “leave the existing alt text alone”. But decorative images—rules, spacers, ornaments—need a way to say “this should have no accessible name”, which is a different thing. Hence a [decorative] marker.

Getting that to survive to the finished HTML took two rounds with Pandoc:

  • If you clear an image’s caption, Pandoc emits no alt attribute at all. Screen readers then fall back to announcing the filename, which is the exact opposite of the intent. rId57.jpg is not an improvement on nothing.
  • If you set it explicitly you get alt="", which is correct—but --embed-resources re-serializes the whole document and rewrites that to a bare alt. The two parse identically per the HTML spec, and some assistive technology treats a valueless alt as a missing one anyway.

I considered a sed pass to put alt="" back and rejected it, because the string alt also occurs inside real alt text, and I’d be corrupting content to fix markup. The filter adds role="presentation" instead, which survives serialization. (--embed-resources later got dropped for unrelated reasons, which fixed this too. I’m leaving role="presentation" in for now.)

Media Resolution, Rewritten

A document turned up whose Markdown still referenced a .so file while the actual file on disk was a .png. The cause was structural: renaming files and rewriting references were two separate passes that communicated only through the filesystem. Either could happen without the other, and the broken result still produced HTML that looked fine in a directory listing.

I had Claude rewrite it as a single Markdown-driven pass that matches on the stem instead of the extension. Because it’s driven by what the Markdown actually references, and it doesn’t care what extension anything currently has, it’s idempotent and self-healing: whatever state a previous run left behind, running it again reconciles it.

Then a verification gate, which was a valuable suggestion from Claude:

if [ "$unresolved" -gt 0 ]; then
  echo "Stopping: $unresolved media reference(s) could not be resolved." >&2
  echo "No HTML was generated." >&2
  exit 1
fi

Stopping is deliberate, and it’s because of a Pandoc behavior I learned about the hard way: given a file extension it doesn’t recognize, Pandoc emits <embed> rather than <img>. So a dead image reference doesn’t produce a broken-image icon or an error. It produces a valid HTML element that renders as nothing at all… in a book with hundreds of pages, inside a zip file, that you then import into an LMS. Broken output that looks fine is worse than no output.

Two bugs cropped up in that new code:

A /media/ pattern that matched URLs. Broadening the reference pattern to accept any extension also cost it its anchor, so a citation in a references chapter—https://www1.nyc.gov/site/dca/media/Face-Masks-in-Short-Supply.page—got treated as a local file that couldn’t be resolved, and the whole run stopped. The pattern is now anchored to each document’s own media directory, with the base name regex-escaped.

Stale copies accumulating. Re-running could leave both rId54.so and rId54.jpg sitting in a media directory, and the resolver picked whichever sorted first, which is not a criterion anyone chose. It now prefers the file the reference actually names, falls back to the newest, and deletes the losers.

When the Filesystem Lies

One of the most annoying problems I ran into had nothing to do with the insanity that was the source documents, but with my hard drive. Does my home computer have a fast SSD drive? Of course. But I need that for important stuff like a copy of the Bitcoin blockchain and model weights for Ollama. All my academia-related work lives on an external USB drive that I sync with the computer in my faculty office via Google Drive. I was mounting that drive in WSL to run my script against the files. (Claude misinterpreted this as a mount of my Drive folder from the “cloud”, and had some snarky remarks, but to be fair my actual setup wasn’t that much more performant.)

Once my new scripts were complex enough, and I took the time to test against a fresh directory with no intermediate files, I started getting errors that moved around between runs and then stopped appearing. That wasn’t a logic bug: that was a storage layer failing to present a consistent view of its own writes. A mv returns success, but the next command doesn’t see the result yet.

The problem wasn’t really my janky setup per se: it was that neither mv nor sed -i was being checked, so the script had no way to tell a janky storage layer from a real failure. Both are now verified after writing, retried once, and counted, and the script says what it thinks happened:

Note: 3 media write(s) had to be retried before the change
  was visible. That is a filesystem symptom, not a conversion one.

When someone else runs this and sees it, the message tells them where to go looking.

A Second, Very Different Book

Everything up to here was tuned to books that OpenStax built. But Business Communication was chapter-per-file with names like BC-01, produced by copy/paste from a web-based editor, and it broke a satisfying number of assumptions.

Pandoc’s line wrapping broke an image path. The conversion crashed on a reference to media/image1.gifif, which is a fun thing to see in an error message. Pandoc’s default wrapping had split the path across two lines:

BC-09/media/image1.g
if

The extractor read image1.g, resolved the stem to image1.gif, and helpfully rewrote it—leaving the orphaned if on the next line to be swept back up. Every grep and sed in the script assumes one complete reference per line, which means --wrap=none isn’t a preference: it’s a correctness requirement. I’d just gotten lucky with the first two books.

I had to undo Part 1’s grid tables fix. In Part 1, I disabled grid tables (-t markdown-grid_tables) to stop Pandoc from chopping up image code. That works right up until you meet a table with multi-block cells, which cannot be expressed in simple Markdown at all. Pandoc’s fallback in that case is to emit a raw HTML block—which passes through a Lua filter completely untouched. No caption, no scope attributes, no wrapper. The table just quietly opts out of every accessibility fix in the pipeline. Grid tables are back on; the original image problem is handled properly now by --wrap=none and the filter.

Captions above tables. This book puts the label above the table, in two different shapes:

**Table 2.1: Message Transmission Mediums**     <- label and title, one paragraph

**Table 7.1**                                   <- bare label
*Sample Code of Conduct*                        <- title, separate paragraph

Both are absorbed now. But adding a captions-above rule promptly broke the economics text, which captions below. Table, Para(label), Table means precisely opposite things under the two conventions, and no amount of local context can tell you which one you’re in. The numbering pass now tallies the convention across the whole document before deciding anything.

Prose that starts with the word “Table”. My first attempt at captions-above swallowed this sentence as a caption and deleted it from the page:

Table 48. 1 provides and example of how to organize a table with categories highlighting your job skills.

(The mangled auto-number and the typo are both in the original.)

I had Claude implement two proposals for telling caption from prose and measure both across all 59 tables in the book, and… both were wrong. Requiring emphasis on the label lost a legitimate plain-text caption. Rejecting long sentences that end in a full stop lost a real 20-word caption. The one signal that turned out to be reliable is the word immediately after the number:

Table 48. 1 provides and example...   -> lowercase verb  -> prose
Table 22.6 Common Formal Business...  -> capital         -> caption
Table 2.1: Message Transmission...    -> colon           -> caption

Being able to cheaply implement and test different approaches was a big win here. I would have settled for an inferior solution if Claude hadn’t been doing the hard work for me.

Spacer GIFs. 274 of the 487 images in this book are 43-byte transparent GIFs used as bullets. Every one of them was a row in the alt text report. They’re stripped now, with the vertical space they were faking replaced by CSS and what was removed written to a log:

images:
  spacer_below: 0.3in
  strip_spacer: true
  spacer_log: spacer-images.csv

Book-specific vocabulary. Tables in this book are labeled Figure, not Table. So captions.table_prefixes became a configuration key rather than a constant.

Tracked changes used as content. This book teaches editing, and its before-and-after examples mark the deleted words using Word’s tracked deletions. This means the content of the lesson is stored in revision metadata. (I might have avoided this issue had I started with the available HTML instead of copying from the online editor. But I’m not sure, and anyway that would have created its own problems.) Anything that resolves revisions destroys it: Word’s own “Accept All”, most converters, and Pandoc, which accepts changes by default. Rather than special-case this in the pipeline, I had Claude write a separate one-shot tool, untrack-deletions.py, that rewrites each <w:del> as an ordinary run with <w:strike/> applied. It touches only word/document.xml and copies every other part of the archive byte for byte.

Files that are not documents. A *.docx glob picks up Word’s ~$Name.docx lock files, which are not zip archives, and one of them aborted an entire run. They’re now skipped three ways: by name, by zero size, and by missing the PK magic bytes at the start of the file.

Positional Keys for Unlabeled Tables

Tables the source never labeled at all still need a key, so a human can supply a caption for a specific one. Without a label, all that’s left is position. Getting that right (in a way useful to a human writing the captions) required three changes:

  1. Filter traversal order is not reading order. Pandoc walks nested block lists before the lists that contain them, so a table inside a Div got a lower number than a table that visibly preceded it on the page. There’s now a separate first pass that walks the document in document order and stamps each table with its number.
  2. Layout tables were being counted. A page with four data tables reported them as #table-1, #table-3, #table-5, #table-6, because five image-only tables were interleaved with them. Only data tables are numbered now, which is also the only numbering a human reading the page could reproduce.
  3. The report couldn’t identify its own rows. #table-3 requires the reader to count tables on a page to find out which one it means. The report gained an Excerpt column holding the table’s first cell, so a row says what it’s about.

Packaging as Common Cartridge

The second half of the problem I wanted to tackle was turning a directory of good HTML into something Brightspace can import. Common Cartridge is the relevant standard: essentially, a zip archive with a .imscc extension containing your content plus an imsmanifest.xml that describes what’s in it and how it’s organized. It’s the evolution of the same Content Packaging used in SCORM, but without a complex run-time engine. I’d been building simple SCORM files by hand to get some of my own content into a widely supported format, and I’d built two CC files by hand for my first pass at the original two OpenStax texts. But that approach was never going to scale.

Dropping --embed-resources

In Part 1, I speculated that going back to embedding images in the HTML might clean things up, since all those media subdirectories clutter the LMS file storage and slow down imports.

I was so wrong. Base64 data URIs inflate every page enormously—preface.html went from 37 KB to 649 KB—and Brightspace doesn’t render them reliably from an imported cartridge anyway. So --embed-resources is gone. The pleasant side effect is the one from the [decorative] section above: with no re-serialization pass, alt="" now survives intact.

Building the Manifest from What the Pages Actually Use

There was no good place to find the hundreds of image files and match them with the hundreds of pages of HTML I was packaging. The first time around, I listed the directories and used some regexes to get the results into a form I could stick in the manifest. I also managed to mess up some references that way. I did have image-alt.csv sitting right there with a list of images in it, but this only contained images that were reported: an image whose alt text was already present and short enough was never reported.

Instead, build-cartridge.py discovers each page’s dependencies by parsing the HTML and collecting the src and href attributes the page actually uses, which is the only list that’s true by construction.

This new script is read-only with respect to page content. That makes it safe to run over and over while you’re fiddling with the organization, and it’s also what makes it useful on any tidy directory of HTML, including hand-authored pages that never went anywhere near convert.sh. Keeping the thing that mutates content and the thing that reads content in separate programs cost me nothing and set me up to evolve this script into a more general tool in the future.

Configuration in YAML

Configuration started out as an XML template but quickly moved into imsmanifest.yaml. (There’s a manifest-to-yaml.py for migrating an existing manifest, which I needed exactly once and kept around in case anyone else needs it.)

I started off with a hardcoded attribution line requested by OpenStax, but with the addition of a book from another publisher it became a general header / footer pair, written in Markdown and rendered once per run:

footer: |
  *Your Textbook Here* is licensed
  [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/).

There’s one subtlety: Pandoc inserts template content after the Lua filter has run. That’s why promoting the H1 to <title> still works: the filter sees the document before the header is prepended. It also means nothing in the header or footer is processed by the filter. Anything you put there is your own responsibility.

Ordering 419 Pages

A cartridge has an organization, and the organization determines what order things appear in when the instructor opens the course. With 419 files, getting this right by hand is a world of pain.

Alphanumeric order isn’t close. Plain sort on the statistics filenames put chapter 1 in positions 1–10 and chapter 2 at position 76, with chapters 10 through 13 in between. Natural sort plus an explicit back-matter order fixed the basic case.

Vocabulary differs between OpenStax books, more than I expected. Statistics ends each chapter with chapter-review, homework, and solutions. Economics uses key-concepts-and-summary, self-check-questions, and problems. My first attempt recognized chapters by looking for known section names, which correctly identified 177 of 419 Economics filenames and then sensibly declined to group anything at all.

Chapters are now detected by shape instead—a numeric prefix, or a filename opening with chapter-12—which doesn’t care what the sections are called. Only the order of back matter still depends on names, and that’s configurable and reported when it hits something it doesn’t recognize.

Appended pages get placed, not piled. If you add a page later, it joins its chapter’s existing group if there is one, or forms a new chapter group if there isn’t. Only pages with no chapter identifiable in the filename land in Unsorted, and groups are added inside whatever container grouping.append_to names.

Reading the Table of Contents from a PDF

Claude came up with a trick that was worth my monthly subscription (technically, paid for by WV HEPC’s grant—thanks, guys!) all by itself. It placed 419 of 419 Economics pages and 169 of 170 Statistics pages.

Every one of the OpenStax books ships as a polished PDF, and that PDF’s bookmarks are its table of contents, in the exact order the publisher intended. So build-cartridge.py takes a --toc book.pdf option: it reads the outline and matches each heading to a page by deriving a filename from the heading text, e.g., “Key Concepts and Summary” to key-concepts-and-summary. One rule, and it works across books with completely different section vocabularies, because it never needs to know what the sections are called.

The first version had a bug worth mentioning, because it’s a design error rather than a coding one: --toc replaced the contents list wholesale. So a carefully curated frontmatter entry got discarded and then turned up in Unsorted, which is a rude thing to do to someone’s deliberate decision. A PDF outline is a source of ordering, not grounds for throwing away curation. It now orders only the pages that contents hasn’t already placed.

Where Things Stand

The toolchain now comprises five scripts:

  • convert.sh (632 lines) — the pipeline. DOCX to Markdown to HTML, media resolution, and the missing-information reports.
  • figures-and-tables.lua (1,093 lines) — the Pandoc filter. Figures, captions, alt text, spacers, table headers.
  • build-cartridge.py (1,191 lines) — manifest and .imscc builder. Read-only, and usable on its own.
  • manifest-to-yaml.py (206 lines) — one-time migration from an existing manifest.
  • untrack-deletions.py (157 lines) — one-time repair of tracked deletions in a docx.

Plus imsmanifest.example.yaml, which documents every configuration key, and README.md documenting usage and known limits.

For the record, here’s what’s actually wrong with the source documents, collected in one place. If you’re struggling with converting and remediating old docx files—and that probably describes someone at every state university right now—it’s a more useful artifact than any of my code:

  • Images stored with a .so extension and an application/octet-stream content type, unusable until renamed by detected content
  • EMF/WMF vector art that no browser renders and that stops conversion cold
  • Tracked deletions used as visible content, destroyed by anything that resolves revisions
  • Equations stored as pictures with MathSpeak-style alt text, read letter by letter
  • Layout tables wrapped around images, producing tables with no header and no caption
  • Missing table header rows, which can’t be automatically fixed downstream
  • Alt text of 300–600 characters, where a long description probably belongs somewhere else (Shepherd is using Panorama to guide accessibility efforts in our LMS. It inflexibly dings alt text longer than 120 characters, which is ham-handed and probably counterproductive. But it’s certainly true that the alt text OpenStax supplied was too verbose.)
  • Broken auto-numbers like Table 48. 1, and ordinary typos, in text that becomes captions
  • Spacer GIFs used as bullets: 274 of 487 images in one book

None of that is exotic. It’s what you get from a decade of well-meaning people using a WYSIWYG word processor as a publishing system.

Principles That Held Up

Here are five lessons I’ll carry to my next project:

  • Report what a human must decide; never invent it. Header text, descriptive captions, and alt text all resisted automation, and every attempt to guess produced worse output than an honest report. The corollary is that the report has to be cheap enough to act on, which is why they’re CSVs and why an empty one deletes itself.
  • Fail loudly rather than produce output that looks fine. A dead image link and a silently dropped page are both completely invisible in a finished cartridge. Refusing to write output is a feature.
  • Measure before adopting a heuristic. I implemented two caption-detection rules that I would have shipped on intuition, measured them across a whole book, and threw both away. AI makes doing this cheap, so there’s no reason not to.
  • Make repair idempotent. Stem-based media matching means any run reconciles whatever state the previous one left behind. This turned a class of bug into a non-event.
  • Separate what mutates from what reads. convert.sh changes content; build-cartridge.py only reads it. That makes the second one safe to re-run and reusable on HTML that never came from the first one.

Using AI Redux

Dillinger: “Now, wait a minute! I wrote you!”

MCP: “I’ve gotten 2,415 times smarter since then.”

About my work back in December/January, I wrote: “even the latest AI models are still prone to confidently giving bad information”. AI saved me work producing sections of code, but it didn’t dramatically change what was possible for me to achieve.

Using Claude just seven or eight months later feels an order of magnitude more productive, and I assume that has less to do with the switch from using (mostly) ChatGPT to using (mostly) Claude and to do with the mind-blowing rate of improvement in the models and the surrounding tooling. (And this even though I’m not letting Claude touch the hard drive on my machine.)

I’m still building my scripts incrementally, making sure each step works as intended and finding new places to go wrong, but I’m doing so at a higher level of abstraction. Not only can I trust Claude to write large blocks of code without detailed guidance, I can ask it for advice on design decisions and even get really useful suggestions unprompted. And Claude being able to run tests in its own sandbox avoided my having to choose between exposing my system to Claude and doing a lot of wasteful pasting of error messages.

I’ve been programming here and there since the 70s, but I’ve never been a Real Programmer. I can’t even think of the last time I wrote a script more than a couple of hundred lines long. Python and shell scripting I can stumble through if I need to, but for me to have also learned Lua and written a couple thousand lines of code would have taken more time than a busy assistant professor has at their disposal. Worse, documenting it would have taken… well, I’d probably never have gotten around to it.

Maybe a few dozen lines of my code survive in these scripts, and it’s fair to call this “my project”. But it’s no longer fair to call it “my code”. That makes me a little sad, truth be told. But that’s more than outweighed by knowing that we’ve saved our students thousands of dollars this semester alone by adopting these four texts.

Next?

There’s probably a lot more to to be done with these conversion scripts, not least adding the option to start with the nice clean Markdown my own textbook is written in instead of nasty little docx files. No doubt I’ll find more cases to handle as I try them on more samples. And I still hope to do more to automate handling table headers.

But what I’m most looking forward to right now is turning build-cartridge.py into a proper solution for packaging content. For example, right now it only knows how to handle HTML pages that appear in a content section, but it would be nice to also handle quizzes and discussion forums. More importantly, it needs a front-end that folks who aren’t prepared to edit YAML and run a shell script can use.

I think that’s doable now, even for an amateur programmer who is busy teaching five classes this semester. Thanks, Claude! /me crosses fingers