The process began with manually restoring a selection of manuscript columns to create training data, then using the resulting model to process the remaining columns. Throughout, the goal was to restore each letter’s original pen strokes in their exact positions.
1. Seed glyph library
Before any automation, I thresholded source columns to separate ink from parchment, traced the result into vector paths (SVG), and manually extracted and cleaned about 30,000 glyphs. Threshold alone shreds thin strokes and picks up stains, so damaged shapes were rebuilt with the pen tool and reshaped from clean examples of the same letters, either from nearby or from another page.
2. Bulk glyph processing with AI
For the training set, I then bulk-processed the glyphs by automatic means: algorithms and glyphs to establish the best baseline that it could, and AI to fill in heavily damaged glyph shapes. Across nearly 370 source columns, this covered approximately 307,470 letter glyphs, which were chosen for their diversity as opposed to using the first 370 columns in order. Once the AI glyphs were established, an algorithm was produced to center the glyphs back onto their original positions. This stage was GPU intensive and required extensive backend setup, but it made the rest of the project feasible for one person.
3. Position matching
AI-restored glyphs cannot be used as finished output. Even when a result looks like the same image with the same shapes, the strokes, shapes, and positions of the glyphs vary wildly from the original. That makes them unsuitable for ML training when the target is to remaster the exact pen stroke. So the processed glyphs worked as a key instead. Each one was shifted to its respective position in the column, anywhere from 1 to 20+ pixels vertically, horizontally, or usually both, then fitted to the surviving ink. When a match wasn’t clean, the 30,000 hand-cleaned glyphs let the code reshape it by finding the replacement that best matched its shape.
4. Per-column cleanup
With the backend in place, cleaning up strokes, paths, and positions took approximately 20-25 minutes per column. Some columns were fast and some were slow. By my estimate, about 85% of glyphs landed in the proper location with the proper shape after all pre-processing steps. A very small portion of those needed a single vertex adjusted, either to smooth the curve or to position the stroke better. Another 10% needed their positions adjusted to match the original glyph location, and fewer than 5% needed multiple vertex adjustments, bending, smoothing, and similar work. These percentages are approximate, based on my observations rather than formal measurements. Without the automation, this would have taken far longer and likely would not have been possible for one person. The approximately 370 finished columns could then be used for training.
5. First training run
After 12 hours the model was blending page artifacts (bleed-through, smudges, ink stains, etc.) into its output. The parchment has heavy bleed-through from the reverse side on many pages and areas, and more; and the model couldn’t tell them from ink. Instead of relying solely on masking areas to weigh training toward good strokes and bad strokes, a different solution was created altogether since the object vector data was already established.
6. Color-burst targets
Instead of training on black markings for both negative and positive weighing, I wrote up the logic for a simple JSX algorithm that gives each object a randomized color gradient fill. Bleed-through, smudges, and other defects stay black, while the positive data that we want now won’t be confused with bleed-through or other rogue strokes.
7. GPU model training
The model, Colored-Carta, was trained for approximately 72 hours on a single NVIDIA A100 (40 GB) GPU. It is built on NAFNet, a neural network architecture designed for image restoration. A script cut the colored columns into small tiles and paired each tile of the original scan with its colored target. The model’s job was to turn a messy scan into clean, colored lettering with the defects dropped. Tile sizes ranged from 64px to 1536px, which let the model learn both individual strokes and how they relate to the surrounding letters, words, and lines. Larger tiles also carried a border of extra surrounding pixels, 16px to 256px depending on the tile size, so the model could see context beyond the edge of the area it was restoring. To keep the model from depending on one kind of image, each tile was randomly shown in different forms: the original color scan, grayscale, contrast-enhanced (CLAHE, which boosts local contrast so faint strokes stand out), thresholded (pure black and white), blurred, or degraded to mimic lower-quality scans and the native resolution of the source images. Paired high- and low-resolution versions and randomized crops added more variety. Every version of a tile was trained against the same colored target. A consistency loss, meaning a penalty whenever the same tile produced different answers in different forms, kept the results stable. Three other safeguards helped. Edge preservation kept stroke edges sharp. Foreground weighting counted mistakes on lettering more heavily than mistakes on bare parchment. Trust masks turned down the weight wherever the target and scan didn’t line up well. The color-burst targets did the main work of separating ink from defects, and this weighting only supported them. I finished with targeted refinement passes to improve results across different image qualities.
8. Finishing
After 72 hours total on the color-burst tile set, a new model learned to drop the black errors and to color only the lettering presented. It handled nearly every case, including some of the worst bleed-through, better than I thought was possible.