OCR in Swift with the Vision Framework

Cookbook Index does one thing that has to work: you point your phone at the index page of a cookbook, and it turns the page into a searchable list of recipes and page numbers. All of that runs on Apple's Vision framework. It's on the device, free, needs no API key, and is very good at printed text.

This post covers what I learned getting from "Vision can read text" to "Vision can read a cookbook index": the basic request, camera scanning, turning lines of text into structured data, and the gotchas.

The basic request

Recognising text in an image takes a few lines:

import Vision

func recogniseText(in image: CGImage) throws -> [VNRecognizedTextObservation] {
    let request = VNRecognizeTextRequest()
    request.recognitionLevel = .accurate
    request.usesLanguageCorrection = true
    request.recognitionLanguages = ["en-GB"]

    let handler = VNImageRequestHandler(cgImage: image, orientation: .up)
    try handler.perform([request])
    return request.results ?? []
}

Each observation is one line of text. It holds several candidate readings, best first, and a bounding box:

for observation in observations {
    guard let best = observation.topCandidates(1).first else { continue }
    print(best.string, best.confidence, observation.boundingBox)
}

A few things that matter more than they look:

• .accurate vs .fast. Accurate uses a neural network and is the one you want for a still photo. Fast is meant for live camera frames, where you'll get another chance in a few milliseconds.
• Language correction helps prose and hurts data. It's good at fixing "Choco1ate" to "Chocolate", but it will also "correct" unusual dish names, ingredient spellings and numbers. Try it both ways on real pages.
• Bounding boxes are normalised and upside down. They run from 0 to 1, with the origin at the bottom-left. Convert before drawing over a UIKit or SwiftUI view, or every box ends up mirrored vertically.
• Pass the right orientation. Photos from the camera are often stored sideways with an orientation flag. Get it wrong and Vision reads nothing, or reads the page as a column of single letters.

On iOS 18 and later there's also a Swift-native version of the API, RecognizeTextRequest, with async/await. The ideas are the same.

Scanning from the camera

You can build live scanning yourself with AVCaptureSession, feeding frames into Vision. If you do:

• Run recognition on a background queue, never the capture callback's thread.
• Throttle it. Recognising every frame at 60 fps melts the battery for no gain. A few times a second is plenty, and you can drop frames while a request is still running.
• Set regionOfInterest to the part of the frame you care about. Less image means faster, more accurate results.

Before building that, look at VisionKit's DataScannerViewController (iOS 16+). It gives you a live camera view that recognises text and barcodes, with highlights and tap-to-select, in a few dozen lines. For many apps it's all you need.

For an index page, I've found it's better to take a still photo and run the accurate recogniser on it. The page doesn't move, and one sharp, high-resolution image beats a stream of slightly blurry frames.

Turning lines into recipes

Vision gives you lines of text. A cookbook index needs entries: a recipe name and one or more page numbers. Most index lines look like one of these:

Apple and blackberry crumble 142
Apple and blackberry crumble, 142
Apple and blackberry crumble ......... 142
Lamb tagine 88, 214
Lemon tart 156–7

A regular expression that takes everything up to the trailing numbers covers most of them:

let pattern = /^(?<name>.+?)[\s,.·…]+(?<pages>\d{1,4}(?:\s*[,–-]\s*\d{1,4})*)$/

if let match = line.firstMatch(of: pattern) {
    let name = String(match.name).trimmingCharacters(in: .whitespaces)
    let pages = match.pages
        .split(whereSeparator: { ",–-".contains($0) })
        .compactMap { Int($0.trimmingCharacters(in: .whitespaces)) }
}

The lines that don't match are interesting too:

• Headings and ingredient groups ("APPLES", "see also Pears") have no page number. Keep them as context, not entries.
• Wrapped lines. Long recipe names wrap, so the page number lands on the next line with the end of the name. If a line has no number and the next line starts in lower case or is indented, join them.
• Page ranges like "156–7" mean 156 to 157, not page 7.

Multi-column pages

Indexes are usually two or three columns, and Vision doesn't promise to return lines in reading order. Sorting by vertical position alone interleaves the columns, mixing "Apple crumble" with "Lamb tagine" from the next column.

The fix is to group observations into columns by their horizontal position first, then read each column top to bottom:

// Cluster by left edge: lines that start near each other share a column.
let sorted = observations.sorted { $0.boundingBox.minX < $1.boundingBox.minX }
var columns: [[VNRecognizedTextObservation]] = []
for obs in sorted {
    if let last = columns.last?.first,
       abs(obs.boundingBox.minX - last.boundingBox.minX) < 0.08 {
        columns[columns.count - 1].append(obs)
    } else {
        columns.append([obs])
    }
}
// Vision's y origin is the bottom, so read each column by descending maxY.
let ordered = columns.flatMap { $0.sorted { $0.boundingBox.maxY > $1.boundingBox.maxY } }

The 0.08 threshold is a fraction of the page width. Tune it on real books, because some indexes indent sub-entries enough to look like a new column.

ISBN barcodes

The fastest way to identify a cookbook is the barcode on the back. An ISBN-13 is an EAN-13 barcode starting with 978 or 979, and Vision reads it with a separate request:

let barcodes = VNDetectBarcodesRequest()
barcodes.symbologies = [.ean13]
try handler.perform([barcodes])

let isbn = barcodes.results?
    .compactMap(\.payloadStringValue)
    .first { $0.hasPrefix("978") || $0.hasPrefix("979") }

Look up the ISBN against a book database, and you have the title, author and cover without the user typing anything.

Gotchas

• Curved pages. Near the spine, lines bend and text compresses. Ask people to flatten the book, or crop to the flat part of the page.
• Small, light type. Indexes use the smallest print in the book. Good light helps more than anything in code. minimumTextHeight can stop Vision from wasting time on specks.
• Handwriting. Vision handles some, but much less reliably than print. Expect notes in the margins to come out garbled.
• Confidence isn't correctness. A high-confidence result can still be wrong ("1" vs "l", "0" vs "O" in page numbers). Validate anything structured, such as checking that page numbers fall within the book's length.

What I'd tell anyone building on Vision

• Let people fix mistakes, easily. No OCR is perfect, and a quick way to correct an entry beats chasing the last 1% of accuracy.
• Scan in batches. An index is several pages. Let people shoot them one after another and review at the end, rather than stopping after every page.
• Use confidence to choose what to show for review. Accept high-confidence lines automatically and highlight the rest, so people check the few that need it.
• Test on real books. Every publisher formats indexes differently, and the edge cases only show up on a pile of real cookbooks.