Final search frontier: SearchInk is teaching machines to read and understand handwritten documents

The future may be digital, but so much of our present and past remains trapped in printed forms.

For several years now, Sofie Quidenus has been focused on solving this disconnect by trying to make printed text readable for computers so the knowledge contained within can be captured and searched. Now she has moved on to what is perhaps the most difficult challenge of this puzzle: handwriting.

With so many documents still being printed and filled out by hand, not to mention letters and notes, she feels it’s essential that computers learn how to decipher a form that can be incredibly varied. And, more importantly, computers need to comprehend the context around these written words so they can truly understand the meaning.

“It’s crazy how many forms are still printed,” she said in a recent interview. “We think we are in the 21st century. But this is still a big problem.”

A native of Austria, Quidenus got interested in the challenge of systems that allow for large-scale scanning of text about a decade ago, while studying economics. She founded Vienna-based Qidenus Technologies (dropping the first “u”) in 2005. The company spent four years researching and developing a V-shaped, robotic, automated book scanner that can scan 2,500 pages per hour. Today, it is used by more than 70 libraries around the world.

But as the book scanning advanced, handwritten documents remained a stubborn problem. Quidenus noted that just the letter “s” presents an infinite number of varieties: cursive or print, male or female writer, right- or left-handed writer, the age of the document, and so on.

Yet, there were some advantages. As Qidenus’ machines scanned more documents — which sometimes included handwritten material — the company was amassing a treasure trove of data it could leverage to begin tackling the problem and teaching its machines.

The problem of handwriting was still particular enough, however, that last year Quidenus decided to launch a spinoff company, SearchInk, to deal with it. In addition to raising money, Berlin-based SearchInk is hiring developers and engineers and has started to make some progress toward a reliable technology, she said.

The company is developing technology it calls “Handwritten Text Recognition,” or HTR. The ultimate goal is to use this technology not just to be able to search a written document for a word, but to train the machine to be able to comprehend its broader meaning and context.

“Our approach is not only to read the document and detect optical patterns,” Quidenus said. “Our approach is to read the document more like a person to get the meaning behind it.”

The first step has been developing a more advanced computer vision algorithm to help the machines recognize handwritten text, analyze the structure of a document, and then begin to organize the semantic information so it can be analyzed. Next, SearchInk is developing a deep learning system so that the machines can take the information and begin to eventually teach themselves how to read, as well as how to learn and continually improve their comprehension.

For now, the process has to be supervised by a human who can offer corrections and improvements and continue to tweak the algorithms. But, eventually, SearchInk wants to move to unsupervised systems, where the machines learn to correct themselves and improve their own algorithms. The latter step would be particularly important in creating a system that can mass-scan handwritten documents.

All of this is expected to take a couple more years of development work and research. The company is far from figuring out a business around all of this, though Quidenus is considering things like making the technology available as a paid API that could allow it to be baked into note-taking apps, like Evernote.

For now, however, Quidenus and her team are enjoying the excitement of tackling a big intellectual challenge that could mean tapping into a reservoir of knowledge that remains tantalizing just out of reach for most of the world.

“Engineers love it if they have a big impact,” she said. “They love if it’s something meaningful. And we are unlocking knowledge that is not visible today.”