Do you see a pattern?

“I’ve seen this movie before and it doesn’t end well,” mutters a VC whose start-up is short of cash.

“We’ll always have Paris,” Rick tells Ilsa near the end of the movie “Casablanca.” 

Both these scenarios illustrate a favorite point of David Waltz, the natural-language expert whom I met while he was senior scientist at Thinking Machines: “Words are not in themselves carriers of meaning, but serve merely as pointers to shared experience.” …or something like that! The meaning should come clear from my examples.

In each case, there’s something complex and familiar that both parties recognize – something well beyond the capacity of words to represent without a sentient, intelligent being to condense them into a pointer at one end and to revive the words into meaning at the other.

When that doesn’t happen, you get scenarios like these: “Let’s have lunch sometime,” says Mr. Big Shot.

“Yes, that would be great!” says Little Worm. “Next Tuesday?”

“Ermm, actually, I’m quite busy next week… In fact, I’m tied up the rest of the month.”

Or “I’d like a red dress that flatters my figure,” says the shopper. She looks at the billowing red tent the saleswoman produces and says, “That is not what I meant. That is not it, at all.”

Natural language rules?

This is all to set up a series of posts on a current fascination of mine, pattern recognition. Pattern recognition means that you recognize a common pattern in a variety of instances…and that you can also produce instances to illustrate the pattern (which of course is exactly what I am trying to do here – illustrate a general theme of pattern recognition with examples).

Most computer programs say “if A, then do B.” Pattern recognition helps you determine whether A is true.

Pattern recognition takes a variety of forms, from object recognition and facial recognition to natural-language processing, which might more aptly be called “meaning recognition.” Pattern recognition ranges from recognizing a person in a crowd (useful to certain government agencies) to recognizing who’s likely at fault in a dispute, who is probably committing fraud, whether Juan’s a good match for Alice next door, which people will like a certain movie, what pitch is most likely to land a new advertising account, who designed a particular dress.

The inputs range from images to descriptions of behavior and numerical data, to natural language. You can do a lot of pattern recognition just with statistics, but only if you have enough data – and outcomes – or models, to start with. (That’s partly why I’m so hopeful about pattern recognition; there is more data everywhere, from people’s buying habits to GPS records of their movements, sensor data about all the things we see, electronic medical records that someday will follow a few standard formats so we can match behavior, therapies, genomes and outcomes.)

Some people are good pattern-matchers without ever articulating what they do; some (yes) recognize and can explain exactly what they are doing. (That is, in tech-speak, some people work like a neural net, producing results from a black box, while others work like an expert system, following explicit rules.)

And other people can read pattern-describing self-help books till they are blue in the face and still not recognize the situations in which they should apply the advice. Consider this piece of advice, for example: “Don’t ever ask a prospect who has said no to change his mind. Just give him a new proposition that he can agree with.” That’s how pattern recognition by a good salesman – and non-recognition (by the prospect who agrees with something that restates what he rejected before) – can work in business.

But back to software. A couple of weeks ago Stefanie Olsen of News.com wrote a