The idea is that you make the center of your product open source, but put the rest under a paid license. This is supposed to make your venture capital backers happy. You gain the benefits of open source but customers aren't "stealing" the software.
This is not a new idea. Eclipse licenses are all about development companies sharing a common store and selling proprietary extensions. The whole idea of an "enterprise" product distinct from the "community" one is a form of open core in action.
Here, the idea gets a new twist. Your community is a community the way a golf club is a small town. (Or, in New York, a "co-op" building.) You have to apply to get in, and you have to sign a contract.
Another difference may be, in practice, that you can call yourself "open core" if what you open source is not really useful, if it doesn't provide something workable, if your extensions are thus necessary for the open source to help a customer make money.
This may please VCs, but I suspect it's going to end up being a niche approach.
That's because, once again, you're putting yourself higher up the "open source incline" in your search for profit. Proprietary extensions are inherent in a variety of BSD-type licenses, from Microsoft's to Eclipse's to Mozilla's. But there's a cost for that.
You can get businesses to ally with you under a BSD license. If you're doing the lion's share of the development yourself, it can even be fair.
But if you want small businesses and ordinary people to stamp out bugs for you, to participate in your forums, to do all the "scut" work proprietary firms have to farm out to India, you have to give them what they see as a fair deal.
That's not to say there aren't many products, even successful products, that are pure business-to-business plays. There are. But they don't fire the public imagination, don't have the mass market appeal, and thus don't have the VC profit potential of something that is truly open source.
There are many ways to skin the open source business model. Support, services, hardware, they're all perfectly valid.
But what I think the open core boomlet is really saying is that open source, as a way to sell software, is slowly falling off the venture capital radar.