ZFS/Flash and the climate debacle

ZFS/Flash and the climate debacle

Summary: One of the striking things about the Oracle presentations this week was the extent to which the new technologies our perception of what can be done within reasonable cost constraints. Five years ago it was possible to imagine storing data about every cubic kilometer of the earth's bio-sphere - but you couldn't actually do it at a supportable cost. Today you can - and there are lots of other projects that just weren't do-able even last year, but are now.

TOPICS: Data Centers, Oracle

Here's something I wrote back in 2005:

It should be possible, for example, to build or modify coal-based power plants to intentionally inject "tuned particulates" into the upper atmosphere, thereby reducing insolation and causing global cooling.

The downside, of course, would be that the effect would be uncontrollable at any but the most aggregate level -and that we couldn't easily turn it off if we notice the other planets, a year or two later, starting to cool. In other words, there are obvious solutions, but grabbing one without understanding the problem might turn out to be even dumber than the anti-nuclear protests of the seventies that led to the carbon economy that's supposedly the cause of global warming.

Fortunately, there is a far better technical solution on the horizon, one that's unambiguously desirable but nicely illustrates how closely technology and politics are linked.

The earth has about 509,600,000 square kilometers of surface area, roughly 71% of it water. Extend our area of interest 3 KM down and 12 up, and we have about 7.644 billion cubic kilometers within which climate is of direct importance to us. Heat and material transfer functions for most of the materials found in that volume are well understood, thus there are no theoretical impediments to modeling the effect that an isolated increase or decrease in solar energy input to a cone cut across these cubes will have on the cubes themselves and thence on their neighbors.

There are a few practical impediments to extending that model to cover the globe, but the theory's all there. What's missing is the both the data and the computing capacity needed. Make those available, however, and it should be possible to fully predict the effect in San Francisco next year of man made cloud cover in Beijing this year.

Getting the data is a matter of being willing to spend the money -the lack of surface differentiation in much of the globe coupled with the availability of space-based sensors make that much less challenging or expensive than it might appear. The problem has been that the combination of processing power and storage needed has not been available at any price - but they soon will be. Both IBM's grid on a chip and Sun's SMP on a chip offer the potential to do this: both directly and in terms of the computation needed to enable data reduction to the point that the required storage becomes practical.

That threshold of feasibility probably won't be crossed this year, but it almost certainly will happen within three or four years.

(Note: I was wrong at the time to trust the data enough to conclude that some warming was occurring, and also wrong about the size of the problem: recent research on energy transfer from solar wind suggests we need to go up another forty kilometers or so.)

At the time, however, my big concern in terms of actually doing this was that the technology needed to efficiently compress, store, and retrieve the data didn't exist.

But did you catch some of the numbers the Oracle team was bruiting about last Wednesday? 60 Terabyte data warehouses? Single tables with three billion rows? sub-second performance on Oracle database machines with multi-terabyte tables?

The contextual bottom line here on what they were saying seems simple: between CMT computing, flash, and ZFS this job doesn't seem out of reach on either practicality or cost grounds anymore.

So why do it? Well, there's an obvious reason: the meltdown taking place in the global warming advocacy business as one alarmist after another proves to have been lying and cheating his way to personal fame and fortune tells us a lot about the people involved, but nothing at all about the reality or otherwise of the threats humanity faces from climate change.

Personally I'd bet an incoming Maunder minimum (mini-ice age) over another MWP (Medieval warming period) - but that's a bet: not a certainty because the combination of theory and data needed to be sure simply doesn't exist.

The best bet, of course, is that nothing significant is happening: that the next hundred years won't be significantly hotter or colder than the last hundred; but, again, discrediting the people phonying up data - and if someone had told me even late last year that the IPCC would be caught selling WWF opinion pieces as refereed science I'd never have believed them - tells us they have no evidence for their warnings, but says nothing about whether those warnings, in either direction, are right or wrong.

What we need, obviously, is good data on which to build, and against which to test, the theories we need to develop before we can understand the problem and make informed predictions.

This project will spin out all the data we need while making it possible to do next hour, next day, next month weather predictions with no more theory than what's already known about energy transfer and material reaction to it. And that, as MArtha would say, would be a very good thing - not to mention, of course, one fantastic opportunity for Oracle to do its stuff.

Topics: Data Centers, Oracle

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.


Log in or register to join the discussion
  • The absence of chaos

    My understanding is that the climate is a chaotic system ... so your gathering and processing increasing amounts of data will be as futile as it has been for weathermen since microprocessors were invented. The butterfly's wing beat in Beijing will send an avalanche to destroy your new hedge.

    Notwithstanding all the punditry and charlatanism, surely the safest bet is to begin a systematic reduction in the consumption of the planet's resources ... to counter the well understood and continuously demonstrated greed of man? Just in case we don't have a complete understanding what effect dumping millions of tonnes of toxic waste will have ;-)
    • Re:The absence of chaos

      No, climate is not chaotic, weather is chaotic. Climate is average weather, and the average of anything, even chaos, is not chaotic.

      The simple fact is that we have spent over a hundred years putting sequestered CO2 back into the atmosphere. CO2 is a greenhouse gas. Rising levels of CO2 will cause atmospheric warming. It is only a question of how much and what to do about it.
      • Sorry guys, but no

        1) Co2 is not a greenhouse gas. In fact the whole idea originated as a metaphor to describe what they thought was happening on Venus - and then the term was used about water vapour, not CO2.

        This belief is starting to look like an urban legend: you can find thousands of reliable sources attesting to it, and no actual evidence. Everyone quotes someone else and the closest to support is a reslt showing that a crock of air holds less heat than a crock of air + extra CO2. Proof! except that CO2 is denser than air and so the crock test is just that: a crock.

        2) the earth is not a lifeboat hurtling through space - it is very nearly a closed system in terms of consumption. When you piss overboard from a lifeboat that water's gone, when you eat the last chocolate bar, you can't get more.

        However, you can't pee overboard in lifeboat earth: you can make water dirty, but you can't get destroy it. You can burn oil, but you get hydrocarbon gases and water - all you've done is extracted some chemical energy from bonds; energy we could put back in if we wanted to.

        Bottom line: the people on a lifeboat can run out of resources, people on earth cannot.
        • Sorrier but you die ...

          "Bottom line: the people on a lifeboat can run out of resources, people on earth cannot."

          Whilst I'm sure the petrochemical industry will be cheered by your erroneous prediction ... and the space programme will be cheered by the fact that you have found a way to effeciently import matter from an extra-terrestrial source (please send them the location of said fossil fules and design of the propulsion method for exo-matter transfer) ...

          ... here on lifeboat Earth (big lifeboat but closed except to the sun's rays nonetheless) this pragmatist views the chances of man destroying 'the natural balance' as the greatest threat to the planet - one we need to start acting on now.

          You can keep pissing until you come within my control ;-)
        • say what?

          "1) Co2 is not a greenhouse gas. In fact the whole idea originated as a metaphor to describe what they thought was happening on Venus - and then the term was used about water vapour, not CO2.

          This belief is starting to look like an urban legend: you can find thousands of reliable sources attesting to it, and no actual evidence."

          Are you serious? Do you know what you're talking about or is this another murphy hatchet job? Look Up Joseph Fourier,Svante Arrhenius, John Tyndall,Gilbert N. Plass, Roger Revelle, David Keeling, etc.

          Perhaps you could list some of these 'thousands' of 'reliable' sources who proved those scientists wrong and demonstrate that CO2 is not a greenhouse gas.
          • Don't Bother

            Murph thinks that C02's density has something to do with its quality as a greenhouse gas, if that tells you anything about the quality of his research or the 'research' that he's been reading.
          • I know

            Yeah, I know. It matches the quality of his writing on just about any subject. Makes you wonder if he chose the name murphy because he's a living example of murphys law.
        • CO2 as a greenhouse gas.

          I think you'll find that the first public speculation of CO2 as a greenhouse gas was voiced in Victorian time by John Tyndall.

          Although there are earlier references I've read but I've failed to note where.
          Otherwise fair comment.
          The world has all the resources we can ever need. All mankind needs is the wit to use, then reuse, and stop polluting the environment.
          Garbage is resources in the wrong place.
      • Crash (like your argument)

        The average of a chaotic system may indeed be reasonably predictable over a time period ... until you come to that feature of chaos which will prove more than unpalatable: the crash.

        I fear you will happily cross the river with an average depth of 1 foot. Unfortunately after wading through the first shelf at 6 inches you will drown in the central 10 foot whirlpool.

        The film 'The Day After Tomorrow' is the sort of thing I have in mind. The problem is that man's tinkering may plunge us into a situation which nature cannot recover from (or where the equilibrium does not sustain life).
        • Chaotic attractors

          Every known chaotic system has a central tendency - so
          if climate is chaotic, it's general trend would show long term consistency.

          However: there's no evidence that climate is chaotic. On the contrary, it's the direct result of some combination of known and unknown forces acting over time; i.e. 100% predictable if we understood all the forces and had data on past actuality.

          One of the fun things here is to reflect on the sad truth that lots of people speak knowledgeably about global temperature change - but no one knows how to measure it, where to measure it, or how to process the resulting time series supposing we could measure it. A lot like AI researchers: trying to recreate inteligence without defining it first.

          PArt of the point of my proposal here is to address this: real data, known measurement biases, known time affixation.
          • Upsetting the trend

            "Every known chaotic system has a central tendency - so if climate is chaotic, it's general trend would show long term consistency."

            There is no problem with the natural trend ... nor that you are going to piss on it ... the problem is that you are going to deplete it to near exhaustion, irradiate it to boiling point, and then some scumbag is going to ckfu it up ... and then I think the atraction may be fatal, or at worst abysmal :-(
          • Slow to understand

            "100% predictable if we understood all the forces and had data on past actuality."
            So if we found out that climate was chaotic then we would be able to say that it could not be predicted beyond a few centuries ... and that we had screwed up so badly between 1900 and 2100 that all was lost?

            Murph, Apple will have the iPad v10 and the situation will be lost before man counteracts his innate greed. It's like computer security - entirely dependent on psychology, not technology.

            I agree that we need more data ... but we KNOW FOR SURE that man is greedy so lets start on that now in case we find its too late. Sorry if your iPad purchase has to be delayed a couple of years. So friggin' what.
  • Wait, wait, wait ... oops too late.

    Your proposal has great potential - of confirming disaster in hindsight. IF GCC is correct, then we have already waited too long. It is against human nature to voluntarily embrace change in a short time period. It does resemble "faith".

    I tend to look at the problem as a statistician would - throw out the extremes and focus on the 80% of data "in the middle". So discounting the envrioloonies and tea-jerkers making the most noise - what does the data say? EVERYTHING I read (and judge) points toward global warming. Seeing whole ice shelves the size of Rhode Island break up, Siberia covered in lakes instead of ice, unchecked trillions of tons of Co2 being released - any reasonable person would come to the same conclusion. There is no doubt.

    "Deniers" come from the perspective that they don't trust liberals and making changes cost too much money - BEFORE they even look at the data (if they ever do). They believe that a single counterexample disproves an entire theory. They look for like-minded people and try to argue for democracy i.e. x number of people don't believe and their numbers are growing SO the theory is wrong. Never mind that only 25% of the population are qualified to objectively examine the data - so having boatloads of Barnum's People making noise about how the theory is wrong holds no weight.

    Hurry up and wait will not work this time. We owe it to our children to get it right now - not to hand them a cesspool and say "good luck".

    What to do about it? Well, THAT'S the REAL issue anyway. A carbon tax is the most efficient way to do this - so of course Congress is doing "Cap and Trade". Hey, NO ONE like more taxes - but Congress enacts new taxes all the time, so what's another one? If you stop this one, there will be another one down the line. It's what you DO with the taxes that is important. If you tax carbon emissions, and then turn around and use that money LOWER those emissions - then the tax pays for itself. A burden on electric generation turns into tax breaks for them to clean up their act.

    In the end we get cleaner air, a new green economy and avoid thermogeddon. Of all the B.S. that my tax dollars are spent on - this will be the best "investment" for mine.
    Roger Ramjet
    • Umm, about that data

      The "data" you cite would reasonably draw your reaction except for two problems:

      - much of that data was faked
      - what wasn't faked, was forced out of context

      Lakes in Siberia, glacial retreats? the annual ross ice shelf flaking off? all faked/exaggerated.

      Context: we have less than forty years of satelite imagery on artic and antartic ice. We know that the NW passage was open in the 1930s - (and because we know the polar bear didn't evolve in the 300 years between the Medievel warming period and the first artic expeditions, we know it's not threatened by much more warming than even the alarmists are predicting) - and we know that in the past atmospheric CO2 has always risen after warming, never before it.

      Consider stuff like this:
      The idea that 2,500 "scientific expert reviewers" provided feedback about [the IPCC's 4th Assessment Report] during its pre-publication phase sounds awesome. But many of those people aren't scientists at all. They're professional activists in the employ of environmental organizations.

      The expert reviewers who had input into just one portion (Working Group III) of the IPCC report are listed in this 8-page PDF. They include three Greenpeace employees, two Friends of the Earth representatives, two Climate Action Network reps, and a person each from activist organizations WWF International, Environmental Defense, and the David Suzuki Foundation.
      This is from Laura somebody at

      I have no idea what credentials she may have, but what she says can be checked, and when checked generally check out - which is more than anyone can say for the IPCC.
      And do you remember that banning DDT cost hundreds of millions of lives (mostly in Africa, of course, so of no account to liberals - Jim Crow, you may recall, was a Democrat; Lincoln a Republican) and was based entirely on faked data and public hysteria? This might (or might not) suggest more of the same:


      Are these charges true? I've no idea - but the point is that you can't believe anybody about any of this stuff without doing extensive checking.

      People (like Laura, above) are now doing that on the climate stuff and what they're finding says that people like you have been deliberately misled - by the same people who got DDT banned, the energy industry hung out to dry, and nearly got nuclear power killed off.

      My proposal here is one possible next step in this debunking process: once enough people understand how bad the deception has been, the next obvious step is to get some real data....
      • source?

        "The "data" you cite would reasonably draw your reaction except for two problems:

        - much of that data was faked
        - what wasn't faked, was forced out of context

        Lakes in Siberia, glacial retreats? the annual ross ice shelf flaking off? all faked/exaggerated."

        There are three working groups WG1 is the one that deals with the science. Can you provide sources showing WG1 data was faked?

        The IPCC Working Group I (WG I) assesses the physical scientific aspects of the climate system and climate change.

        The IPCC Working Group II (WG II) assesses the vulnerability of socio-economic and natural systems to climate change, negative and positive consequences of climate change, and options for adapting to it.

        The IPCC Working Group III (WG III) assesses options for mitigating climate change through limiting or preventing greenhouse gas emissions and enhancing activities that remove them from the atmosphere.
      • Let me summarize Murph's argument

        If someone on the Internet says global warming is a scam, it must be true. Countless scientific journals got it all wrong! 'Donna Laframboise' says so!

        Don't worry Murph. There are a lot of loonies out there on the interwebs. you'll have an endless supply of unqualified folks like Anthony Watts to keep an endless supply of BS flowing into your brain.
        • You're getting at (or illustrating) something important here

          I don't know who "Donna Laframboise" is - I do know who Tony Watts is - but I don't think it matters because there's a key difference between these two and most of those 'scientific journals."

          The difference is this: follow up on the claims made by the debunkers, and they are demonstrably true. Watts made his claim to fame via the surface stations project - and when you read his stuff you can verify what he found from numerous sources - you can even go look at the local weather station to see it for yourself. When laura lists WWF or Greenpiece writers quoted by IPCC as reliable science you don't have to believe her: you can get the IPCC stuff and check for yourself.

          In contrast one of the major revealations from the hadley files is that the bad guys poisened the climate journals - and did neat things like citing their own unpublished papers as if these were peer reviewed science and then using the clout this gave them with outsiders to deny publication to contrary voices.

          Take anything Laura says, whoever or whatever she is, and check it - take anything Hansen says and check that - and you'll discover an important difference: the stuff from the nobody checks out, the stuff from hansen does not.

          This, I think, is an important difference: and not just here; it's true across the board and represents a new era in effective citizen journalism.
          • if what you say is true, it's nothing special, oil companies do the same

            I'm pretty convinced with the concept of human CO2 caused global
            warming threatening our existence.

            If scientists have acted this way, (in either direction, for or against),
            they should be barred from the profession.

            It's part of a much wider problem of a lack of integrity. Look at banks,
            politics. Why should science be any different.

            Imagine being faced with Nazi germany with the incompetence and
            self-serving arrogance that abounds today.
            We really don't have a chance.

            We need to fix society from top to bottom in terms of morals. Maybe
            this is our new Sodom coming in fast.
          • Two wrongs do not make a right

            biut the the zinger comes from your statement:

            "I'm pretty convinced with the concept of human CO2 caused global warming threatening our existence."

            Ask yourself why you think this -then check your sources for this certainty.

            Remember, please, that CO2 is a critical plant food - people pump it into green houses to get happier
            plants - and that we routinely get better corn crops downwind from big coal plants than upwind of them.
          • you can now sail over the north pole without an icebreaker

            I've read so much of the climate science and evidence, the reports and
            so on.
            Events seem to be outstripping even the worst predictions.
            That's why I am certain.

            "Remember, please, that CO2 is a critical plant food - people pump it
            into green houses to get happier plants - and that we routinely get
            better corn crops downwind from big coal plants than upwind of them"

            This is completely irrelevant. If the Sun's output doubled over 10
            years, for the first few days, plants would be growing well, and we'd
            all be going to the beach. Sometime in the first year we'd be running
            around on fire.