Cambridge speedy software brings out the stars
Case study: How analytical software is helping the astronomy department
Cambridge University is speeding up astronomical discoveries by using analytical software to help trawl through its mountains of data.
The Cambridge Astronomical Survey Unit (Casu) - part of the university's Institute of Astronomy - is testing the Kognitio WX2 database system with legacy data.
Getting information out of a telescope and into a usable format is a laborious task that can take months, delaying the availability of data for analysis and research.
A standard data-set or image from an astronomical observation could contain anything from dozens to billions of celestial objects, such as stars or galaxies. Each of those objects will then have between 80 and 100 measurements assigned to it - such as its position in the sky, brightness and so on.
The Casu team analyse all of this data by searching through the database to pick out certain objects - for example using a search to find the number of stars of a certain temperature.
silicon.com Public Sector
Get the latest public sector news straight to your inbox. Sign up for the PS newsletter today!
Jim Lewis, senior research associate at Casu, told silicon.com: "When you have so much information then those sorts of searches take hours, if not days, and it slows down the discovery process."
In one test, WX2 searched through a database of between 600 million and 700 million objects to identify binary star systems where the two stars have very different surface temperatures.
Lewis said the current database would "just turn its nose up and die if I asked it to do this search" but WX2 managed it in 29 seconds.
Another advantage of the WX2 is it does away with the lengthy task of indexing astronomical measurements - a process that makes finding information within the astronomy department's databases a lot easier.
Lewis said: "You may have the data in there [the database] but it's totally useless until you can actually create the indexes, and that may take you weeks, if not months. But with WX2 you don't have to index anything because once the data is in the database it's ready to use."
By the end of 2007 Casu will be collecting up to 500GB of data every night and has a space mission due for launch in 2011 or 2012 that is estimated to generate up to one hundred trillion observations of individual stars.
Lewis added in the next 10 years Casu is running the risk of drowning in data and the current database management system will not be able to cope unless a technology - similar to WX2 or otherwise - is implemented.