Performance problems?
Contents  Introduction Mercury Load Runner IBM Rational Tester Segue SilkPerformer NeoLoad Specifications How we tested Editor's choice About RMIT | ||||
We have found that most of the issues associated with an application are related to configuration rather than hardware. So by increasing the CPU speed, adding more memory, or increasing the bandwidth you're not necessarily going to shorten your response times, and by only benchmarking the actual application you're not going to be able to pinpoint the root cause of any issues. You have to use diagnostics to find the root cause of bottlenecks. Diagnostics usually come built in with the tool or you can get them as add-ons. Some even interface with third-party diagnostic tools.
Most of these products are a part of a lifecycle management product. They employ a distributed infrastructure where programmers and testers can come closer. This is also a way of encouraging testing in early stages rather than leaving it till the end when it's too late. Functional testers can also develop scripts that will be able to be executed by performance testers, cutting down the time it takes to unit test or test a system from end to end.
To make it possible to perform end-to-end testing across every aspect of your enterprise application, you need a tool that can support your application's environment, Web services, and development frameworks. All the tools tested here support a large range of technologies but you will want to confirm with the vendor.
Scripting
Scripting used to be the most time consuming part of performance testing but now it is easy. These tools are supporting more and more protocols, which makes it simple when testing a range of different technologies, all you have to do is point and click to record a user action. Features like auto-correlation automatically detect and handle dynamic values that may have been recorded. Dynamic values such as usernames and times and dates can be parameterised with variables by a simple replace. There's no coding involved, and the tool takes snapshots of every page so you can go back and look at what you just recorded. One thing you can't do is view the script being played back just the way you recorded it -- you can only do this with functional testing tools.
Runtime
The main factors associated with running the scripts include the type of load you are going to place on the application. One can be a ramp up of load so you can see when the application is going to break. Sometimes this is not the best way to test your application as it will exhaust your hardware resources and mask the real problem.
Another approach is ramp up ramp down -- say you ramp up the number of users to 200 then sit on 200 users for 30 minutes then ramp down again. You could then separate the results into three lots -- the ramp up, the steady flat load, which ran for 30 minutes, as well as the ramp down. You would then be able to extract the information, which is most important. In most cases it's the steady load of 200 users you would be most interested in.
You can also do volume testing or run test scripts over and over with a predefined number of virtual users for a predefined time. Rational's product allows you to dynamically throw additional users onto an already running test. The tools are very flexible in how you can load up your applications. While the test scripts are running it's important that you monitor your hardware to see how it is coping. Whether the tools use agent or agentless monitors, it's essential to know the health of critical machines.
You're also going to want to be able to keep an eye on the health of the test machines which are generating the loads, just in case they are maxing out. If they are working too hard you're going to want to introduce another test machine to reduce the load.
Diagnostics
Diagnostics are crucial when you need to discover why your application is not performing and what the root cause is. It basically gives you an insight into your application, allowing you to drill down on the Web transaction that was returning the longest response time, then drill deeper to the method that the application is using to carry out the task, and then further to the SQL call that is the root cause of the delays. But beware, you will only find these capabilities in the more expensive tools.
Reporting
Once all the testing is done you will want to create detailed graphs and explanations. It's also good to be able to create your own templates so you can produce the same report without having to go through the whole process of setting up a report. Being able to overlay graphs and add annotations hence a good thing to do, though some of the graphs produced by these tools can be hard to understand. And when it comes to publishing the results your going to want reports in HTML or PDF.
From past experience, we have spent most of our time analysing the data generated from these tools. You don't want to spend too much time setting up and configuring the product and then spending hours upon hours creating scripts, you want to fast track past all this and get to what matters most -- the performance of your application.
Contents Introduction Mercury Load Runner IBM Rational Tester Segue SilkPerformer NeoLoad Specifications How we tested Editor's choice About RMIT | ||||
Mercury is without a doubt the biggest player in software automation. They won more than 50 percent of the market with the company recording revenues of $685 million last year. None of the other vendors in this space even came close.
We had a Systems Engineer come to the TestLab to help install Load Runner, but it's actually a piece of cake (it was the only way Mercury was going to be involved in this evaluation). They also recorded the scripts for us, played them back, found the faults on our Web server, did the analysis and then created a report for us. They pretty much did our job then uninstalled Load Runner from our PC before leaving the Test Lab.
This made it a bit hard for us to run some more tests later on as we were restricted to five users on the 10-day trial version you can download from their site. But luckily we've used Load Runner on a number of occasions before so we can make some fair comments on this product based on previous experience.
Load Runner gives you three main options when you first start the application. You can choose to create/edit scripts, run load tests or analyse load tests. Clicking on create/edit scripts opens up the Mercury Virtual User Generator, which gives you a variety of protocols or technologies to choose from. We chose the Web (HTTP/ HTML), which emulates the communication between a browser and Web-server.
The next thing was to record a user action. All we did was hit the start record button and enter the destination URL. An Internet Explorer browser opened and from here we booked a flight. Once we finished our transaction all we had to do was stop the recorder -- the application displays the results. The next step was to check for correlations then parameterise variables like login and password so we could simulate different user names logging into the system to book a flight. To make sure there were no errors in the recording we replayed the test with only one user.
We opened up the controller, which is used to execute and manage the test scripts. From the Design screen we were able to set the run time settings like the number of iterations we wanted the script to run through.
From the Controller we set up a schedule for the load test, deciding to take the ramp up approach. There are other options available like ramp down or load all users simultaneously, run the test for a set duration, or set the users to run indefinitely. Again there is a good amount of flexibility here.
Clicking the start scenario button kicks off the virtual users and from the run screen you can watch in real time where the tests are up to and how your system is performing under the load. The layout of the run screen was excellent. Every bit of information you need to monitor is on this screen. We were mainly concentrating on the response times for our server. By clicking on these graphs or on a particular transaction we could drill down to see what was causing the long delays. You can get down to the method and even SQL call.
We could quickly see that after 10 users the performance of the system dropped dramatically, and after examining the information that was coming back from the Apache monitors we realised the number of threads was limited on the Web server. We were able to correct this configuration and after we ran the test a second time discovered the database was a bottleneck. More is explained about this in the "How we tested" section.
The next step was to analyse the results. We were able to build custom graphs, adjust the granularity, scale, apply our own filtering, and create a Web page breakdown of the tests. We annotated the graphs and saved the results as a template so next time we wouldn't have to customise the report again. We could export the results into Word or in HTML.
The great part was how each of the graphs was supported by a brief explanation. The report also provided a glossary of the technical terms used.
Mercury indicated that there are options for more comprehensive type reporting but we were impressed with what we saw in the standard package.
| |||||||||||||||||||||||||||
Contents Introduction Mercury Load Runner IBM Rational Tester Segue SilkPerformer NeoLoad Specifications How we tested Editor's choice About RMIT | ||||
The IBM Rational Software Development Platform uses an Eclipse-based common development environment that is shared by number of developer-, functional-, and performance-testing applications. All of these products have the same UI, called a workbench, and each product adds functionality to the workbench by contributing plug-ins.
IBM was a little more flexible than Mercury giving us a longer licence and 500 users to throw at our application. A company person performed the installation, which was a little painful. We installed the licence server on the same machine we installed the performance tester. It had us puzzled why we couldn't activate the 500-user licence -- we later worked out that we should have installed the licence server on a different server while pointing the performance tester at the licence server.
Creating a script isn't too difficult but it was a little easier using Mercury or Segue. First we created a project, then a performance test, and then we were able to record a performance test. The application connects to the licence server first and opens up a Web browser -- all you have to do is point and click to record your test. Once you have completed the task you can close the browser and the application generates the script code.
There is a tree-like menu in the centre of the workbench, which displays each of the pages we visited and transactions, as well as an element viewer on the right which details all the elements of the page and includes any variables like logins and passwords.
All variables are highlighted but only text is displayed, not the actual HTML page. Right clicking on the highlighted data allows you to add a data pool so you can vary your test data. The next thing to do is add the test to a schedule to represent a workload. We thought this would have been an easy task but had to search around in order to work out how to do it.
We got a little stuck but Rational offers and excellent wizard that can guide you through the steps. There are tutorials and samples built in as well as Web resources that should get you out of any confusion.
We had to create a schedule and add users to it. From here we customised the load test by adding a delay between starting each user. After an elapsed time we stopped running the schedule and modified the duration of the think times. We then configured the behaviour of the users. Rational relies heavily on loops and iterations to carry out tests which can be a little confusing at first. In Rational's case there are a few extra steps you have to do before you can kick off your test.
During test execution, you can dynamically throw additional users at the test. This is great, especially during ramp up, and saves you from rerunning tests. We didn't find this feature in any of the other applications. The user interface was good. If you're familiar with Eclipse then you won't have any problems getting a feel for this tool. The layout of the controller wasn't too bad, the only concern was that we had to run our screen at a higher resolution to see everything without having to scroll up and down.
By running the performance schedule we started the load test. We could then view the test summary, page performance and throughput, response times, server health, and more by clicking on the tabs to go from one screen to another.
The page performance graph was the most useful during our test, allowing us to monitor which specific page was taking the longest and in our case, since we placed a bottleneck on our Apache server, it was the main welcome page that recorded the longest delays as the apache server would take on more than 20 connections.
You can't drill down further from the real-time graphs like you can with Load Runner, and we preferred the real-time graphs that were produced by Mercury. To look at the reports you view the completed runs and select up to six different reports -- all can be customised. There is what is called -generic counters" which allow you to add overlays to existing graphs. You can filter data so you only display data with the highest or lowest values and more.
If you want to create some pretty reports to give to your boss you can but this product doesn't do it as well as Mercury and Segue. We also found creating a full-blown report takes a quite a while.
| |||||||||||||||||||||||||||
Contents Introduction Mercury Load Runner IBM Rational Tester Segue SilkPerformer NeoLoad Specifications How we tested Editor's choice About RMIT | ||||
Segue has been a long-time player in this area. Unfortunately we couldn't find any info on Rational to compare the two in terms of revenue but Segue recorded net revenues of an US$8.9 million in the second quarter of 2005, which was an 11 percent increase over the same time last year. If we are a bit generous and say total sales for the year nears $40 million, that's still less than six percent of Mercury's total sales last year.
We didn't have an engineer available to visit our Lab and help us with the install, because currently Segue don't have any real representation in Australia but will by the end of the year. It has a great support structure. Out of Belfast, Ireland, they provide support team between the hours 18:00 and 12:00 AEST. They also offer user forums and knowledge-base access for many resolutions as well as an online case logging service.
We were able to download SilkPerformer from the Segue Web site and pretty much ran the install without any help from Segue.
All up it took around 10 minutes to get it up and running and to run our first test script which says a lot about how easy it is to install, configure, record, and run a test. There were no proxy issues like with Rational. SilkPerformer automatically checks to see if you are going through a proxy server by using the details from your Internet Browser.
There were two main components of SilkPerformer -- the workbench and developer workbench, both similar. The Workbench is where you run your actual tests, while the developer workbench is where you can allow your developers to create and modify scripts as well as test them with up to five virtual users.
The Workbench provides an easy-to-use workspace and we must say the GUI does somewhat resemble the LoadRunner GUI which is good. It would also be easy to move from LoadRunner to SilkPerformer and vice- versa. When we clicked on the model script button, our default browser launched and we were able to start recording our script. A little recorder appeared on the top right-hand side of your screen and from here you control the recording session. Again, this was done in a very similar fashion to LoadRunner. As soon as we finished recording our script SilkPerformer displayed the source code for the script.
We checked the script by playing it back for one user. Then by using the true-log explorer we could analyse and customise the script. This is a very powerful area where you can find differences between your test and recorded session. From here you can also customise user data with a simple right click on the field you want to make variable. You can add verifications here to verify whether or not your application is responding with incorrect data values or error messages. This does, however, place a small overhead on the performance of your application, which is something to be wary about.
From the Workbench we configured the workload, allowing virtual users to have different connection speeds and emulate different browsers. We only saw this sort of customisation with LoadRunner. Before we ran the test with multiple users we checked the health status of the agents, which are the hosts that generate the load. In this case it was going to be the local host or the machine we installed SilkPerfomer on. By clicking on the agent's icon we were able to view a system summary for that particular agent, as well as view the maximum number of virtual users this agent can generate.
We thought this was a great way of knowing when you need to install a new agent on another machine. With most of the other vendors you have to monitor the CPU usage and memory and make a decision whether or not to employ another machine to generate the load.
There are a few ways we could distribute the workload -- by using an increasing load, steady state, dynamic, all-day, or queuing. Again this is very easy to use and the images that display each type of workload are great in helping you understand how the load really works. By running the test this opens up the performance explorer window. Here you will find a set of windows, which give you real-time information during the test.
We kept having to flick back and forth between the workbench and performance explorer window, which was a little annoying as both screens display important real-time information. It would be ideal if this could be combined into one screen. As far as diagnostics go, SilkPerformer takes a less intrusive approach, if you need to drill down to the source code, for example, you have to purchase an add-on application.
To open up a report simply click on the explore results button and export to HTML. The report is comprehensive and you can add text next to each bar graph. There are various ways of customising reports; you can do overlays, set time bound histograms, and templates.
| |||||||||||||||||||||||||||
Contents Introduction Mercury Load Runner IBM Rational Tester Segue SilkPerformer NeoLoad Specifications How we tested Editor's choice About RMIT | ||||
Okay, we've covered probably 70 percent of the market with the previous three. Unfortunately Compuware and Radview declined to take part in this evaluation; we would have had the top five in the market otherwise.
We've read some good things about NeoLoad so decided to try it out. Our first impression was it's a little light and doesn't quite look like an enterprise tool -- the main reason we didn't include it in the main review.
Neoload, however, doesn't try and position itself up there with expensive high-end testing products, nor does it compare itself with the really cheap tools which are limited in features -- it sits somewhere in the middle. We think it should do well in this space.
The tool is perfectly marketed for businesses that don't have a large budget but want the confidence that their applications can scale well.
Leasing a 500-user licence for a week, which may be all the time you need to perform your testing, will only cost around AU$900. Pretty cheap but how good is it? It's a piece of cake to install and if you have never done any form of load testing before you will be surprised how quickly you can become expert using this software. It supports the most popular Internet technologies but does lack in diagnostics.
The reporting is excellent -- we were surprised at the professional PDF reports NeoLoad can generate. It can also export your reports to HTML.
The annoying thing was that we couldn't launch a load test for a predefined number of iterations. We had to end the test after a certain time but we were advised just before going to print that the new version v1.1, which only just recently came out, features an option to run your tests by iteration.
There are a few other new things you can read about here, but the most important feature added is the iterations.
It doesn't have any local support but we doubt you will have any major problems getting up to speed with this product. It does, however, offer international phone support as well as e-mail support which is probably all you will ever need.
| |||||||||||||||
| Product | IBM Rational Performance Tester 6.1 | Mercury LoadRunner 8.0 | NeoLoad v1.0 | SilkPerformer |
| Company | IBM | Mercury | Neotys | Segue |
| Phone | 02 9354 4000 | 02 8273 1999, 1800 837 848 | +33 (4) 91 82 84 90 | +1 781 402 1000 |
| Website | www.ibm.com /software/rational/ | www.mercury.com/au | www.neotys.com | www.segue.com |
| RRP | Base Performance Tester (incl 5 virtual users) is around AU$3000 for authorised user licence. VU Packs start at around AU$4200. | Project based pricing starts from AU$25,287 for min three-month term | AU$2790 for one month lease with 500 virtual user licence | US$8780 for 500 VUs per month + Add Ons |
| Support | Web support free. E-mail and telephone is included in maintenance subscription during normal business hours. High priority issues and e-mail available 24/7 | 18 percent of list price | Australian Time Zone coming soon (September) with a local partnership. No additional cost | 18 percent of list price: hours are 9am GMT - 3am GMT Mon-Fri. Includes an unlimited technical support assistance, support Web site |
| Visual scripting | Yes | Yes | Yes | Yes |
| Auto correlation - dynamic values | Yes | Yes | Yes | Yes |
| Ramp up/Ramp down | Yes | Yes | Yes | Yes |
| Dynamically add users to already running load test | Yes | Yes | No | Yes |
| Simulate link speeds | No | Yes | Yes | Yes |
| Breakdowns of Web transactions | Yes | Yes | Yes | Yes |
| Diagnostics for J2EE .NET | Yes | Yes | No | Yes |
| Templates | Yes | Yes | No | Yes |
| Export to PDF/ HTML | Yes | Yes | Yes | Yes |
| Support for ERP/CRMâ€"SAP, PeopleSoft, Siebel, Oracle | Yes | Yes | No | Yes |
| Support for Web browserâ€"HTTPS/ HTML, Macromedia Flash | Yes | Yes | Yes | Yes |
| Support for Terminal Servicesâ€"Citrix Metaframe | Yes | Yes | No | Yes |
| Support for middlewareâ€"CORBA, J2EE, .NET | Yes | Yes | No | Yes |
| Multi platform (Windows, Linux, Solaris, Mac) | Yes | Yes | Yes | Yes |
| Target market | Model is structured to cater for any organisation | Enterprise | SMB/Enterprise | Medium to enterprise accounts; SilkPerformer Lite -- smaller accounts |
Contents Introduction Mercury Load Runner IBM Rational Tester Segue SilkPerformer NeoLoad Specifications How we tested Editor's choice About RMIT | ||||
We setup two Pentium 4 2.6 GHz machines with 1GB of RAM running Windows 2000 Professional. These machines were also connected to each other through a 10/100 switch.
On PC 1 we installed an Apache Web Server, which powered a mock travel Web site. While on PC 2 we installed each of the Load Testing Tools. We had the vendors send down an engineer to help us install and run up some basic load tests just so we could shorten the time spent to learn each product. From the mock tours Web site we registered a couple of user accounts. These accounts allowed us to later book a flight through the tours system. When creating a new account we had to fill out a form with some basic information about ourselves such as first name, last name, phone, e-mail, address, city, state, post code, country, and we choose a user name and password. Once we set up these users we were able to book a flight. We did this using the record feature in each of the applications. Hitting the record button on each of the tools launched an IE browser, which we used to navigate to the destination URL.
We could then see the main page on the tours Web site -- we logged into the system and booked a return flight from Sydney to San Francisco. We then filled out our credit card details, confirmed the flight, and logged out of the system.
Up until now it was quite easy, so to make it more interesting we edited each of the test scripts or recorded sessions and looked at the way each of the apps displayed the script and checked to see correlations they made.
We parameterised the user name and password and randomised the credit card field. We sourced a list of user names and passwords, which the script used to simulate a large number of different users logging into the tours system.
When we played back the script every user booked the same flight on the same day. On a real system we would run into loads of problems simulating 500 users booking the same flight. The flight would quickly get full and users would receive different HTML pages from what they would expect -- a page might come back saying there were no flights on this day, and the script would hang because it wouldn't have an exit point. So what you would have to do is insert conditional statements (IF ELSIF) so when that did happen, the script would try another date (for example) in order to continue with transaction. You might also use a form verification just to make sure the script is in fact going to the right page.
As part of our test we had to decide how to place the load on the test server. The vendors each have their own opinion on how this should be done. We took the ramp up approach where we slowly increased the load by one every 10 seconds until we got to 50 users, then we ran the test for another five minutes.
We quickly discovered that we only had to simulate 40 users before we found the tours Web site started to fall down. We didn't get any failed transactions,which was good, but the delays were too long to consider this site acceptable for 40 users to use at the same time. The tours Web site uses flat files instead of a proper database to store booked flights -- this became the bottleneck. It particularly occurred when we had to secure the purchase of the flight or when the application was writing our flight details to the flat file that the response times increased to around about 70 seconds. Things like logging in and getting to the main page took up to 45 seconds.
Before we moved on to pinpointing the database as being the bottleneck we preconfigured the Apache Web Server so it would only accept 20 concurrent threads (ie. requests) at any given time. In Internet Explorer, when one window is open it uses two threads, so this means that only after 10 users will the system start to drop off in performance, leaving many users would get stuck on the main page. This is exactly what happened to us -- there was a large delay trying to access the main URL. As more users entered the mix, the delays became longer. We put this in there just to see if the vendor's software could pick up this problem -- we also ran the tests with 200 supported threads and checked out the reporting. We looked for any unique features that separated them and evaluated the platforms and type of applications they can support, the general setup of the controller, virtual clients, monitors, and ease of use.
Contents Introduction Mercury Load Runner IBM Rational Tester Segue SilkPerformer NeoLoad Specifications How we tested Editor's choice About RMIT | ||||
This large company is suspicious that bottlenecks occur during heavy transaction periods. The IT manager wants to monitor system loading and is not sure which product to choose.
Concerns: Recording and playing back transactions, supported platforms, reporting, pinpointing of bottlenecks, test script management, leasing arrangements with virtual clients.
Scenario winner and Editor's choice: Mercury Load Runner 8.0With standard Load Runner licensing, the licence is locked to a particular node but there is no charge to move around. With the enterprise-wide Load Runner licensing (called Performance Center), licences are pooled and each virtual user type is licensed separately. This can work out very expensively.
IBM's Rational Performance Tester doesn't cost as much as Mercury and it is more flexible with their licensing -- you don't pay extra for different protocols. Segue were also very expensive but are flexible with their licensing like Rational. Additionally Segue customers can "check out" a licence from the SilkMeter server onto a laptop for mobile computing (as they may want to work from home on a weekend or have to travel to a remote office).
Contents Introduction Mercury Load Runner IBM Rational Tester Segue SilkPerformer NeoLoad Specifications How we tested Editor's choice About RMIT | ||||
This article was first published in Technology & Business magazine.
Click here for subscription information.