Today we are inundated with articles on how to test this and test that. The one thing that a lot of automated testing framework makers don’t cover is the data. Some leave you to insert thousands of rows of data in order to simulate a realistic scenario? Another approach commonly used in the industry is to use customer data. “If it works for customer X, and they are your biggest customer, then it works fine.” This kind of thinking won’t cut it for very long in the real world. There are also other things to consider. For example:
- What if your customer didn’t want their data used in this way?
- Why wait until your big customer complains about the responsiveness of your application before making changes?
- What can we do ahead of time in order to generate the data needed for realistic testing?
Automated Data Population
That’s right we automate the data population. Let’s write a little bit of code that will allow us to create the data for us so that when the need arrives to test big data we don’t have to spend hours setting up an environment. Let’s face it, programmers are very hard-working forward thinking individuals. If we can work smarter, then we are going to. After all necessity is the mother of invention right?
Just like our Cylon friend here we need to construct something that will populate our data for us, and preferably not try to kill us in the process.
Time To Hire
Let’s begin by taking a look at a simple employee sample. We are going to hire a few thousand staff so that we can fill our employee table in the database. I’ll use the good old Northwind database which can be downloaded from CodePlex here.
If you look at the employee table in the database there are several columns which must be filled in. We will need to come up with values for those and I’ll show you how in the code below. The tool of choice here today is called LINQPad. The tool is free for basic use and you can download it from http://www.linqpad.net.
If you find this tool useful, I highly encourage you to purchase it. I have found that it is an invaluable sandbox to test ideas in before putting them into the solution. To follow along you will also need to download the HtmlAgilityPack. You can use NuGet to download the dll’s or get the HtmlAgilityPack from http://htmlagilitypack.codeplex.com. This is an agile HTML parser that builds a read/write DOM and it made the job of creating a code sample really easy. It’s what I used for the HtmlDocument on-line 27. Enough of a sales pitch, lets take a look at the code to make it happen.
I’ve created the following LINQ query file . It is a small C# program that uses the “.\NORTHWIND” database connection.
As you can see the code in its entirety consists of only three methods. I have a method that will create the web request, CreateRandomNameRequest. I have a method that will add the staff members to the database in a batch update, AddStaffBatch. Finally the Main method that controls the actions.
This method will create the web request with the POST information. As you can see on-line 57, I’ve elected to tell the web server that I’m Opera mobile on an Android device. I did that to hopefully eliminate some of the unnecessary information from the server. I then write the posting data to the web requests data stream. This is how you push the post information into the web request.
This method will get the response from the web server using the request generated from the first method. It will then parse the stream into the HtmlDocument. After the document has been parsed, I look for the input element named “exportL“. The value attribute on this element contains the array of names (and genders) that were generated by the server. I simply break this apart and then start creating Employee entities. Once I have created the “batch“, then I will insert them all and submit my changes.
Vioala, you have just populated your database with 6000 employee records! This process is repeatable and configurable so that you can adjust it to meet your needs. Also consider how long it takes to generate this information. I’m willing to bet you couldn’t add 20 records in the amount of time it takes for the LINQPad query to add 6000 to your database. This kind of savings is nice once, but think of it from a quality perspective where you need a repeatable process that you can run over and over again. This can add up to REAL savings!
I want to thank Joe Apple for creating http://listofrandomnames.com. This is an incredibly cool website site and I wouldn’t have been able to make this demo without him.
Until next time Code Hard…