Automation In Testing

Today we are inundated with articles on how to test this and test that. The one thing that a lot of automated testing framework makers don’t cover is the data.  Some leave you to insert thousands of rows of data in order to simulate a realistic scenario? Another approach commonly used in the industry is to use customer data. “If it works for customer X, and they are your biggest customer, then it works fine.”  This kind of thinking won’t cut it for very long in the real world.  There are also other things to consider.  For example:

  • What if your customer didn’t want their data used in this way?
  • Why wait until your big customer complains about the responsiveness of your application before making changes?
  • What can we do ahead of time in order to generate the data needed for realistic testing?

Automated Data Population

That’s right we automate the data population.  Let’s write a little bit of code that will allow us to create the data for us so that when the need arrives to test big data we don’t have to spend hours setting up an environment.  Let’s face it, programmers are very hard-working forward thinking individuals.  If we can work smarter, then we are going to.  After all necessity is the mother of invention right?
By your command...

Just like our Cylon friend here we need to construct something that will populate our data for us, and preferably not try to kill us in the process.

Time To Hire

Let’s begin by taking a look at a simple employee sample.  We are going to hire a few thousand staff so that we can fill our employee table in the database.  I’ll use the good old Northwind database which can be downloaded from CodePlex here.

If you look at the employee table in the database there are several columns which must be filled in.  We will need to come up with values for those and I’ll show you how in the code below.  The tool of choice here today is called LINQPad.  The tool is free for basic use and you can download it from

If you find this tool useful, I highly encourage you to purchase it.  I have found that it is an invaluable sandbox to test ideas in before putting them into the solution. To follow along you will also need to download the HtmlAgilityPack. You can use NuGet to download the dll’s or get the HtmlAgilityPack from  This is an agile HTML parser that builds a read/write DOM and it made the job of creating a code sample really easy.  It’s what I used for the HtmlDocument on-line 27. Enough of a sales pitch, lets take a look at the code to make it happen.

Sample Code

I’ve created the following LINQ query file .  It is a small C# program that uses the “.\NORTHWIND” database connection.

void Main()
	var totalNamesWanted = 6000;
	var batchSize = 1000;

	var currentTotal = 0;
	while (currentTotal < totalNamesWanted) {
		currentTotal = Employees.Count();

// Define other methods and classes here
private void AddStaffBatch(int sizeOfBatch) {
	const string randomNamesUrl = "";

	var staffId = (Employees.Max(staff => staff.EmployeeID) + 1);

	var req = CreateRandomNameRequest(randomNamesUrl, sizeOfBatch);
	using (var webResponse = req.GetResponse()) {
		using (var responseStream = webResponse.GetResponseStream()) {
			if (responseStream == null)
				throw new Exception("No response stream found for the given url");

			var streamReader = new StreamReader(responseStream, System.Text.Encoding.UTF8);
			var responseData = streamReader.ReadToEnd();
			var htmlDoc = new HtmlDocument();
			var nameToGet = "exportL";
			var nodes = htmlDoc.DocumentNode.SelectNodes("//input[@name='" + nameToGet + "']");
			if (nodes.Count() == 1)
				var rawNameArray = nodes.First().Attributes["value"].Value.Split(',').ToArray();
				var staffToInsert = new List();
				foreach (var rawName in rawNameArray)
					var staffName = rawName.Split('|');
					staffToInsert.Add(new Employees {
							EmployeeID = staffId++,
							FirstName = staffName[0],
							LastName = staffName[1],
							Notes = "Created using LINQPad and",

private HttpWebRequest CreateRandomNameRequest(string url, int numberOfNames)
  var webReq = (HttpWebRequest)WebRequest.Create(url);
  webReq.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
  webReq.ContentType = "application/x-www-form-urlencoded";
  webReq.UserAgent = "Opera/12.02 (Android 4.1; Linux; Opera Mobi/ADR-1111101157; U; en-US) Presto/2.9.201 Version/12.02";
  webReq.Referer = "";
  webReq.Method = "POST";  

  var postData = string.Format("action=main.generate&numberof={0}&nameType=na&fnameonly=0&allit=0", numberOfNames);
  var encoded = Encoding.UTF8.GetBytes(postData);
  webReq.ContentLength = encoded.Length;
  var dataStream = webReq.GetRequestStream();
  dataStream.Write(encoded, 0, encoded.Length);

  return webReq;

As you can see the code in its entirety consists of only three methods.  I have a method that will create the web request, CreateRandomNameRequest.  I have a method that will add the staff members to the database in a batch update, AddStaffBatch.  Finally the Main method that controls the actions.


This method will create the web request with the POST information.  As you can see on-line 57, I’ve elected to tell the web server that I’m Opera mobile on an Android device.  I did that to hopefully eliminate some of the unnecessary information from the server.  I then write the posting data to the web requests data stream.  This is how you push the post information into the web request.


This method will get the response from the web server using the request generated from the first method. It will then parse the stream into the HtmlDocument. After the document has been parsed, I look for the input element named “exportL“. The value attribute on this element contains the array of names (and genders) that were generated by the server. I simply break this apart and then start creating Employee entities. Once I have created the “batch“, then I will insert them all and submit my changes.


Vioala, you have just populated your database with 6000 employee records! This process is repeatable and configurable so that you can adjust it to meet your needs. Also consider how long it takes to generate this information. I’m willing to bet you couldn’t add 20 records in the amount of time it takes for the LINQPad query to add 6000 to your database. This kind of savings is nice once, but think of it from a quality perspective where you need a repeatable process that you can run over and over again. This can add up to REAL savings!

I want to thank Joe Apple for creating This is an incredibly cool website site and I wouldn’t have been able to make this demo without him.

Until next time Code Hard…


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s