Quick overview on Cobertura (Testing Coverage Tool)

I have mentioned Cobertura in this blog.

Cobertura is a tool similar to Coverlipse mentioned in this blog. Cobertura is a measurement tool to measure your unit tests’ coverage on your tested codes. I think that it is really a good tool. It is a bit difficult for a beginner, because it requires some configuration. You have to know how to use Ant (a build tool).  Here, I just provide a tutorial to get a quick overview of the Cobertura on Windows.

1. You need to have Ant installed on your computer. You can download from this page. Select apache-ant-1.7.1-bin.zip.  See this page for installing instruction. Be sure to add Ant to your environmental variable called PATH as illustrated in “Windows / OS 2″ section in the installing instruction page, so you can call Ant from any folders. After you install ant, try running just command “ant” in any folder in your command promt. If it displays something like Ant is not recognized as a command, it means that you have not set up the environment variable PATH correctly. See this page for instruction on how to set an envrionmental variable on Windows.

2. download the Cobertura from this page. Select cobertura-1.9-bin.zip.

3. Unzip into your folders. You will get cobertura-1.9 folder

4. Put your Java source files into the example folder. There is an example found in \cobertura-1.9\examples\basic. I put my Java source files which have been used as an example in this blog and this blog. It consists of one Java file, DoSomething.java, and its test case file, TestCountCharacter.java. Basically, put your codes in \cobertura-1.9\examples\basic\src folder.

Put your java files (unit test case and class to be tested) in the cobertura-1.9examplesasicsrc folder

5. Put JUnit.jar into \cobertura-1.9\lib.

See this blog for how to get JUnit.jar.

If you do not put this Jar file here, then build will be failed, because it can not compile some classes which use come classes found in JUnit.jar. The error is shown below. If your Java compiler can find JUnit.jar, then you do not need to add JUnit.jar into the lib folder.

Ant build is fail because JUnit has not been added into the lib function.

6. Modify the ant build file found in \cobertura-1.9\examples\basic\build.xml.

On the line around 90, add your java test file into it. In my case, I have to add TestCountCharacter.java which contains the JUnit test cases, so I put “<include name=”TestCountCharacter.java” />” as shown below.
<batchtest todir=”${reports.xml.dir}” unless=”testcase”>
<fileset dir=”${src.dir}”>
<include name=”**/*Test.java” />
<include name=”TestCountCharacter.java” />
</fileset>
</batchtest>

7. Run the ant build file simply by typing “ant” in the \cobertura-1.9\examples\basic.

Behind the scene, Cobertura compiles, run test cases and produces the report on your test case coverage. See this page for full instruction.

Ant build success

8. View the report. Reports are automatically generated after the ant build. There are two reports: report on JUnit test case and report on test coverage.

You can view the report of your JUnit test case by opening up the index file in cobertura-1.9\examples\basic\reports\junit-html.

You can view the report of your test coverage by openning up the index file in cobertura-1.9\examples\basic\reports\cobertura-html.

Report on JUnit test cases is automatically generated and can be found in this folder

Report on your test cases

As you see from the figure below, the line coverage is not 100% achieved because there some statements which have not been executed. The branch statement does not achieve 100% coverage, because there is a branch which has not been executed.

Cobertua report displays the test cases

If you furhter click into the class DoSomething.java, it shows you which line has not been executed and how many time that line has been executed as shown below.

Cobertura shows test coverage (line (or statement) and branch)

How to get JUnit.jar

If you do not have JUnit.jar in your computer, you can download from here. Just download any junit-x.x.jar where x is a version number. The Junit.jar is in the downloaded zip file.

Testing coverage with Cloverlipse in Eclipse 3.2

You want to make sure that your test cases have executed every thing in your function. You can use test coverage criteria (eg. statement, brance, data flow, etc) as a criteria to measure the quality of your test cases.

Or, you can use it to study old codes when you want to see which lines or functions are used without going through a cumbersome debugging process.

This article is continued from the previous tutorial of using JUnit in Eclipse. If you want to see the testing coverage, you can click as shown below.

Run JUni test with Coverlipse

I think that Coverlipse is based on statement coverage. There are many test coverage or aquadecy criterion. More info on test coverage can be found in wiki. Basically, you want to test everything in your code. To satisfy statement coverage, you need to execute all statements. To satisfy branch coverage, you need to execute all brances.  Coverclipse shows you which statement is executed by putting a green check on that line as shown below.

After running JUnit with coverclipse, the Coverlipse the line of code which you test case has executed by putting a green check on that line

To see a test coverage clearly, I have modified the function that I tested. The function does not mean to do anything make sense. It just serves as an example for code coverage. As you can see below, there are some codes which are not excuted by test cases. It is shown by a red exclamation side on that line.

Coverlipse shows you which lines are not covered by your test cases by putting a red exclamation sign (!) on that line

With Coverlipse, you can use to see which codes are used when you have to study bunch of old codes. It can give you a quick overview than going through a debug processes.

Unit testing with Eclipse and JUnit

Software testing is necessary to ensure the quality of the software. I would like to how can you make  a quick unit test in Eclipse using JUnit.

First of all, there are 2 main kinds of testing : blackbox testing and whitebox testing. Blackbox testing is when you derive a test case for a function without considering codes inside the function. You only care that you provide all possible inputs. There are many blackblox testing techniques. One of them is Category Partition Testing. Bascially, you simply divide all inputs into classes, and then just test one representive input of each class. For example, if a function X has an input of type Integer. You may partition this input into three classes which are negative numbers, zero and postive numbers. This is just an example. Partitioning inputs is depended on your function. Then you derive three unit test cases. One for each partition. If you have 2 inputs. Then you may have 2*3 = 6 combinations of inputs. Sometimes, you may have a lot of combination than you can possibly test. The slides (slides 15) and the paper Category Partition Testing suggest the use of constraint to limit some combinations.  There could be some possbile combination which will not happen for sure, so it can be considered as a constraint and removed out from testing.

Enough with the theory.

I use Eclipse 3.2 and it is already equiped with JUnit. We are going to use some classes from JUnit.jar. You may need to add the JUnit jar file into your Eclipse project if it does not automatically add JUnit into your project build path. (If Eclipse does not add JUnit into your project build path, then your Java class which requires classes found in JUnit.jar can not be compiled.) This  page gives you some idea of how to add JAR file in Eclipse. See this blog for how to get JUnit.jar.

You need to have a Java function that you want to test. A figure below shows an example of Java function.

a simple java function that you want to test

Next, you can add a Junit test as shown below.

Adding a new Junit test case

Then the “New JUnit Test Case” dialog is shown as illustrated below. You can select a class that you want to test in the textbox “Class under test.”

Next, then you can select a function that you want to test.

Selecting a function that you want to test

You will see a new Java class as shown below. Eclipse  may not be able to recognize some classes as shown below.

Eclipse does not recognize some classes found in JUnit.jar

So you need to either add a JUnit jar file into your project. Or, eclipse may ask you whether you want to add Junit.jar into your classpath as shown below.

Eclipse asks whether you want to add JUnit.jar into the build path

You may add more test cases. In the figure below, I have 4 test cases just for one function. Each test case is represented by one function. A function name needs to be started with the word “test.”  You can simply runs the JUnit as shown below. Normally, you use the assert something function to assure the result from the function. For example, if you expects that a function returns 3 given your inputs, then you assert like following:

assertTrue(returned_value == 3) ;

If the function returns 3, then this test case is passed. Otherwise, it is failed. Deriving test cases and keeping it for further use is a good idea. Developers usually have a version. With JUnit test cases, they can see if a new version of the function still working properly.

The figure below shows when all unit test cases are passed.

result of running JUnit test cases. In this case, all test cases are passed

Virtualization tool

I think that the trend nowsaday is moved toward virtualization where one can run one OS on top of another OS. There are a lot of server which has low utilization. Each server consumes power. Producing power may produce CO2. Running low utililization server is not sustainble for economic and environment. It would be a good idea to move a lot of low utilization servers into one server using the virtualization technology.

Personally, I do not have a data center, so I don’t read need to care about saving power. However, I use virtualization technology usually for testing software running with different configuration from my default windows. I usually use VMware server (the free version) to create a virtualized OS. However, VMware server consumes some resouces (memory). I have seen VMware Player for quite a long time, but I have not tried. I tried today. It is really impressive. It lets me run the virtualized OS that I created with VMware server. It seems to me that VMware player does not consume much computer resource.

Cobertura

Today, I have tried Cobertura which a free test coverage calculation tool for my first assignment for the software testing class.

You can either use this tool via a command line or an ant build file..

At first, I was trying using a command line, but it did not seem that there is any happened. Perhaps, I typed in the wrong command.
I don’t have much time to construct the ant build file. Luckily, I found an example of ant build file.

Cobertura helps measuring the coverage of my Junit tests. It measures the coverage by looking the percentage of statments and branchs covered. Cool tool.

Software Testing Course in Edinburgh

Previously, before I took Software Testing class, I unit-tested my work unsystematically and mannually. When I first study about testing, some books say that you should use “assert” (such as assertEqual in JUnit or Assert.AreEqual in .Net Framework) to reassure the value in the variables. At first, I did not understand why. Later on, I understood that it would make it easier for me to track when there is something wrong rather than debugging. I still remembers the old day when I spent time debugging in Eclipse or Visual Studio. Going line by line debugging took so much time and energy.

From the Software Testing class, http://www.inf.ed.ac.uk/teaching/courses/st/, I learned that I can do a Junit test. At first, I was thinking whether it would be benefitial if I make a test suite for my funtions. I realized that it is very useful. I remembered the old day, when there was a bug, I have to fix. Fixing a bug can cause a new bug. Having a suite test ready will make your life easier. I can test my bug fixing code just within a second to see whether it has an impact to other codes.

I also need to design my software to be easy to test as well.

If you want to see a sample of software testing sylable, you can take a look at the Edinburgh’s Software Testing course.