Quick News

ObjectLab Open Source News

Grab this Headline Animator

2006-11-30, if you intend on using QALab with Maven 2, please do read the plugin FAQ, thanks.

2006-11-27, QALab is integrated in JBuilder 2007 as part of the new TeamInsight team-RAD feature.

"We are really excited at the collaboration between ObjectLab and CodeGear" said Joe McGlynn, Product Manager for JBuilder. "Our engineers were searching for an innovative way to consolidate and trend key project metrics and we found a truly wonderful solution in QALab".

ObjectLab has worked with Borland's Developer Tools Group ("DevCo/CodeGear") to make the transition to Maven 2 smooth and rich 'out-of-the-box'. QALab itself will remain independent and carry on supporting all platforms

2006-11-12, version 1.0 is out! After over 3 years of internal use and 15 months as an open source project, we finally ship a version that we consider good enough to be named '1.0'. We are very proud of this little project, it has generated a fair amount of press. We would like to thank our colleagues, friends and users who have helped us reaching this stage. We will also try to help and improve the product and we already have a few more ideas... and some very interesting news coming soon...Watch this space!

QALab Introduction

QA Tools like checkstyle, pmd, pmd-cpd, findbugs, cobertura (cobertura-branch and cobertura-line) and simian are great build tools but they only take a snapshot of the state of your project. You do not get a sense of the trend of your project.

This project collects and consolidates data from several QA tools and keeps track of them overtime. This allows developers, architects and project managers alike to be presented with a trend of the QA statistics of their project.

The following tools are currently supported:

  • Checkstyle: code style validation and design checks. QALab keeps track of number of violations per file and overall.
  • PMD: Code checks (possible bugs, dead code, sub-optimal code, etc). QALab keeps track of number of violations per file and overall.
  • PMD CPD: Duplicate code (always a bad idea) detection. QALab keeps track of number of the overall number of duplicated lines.
  • FindBugs: fantastic tool to detect potential bugs (really!). QALab keeps track of number of violations per file and overall.
  • Cobertura: Coverage tool. QALab keeps track of percentage of branch and line coverage.
  • Simian: excellent duplicate code detection (non-open source). QALab keeps track of number of the overall number of duplicated lines.

This project can be used via ant or Maven. There are three main steps to using QALab:

  1. Collect Data from QA Tools into a generic qalab.xml format. This is a necessary step to consolidate data. Refer to the documentation from Checkstyle, PMD, PMD CPD (Copy Paste Detector), FindBugs, Cobertura and Simian in order to generate the xml reports.
  2. Generate Charts from qalab.xml for each file; these will show the trends over time for each file and the overall project.
  3. Generate a summary xml and html pages with the files that have seen a change in their QA statistics over the last n days. This is particularly useful for developers who can see the impact of the latest code checked in.

Here is a highlight of the steps available, in green are the input files, generated by the respective QATools. In yellow, the files generated by QALab and each transition is a task in QALab.

Data Consolidation

PMD, Checkstyle, Cobertura, PMD-CPD, Simian and FindBugs have each their own XML format. This task is to extract some information from each file, typically the number of violations per file and typically to add this information to a qalab.xml file. QALab is gathering statistics info and therefore, we are not interested in the exact 'violation' or error (see checkstyle, pmd, etc for that).

Please note that the qalab.xml will keep growing and should be kept in your source repository or backed up regularly. The more info you have in it, the more valuable it becomes. As such, it is recommended to use QALab only say, once a day when the whole set of checkstyle, pmd, pmd-cpd, Findbugs and Simian statistics are generated.

The QALab team realise that some project may generate very large qalab.xml files if they have lots of classes and lots of violations. QALab was used for over 2 years on a 450,000 lines of code project without any issue. Having said that, QALab offers you the ability to define how to save the consolidated statistics; simply implement the QALabExporter interface, say to a DB? Most of the examples are shown with the XML file.

In its default configuration, QALab will will create a new summary and result entry for each run and file each time you run the 'merge' task. This can be controlled by using a given timestamp (or forcing it to use only a 'date') and an action 'replace', in which case, each run of QALab 'merge' will replace the previous one with identical timestamp and type. This is very useful if you want to run merge several times per day but want to keep only the latest set of stats per day.

The XML format uses primarily attributes as it tries to limit the number of elements in order to save some space. The name of the attributes though is quite explicit.

Please note that Cobertura generates 2 types of statistics in QALab: cobertura-line and cobertura-branch, corresponding to the line and branch stats in the coverage.xml file

<qalab version="1.1">
    <summary>
        <summaryresult date="yyyy-mm-dd HH:MM:ss" statvalue="n" filecount="n"
                       type="checkstyle|pmd|pmd-cpd|simian|findbugs|cobertura-line|cobertura-branch"/>
    </summary>
    <file id="path/to/File1.java">
        <result date="yyyy-mm-dd HH:MM:ss" statvalue="n"
                type="checkstyle|pmd|findbugs|cobertura-line|cobertura-branch"/>
    </file>
    <file id="path_to_File2.java" path="path/to/File2.java">
        <result date="yyyy-mm-dd HH:MM:ss" statvalue="n"
                type="checkstyle|pmd|findbugs|cobertura-line|cobertura-branch"/>
        <result date="yyyy-mm-dd HH:MM:ss" statvalue="n"
                type="checkstyle|pmd|findbugs|cobertura-line|cobertura-branch"/>
    </file>
</qalab>
        

The date attributes will be the date and time when the data was collected, this is due to the fact that most original XML (checkstyle, pmd, etc) do not have a date representing when they ran.

The DTD for qalab.xml is as follows:

<!DOCTYPE qalab [
<!ELEMENT qalab (summary, file*)>
<!ATTLIST qalab version CDATA #REQUIRED>
<!ELEMENT summary (summaryresult*)>
<!ELEMENT file (result+)>
<!ATTLIST file
	id ID #REQUIRED
	path CDATA #REQUIRED
>
<!ELEMENT result EMPTY>
<!ATTLIST result
	date CDATA #REQUIRED
	statvalue CDATA #REQUIRED
	type CDATA #REQUIRED
>
<!ELEMENT summaryresult EMPTY>
<!ATTLIST summaryresult
	date CDATA #REQUIRED
	statvalue CDATA #REQUIRED
	filecount CDATA #REQUIRED
	type CDATA #REQUIRED>
]>
        

Generating Charts

Once you have a qalab.xml, it is possible to generate some charts:

  • Summary charts for all or a subset of statistics types (checkstyle|findbugs|pmd|pmd-cpd|simian|all)



  • Charts for each file, typically a class, with a statistics and then again, you can select the set of statistics)

Generating Movers and Shakers Summary

As a project grows and the number of developers increases or changes, it is becoming more and more difficult to keep track of the changes impacting the code quality. The best way to deal with this is to deal with any deviation as early as possible. Some style violations may remain in your code for months and are accepted as such.

This report will only show the changes over a given time window, it will highlight the Up (baaad!) or Down impacts (good!) on the code.