Effective QA

Configuring SQL Server for external access on Windows 2003

2010-05-15T08:45:00.002-06:00

Recently I was trying to expose SQL Server for external access on a Windows 2003 server. I ran into several issues:

Port 1433 was blocked by the firewall. This serverfault article explains how to detect and correct this

The TCP/IP filters for inbound access had to be altered. This Microsoft article describes doing this

In retrospect neither of these issues are surprising, but especially the TCP/IP filter issue took me some time to track down, so perhaps this will help someone else.

Windows installer clean up utility

2010-05-15T08:36:00.003-06:00

I recently found myself with a program that did not uninstall cleanly and it left enough of a mess that I could reinstall it. After some searching around, I found this tool from Microsoft:

http://support.microsoft.com/default.aspx?scid=kb;en-us;290301

It discusses cleanup of Office installs on XP, but I found it to work fine on other programs on 64-bit Vista. Give it a try the next time you have a program that won't uninstall.

A minimal Erlang script for load testing

2009-12-06T14:14:00.007-07:00

Recently I became interested in Erlang, a functional programming language that excels at developing concurrent applications. From a SQA perspective, concurrency immediately made me think of performance and load testing. I am not the first one to think this - see the open source application Tsung to see just how far this idea has been taken.

I set about to see how minimal an Erlang program I could write that would implement a bear-bones load tester for web apps. Here is what I came up with:

 1: -module(lmin).
 2: -export([example/0, ramp/4, vclient/2]).
 3:
 4: vclient(Supervisor, Script) ->
 5:    inets:start(),
 6:    Total = vclient_acc(Script, 0),
 7:    Supervisor ! {self(), Total / 1000000}.
 8:
 9: vclient_acc([Cmd|Remaining_cmds], Total_time) ->
10:   Fetch_time = timed_request(Cmd),
11: vclient_acc(Remaining_cmds, Fetch_time + Total_time);
12: vclient_acc([], Total_time) -> Total_time.
13:
14: timed_request({Delay, Url}) ->
15:    timer:sleep(Delay + random:uniform(Delay)),
16:    Start_time = now(),
17:    {ok, _Result} = http:request(Url),
18:    timer:now_diff(now(), Start_time).
19:
20: ramp(Nstart, Nend, _Nstep, _Script) when Nstart > Nend -> done;
21: ramp(Nstart, Nend, Nstep, Script) ->
22:    launch_vclients(Nstart, Script),
23:    collect_results(Nstart),
24:    timer:sleep(1000),
25:    ramp(Nstart+Nstep, Nend, Nstep, Script).
26:
27: launch_vclients(0, _Script) -> true;
28: launch_vclients(N, Script) ->
29:    spawn(?MODULE, vclient, [self(), Script]),
30:    launch_vclients(N-1, Script).
31:
32: collect_results(Nexpected) ->
33:    {Sum, _Results} = collect_results(Nexpected, Nexpected, [], 0),
34:    io:format("~p: ~.2f average: ~.2f~n",
35:    Nexpected, Sum/Nexpected]).
36:
37: collect_results(0, _Nexpected, Results, Sum) -> {Sum, Results};
38: collect_results(Nremaining, Nexpected, Results, Sum) ->
39:    receive
40:       {Vclient, Time} ->
41:          collect_results(Nremaining-1, Nexpected,
42:          [{Vclient, Time}|Results], Sum+Time)
43:    end.

I am not going to write a Erlang tutorial here; I am a complete Erlang newbie and would not do it justice. The Erlang documentation section is a good place to start for those who are interested. What I will do is provide a brief overview of this application.

Lines 4-18 are the virtual clients that will send requests to the SUT (System Under Test). timed_request (lines 14-18) makes the actual request to the server; vclient_acc accumulates the response times for each command in the script. A script is a list of commands of the format {Delay, URL}. For each command in the script, the virtual client waits for a random time in the range Delay..2*Delay and then makes a request for URL.
Lines 2o-30 launch the the virtual clients. Each client runs the same script and starts executing immediately when it is spawned. The key is ramp/4, which in turn hits the SUT with Nstart, (Nstart+Nstep), (Nstart+2*Nstep)... Nend virtual clients.
Lines 32-42 collect and report the results.

So is this proof of concept load tester useful for anything? See my previous article, "Results using a simple Erlang load tester" to find out.

Results using a simple Erlang load tester

2009-12-05T17:43:00.013-07:00

Having written a minimal load tester in Erlang, the next step was to try producing useful results with it. I downloaded the latest version of Hudson, my favorite continuous integration tool. It comes "pre-loaded" with Winstone, a lightweight server container. What kind of load can Hudson handle? I started by devising a short script. After a random time within the specified range, the client requests a specific page:

Clearly this faster than most users will navigate around Hudson, so this script is something of a worst-case.

I then ran the script 10 times, recording the results:

The red cells in the spreadsheet indicate where Hudson became unresponsive. The red cells containing "10.00" actually did not respond; I added those values for the purpose of making this graph:

Pretty clearly, Hudson is comfortable with up to 9 simultaneous clients. When moving to 12, performance starts to degrade, but is still acceptable. At 15, there were some outright failures. If this Hudson instance is intended to handle more than 12 simultaneous clients, adjustments are necessary.

With a quick look through the log files, I found the problem almost immediately:

Caused by: java.lang.OutOfMemoryError: Java heap space

So it may be pretty easy to increase load capacity for Hudson. Good news - my tiny little script found a problem before my customers did.

In a future post, I'll see what a little parameter tuning does for Hudson load handling.

"The Parallel port driver service failed to start" when booting in Hyper-V

2009-11-24T05:50:00.003-07:00

At work I've been moving our 2003 Enterprise virtual machines from Microsoft's Virtual PC 2007 to their Hyper-V platform. Generally this process went smoothly, with one exception; when booting, an error dialog was displayed:

The Parallel port driver service failed to start due to the following error:
The service cannot be started, either because it is disabled or because it has no enabled devices associated with it.

After a bit of searching, I discovered there is a registry setting that will turn off this message. Using regedit, navigate to:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Parport

and set:

Start = 4

Viola, no more error message during boot up.

Reference: Microsoft Support (this article references Vista / 2008 Server, but it worked fine with 2003 Enterprise SP2)

Using virtualization to improve quality and shorten QA time

2009-02-13T07:44:00.008-07:00

Michael O'Leary at Robert Half Technology asked me to speak to some of their clients on virtualization and how it has helped me in my QA efforts. Here are the handouts and slides:

handouts (doc format)
slides (pdf format)

I used a variety of online resources to prepare for my presentation:

Introduction

Resources

x86 Hardware Support for Virtualization

Virtualization Providers
(extremely abbreviated; see http://en.wikipedia.org/wiki/Comparison_of_platform_virtual_machines)

Automated Testing Tools
Continuous integration (Hudson): https://hudson.dev.java.net
Technical debt: http://sonar.codehaus.org
Automated testing: http://wtr.rubyforge.org/ http://seleniumhq.org/

Cloud Computing

My thanks to Michael, Robert Half Technology and the attendees.

Criteria for Selecting a Technical Debt Calculator

2008-08-28T08:23:00.009-06:00

In "Calculating Technical Debt", I proposed a set of criteria for selecting a technical debt calculator. This list was:

Plugin architecture
Flexible roll-up rules
Current perspective
Trend perspective
Languages supported
Environments supported
Supports qualitative data
Build environments
Custom dashboards
Aggregates multiple projects
Continuous integration servers
User community

To this list I added a few more criteria relevant to using the tools:

Tool quality (is the quality calculator buggy?)
Latest release (is this project alive?)
Documentation (can I figure out how to use it?)

In "Characteristics of Technical Debt", I came to the conclusion that the McConnell internal quality characteristics were well suited to calculating technical debt. His criteria belong on the list then:

Calculates maintainability
Calculates flexibility
Calculates portability
Calculates reusability
Calculates readability
Calculates testability
Calculates understandability

I make no claim this list is exhaustive, but it covers enough of the issues important to me that I am comfortable moving forward with it.

In making these evaluations, I need some projects to run through the tools to see how they perform. The obvious choice is some open-source projects. These projects should:

Be developed in Java
Have their development history captured in a version control system
Have high development activity
Have JUnit tests
Some built with Ant, some with Maven 2

A trip around a few open-source repositories turned up a few likely candidates:

Jena - A framework for building Semantic Web applications. I'll focus on the ARQ component. (Ant)

Jena (ARQ module) has eight releases in its Subversion tags folder:

$> svn ls https://jena.svn.sourceforge.net/svn/root/jena/ARQ/tags

ARQ-2.0/
ARQ-2.0-RC/
ARQ-2.0-beta/
ARQ-2.1/
ARQ-2.1-beta/
ARQ-2.2/
ARQ-2.3/
ARQ-2.4/

Tiles - A Web templating framework. (Maven 2)

Tiles has eight releases in its Subversion tags folder:

$>svn ls http://svn.apache.org/repos/asf/tiles/framework/tags

tiles-2.0.0/
tiles-2.0.1/
tiles-2.0.2/
tiles-2.0.3/
tiles-2.0.4/
tiles-2.0.5/
tiles-2.0.6/
tiles-2.1.0/

This would seem a trivial matter to pick out some projects to use as a basis for our tool evaluation. It turns out this is not so. I won't bore you with the reasons, but this took WAY longer than I expected.

With criteria and test cases in hand, I'll move on to evaluating the options. Sonar seems like a reasonable place to start.

Characteristcs of Technical Debt

2008-08-25T16:54:00.013-06:00

I was planning to launch into an evaluation of the tools identified in "Calculating Technical Debt" (please read that first if you have no idea what I am talking about), but in preparing for the evaluation, I discovered there are large bodies of work addressing the characteristics of interest for determining code quality. There is even an ISO standard (9126) defining one such set of quality characteristics.

I guess it was pretty naive of me to think that quantifying software quality was a new undertaking. There are five models that appear repeatedly in the literature:

Boehm
McCall
FURPS
ISO 9126
Dromey

Each model has a different focus and thus includes different characteristics to assess quality (see reference page 38). The following table summarizes these top five models:

Reference: Ortega, Maryoly; Pérez, María and Rojas, Teresita. Construction of a Systemic Quality Model for evaluating a Software Product. Software Quality Journal, 11:3, July 2003, pp. 219-242. Kluwer Academic Publishers, 2003

In "Quantifying the cost of expediency", I proposed using McConnell's "internal quality characteristics" from Code Complete. Does this still hold up in light of these large bodies of research? Lets see.

In a perfect world, I'd take the time research all of these models along with the various hybrids that have been devised. Perhaps I'll formulate my own model one day and become rich and famous. My goal for now is not to devise the optimal model, but rather to create a proof-of-concept for calculating the cost of expedient design and development choices.

From a cursory literature search, it appears that ISO 9126 is the most widely used model. I'll use that as the basis for comparison.

The ISO 9126 standard is summarized by the following graph:

(from http://www.cse.dcu.ie)

As you can see from the graph, the ISO model defines six major characteristics of quality:

Functionality
Reliability
Efficiency
Usability
Portability
Maintainability

Each is further decomposed. Comparing ISO 9126 with McConnell, you'll notice that McConnell is missing characteristics like accuracy and efficiency. He actually presented a separate list "external quality characteristics". These refer to customer-facing issues rather than developer-facing issues. Most of the ISO 9126 characteristics missing in his internal list appear there. Intuitively external characteristics are fundamental to understanding technical debt. They are also more difficult to determine automatically. I am going to choose to postpone including them in the technical debt evaluation for now - it is the expedient thing to do. ;-)

The characteristics on McConnell's list that don't appear in ISO 9126 are flexibility and reusability. These characteristics seem relevant to the technical debt of a project. Flexibility and reusability do appear in the McCall model (see table). McConnell provides the following definitions (p. 558):

Flexibility - The extent to which you can modify a system for uses or environments other than those for which it was specifically designed.

Reusability - The extent to which and the ease with which you can use parts of a system in other systems.

After all this, I am back to where I started. That is not where I expected to be. The original title of this entry was something like "An ISO model for technical debt". It was not until I started comparing ISO 9126 with McConnell that I came to the conclusion that McConnell better serves my purpose.

In the next entry I'll return to looking at the available tools for calculating technical debt.

Calculating Technical Debt

2008-08-20T15:25:00.008-06:00

In "Quantifying the cost of expediency", I describe the problem IT management face when they trade off shorter development time with robust code. To recap, my conjecture is that a solution to understanding such trade offs can be reached by sub-dividing the problem into two phases and addressing each individually. The phases are:

Evaluate the code base against a set of quality criteria and track trends
Convert these metrics into an easily digestible form

At the AgileNM meeting today, I learned there is a term for the code degradation that often occurs in a software project: technical debt. We had an excellent discussion of qualitative ways attendees assess this cost. These included:

Decreases in team velocity over time
Developer time to understand unfamiliar code
Querying developers

How exciting to have this group interested in the very topic I've been pondering and writing about! In looking for quantitative measures of technical debt, my research turned up several candidates for calculating the current debt and tracking trends. This list is shamelessly lifted from the Sonar Related Tools page:

Open Source

Commercial

The commercial tools don't appear to have evaluation downloads, so I'll focus on open source tools for now. Perhaps I can get evaluation copies in the future.

The open-source tools all leverage other open-source tools that determine one or more pieces of the technical-debt picture. Checkstyle, PMD and CPD are static analysis tools while JUnit and Cobertura are runtime tools which all contribute to understanding the debt .

The open-source tools are all Java-centric. I'll search for similar .Net tools.

Originally, I had planned to jump into Sonar but, given the alternatives, it seemed prudent to consider what features best support calculating technical debt. I haven't looked at any of them closely yet, so hopefully this list isn't overly biased:

Plugin architecture for adding new analysis tools
Flexible rules for rolling up results into characteristics
Current and trend perspectives
Support for multiple languages and environments
Able to consider qualitative data in calculating characteristics
Support for multiple build tools
Flexible dashboard creation to display results
Interfaces to continuous integration servers
Vigorous user community

In the next installment I'll present my take on how the open source tools stack up against this feature set.

Quantifying the Cost of Expediency

2008-08-13T17:44:00.013-06:00

Software project managers often ask teams to proceed at the greatest possible speed without regard for the long-term consequences to code quality. Lets call such choices "expedient". If expedient choices are made infrequently, the development team can recover by refactoring the tainted code. For many shops though expediency becomes the norm rather than the exception. The downside of expedient choices is not initially apparent. There is an uneasy feeling in our stomach, but nothing tangible to explain why. The upside usually is apparent since it can (with some accounting magic) be measured in dollars. My contention is that to make informed decisions managers need the ability to measure the "expediency cost" in dollars too.

We have some idea of this cost to the industry as a whole. Capers Jones estimated that almost two-thirds of developer time is spent repairing software. In Code Complete, he states:

Projects that aim from the beginning at achieving the shortest possible schedules regardless of quality considerations, tend to have the fairly high frequencies of both schedule and cost overruns. Software projects that aim initially at achieving the highest possible levels of quality and reliability tend to have the best schedule adherence records, the highest productivity, and even the best marketplace success.

So evidence suggests we're heading down the wrong path when we attempt to be expedient, but how do we quantify this cost? What should high quality code look like?

In Code Complete McConnell lays out the following categories for internal quality characteristics (descriptions are paraphrased):

Maintainability - Can the software be modified?
Flexibility - Can the software be repurposed?
Portability - Can the software be ported to new environments?
Reusability - Can the software be used in other systems?
Readability - Can the source code be read?
Testability - Can the software be verified correct?
Understandability - Can the software be understood at the system-organizational level?

I propose then the task is two-fold. First, evaluate the code base against these criteria. This includes tracking metric changes over time. Second, convert these metrics into information management can use to make informed trade-offs between quality and expediency. If this information is ultimately mapped to dollars, a real apples to apples comparison of the cost of expediency can be made.

Practice what I Preach - Code Reviews, TDD and RDoc

2008-07-05T21:41:00.007-06:00

The first internal release (0.1) of my ongoing attempt to "practice what I preach" is complete. The program works well, has high coverage and acted as a catalyst for many of the quality tools and practices I've been discussing. Three things stuck out in my mind as I completed the release:

(1) Even on a one-man project, code reviews are beneficial. One night I spent hours in the debugger plus adding puts everywhere to complete the refactoring to accommodate changes. It was one of those late nights when I should have probably just walked away. What I realized when it was finished is that about 10 minutes of inspecting the code would have produced the same results as my hours of debugging. Perhaps all of you are smarter than me and wouldn't have this outcome, but give it a try sometime and see if inspection isn't faster. Or better yet, try an informal code review with a co-worker.

(2) I was shocked at how much completing the RDoc improved my code. When I started to write about something and it was complete hogwash, I could either try and explain why I had written complete hogwash or I could clean it up. It was about the same amount of work, so why not clean up the code? An example was the large increase in private methods. If they are private, I don't need to write docs on them. :-)

(3) I wrote most tests and competed the RDoc after the coding was complete. They came out well, so why not keep doing this? Because for my project, these activities are the point. So of course I will finish them. At my day job, the code is (mostly) the point. So while I took the time to do the "right thing" at home, at work I would've been under extreme pressure to move onto the next task. Even if you don't believe TDD will deliver higher quality code, it is worth doing just so testing (and doc) get done at all.

I am researching how to quantify "code rot". My management can quantify the gains of pushing code out the door half-baked if they can attract new customers as a result. But they don't have the foggiest idea what it has cost them in terms of their major asset - the code base which is essential to the viability of the business. If there were some code rot number(s) which were convertible into dollars, the number of jackass decisions may well decrease. Anyone have ideas on this?

Test::Unit, Rake and Hudson status

2008-06-25T13:36:00.007-06:00

All the Hudson Test::Unit tests were passing. A sea of success. Then I started looking at the result files. All was not good after all. The unit tests were not really passing, even though Hudson said they were:

I spent a long time trying to figure out who was dropping the ball on the status. Was it Test::Unit? Rake? Windows? Hudson? I never found a definitive answer, though it was starting to look like it might be Test::Unit.

I did find this post and gave it a try. Unfortunately, this solution works when there is a non-zero exit status, but this is not the case when running Unit::Test from rake.

The solution turned out to be much simpler than trying to figure out how to correct Rake or Test::Unit. Hudson to the rescue. Hudson supports a plugin architecture. The repository has a plugin named Text-finder Plugin (download it here). It determines the success or failure of a build by looking for a regular expression in the build artifacts or system output. I added a rule which looks for the expression /\d+ tests, \d+ assertions, 0 failures, 0 errors/ in the console output. If the expression is found, the build passes; otherwise it fails. Once installed the configuration panel has this section:

Yet again Hudson solves the problem where other players are dropping the ball. The Hudson status now accurately conveys the result of running the unit tests.

Ruby - Making a Mockery of Testing

2008-06-14T13:21:00.012-06:00

Having gotten rcov working well, I now had to eat the bitter fruit of my low coverage. The main code talks to a news server, so testing would not be so simple as calling a few methods and checking the results. Perhaps I still need to refactor into smaller, more easily testable methods. I'll visit that at another time.

The solution is to "mock" the news server. A mock is code that provides the same interface (or at least enough of it for the tests) as a "real" object or class, yet return expected data each time the test is run. Mocks also can be configured to know how many times a method should be called, in what order they are called, the values that should be passed in and so on. If these conditions are not met, an error is thrown and the test fails.

Ruby has several tools for creating mock objects. I settled on Flex Mock after reading a few blogs. Others were passionate about Mocha - I'll take a look at that another time.

For the purposes of my test, a Ruby newsreaderAPI needs to provide the following methods:

Net::NNTP.new(host, port, timeout) - Specify the host, port and timeout values to use when connecting (class method)
Net::NNTP.connect() - Connect to the host with parameters set in new.
Net::NNTP.xover(groupname, :from=>low, :to=>high) - Retrieve the headers from group groupname in the range low..high
Net::NNTP.group(group) - Set the group to fetch articles from
Net::NNTP.article(id) - Retrieve the article with the given id

Notice there is one class method to mock. After a little looking, I found the answer in the README file (see section "Mocking Class Objects) for FlexMock. The answer is to have the class method return another flexmock object which is the instance.

nntp_mock = flexmock
... more stuff here to define nntp_mock...
flexmock(Net::NNTP).should_receive(:new).and_return(nntp_mock)

Now when the Net::NNTP.new method is called, it will return our flexmock instance which handles the rest of the test.

Lets look at the flexmock instance object now:

nntp_mock = flexmock
nntp_mock.should_receive(:connect).once.
with_no_args
nntp_mock.should_receive(:group).once.
with(String).and_return(group_mock)
nntp_mock.should_receive(:xover).once.
with(String, Hash).and_return([summary1, summary2])
nntp_mock.should_receive(:article).twice.
with(String).and_return(article1, article2)

One interesting feature is that parameters may be chained together. So for example the last line specified that:

It responds to the method call 'article'
It should be called exactly twice during the test (fail otherwise)
It must receive exactly one parameter which is a string
It returns the specified articles

So with just a few lines of code we've written a newsgroup reader application which is sufficient for our test. Its behavior is deterministic and further our test will fail if any of the expectations set for it fail. That is a lot of value for a small effort.

The README file is extensive and gave me enough information to write my mock.

The next time you need to write a test for code which references an external resource, mock it instead. You'll be happy you did.

Walking the Talk, a few days later

2008-06-12T11:56:00.018-06:00

The "Walking the Talk" project is well underway. My progress has been brisk and I'm learning quite a bit along the way. The simplicity of Ruby is a gift, allowing me to understand the concepts involved without getting bogged down by endless details.

Several tools were left out of the quality sandbox sandbox last time. RDoc (source-embedded documentation ala JavaDoc) and Log4r (logfile support ala Log4J) are both useful tools for the agile developer. I'm still learning some of the finer points of each, but both were up and functioning at a basic level in no time. Expect to hear more about these in a future post.

Hudson
Hudson continues to be awesome. I just read on Kohsuke Kawaguchi's Blog that Hudson and his other projects he's been doing in the background are going to become his day job. This is exciting news for Hudson fans! The brisk development pace will likely accelerate further.

TDD
I had an epiphany several months ago in our AgileNM meeting about TDD. A member said that TDD significantly reduced the cyclomatic complexity of the code. Ah, TDD is not about having better tests, it is about writing better code! I'll relate my personal experience with this a little later, but let's just say for now that my personal experience has only increased my enthusiasm! Take a look at this paper from David Jansen of the University of Kansas for more background.

rcov
rcov pretty much worked as expected. There were only two ways that it fell short of the standard set by Cobertura for me. The first was the lack of hit counts for each line. I found this interesting to understand where the hot spots in the code might be even before firing up a profiler.

The other shortcoming of rcov compared to Cobertura is that it does not include cyclomatic complexity analysis. After a little Googling, it appears that Saikuro fills that gap. I will report on it as soon as I get a chance to play with it.

Trac
Trac was the biggest disappointment of the bunch. One area where Ruby leaves Python in the dust is in package management. I had to install so many different packages and take so many manual steps just to get something that sort of worked. Compared to RubyGems projects, this is an absolute joke. I couldn't get either of the plugins I installed to work.

The good news is there will soon be an excellent issue tracker/wiki available. Confluence, the wiki we use at work, already has a personal edition available. With the 4.0 release of JIRA (date not announced yet) there will be a personal edition of JIRA as well. Both are from Atlassian, who are fantastic to work with. Goodbye Trac, it was fun knowing you, but I want to spend more time developing my app and less time screwing around with plugins.

Walking the Talk

2008-06-07T08:33:00.004-06:00

At work, I am the champion of many technologies which further software quality, including:

Continuous integration (Hudson)
Version control (Subversion)
Test Driven Development
Unit testing (JUnit)
Defect tracking (JIRA)
Light documentation via a wiki (Confluence)
Coverage analysis (Cobertura)
Profiling (YourKit)
Automated builds/testing (Ant)

Recently I started playing around writing a newsgroup reader in Ruby. So I immediately started using all these tools, right? Wrong. I just started hacking away. I was able to establish a proof of concept pretty quickly and learn about some of the gotchas. But after being at this a while, I started feeling lost. I'd forget how something worked and have to rediscover something I'd figured out previously. I'd have ideas for things that I wanted to do, but no place to write them down and think about priorities. Perhaps it was time for me to start walking my talk.

At home I have several constraints I don't have at work. I am writing in Ruby, not Java. I am not willing to pay the money for commercial tools (JIRA, Confluence, YourKit). I think Ant is a silly way to do builds and automated tests. So I started thinking about what my own tool set might look like.

Continuous integration: Hudson. I have written about this before, but it is a fantastic continuous integration tool. Kohsuke Kawaguchi is constantly improving it, so it is just getting better and better.

Version control: Subversion. I see no reason to switch here either. Version control with only one developer is dead-simple anyway, but why bother learning something new when Subversion works so well?

Unit testing: Test::Unit. It ships with Ruby and is the spiritual equivalent to JUnit. ZenTest also looks very interesting - promising to go far beyond the simple XUnit frameworks. I'll take a look at this as well.

Defect tracking: Trac. There are endless open source defect trackers out there. Maybe some are better than Trac. But I like that it is light weight, has an integrated wiki and has bazillions of plugins, including Hudson integration. It is also written in Python, my previous favorite language before Ruby.

Wiki: Trac. See Defect Tracking above.

Coverage Analysis: rcov. To the best of my knowledge, this is the only game in town. The outputs look very similar to Cobertura. I haven't played with this yet, but it is on my short list.

Profiling: ruby-prof. This seems to be the clear winner over Profile. I could not find a single reference comparing the two that didn't favor ruby-prof.

Automated builds/testing: Rake. There are quite a few Ruby-based build tools available. Since I am planning to use Rails in my implentation, Rake seems like the obvious choice.

That is the plan. I'll see how it all works out and report back on my progress. Even if it is a big failure, I will have taken a sip (gulp?) of my own medicine.

Google calendar plugin for Hudson

2008-03-08T07:05:00.003-07:00

For some time now I have been a major fan of Hudson, an open source continuous integration server. I recently gave a presentation to my group on using Hudson not only to run our continuous integration effort, but also to routinely check for anomalies in our database and environment.

One question that my boss posed was how we could monitor results from home, since the server Hudson runs on is not exposed to the outside world. At first I did not have a good answer, but a quick trip to the Hudson plugin wiki. First of all, I was shocked to find there were nearly 50 plugins available. Hudson is clearly catching on. The one that seemed like the best solution for our team is the plugin to publish results to Google Calendar. With some simple setup in Hudson, build results are published to a shared Google Calendar. None of the build artifacts are published, so there is no risk of loosing important resources. At the same time, we can stay current with the health of our system from home.

We could also use email for this, but this seems like a more elegant solution. If you are looking for a way to access your Hudson results from outside your firewall, this just might be your ticket!

Using JRuby when package names include capitals

2008-01-19T07:33:00.000-07:00

I discovered solutions to a few problems I was having using JRuby with our Ruby libraries at work. The first relates to our domain name and "magic" names in JRuby. For example, you can write something like:


org.junit.Assert.assertEquals("Silly example", 4, 4)

This code will run just fine. But now consider my company - samba.biz. From the example above, I expected to be able to write:


biz.samba.Assert.assert_equals("Silly example", 4, 4)

But this gives me "NameError: undefined local variable or method 'biz' for main:Object". It turns out that 'com' and 'org' (and perhaps others?) are "magic" names that support this syntax, but 'biz' is not. After some fooling around, I was able to get the above example to work like this:


include_class 'biz.samba.Assert'
Assert.assert_equals("Silly example", 4, 4)

This also solved another problem I was having at work. For some reason, we have some package names that include capital letters (why?). For example


biz.samba.backend.states.CA.utils.ProcessEpnsByList

JRuby wanted to think that CA was a class name, since it starts with a capital letter. Again, the syntax above solves this issue.

http://blogs.sun.com/coolstuff/entry/using_java_classes_in_jruby
was a big help in helping figure this out.

Correcting blank image and link dialogs in Joomla

2007-12-15T12:53:00.000-07:00

I was going nuts! When I worked on my friend's Joomla site from home, the Image and Link dialogs appear just fine:

But when I tried it from work or when my friend did it from her house, the dialogs came up blank:

I tried every combination of browser setting and browser (Firefox, Opera, IE). Nothing. What was so magical about my machine at home?

I found the answer on Alex Le's blog. It turns out that my site home (as defined by $mosConfig_live_site) was set to www.site_name.com while I was firing up the administration console with http://site_name.com/administrator (notice the lack of www). Once I launched from http://www.site_name.com/administrator, the edit dialogs are dandy again, just like they are from home. Thanks, Alex!

Getting JRuby gem utility to work

2007-12-15T12:27:00.000-07:00

I've been cursing my brains out trying to get the JRuby version of gem to work. After much hunting around, I found that a Java memory size issue was causing the problem. Things have been working fine since I started installing gems with:

jruby -J-Xmx512M -S gem install

My apologies for having lost the original reference to this information. This bug entry might have been where I first found the solution.

JRuby as a system programming language

2007-12-15T11:17:00.000-07:00

At work, we are starting to refactor some of our system utilities. The scripts interact with these basic components:

Postgres databases
FTP servers
Files on various servers
Java applications
Specific system calls

My boss, being from an "older generation" ;-), favors Unix scripting (sed, bash, etc.) . Being a Ruby/JRuby enthusiast myself, I wanted to show my team how easy these tasks are in JRuby. While sed might take fewer characters in some cases, I think JRuby strikes an ideal match between terseness and readability.

Note that the following code will not run on your computer, at least not without setting up a Postgres database and changing the FTP server info, etc. But it will give you a good flavor for system programming in JRuby.


#

# Copyright 2007 Effectiveqa.com. All rights reserved

#



def postgres

# Accessing Postgres from JRuby

# http://ruby.scripting.ca/postgres/rdoc/



  $LOAD_PATH << 'C:/jruby-1.0.1/lib/ruby/gems/1.8/gems/postgres-pr-0.4.0/lib'
   
  require 'postgres'

  con = PGconn.connect('localhost', 5432, nil, nil, 'foo', 'postgres', 'password'

  res = con.query("select * from bar")

  puts res[0][0]

end



def ftp

  # FTP Access

  # http://ruby-doc.org/stdlib/libdoc/net/ftp/rdoc/index.html

  require 'net/ftp'

  Net::FTP.open('effectiveqa.com') do |ftp|

     ftp.login('username', 'password')

     ftp.chdir('public_html')

     ftp.list('*.css').each {|file| puts file}

     ftp.gettextfile('main.css', 'local_main.css')

  end

end



def ftp_password

  # Show how easily JRuby can handle password problem

  require 'net/ftp'

  begin

     ftp = Net::FTP.open('effectiveqa.com')

     ftp.login('effectiv', 'BADPASSWORD')

  rescue

     puts "Here is where I fix the password"

     ftp.login('username', 'password')

     ftp.chdir('public_html')

     ftp.list('*.css').each {|file| puts file}
  end

end



def file_tricks

  # Different file/directory manipulations

  # http://www.ruby-doc.org/stdlib/libdoc/fileutils/rdoc/index.html

  require 'fileutils'

  include FileUtils



  begin cd('dfsdf') rescue puts 'Oops, had a problem...' end

  chmod(0777, 'local_main.css')

  puts "Don't match!" if ! cmp('local_main.css', 'local_main_1.css')

  cp('local_main.css', 'local_main_copy.css')

  puts "Do match!" if cmp('local_main.css', 'local_main_copy.css')

  puts pwd

  rm('local_main_copy.css')

  begin rm('DOESNT_EXIST') rescue puts 'Caught one' end

end



def file_tests

  # Very nice tests

  # http://ruby-doc.org/docs/ProgrammingRuby/html/ref_m_filetest.html



  puts FileTest.directory?('example.zip') # false

  puts FileTest.size('example.zip') # 1723

  puts FileTest.file?('.') # false

  puts FileTest.file?('example.zip') # true

  puts FileTest.readable?('example.zip') # true

  puts FileTest.writable?('example.zip') # true



  include FileTest

  puts directory?('.') # true; needs include

  puts file?('.') # false

end

def use_java

  # Show off how easy it is to use Java from JRuby

  include Java



  zf = java.util.zip.ZipFile.new('example.zip')

  puts "There are #{zf.size} files in the zip."

  zf.entries.each {|entry| puts "  file: #{entry}"}

end



def split_files

  # Easy processing of DB dump files



  lines = IO.readlines('pipeFile.txt')

  lines.each {|line| puts line.split('|')[1]}

end



def system_calls

  # JRuby has full access to OS calls

  lines = `df`.split("\n")

  lines.shift   # Read off header line

  lines.each do |line|

    fields = line.split

    next unless fields[0] =~ /C:$/   # Only care about the C: drive

    percent = fields[4].to_i

    puts "Holy cow, disk almost full" if percent > 95

    puts "Disk > 50% full" if percent > 50

  end

end



# Some interesting Ruby libraries

# http://beaver.net/slides/ruby/10-easy-pieces.html

# Library doc: http://www.ruby-doc.org/stdlib/

# Pathnames: http://www.ruby-doc.org/stdlib/libdoc/pathname/rdoc/index.html

# MD5, sha1: http://www.ruby-doc.org/stdlib/libdoc/digest/rdoc/index.html

# Waking trees: http://ruby-doc.org/stdlib/libdoc/find/rdoc/index.html



puts "\n**** I can access postgres\n"

postgres

puts "\n**** I can access FTP\n"

ftp

puts "\n**** I can tell if an FTP password failed\n"

ftp_password

puts "\n**** I have all kinds of *nix file commands\n"

file_tricks

puts "\n**** I also have all kinds of cool file tests\n"

file_tests

puts "\n**** I can easily use Java code\n"

use_java

puts "\n**** I can split up Dave's db dump files\n"

split_files

puts "\n**** I can make OS calls with ease\n"

system_calls

The only shortcoming I could find with JRuby is that it doesn't support all the command line options that Ruby does. So you can see that JRuby falls behind in the terseness of its one-liner to rip a particular field out of a pipe-delimited dump file, though I would argue it is more readable.

#> ruby -n -a -F"\|" -e "puts $F[1]" pipeFile.txt

#> perl -n -a -F"\|" -e 'print "$F[1]\n"' pipeFile.txt

#> jruby -e "IO.readlines('pipeFile.txt').each {|ln| puts ln.split('|')[1]}"

Using Subversion svn+ssh protocol without typing passwords

2007-12-05T09:28:00.001-07:00

I have been having more and more problems using the svn: protocol for Subversion. So I've moved to using the svn+ssh: protocol. The only frustration was having to constantly enter my password (sometimes many times) for each svn command. It turns out that svn+ssh: is using standard SSH protocols, so SSH key authentication solves the problem.

I found a nice write-up on Aaron Toponce's blog on how to set it up.

It is a little more complicated for Windows. I will address that in a future installment.

New Mexico Information Technology Infrastructure & Architecture group forming

2007-11-28T20:26:00.001-07:00

Today, I attended the first meeting of "New Mexico Information Technology Infrastructure & Architecture," the brainchild of Thierry Thelliez, oft contributor and board member of the AgileNM group that I run. Thierry envisions the group positioned somewhere between the AgileNM/SPIN groups and the technology-focused groups (like the New Mexico Ruby and Java users groups) in terms of where it fits on the technology vs. business continuum. Some of the topics discussed that caught my eye:

Server virtualization
Web frameworks
Open source
Scripting languages (e.g. Ruby, Python)
Wikis
Distributed architectures

The next meeting is scheduled for January, 2008. Visit the NMITIA site for more details and to add your input for topics you'd like to see. Like the AgileNM group, Thierry's vision is that group members will largely drive the agenda. So help drive the group in a direction that will meet your professional needs.

Flex Functional Testing with Ruby

2007-11-24T07:33:00.000-07:00

I have been playing with Flex the last few days, creating an application to screen scrape audio equipment sales sites on a periodic basis and send me email when a component I am looking for is posted. Currently, I am a little hung up trying to use WebORB interface between Flex 3 Beta 2 and Ruby on Rails. The examples appear to all be written for Flex 2. So far, I have found some references that make it sound doable, but I am not quite there yet.

In the meantime, I ran across FunFX, a Ruby library for functional testing of Flex applications. This is very exciting to me since, until now, I was only aware of Mercury (oops, HP) QuickTest Pro as a testing option for Flex. To be able to test Flex apps with code very similar to the Watir tests I write now is very exciting. I have not fired it up yet, but as soon as I have some experience with it, I'll report back.

Firefox 3 Beta Available!

2007-11-20T20:09:00.003-07:00

I just read in the New York Times technology section that the Beta for Firefox 3 (Minefield) is now available. Being too curious, I had to give it a try.

It doesn't work with most of my plugins yet (although I did manage to load Adblock Plus!). The first big problem I found was editing this post. The Javascript editor for Blogger did not respond particularly well, so I am finishing this up with Firefox 2. And, so far, I have not noticed any killer features I can't live without.

Even so, I still plan to use Firefox 3 for most of my casual browsing. The reason I plan to stay with it is speed. Except for Opera, this is the fastest browser I have seen. Mozilla also claims that it has fixed 300 memory leaks.

Killer CMS - Joomla

2007-11-10T11:18:00.000-07:00

I hadn't had much need for a Content Management System (CMS) until, recently, I offered to help a friend get her site up and running. She has written professionally for the Web for years, but the actual web design is handled by the organizations she works for. I was going to whip something up for her myself but with zillions of CMS systems available, I didn't see any reason to start from scratch.

She doesn't have a hosting service yet, so I started looking around at some of them. Many hosting services include CMS support. It seemed like it would be easier to start with one that was pre-installed, so I first looked for services with pre-installed CMS. Several providers supported Joomla!, so I decided to give that a try.

Joomla!'s demo site was down, so I went ahead and installed it on my local machine. It was literally installed in five minutes. My home box is a Windows machine and I already had the Apache/MySql/PHP stack installed using XAMPP. If you haven't tried XAMPP, it is just about the easiest way to get the stack up and running on Windows. All I did was drop the .zip files contents into my xampp\htdocs directory, point my browser at http://localhost/joomla/install, answer a few questions and I was done. So many things like this I get frustrated just getting things set up that I give up before I ever start. Not so with Joomla!.

After the install process completes, you have to remove the install folder in order to continue. This is a security feature. Once it is removed, pointing my browser to http://localhost/joomla produced a page that looked like:

So far, I have found Joomla! to be intuitive and easy to learn. But what really makes it exciting to me is the depth of the community behind it. For example, Googling on "Joomla templates" returns 7.9 MILLION hits. The first template site I went to has 1,500 free templates available! Hundreds of plugins are also available. I guess I am the only one in the world who didn't know about Joomla! until now.

My comments are based on the 1.5 RC3 (codename Takriban) release. The current production version is 1.0.13, which I have not tried.

One thing that frosts me is that it is written in PHP. I consider this a hack language, not worthy of real software efforts. Yet people are getting cool things done with it. Too bad it isn't written in Rails or Django. ;-)

Using Watir and Hudson to Monitor Performance

2007-10-30T12:03:00.003-06:00

Currently, I have a full suite of Watir regression tests that I use routinely during the development of our web app. For those who may not be familiar with it, Watir is a Ruby library which allows programmatic control of Internet Explorer to navigate through an application under test (AUT) and determine if the application is performing properly. I'll have more to say about Watir another time.

One of the challenges with our application is performance. I wanted to have a way of determining which changes to the app, network, hardware, etc. were having a positive or negative impact on performance. By wrapping some timing functions around the most timing-sensitive issues in the regression library, I created a performance benchmark.

The Timing Wrapper

The simplest timing wrapper could be as simple as this:


start = Time.now
# Do your operations here
elapsed = Time.now - start

I started this way then decided I wanted a little more capability, so I came up with a hierarchical timer, to easily time activities within larger activities. A likeness of the library I use is available here.

Watir

Watir is a very nice, open-source Ruby library that allows driving an IE session from a Ruby program and then scraping the pages to determine if a proper result was generated. I started with Selenium, but ran into problems with https and a few other things that Watir was able to handle. The one area where Selenium really beat Watir was in the quality of the recorder. But pretty quickly you learn to write the scripts by hand, with a little help from the IE Developer Toolbar. Watir does have a recorder, but I did not find it very useful.

A simple test might look something like this:


def list_page_to_page
 # Try some list features
 [['User 1', 'List 1', 'Company 1 page 1 -> page 2'],
  ['User 2', 'List 2', 'Company 2 page 1 -> page 2'],
  ['User 3', 'List 3', 'Company 3 page 1 -> page 2'],
 ].each do |user_name, list, label|
    login(user_name, $password, @test_site)
    @ie.link(:text, list).click

    # Now try paging...
    time(label) { @ie.link(:url, /page=2/).click }
    logout
 end
end

This example logs in as three different users, clicks on a link with particular text in it and then times how long it takes for the page to return when clicking a link that goes to page two of a multi-page list.

Hudson

For a test like this to be valuable, it needs to be run on a regular basis. While cron or Scheduled Tasks can do this, I much prefer Hudson, the continuous integration engine we use for verifying our commits do not break our JUnit tests. Hudson normally fires on actions like Subversion commits, but it can also schedule builds on a regular basis. So I set the performance task to run every hour and it just cranks away, sending me an email if there are any problems. While there are a lot of continuous integration engines available, both open-source and commercial, I am extremely pleased with both the quality of Hudson and the rate of development. I'll be writing a lot more about Hudson as soon as I get a chance. Check it out!

Give Me My Screen Back!

Since Watir actually fires up an instance of IE to do its work, it can be very disconcerting when a test fires off on your desktop machine while you are working on something else. I did have Watir minimize IE as soon as it started, but it would still be stealing focus from me and sometimes things I was trying to type in an editor would end up in some form field in IE and break the test.

I tried using another box, but it turns out it was slow enough that the tests didn't run very well. What finally worked out was using a virtual machine via VMware. Their VMWare Server is free, though you do have to sign up to get activation codes. So far, the amount of spam has been quite reasonable. I have a fairly beefy machine at work (Intel dual core, 2GB RAM), but I don't notice any degradation in performance.

Graphing the Results

While I could export the results to Excel (or better yet, IBM Lotus Symphony), I ended up using the powerful JFreeChart package. It offers an amazing amount of control in creating just the graph you want. I looked for a Ruby graphing package, but everything I found was pathetic compared to JFreeChart. Since I am not much of a Java fan, I wrote the code in JRuby. This also allowed me to pull in information from other sources, including our Apache logs and Zabbix. Here is a sample chart:

This particular graph shows that on a weekend with almost no traffic to the site (pink line), there were still performance hits on both staging (blue line) and production (red line) around 4 AM and 4 PM. These periods also corresponded to increased CPU load on the server (black line).

Summary

By using several open-source tools and a few snippets of code, I am able to record and graph the performance of our system over time and understand the impact changes to our system and operating environment have on our user experience. I recommend adding performance testing to your standard regression test suite. The immediate feedback is very helpful in heading off design decisions with negative performance implications.

Open Source Load Testing with WebLOAD

2007-10-19T15:17:00.000-06:00

Introduction

For some time now I've been playing with Grinder as a load-testing tool. There was much to like about Grinder - it seemed like one of the more advanced open-source load/capacity testers. The scripting language is Jython, which I love as a result of long being a fan of Python. Still, it is clearly underpowered compared to commercial tools. Then, one day when I was visiting Open Source Testing, I ran across WebLOAD and I knew that I was on to something special.

WebLOAD claims to contain over 250 man-years of development. It was converted to open source in April 2007. It was created by Ilan Kinreich, a co-founder of Mercury Interactive. In my opinion, this is a commercial quality application. One area that really shows is the documentation. After trying to piece together how some open-source tools work by reading forum entries, having real documentation is a godsend.

Getting Started

WebLOAD has an excellent Quick Start guide. I highly recommend starting there. The basic steps for running a load test are:

Create an Agenda
Configure a load template
Configure session options
Run the test
Analyze the results

Create an Agenda

Creating agendas are easy. WebLOAD comes with a proxy recorder. You simply start the recorder and execute a user scenario in your browser. The proxy is set up automatically - no manual steps in your browser options. The commands are stored as Javascript. I really appreciate not having to learn yet another proprietary language.

The recorder also has playback and debug capabilities. It is really nice to get your agenda working cleanly before running 1,000 concurrent versions of it!

Configure a Load Template

A load template is when you actually configure the test to be run. What agenda(s) will be used? What statistics will be collected? How many simultaneous users will you simulate? Will they hit the site all at once or ramp up linearly? What host or hosts will provide the virtual clients?

All of this is done from the main WebLOAD console, not the IDE that you used to record the Agenda.

I highly recommend using the wizard that runs through the configuration options to set up the template.

Setting Session Options

Under Tools > Current Session Options, there are a large number of controls you can set, like the maximum time it can take for a page to return before it is considered a failure. In the Quick Start guide, they set a 20-second maximum time and verify 5% of the sessions.

Run the Test

From the Console, you just click the the Start Session icon and you are under way.

I love watching the various stats displaying in real time. Very cool! When running a stress test, monitoring the log view gives a good indication of when things start to break down.

Analyze the Results

WebLOAD has the ability to graph any number of parameters and statistics, including things like CPU and disk utilization on the server(s). It also supports export of data to popular formats such as Excel.

Hurdles

I did have to overcome several problems before WebLOAD would work with the main application I test. The first is that my application uses URL rewriting to track user sessions. So any URL recorded with the IDE will be incorrect at testing time, since each session is assigned a new sessionid which is part of the URL. This is where WebLOAD's professional documentation comes in again. On their forum, a white paper called "Session Management in Performance Testing [How-to]" describes handling URL re-writing along with cookies and hidden-form fields. You've got to love professional documentation!

The second issue I haven't figured out yet is why I was not able to record from our test site. I had to build sessions from our production site, then went in and modified the URLs to reference our test server. I'll let you know what I figure out with this one.

Gotchas

I learned a lesson the hard way. Do not run a stress test on your production system during business hours! (Notice issue number two in "Hurdles.") I managed to run an Agenda that hadn't been converted to the testing server's URLs. Ouch. It brought down the server at one point. Not a good thing to be doing at 10 AM! There is one good thing that came from this - it convinced me that WebLOAD was actually doing something. It's not just a program that draws pretty graphs. ;-)

Summary

Indeed, I am impressed with WebLOAD. It comes closest to an "ideal" load-testing solution than anything I've seen in the open source world. I especially like the professional quality documentation. With most open-source load testers, I give up in frustration before I ever get anything working. With WebLOAD, I had things working in a morning.

Perhaps others who have experience with commercial tools such as HP's LoadRunner will chime in on how it compares to the "big boys".