A minimal Erlang script for load testing

Recently I became interested in Erlang, a functional programming language that excels at developing concurrent applications. From a SQA perspective, concurrency immediately made me think of performance and load testing. I am not the first one to think this - see the open source application Tsung to see just how far this idea has been taken.

I set about to see how minimal an Erlang program I could write that would implement a bear-bones load tester for web apps. Here is what I came up with:

 1: -module(lmin).
 2: -export([example/0, ramp/4, vclient/2]).
 3:
 4: vclient(Supervisor, Script) ->
 5:    inets:start(),
 6:    Total = vclient_acc(Script, 0),
 7:    Supervisor ! {self(), Total / 1000000}.
 8:
 9: vclient_acc([Cmd|Remaining_cmds], Total_time) ->
10:   Fetch_time = timed_request(Cmd),
11: vclient_acc(Remaining_cmds, Fetch_time + Total_time);
12: vclient_acc([], Total_time) -> Total_time.
13:
14: timed_request({Delay, Url}) ->
15:    timer:sleep(Delay + random:uniform(Delay)),
16:    Start_time = now(),
17:    {ok, _Result} = http:request(Url),
18:    timer:now_diff(now(), Start_time).
19:
20: ramp(Nstart, Nend, _Nstep, _Script) when Nstart > Nend -> done;
21: ramp(Nstart, Nend, Nstep, Script) ->
22:    launch_vclients(Nstart, Script),
23:    collect_results(Nstart),
24:    timer:sleep(1000),
25:    ramp(Nstart+Nstep, Nend, Nstep, Script).
26:
27: launch_vclients(0, _Script) -> true;
28: launch_vclients(N, Script) ->
29:    spawn(?MODULE, vclient, [self(), Script]),
30:    launch_vclients(N-1, Script).
31:
32: collect_results(Nexpected) ->
33:    {Sum, _Results} = collect_results(Nexpected, Nexpected, [], 0),
34:    io:format("~p: ~.2f average: ~.2f~n",
35:    Nexpected, Sum/Nexpected]).
36:
37: collect_results(0, _Nexpected, Results, Sum) -> {Sum, Results};
38: collect_results(Nremaining, Nexpected, Results, Sum) ->
39:    receive
40:       {Vclient, Time} ->
41:          collect_results(Nremaining-1, Nexpected,
42:          [{Vclient, Time}|Results], Sum+Time)
43:    end.

I am not going to write a Erlang tutorial here; I am a complete Erlang newbie and would not do it justice. The Erlang documentation section is a good place to start for those who are interested. What I will do is provide a brief overview of this application.

• Lines 4-18 are the virtual clients that will send requests to the SUT (System Under Test). timed_request (lines 14-18) makes the actual request to the server; vclient_acc accumulates the response times for each command in the script. A script is a list of commands of the format {Delay, URL}. For each command in the script, the virtual client waits for a random time in the range Delay..2*Delay and then makes a request for URL.
  
• Lines 2o-30 launch the the virtual clients. Each client runs the same script and starts executing immediately when it is spawned. The key is ramp/4, which in turn hits the SUT with Nstart, (Nstart+Nstep), (Nstart+2*Nstep)... Nend virtual clients.
  
• Lines 32-42 collect and report the results.

So is this proof of concept load tester useful for anything? See my previous article, "Results using a simple Erlang load tester" to find out.

Results using a simple Erlang load tester

Having written a minimal load tester in Erlang, the next step was to try producing useful results with it. I downloaded the latest version of Hudson, my favorite continuous integration tool. It comes "pre-loaded" with Winstone, a lightweight server container. What kind of load can Hudson handle? I started by devising a short script. After a random time within the specified range, the client requests a specific page:



Clearly this faster than most users will navigate around Hudson, so this script is something of a worst-case.

I then ran the script 10 times, recording the results:



The red cells in the spreadsheet indicate where Hudson became unresponsive. The red cells containing "10.00" actually did not respond; I added those values for the purpose of making this graph:

Pretty clearly, Hudson is comfortable with up to 9 simultaneous clients. When moving to 12, performance starts to degrade, but is still acceptable. At 15, there were some outright failures. If this Hudson instance is intended to handle more than 12 simultaneous clients, adjustments are necessary.

With a quick look through the log files, I found the problem almost immediately:

Caused by: java.lang.OutOfMemoryError: Java heap space

So it may be pretty easy to increase load capacity for Hudson. Good news - my tiny little script found a problem before my customers did.

In a future post, I'll see what a little parameter tuning does for Hudson load handling.