While creating a proof of concept of a new, cloud based environment of a big news platform, we used chronicreplay to test the behaviour of the new environment under a realistic load, similar to the productive system.
chronicreplay replays Apache (or any other) log files in the correct time and order, see chronicreplay's README for more information. As we had the time of the request in the original Apache log files, we were able to plot and analyze the time differences.
The peaks in this graph are caused by Java's default garbage collector cycles. Switching to a parallel way those peaks disappeared (visible in the data of Server 1). FasterJ has an excellent article explaining Java's garbage collectors.
I updated chronicreplay to write more logfiles, better structured than before, so that it is much easier to use gnuplot to visualize the data.