Upgrading legacy libraries of a (server-side, running on Node) legacy JavaScript application, I had to upgrade  (and replace) following libraries:

• cache-manager
• cache-manager-redis
• cache-manager-redis-store

As there is missing documentation (node-cache-manager-redis-yet does not have any at all) and there is no current example using a memory cache and Redis at the same time, I spend some time finding the solution. So it might be a useful shortcut for you, if you need to do the same task.

TL;DR

The code I eventually came up with (in short, for more details read further):

The problems and my solutions

• cache-manager's API changed. It changed for better, so this is a good thing, just a bit of work and restructuring of code, as it relies on Promises now and the application I am working on is already relying on a working cache object on startup.
  So I created a wrapper object taking care of the awaits.

• While the memory cache accepts null and undefined as result values of a wrapped function, cache-manager-redis-yet does not, and instead of omitting it, it throws an error. If you don't want this error to pop up in your application, you have to catch it (see line 24).
• cache-manager-redis is not maintained any more, so I removed it completely.
• cache-manager-redis-store is not "officially" supported by cache-manager, so I used the recommend one: node-cache-manager-redis-yet, which unfortunately has absolutely no documentation and no example.
  It is a fork of cache-manager-redis-store, so we can use their example to create a Redis cache. Keep in mind, we need the full Redis URL here, including the schme redis://, just the host won't work.

• Inconsistencies in the API using cache-managers multiCache: While creating a cache with caching(...), which returns a Promise<MemoryCache>, multiCache(...) is only accepting Cache objects as parameters, no promises. So we either have to wait until the caches are created, or wrapping it into Promise.all(...)

What does this mean in practice?

If you have, for example, a docker image containing any kind of data storage (in my case a container used for development and testing purposes of Sophora CMS server), where the volumes containing data are declared as docker volumes in the original image, it is not possible to create a new image (with your personal, project specific test data) using docker commit.

But there is a way to put content into directories declared as volumes: Create a Dockerfile that is ADD ing this content. Of course, this means that you have to copy the content from your prepared container somwhere where your new Dockerfile has access to.

FROM baseimage

ADD /path/to/data /path/to/data/volume/directory/in/container

The resulting image will contain data in the directories, even if they are still declared as volumes.

Developing Java applications accessing data from Solr is easier with spring-data-solr. You can just use Repository interfaces, auto generated findBy... methods or @Query annotations to define your data sources. Unfortunately using this method it is not possible to configure the name of the Solr core (or collection). The name of the collection is automatically defined based on the name of your Document class or by the @SolrDocument annotation.

@SolrDocument(collection = "collection1")
public class YourSolrDataClass {
    @Id
    public String id;
    public String text;
    // ...
}

While your Repository can be just an interface. See Spring-Data-Solr's showcase implementation on Github, which is downloadable from the "Getting Started" page as well.

Configuring the collection name is not possible. However, there is a workaround with a bit of code. Unfortunately this means that you have to implement the methods which are actually querying Solr, you can't use the @Query annotations and auto generation magic (which is, by the way, explained very well in this article by g00glen00b) in your Repository interface any more.

First step, reimplement org.springframework.data.solr.repository.query.SolrEntityInformation. An implementation of this interface is handed around internally in SimpleSolrRepository, which is used commonly as implementation of your repository.

Then, implement your Repository, extending from SimpleSolrRepository.

That's all. After that, you can setup your Spring context.

If there is a simpler solution, I am looking forward to it, drop me a line. I'd wish to see this feature (or more simple) included in Spring Data Solr.

Within the last years developing, prototyping and designing software projects I collected a set of best practices for setting up and developing software projects. I wrote down a more general overview in my article about about defining key factors for successful software development.

The following (opinionated) principles are not limited to new projects. They are valid for yet created and legacy projects as well. Following them will make the maintenance of your projects and onboarding of new developers much easier and though, cheaper and faster.

I made my experiences mainly in the Java world, creating libraries, command line applications and web applications with Java or Groovy, using the Spring framework very often. So especially part 3 and 4 will be tied to the Java ecosystem. The examples may not, but the general principles will be valid for other ecosystems as well.

• Part 1: The build and dependency management tool
• Part 2: The version control system
• Part 3: Context Definition and Dependency Injection
• Part 4: Modularisation
• Part 5: Documentation and the README
• Part 6: Executable documentation and deployment prototype (aka "The Dockerfile", or similar)

Part 1: The build and dependency management tool

Dependency managenent and build configuration

Building your project and dependency management are two different tasks. You might want to use (or be using already) different tools. This is quite common (but not necessary) in front end projects, where npm and bower (npm for development dependencies, bower for front end, runtime dependencies) is used as dependency management and webpack, grunt or gulp as build tool.

In the back end world both tasks are commonly solved by the same tool.

Use a build tool

First of all: If your software needs to be prepared or build, yes, please use a build and dependency management tool (or more of them, if needed). Don't rely only on your IDE configuration^[1]. There are several reasons for that.

• It will save you time^[2]. The build tool offers solutions to most issues you will have to address manually otherwise.
• Independent instance of truth: The build of your software is ok, if it succeeds with your build tool, not if "it works on my machine".
• Automation and Continous Integration: Build tools are, in contrast to IDEs, runnable by automated tasks on server systems. This enables Continuous Integration and Continuous Delivery.

Choosing a build tool

So which build tool you should choose? While it fits your needs, it does not matter. I encourage you to try out several ones, not only experimenting, but with real projects, too. With this experience you will be able to decide in the future which build tool suites best for a new project.

Besides that, consider these points choosing a build tool:

• Does it fit in your technology stack? This means, for example, that you should use something able to communicate with Maven repositories if you are shipping your software in Maven repositories, and that it should be able to fetch your dependencies from where they are.
• It is easier, especially if you are trying a new tool, if you know the technology or program language the tool is using. This will help you interpreting error messages, debugging and writing build scripts.

If you want to dive in deeper in build tools and their characteristics, read Neal Ford's article about hating or leaving Maven and the difference between composable and contextual tools.

The aim of this article is not a detailed overview over existing build tools (there are other articles specialized in this topic), it should encourage you to use a build tool as primary build authority and give you some hints how to choose one. Nevertheless I will share a few experiences I made with build tools.

Running your project

I met a lot of projects without any hint on how to start the software. Every developer had the run configuration saved in his IDE. This works... at least on "my machine" (from the developer's point of view).

If technically possible, provide a run task in your build tool configuration for development purposes. This is quite common^[3] for Maven (e.g. mvn tomcat:run for web apps, Gradle (e.g. ./gradlew bootRun for Spring Boot applications, or npm build && npm serve, to assemble and serve front end applications).
Providing this, you will have the most important part of "The run script"^[4].

My personal experience with concrete build tools

I am mainly developing in the Java world in Java and Groovy, using Maven and Gradle as build tools.

I personally used Maven a lot and very, very extensively and at some point I started to agree Kent R. Spillner in "Maven is a horrible implementation of bad ideas". But that's not completely true. I agree, that Maven is far away from being the best build tool in the world. It is horribly unflexible, creating custom tasks is only possible with workarounds, writing own plugins is unnecessarily complicated and your pom.xml files will get terribly long.

But the dependency management just works, it is fast, and it does it's job: compiling, testing, building and even running or deploying the software. There Maven's features end. If you want Maven to do something else, it often gets very complicated up to impossible, because you are bound tightly to Maven's defined project life cycle.

Gradle seemed to be the solution: flexible, enables you to create custom tasks and a relationship between those and using the well known dependency infrastructure of Maven. After using Gradle for a few years now, in some cases I am considering switching back to Maven again. I love the flexibility of Gradle. But it is slow (although Gradle tells us the opposite, but that's not the experience I made), and I found me fighting with configuration problems Maven solves out of the box.

Maven does a lot of things which need to be configured manually in Gradle. On the other hand: If you need some custom tasks not fitting in Maven's tight build life cycle of Java projects, prefer Gradle.

Lately I tried Kobalt, but had to discard it as I worked on a project depending on libraries provided in an old maven1 repository, which is not supported by Kobalt, and because of lacking IDE support^[5]. I will try again in my next project, hoping that the IDE support will improve.

Using the Maven dependency ecosystem, there are still SBT from the Scala world, Ivy, Leiningen (which mainly targets Clojure projects) and Buildr to try.

I did not try Rake for Java projects as Spillner suggested in 2009, Thoughtwork's Technology Radar recommended in 2012 and Martin Flower wrote about, but it does not seem to be very common in the Java world, though.

For front end projects I used npm, Bower, Grunt and Gulp, but made positive experiences with npm as a single dependency management and "build coordination" tool, without any others, as Keith Circle recommended in 2014 already. I call it "Build coordination tool", because npm actually does not build my project, but manages and runs the scripts that will do so. Typically this includes minifying und bundling sources. As build tool (or "module bundler") I recommend webpack^[6].

I recommend Addy Osmani's npm scripts boilerplate gist to pick the npm scripts you need for your project.

To bring the backend and front end worlds together, there is maven-frontend-plugin for Maven, and gradle-node-plugin for Gradle, which just execute your npm/Grunt/Gulp tasks.

Update: Unfortunately this solution is not working any more with recent versions of Firefox and FirefoxDriver.

I spent some time currently to automate integration tests and webtests, mainly using spock and, for the web test part, geb.

Unfortunately I did not find any (simple) working example with basic auth. I spent quite a few hours trying it and found a solution now. As stated here, it should be possible to pass username and password in your url: http://user:pass@host.tld.

To avoid an alert box warning the user that the browser is going to login with the given credentitals, Firefox' property signon.autologin.proxy has to be set to true.

This can be achieved initializing FirefoxDriver (Groovy example) with:

driver = {
	FirefoxProfile profile = new FirefoxProfile();
	// this avoids the "Your are signing in to..." alert
	profile.setPreference("signon.autologin.proxy", true);
	
	DesiredCapabilities desiredCapabilities = new DesiredCapabilities();
	desiredCapabilities.setCapability('acceptInsecureCerts', true);
	
	FirefoxOptions options = new FirefoxOptions().setProfile(profile)
            .addCapabilities(desiredCapabilities);
	
	new FirefoxDriver(options)
}

There is an excellent example project by geb which uses the latest versions of selenium and FirefoxDriver (marionette).

I created a pull request on this example project for documentation purposes implementing this change.

A while ago I was asked about my opinion of key factors for successful and quality software development. There are a few lists published from more or less famous programmers, like this excellent collection from Jens Rantil or this from computer game development legend John Romero (scroll down to the bottom of the article).

Both of these lists do not separate more general, organisational topics from detailed, programming design specific guidelines. Both aspects are important, but the latter ones can be implemented within a development team, while the former topics might need to be addressed to a higher organisational level and though might be, depending on your organisation, more difficult to implement.

I won't mention evident best practices, such as "use a version control system", "set up a continuous integration", and so on, I assume nobody will argue about those points.

Organisational key factors

• Code Reviews^[1]: As a development team member, review code, as much as possible, not to control your team members, but mainly for knowledge transfer. If several people are working on a system, it will help to prevent re-inventions and will save time on further development if the team members are aware of the changes done to the code they are working with.
• Mentor-ship in heterogeneous teams: Heterogeneous is a complicated word for a simple fact that you will face in most teams: there are developers with more overview able to write better code with less errors in less time than others. One of the reasons is the varying amount of experience, so take measures to keep all developers on (your) track, using e.g. pair programming and code reviews (with at least two developers in front of a screen forcing them to talk directly, unlike above, without code review tools), short training or live coding sessions, ...)
• Be pragmatic: No ideological discussions. Strive for pragmatic solutions. Software development is a business, not an art^[2], you earn (or your company earns) money selling solutions, not the most beautiful piece of code or technology.
• UX: If your software needs an UI, especially a web based UI, try hard to get an UX expert which takes the principle UI decisions, not the so called designers. Most designers (all I met) are perfectionist about their layout and design^[3], but they often don't think beyond it. This means that they don't think about what should actually happen if the user clicks on their so carefully designed button or what should happen if something goes wrong^[4].

Software design best practices

• Unit Tests and Test Driven Development: Unit testable code leads to good software design and allows refactoring
• 'The Run Script'^[5]: Instead of documenting extensively and describing how to run your project, create a script running your software on a developer's machine (I recommend container or virtualization technologies (Docker^[6], Chef, Ansible, ...) over build systems (Maven, Gradle, ...), if possible, out of two reasons: First, you have a running example of a possible deployment scenario; and second, a virtualized container avoids side effects with your local system.
  I ususally set up both.
• Development environment: Make sure your development environment (if the software needs more than the run script), can be set up from scratch automatically. Minimize or avoid ramp up time, developers need to start the project. Whatever is important to know to get started, should be documented shortly in a kind of README^[7].
• Further Automation: automate all tasks you have to do more than once, this includes acceptance tests/web tests. Automating will not only save you time, it will make your environment more maintainable (testable). This implies automation of release todos.
  
  The three bullet points above, unit tests, 'the run script' and setting up the development environment are parts of automation, as well.
• Encourage innovation: With good and motivated developers innovation will happen, if the technological framework and project set up won't stop it. Assure that it is...
  • easy to switch development branches (use a clean repository^[8])
  • easy to rollback your system
  • easy to set up a new environment from scratch (see above)
  • possibility to use whatever IDE to develop, do not force your team to use one OS/software/IDE
• Avoid inheritance...: Inheritance causes way more software design issues than you might think. There are lots of articles out there about this issue^[9], consider Robert Martin's Book and Gang of Four's Design Patterns, as well.
• Operation awareness: You may call it DevOps or not, keep your developers aware of or make them take part in the deployment and operations of the software they write, this will help them to keep in mind topics like caching, distribution, synchronization, sessions, monitoring, performance, redundancy, concurrency, ...
• Keep your Code clean: Read, understand and follow Robert Martin's principles in his Books Clean Code and Clean Coder.
  As developers spend about 10 times more time reading code than writing code^[10], you can imagine that speeding up the reading process will help your efficiency.
• Frameworks: For the same reason: choose a framework not (only) considering speed of development (unless you are prototyping), but considering the cleanness of it's architecture and it's principles to keep your code clean.
• Be aware of the Pareto principle, which is so true in software development. Keep this in mind when estimating and when communicating that "80% of the task is done".
• Be pragmatic: Again.
• No ideological discussions about technology.
• If the 20% of the work of a certain task which needs 80% of the time is not really necessary^[11], don't do it. Strive for simplicity. Remember: Quality, Tests and clean code is no option!

Thanks to Don Swinscoe (@dswinscoe) for re-reading and correcting this text as a native speaker.

A generated styleguide catalogue has many advantages in every day project live (see below), but as most of the available tools did not suite my needs (difficult to set up, too much tool dependencies, bugs, ...) I created my own solution: style-code.

How a generated styleguide will help developing web project teams

It is quote common in web projects, even in bigger ones, that the "Layout" people are responsible for all appeareance topics. Most of the times this is leading to a kind of "PDD": Photoshop Driven Development. Especially (but not only) in these projects it is important to create a kind of "Layout component catalogue".

This won't solve the usability issues (Layout is not usability, as we know), but, if established as part of the layout development, encourages designers and frontend developers (those defining the styles technically, in CSS or any preprocessored technology like SASS, LESS, ...) to think in independent components.

The resulting creation of an inventory of layout definitions (definition of standard font sizes, appearance of links, paragraphs, buttons) or components (visual elements containing more atomic elements, for example teasers, menus, ...) has several advantages:

• Overview over all used styles ("Do we really need 4 different link styles? And 6 different font sizes with 10 different line-height definitions? And, if yes, why?")
• Explaining all the styles and naming them forces designers to think about the reason why this style is needed and should be applied in which cases.
• Consolidation of styles
• CSS style definitions will be more independent as they are designed for each component (avoidance of definitions like .main .right-content .marginal-element:first-child .header {...}).
• First acceptance of visual elements won't depend on a working system, feedback from the designers will (hopefully) focus just on this element, and not on any bug or feature request elsewhere on the site (this might sound strange, but will save a lot of time, according my experiences)
• Reuse of already defined components, technically, which leads to cleaner style definitions

Why style-code?

There are some tools out there to generate styleguides, like StyleDocco,
documentCSS, KSS Stylesheets or Devbridge's Styleguide.

No prerequisites (except java)

All of them do have one requirement in common: The need of (at least) npm, others need even more installed tools, like Ruby, Ruby DevKit. That makes it quite difficult to integrate the styleguide generation into an highly automated and platform independent environment (=> java world). Did you try to run e.g. documentCss from a Jenkins, without needing root access to the underlying operating system?

A lot of the npm tasks can be automated, for example with the maven-frontend-plugin. This works fine until the npm task you use needs a locally installed Ruby. StyleDocco, for example, works, but has bugs with background graphics (the frames used there are no real iframes, but javascript generated iframes, where Firefox is not able to resolve the resource URLs of the graphics).

style-code needs just java.

Separation from productive CSS code

Most of the existing tools are aiming to document the existing CSS code with examples, similar to JavaDoc. That is not what I want in my styleguide. The styleguide should be a separate document, an implementation reference showing how the elements should look like (in any browser, real HTML/CSS/JS) including implementation examples.

At the same time it needs to work with the productive resource files (CSS, JavaScript, Fonts), to be able to develop, test and find regressions without depending on the running system.

The documentation made with style-code is not embedded in any CSS files, it is a separate markdown file, which will be enhanced by iframes and live editing support by some injected JavaScript.

style-code gradle plugin and demo project

There is a demo project using the Gradle plugin to show how a styleguide project can be set up.

There are various possibilities, I am still working on the demo project and will add some more demo projects as well to create showcases for different use cases.

After using the Grok Debugger often in the last weeks, I searched for a similar service for joda's time patterns to parse strings. I did not find anything, so I created one, for you and for me.

Update on 2017-09-26: Just updated this application to use Java 8's DateTime API, removing JodaTime, using only Java 8's DateTime API.

• https://java-time-parse-debugger.herokuapp.com/

The former URL joda-time-parse-debugger.herokuapp.com ist not reachable any more.

You can find the source code on github.

While creating a proof of concept of a new, cloud based environment of a big news platform, we used chronicreplay to test the behaviour of the new environment under a realistic load, similar to the productive system.

chronicreplay replays Apache (or any other) log files in the correct time and order, see chronicreplay's README for more information. As we had the time of the request in the original Apache log files, we were able to plot and analyze the time differences.

The peaks in this graph are caused by Java's default garbage collector cycles. Switching to a parallel way those peaks disappeared (visible in the data of Server 1). FasterJ has an excellent article explaining Java's garbage collectors.

I updated chronicreplay to write more logfiles, better structured than before, so that it is much easier to use gnuplot to visualize the data.