June 26, 2015 / by Torben Greulich / Software engineer / @mr_tege

Feedimport with jenkins

One of our daily challenges is to download and import plenty of product data in different forms. When we were planning a new tool to do this, we came up with Jenkins. This post shows why we chose jenkins and how we implemented a Jenkins plugin.

Tool requirements

The main requirements for our new tool:

• download product data from different sources at different times
• parallel download from different sources
• convert and store product data
• show progress of each running process in a UI
• show statistics about past processes in a UI
• multi-client capability

Why Jenkins?

Technically Jenkins is an open source build and continuous integration server with several features. Out of the box it supports features like integration of LDAP as security, version control by git, master / slave, it offers easy scheduling of jobs and parallel execution of different jobs, it offers a REST API and it comes with a build in UI - to mention just a few features.
Furthermore, Jenkins can be easily extended by plugins.

With regard to our requirements of a new tool, we see, that a plenty of our requirements are satisfied by Jenkins. Especially the execution handling of Jenkins satisfies the main requirements. Due to this, we decided to use Jenkins as a basic tool and develop a plugin that handles the converting and storing of product data.

Jenkins Plugin

Basic Build

It’s very easy to start with a Jenkins plugin, just read the manual for basic steps.

In order to have a simple BuildStep in Jenkins, which you can add to every build, you just need to implement the prebuild() Method of the abstract class hudson.task.Builder. You can also edit the associated config.jelly for this build, to configure the UI.

src/main/java/.../HelloWorldBuilder.java
src/main/resources/.../HelloWorldBuilder/config.jelly

Project

It becomes a little more complex, if you want a custom build in Jenkins. You have to use the hudson.model.project and hudson.model.Build Interfaces in Jenkins. The Build interface is very simple, you just have to set the correct generics in your custom build. e.g.

public class CustomBuild extends Build<CustomProject, CustomBuild> {
  public CustomBuild(CustomProject project) throws IOException {
    super(project);
  }

  public CustomBuild(CustomProject project, File buildDir) throws IOException {
    super(project, buildDir);
  }
}

Afterwards you have to implement the project. The most important method is getBuildersList(). This configures, which builder will be preselected by default.

public class CustomProject extends Project<CustomProject, CustomBuild> implements {
   
...
     
  @Override
  public DescribableList<Builder,Descriptor<Builder>> getBuildersList() {
     DescribableList<Builder, Descriptor<Builder>> result = super.getBuildersList();
     if (result.isEmpty()) {
        result.add(new CustomBuilder( ... ));
        ...
     }
   
     return result;
  }

...

Like the hudson.task.Builder you can also configure the jelly files in:

src/main/resources/.../CustomProject/main.jelly

Additional

A nice interface we used in our implementation is the hudson.model.Action. You can add any action to every hudson.model.Build. Furthermore, you can edit the UI behaviour by editing the *.jelly file, like everything else in Jenkins. There are already several interesting implementations of action. We implemented a ReportingAction to transport some reports and display them in the UI.

Installation

Just package the project with maven and you will find a *.hpi file in your target folder.

Deployment

We are not using the default installation by adding the plugin as *.hpi to an existing Jenkins server. We rather add the plugin directly to the jenkins-server.war. First we add the jenkins-server as dependency in our maven pom.xml. Secondly, within a maven build step, we unpack the original jenkins-server.war, add the *.hpi into the plugins folder of the unpacked Jenkins and pack the Jenkins server as *.war again. We use different maven build plugins like ant, maven-dependency-plugin and maven-war-plugin for that. Afterwards just run the jenkins-server.war in a tomcat.

Conclusion

We have run the Jenkins with our plugin for several months now and are very pleased with the result. The authorization, REST API and the UI fulfils our expectations. But the biggest feature for us was the built in execution handling of jobs. We can easily add, edit and remove or start and stop our custom jobs and the execution is fully handled by Jenkins. Furthermore an easy configurable scheduling is integrated in Jenkins.

This was a big advantage in building the software, because we could concentrate on the main features and did not care about the things Jenkins supports out of the box.

As might be expected, not everything worked perfect. We ran into problems with Jenkins slaves.

To run a plugin, the Jenkins slave needs the plugin and all its sources. Either all your classes in the plugin must be serializable, so that the plugin can be transfered from master to slave by Jenkins itself or the plugin must be installed on the slave manually.