Update to the Hudson book

News from the Hudson by Simon Wiest book I reviewed: it is now in the press and if you order it (e.g. at Amazon), you should be able to read it by Xmas 😉 (if you can read German :-O )

Werbung

Hudson: Review and ideas

In June I had the pleasure to do a review of the first German book about CI with Hudson by Simon Wiest. The book is announced for December 2010 and it will cover topics from basic (setup, first jobs) up to deep knowledge like writing plugins for Hudson.

As with earlier reviews I had printed the book (draft), read it and made comments on the side (and Simon had to listen to them for hours 😉

But some comments weren’t for the author – they „just“ ideas which came into my mind. But because I won’t have the time for implementing them, I’ll write about them …

Capability Plugin

The main idea is automatic dispatching of jobs to slaves according to their needs. If I remember right, TeamCity has this feature too …

The Slaves

The slaves have some capabilities. There are two kinds of them:

1. automatic capabilities: result from tool configuration, operating system, jdk versions, build tool name+version, …
2. manual capabilities: the admin of the slave can define key-value pairs

The Jobs

For jobs you define requirements according to the provided capabilities. And like them there are two requirements:

1. automatic requirements: selected jdk, build tool, operating system (if you use e.g. a Windows Batch …)
2. manual requirement: define additional requirements (or overwrite the complete formula). Use key=value (aspectj.available=true) or comparisons (ant.version >= 1.8) and combine them (and, or, xor, …)

Dispatcher strategy

The dispatcher must select all slaves which fit the defined requirements. From this list it selects the one by looking at the build queue of the slave (does it have time for building?) and net-response-time (don’t forget that much data has to be transfered …)

 

Not sure if there is something like this now …. 😉

Another Admin job Hudson: ensure that there are not too many old builds …

Especially if you have a large number of jobs and they are running more often, you’ll come to a point, where your disk is full of old builds.

Hudson provides a configuration parameter for that: „discard old builds“. This will delete old builds according to the specified number of days or number of builds.

For Apache I wrote a script which ensures, that all jobs have „discard“ setting and that existing values are not higher than a defined maximum value.

/** Default-Setting for the "number of old builds" */
numberOfOldBuilds  = 10

/** Maximum of "number of days" */
maxDaysOfOldBuilds = 14

/** Should we override existing values? */
overrideExistingValues = true

/** Closures for setting default 'max number' */
setMaxNum = { job ->
   job.logRotator = new hudson.tasks.LogRotator(-1, numberOfOldBuilds)
}

/** Closures for setting default 'max number' */
setMaxDays = { job ->
 job.logRotator = new hudson.tasks.LogRotator(maxDaysOfOldBuilds, -1)
}

// ----- Do the work. -----

// Access to the Hudson Singleton
hudsonInstance = hudson.model.Hudson.instance

// Retrieve all active Jobs
allItems = hudsonInstance.items
activeJobs = allItems.findAll{job -> job.isBuildable()}

// Table header
col1 = "Old".center(10)
col2 = "New".center(10)
col3 = "Job".center(50)
col4 = "Action".center(14)
header = "$col1 | $col2 | $col3 | $col4"
line = header.replaceAll("[^|]", "-").replaceAll("\\|", "+")
title = "Set 'Discard old builds'".center(line.size())

println title
println line
println header
println line

// Do work and create the result table
activeJobs.each { job ->

 // Does the job have a discard setting?
 discardActive = job.logRotator != null

 // Enforce the settings
 action   = ""
 newValue = ""
 oldValue = ""
 if (!discardActive) {
 // No discard settings, so set the default
 setMaxNum.call(job)
 action   = "established"
 newValue = "$numberOfOldBuilds jobs"
 } else {
 // What are the current settings?
 oldDays = job.logRotator.daysToKeep
 oldNums    = job.logRotator.numToKeep

 if (oldNums > 0) {
 // We have a set value for 'numbers'
 if (oldNums > numberOfOldBuilds && overrideExistingValues) {
 // value is too large so set a new one
 setMaxNum.call(job)
 action   = "updated"
 newValue = "$numberOfOldBuilds jobs"
 oldValue = "$oldNums jobs"
 } else {
 // Correct value or we arent allowed to override.
 oldValue = "$oldNums jobs"
 }
 } else {
 // we have a value for 'days'
 if (oldDays > maxDaysOfOldBuilds && overrideExistingValues) {
 // value is too large so set a new one
 setMaxDays.call(job)
 action   = "updated"
 newValue = "$maxDaysOfOldBuilds days"
 oldValue = "$oldDays days"
 } else {
 // Correct value or we aren't allowed to override.
 oldValue = "$oldDays days"
 }
 }
 }

 // String preparation for table output
 oldValue = oldValue.padLeft(10)
 newValue = newValue.padLeft(10)
 jobname  = job.name.padRight(50)

 // Table output
 println "$oldValue | $newValue | $jobname | $action"
}
println line

// Meaningful output on the Groovy console
// (the console will output the result of the last statement)
printout = "Number of Jobs: $activeJobs.size"

In the first section I define the „constants“ (line 001-008). After that I define two closures which update a given Hudson job (line 010-018).
The basic structure is the one I used in earlier scripts …
The work here is in lines 053-088. But that’s pretty easy: check the given values and eventually set new values using the pre defined closures.
New is the last line: I dont use a >x = „“< instruction for suppressing the output. I use a more meaningful message: the number of jobs.

Hudson: Overview of the suggestd timeout settings

In my last post I explained why and how to check the timeout settings for Hudson jobs.

On our mailinglist for Hudson users at Apache there was a suggestion to get an overview of (computed) suggested timeout settings.

So here is the follow up to my earlier code …

hudsonInstance = hudson.model.Hudson.instance</pre>
allItems = hudsonInstance.items
activeJobs = allItems.findAll{job -> job.isBuildable()}
wrappableJobs = activeJobs.findAll{job -> job instanceof hudson.model.BuildableItemWithBuildWrappers}

jobsWithoutTimeout = wrappableJobs.findAll { job ->
 job.getBuildWrappersList().findAll{it instanceof hudson.plugins.build_timeout.BuildTimeoutWrapper }[0] == null
}

println "Suggested timeout values for jobs without any ($jobsWithoutTimeout.size in total):"
jobsWithoutTimeout.each { job ->
 defaultTimeout = Math.round(job.estimatedDuration * 2 / 1000 / 60)
 if (defaultTimeout < 10) defaultTimeout = 10
 String s = defaultTimeout
 s = s.padLeft(4)
 println "$s | $job.name"
}

x = ""

The new stuff is only the creation in the last few lines. Nothing special – apart from the conversion from Long to String for getting padLeft() work 😉

The result is a „table“ like this:

How to check if all Hudson jobs have a timeout?

At Apaches Hudson installation I have sometimes seen the situation where too many builds are stuck and therefore blocking the executors. And sadly for me – the executors my own jobs require …

Having a policy to use the „build timeout plugin“ and kill jobs which are running too long (thinking more of „not running any more“ 😉 is good. But having a program which checks this is better …

So I tried a little bit Groovy’in for the Groovy console:

hudsonInstance = hudson.model.Hudson.instance
allItems = hudsonInstance.items
activeJobs = allItems.findAll{job -> job.isBuildable()}
wrappableJobs = activeJobs.findAll{job -> job instanceof hudson.model.BuildableItemWithBuildWrappers}

jobsWithoutTimeout = wrappableJobs.findAll { job ->
 job.getBuildWrappersList().findAll{it instanceof hudson.plugins.build_timeout.BuildTimeoutWrapper }[0] == null
}

println "There are $jobsWithoutTimeout.size jobs without timeout:"
jobsWithoutTimeout.each { println "- $it.name" }

x = ""

In line 1 we get the reference to the Hudson singleton. Then we get the list of all item in line 2 which we filter in line 3 to get only buildable items, like our jobs. The line 4 contains the first thing special to this requirement: the item must be able to have a BuildWrapper.

But the most thing is done in line 5 which filters again with a closure: get all BuildWrappers for the job, but only if it is our TimeOut-Plugin. Because it can be registered only once, I check the first element of that list. It must be null for being a problem. Otherwise the job has a timeout setting.

After that, the last two lines are simply out … and the last line supresses the result output in the console.

Update:

Antoine Tulme had consulted Kohsuke Kawaguchi and he sees three possibilities of forcing the timeout setting:

1. We cannot make the timeout field mandatory.
2. We can create a plugin that presets the timeout field.
3. We can iterate over the projects and set a value for the timeouts en masse.

Good, so I evaluate my „iteration solution“ a little more.

We have a list of all jobs without settings and so we only have to iterate over this list, instantiate and initialize the BuildTimeoutWrapper and add it to the jobs wrapper-list:

jobsWithoutTimeout.each { job ->
 defaultTimeout = 180
 defaultFailBuild = false
 plugin = new hudson.plugins.build_timeout.BuildTimeoutWrapper(defaultTimeout, defaultFailBuild)
 job.getBuildWrappersList().add(plugin)
}

BTW – If you want to work with a plugin, you could start with the Create Job Advances Plugin – maybe this requires code enhancement … and it will only for future jobs, not for existing one.

Update:

The last update of the script for setting the timeout value is this:

hudsonInstance = hudson.model.Hudson.instance
allItems = hudsonInstance.items
activeJobs = allItems.findAll{job -> job.isBuildable()}
defaultFailBuild = true

println "Cur   |  Est  | Name"
activeJobs.each { job ->
 // Get the Timeout-PlugIn
 wrapper = job.getBuildWrappersList().findAll{it instanceof hudson.plugins.build_timeout.BuildTimeoutWrapper }[0]

 // Get the current Timeout, if any
 currentTimeout = (wrapper != null) ? wrapper.timeoutMinutes : ""

 // Calculate a new timeout with a min-value
 defaultTimeout = Math.round(job.estimatedDuration * 2 / 1000 / 60)
 if (defaultTimeout < 10) defaultTimeout = 10

 // Update the timeout, maybe requires instantiation
 action = (wrapper != null) ? "updated" : "established"
 if (wrapper == null) {
 plugin = new hudson.plugins.build_timeout.BuildTimeoutWrapper(defaultTimeout, defaultFailBuild)
 job.getBuildWrappersList().add(plugin)
 } else {
 wrapper.timeoutMinutes = defaultTimeout
 }

 // String preparation for table output
 String defaultTimeoutStr = defaultTimeout
 defaultTimeoutStr = defaultTimeoutStr.padLeft(5)
 String currentTimeoutStr = currentTimeout
 currentTimeoutStr = currentTimeoutStr.padLeft(5)
 String jobname = job.name.padRight(40)

 // Table output
 println "$currentTimeoutStr | $defaultTimeoutStr | $jobname | $action "
}

x = ""

This updates all timeout settings and reports this like here:

Hudson: start a list of jobs using Groovy console

Recently I wrote how to get a list of failed jobs in Hudson.

Rob Whitlock asked how to restart that list.

Here is the code:

joblist = hudson.model.Hudson.instance.items.findAll{job -> job.isBuildable()}  

startServer = "admin computer"
startNote   = "bulk start"
cause = new hudson.model.Cause.RemoteCause(startServer, startNote)
joblist.each{run -> run.scheduleBuild(cause)}

In the first line I just get a list of jobs from somewhere.

The interesting part is line 6: here I start the build or more precise – reschedule it. Hudson starts it somewhere in the future.

There is a scheduleBuild() method without argument, but it is deprecated. That’s why I create a ‚cause‘ first. So the build knows why it is run. Usually you have a „started by user XYZ“ or „started by upstream project“ here.

Hudson: how to get a list of all failed jobs?

If you have a lot of job configured on your Hudson installation, you maybe want to have a list of job which last run fails. Sadly there is no view to that, but you could get the list via Hudsons script console and a three lines of Groovy:

activeJobs = hudson.model.Hudson.instance.items.findAll{job -> job.isBuildable()}
failedRuns = activeJobs.findAll{job -> job.lastBuild.result == hudson.model.Result.FAILURE}
failedRuns.each{run -> println(run.name)}

or a little bit longer (for explanations):

hudsonInstance = hudson.model.Hudson.instance
allItems = hudsonInstance.items
activeJobs = allItems.findAll{job -> job.isBuildable()}
 failedRuns = activeJobs.findAll{job -> job.lastBuild.result == hudson.model.Result.FAILURE}
 failedRuns.each{run -> println(run.name)}

Line 1: This is the main entry point to the running Hudson instance.

Line 2: Jobs are a special kind of item. Here are all stored items.

Line 3: Now we filter the whole list. We dont need disabled jobs as we dont want to fix them. So the closure checks if the job is buildable and therefore active.

Line 4: The next filter removes all jobs which were ok. The closure checks the last build result and selects the job if it was a failure.

Line 5: Now we pass an additional closure to the list for getting a nicer output (than a println(failedRuns) )

(Sorry for bad source code formatting – WordPress doesnt support Groovy.)

My first Hudson PlugIn

I set up a Hudson instance in my company. But after adding the 20th job I lost the overview: which job depends on which other job? I need a visualization.
I need a plugin which reads the job dependencies and creates a dependency graph.

This problem consists of two parts: getting te dependencies and creating the graph. And I need to solve both for getting the plugin work …

I start with creating the graph. Shouldn’t be a problem – I am sure there are some libraries I could use as dependency graphs are an old and common ‚problem‘. And searching for „java create dependency graph“ gave me a nice hit: yFiles.
But this lib has an enourmous drawback: it is commercial and the plugin should be published as OSS.
But I remembered another tool: Graphiz. And via its homepage I found a Java layer on top of it: Linguine Maps.

First played a little bit:

String dotExeFileName = "C:/temp/ec-dep/Job Dependency Visualizer/lib/graphiz/dot.exe";
 // create the board for the graph
 Graph graph = GraphFactory.newGraph();
 // create the nodes
 GraphNode child   = graph.addNode();
 GraphNode me      = graph.addNode();
 GraphNode wife    = graph.addNode();
 GraphNode father  = graph.addNode();
 GraphNode mother  = graph.addNode();
 GraphNode grandma = graph.addNode();
 GraphNode grandpa = graph.addNode();
 GraphNode friend  = graph.addNode();
 // connect the nodes according to the dependencies
 graph.addEdge(me, child);
 graph.addEdge(wife, child);
 graph.addEdge(father, me);
 graph.addEdge(mother, me);
 graph.addEdge(grandma, father);
 graph.addEdge(grandpa, father);
 // nodes have a caption
 child.getInfo().setCaption("Child");
 me.getInfo().setCaption("Me");
 wife.getInfo().setCaption("Wife");
 father.getInfo().setCaption("Father");
 mother.getInfo().setCaption("Mother");
 grandma.getInfo().setCaption("Grandma");
 grandpa.getInfo().setCaption("Grandpa");
 friend.getInfo().setCaption("Friend");
 // highlight some nodes
 me.getInfo().setFillColor("red");
 grandma.getInfo().setModeDotted();
 grandpa.getInfo().setModeDotted();
 // save as gif
 File dotFileName = File.createTempFile("tmp.graph.", ".dot");
 GRAPHtoDOTtoGIF.transform(
 graph,
 dotFileName.getAbsolutePath(),
 "C:/temp/graph.demo-old.gif",
 dotExeFileName
 );

Oh, this is the library I want to use. So I wrote a class tayloring the graph creation (Java 1.4) to dependency graphs with use of Java >=5 and a fluent interface.

DependencyGraph
 .create()
 .addDependency("Child", "Me", "Wife")
 .addDependency("Me", "Father", "Mother")
 .addDependency("Father", "Grandma", "Grandpa")
 .addDependency("Friend")
 .nodeBorder("Grandma", BorderStyle.BORDER_DOTTED)
 .nodeBorder("Grandpa", BorderStyle.BORDER_DOTTED)
 .highlight("Me", Style.FILL_COLOR, "red")
 .toGif("C:/temp/graph.demo.gif");

My base concepts are:

• a string could have dependency on other strings (me–>father,mother; wife:<none>)
• the string is the caption of the graph node
• the string is the unique node name
• two strings can only be connected once

So I could store the dependencies as a map: Map<String,Set<String>>

public class DependencyGraph {

 /** Path to the Graphiz executable. */
 private String dotExeFileName = "C:/temp/ec-dep/Job Dependency Visualizer/lib/graphiz/dot.exe";

 /** The graph to create. */
 Graph graph = GraphFactory.newGraph();

/** All already created GraphNodes. */
 Map<String, GraphNode> nodes = new Hashtable<String, GraphNode>();

 /** Already created dependency lines. */
 Map<String, Set<String>> dependencies = new Hashtable<String, Set<String>>();

 public static DependencyGraph create() {
 return new DependencyGraph();
 }

/**
 * Adds a node and its dependencies to the graph. If the node exists
 * already, it is not created a second time.
 */
 public DependencyGraph addDependency(String node, String... parents) {
 Set<String> list = new HashSet<String>();
 Collections.addAll(list, parents);
 addDependency(node, list);
 return this;
 }

public DependencyGraph addDependency(String node, Set<String> parents) {
 createGraphNode(node);
 for (String parent : parents) {
 createGraphNode(parent);
 if (!dependencies.get(node).contains(parent)) {
 // create the link
 graph.addEdge(nodes.get(parent), nodes.get(node));
 // register the dependency
 dependencies.get(node).add(parent);
 }
 }
 return this;
 }

/**
 * Helper method for creating graph nodes and store them in the cache if the
 * don't exist before.
 */
 private void createGraphNode(String nodeName) {
 if (!nodes.containsKey(nodeName)) {
 GraphNode node = graph.addNode();
 node.getInfo().setCaption(mask(nodeName));
 nodes.put(nodeName, node);
 dependencies.put(nodeName, new HashSet<String>());
 }
 }

/**
 * Modify the shape style of a node.
 *
 * @param node the name of the node to style
 * @param style the new style
 * @return self
 */
 public DependencyGraph nodeStyle(String node, ShapeStyle style) {
 nodes.get(node).getInfo().setShape(style.value);
 return this;
 }

/**
 * Stores the graph as GIF image.
 * @param file the file to new image file
 * @return self
 * @throws IOException while file access
 */
 public DependencyGraph toGif(File file) throws IOException {
 File dotFileName = File.createTempFile("tmp.graph.", ".dot");
 GRAPHtoDOTtoGIF.transform(
 graph, dotFileName.getAbsolutePath(),
 file.getAbsolutePath(), dotExeFileName);
 dotFileName.delete();
 return this;
 }

}

public enum ShapeStyle {
 RECORD(0),
 BOX(1),
 ELLIPSE(2),
 CIRCLE(3),
 DIAMOND(4),
 TRIANGLE(5),
 INVTRIANGLE(6),
 HEXAGON(7),
 OCTAGON(8),
 PARALLELOGRAM(9);

 /** the LinguineMaps style value */
 int value;
 private ShapeStyle(int value) {
 this.value = value;
 }

}

After solving the second part of my problem I have to solve the first part too. I need information from Hudson:

• List<Job> getAllJobs()
• List<Job> getParentJobs(Job)

For creating a graph per view I also need

• List<View> getAllViews()
• List<Job>  getJobs(View)

And for keeping the graphs up to date I need to be informed about configuration changes.

• void onJobConfigChange(Job)

I have to search for these…

I am working on the plugin part now – so I’ll write more if I have finished that.