Update to the Hudson book

News from the Hudson by Simon Wiest book I reviewed: it is now in the press and if you order it (e.g. at Amazon), you should be able to read it by Xmas 😉 (if you can read German :-O )

Werbung

Hudson: Review and ideas

In June I had the pleasure to do a review of the first German book about CI with Hudson by Simon Wiest. The book is announced for December 2010 and it will cover topics from basic (setup, first jobs) up to deep knowledge like writing plugins for Hudson.

As with earlier reviews I had printed the book (draft), read it and made comments on the side (and Simon had to listen to them for hours 😉

But some comments weren’t for the author – they „just“ ideas which came into my mind. But because I won’t have the time for implementing them, I’ll write about them …

Capability Plugin

The main idea is automatic dispatching of jobs to slaves according to their needs. If I remember right, TeamCity has this feature too …

The Slaves

The slaves have some capabilities. There are two kinds of them:

1. automatic capabilities: result from tool configuration, operating system, jdk versions, build tool name+version, …
2. manual capabilities: the admin of the slave can define key-value pairs

The Jobs

For jobs you define requirements according to the provided capabilities. And like them there are two requirements:

1. automatic requirements: selected jdk, build tool, operating system (if you use e.g. a Windows Batch …)
2. manual requirement: define additional requirements (or overwrite the complete formula). Use key=value (aspectj.available=true) or comparisons (ant.version >= 1.8) and combine them (and, or, xor, …)

Dispatcher strategy

The dispatcher must select all slaves which fit the defined requirements. From this list it selects the one by looking at the build queue of the slave (does it have time for building?) and net-response-time (don’t forget that much data has to be transfered …)

 

Not sure if there is something like this now …. 😉

Another Admin job Hudson: ensure that there are not too many old builds …

Especially if you have a large number of jobs and they are running more often, you’ll come to a point, where your disk is full of old builds.

Hudson provides a configuration parameter for that: „discard old builds“. This will delete old builds according to the specified number of days or number of builds.

For Apache I wrote a script which ensures, that all jobs have „discard“ setting and that existing values are not higher than a defined maximum value.

/** Default-Setting for the "number of old builds" */
numberOfOldBuilds  = 10

/** Maximum of "number of days" */
maxDaysOfOldBuilds = 14

/** Should we override existing values? */
overrideExistingValues = true

/** Closures for setting default 'max number' */
setMaxNum = { job ->
   job.logRotator = new hudson.tasks.LogRotator(-1, numberOfOldBuilds)
}

/** Closures for setting default 'max number' */
setMaxDays = { job ->
 job.logRotator = new hudson.tasks.LogRotator(maxDaysOfOldBuilds, -1)
}

// ----- Do the work. -----

// Access to the Hudson Singleton
hudsonInstance = hudson.model.Hudson.instance

// Retrieve all active Jobs
allItems = hudsonInstance.items
activeJobs = allItems.findAll{job -> job.isBuildable()}

// Table header
col1 = "Old".center(10)
col2 = "New".center(10)
col3 = "Job".center(50)
col4 = "Action".center(14)
header = "$col1 | $col2 | $col3 | $col4"
line = header.replaceAll("[^|]", "-").replaceAll("\\|", "+")
title = "Set 'Discard old builds'".center(line.size())

println title
println line
println header
println line

// Do work and create the result table
activeJobs.each { job ->

 // Does the job have a discard setting?
 discardActive = job.logRotator != null

 // Enforce the settings
 action   = ""
 newValue = ""
 oldValue = ""
 if (!discardActive) {
 // No discard settings, so set the default
 setMaxNum.call(job)
 action   = "established"
 newValue = "$numberOfOldBuilds jobs"
 } else {
 // What are the current settings?
 oldDays = job.logRotator.daysToKeep
 oldNums    = job.logRotator.numToKeep

 if (oldNums > 0) {
 // We have a set value for 'numbers'
 if (oldNums > numberOfOldBuilds && overrideExistingValues) {
 // value is too large so set a new one
 setMaxNum.call(job)
 action   = "updated"
 newValue = "$numberOfOldBuilds jobs"
 oldValue = "$oldNums jobs"
 } else {
 // Correct value or we arent allowed to override.
 oldValue = "$oldNums jobs"
 }
 } else {
 // we have a value for 'days'
 if (oldDays > maxDaysOfOldBuilds && overrideExistingValues) {
 // value is too large so set a new one
 setMaxDays.call(job)
 action   = "updated"
 newValue = "$maxDaysOfOldBuilds days"
 oldValue = "$oldDays days"
 } else {
 // Correct value or we aren't allowed to override.
 oldValue = "$oldDays days"
 }
 }
 }

 // String preparation for table output
 oldValue = oldValue.padLeft(10)
 newValue = newValue.padLeft(10)
 jobname  = job.name.padRight(50)

 // Table output
 println "$oldValue | $newValue | $jobname | $action"
}
println line

// Meaningful output on the Groovy console
// (the console will output the result of the last statement)
printout = "Number of Jobs: $activeJobs.size"

In the first section I define the „constants“ (line 001-008). After that I define two closures which update a given Hudson job (line 010-018).
The basic structure is the one I used in earlier scripts …
The work here is in lines 053-088. But that’s pretty easy: check the given values and eventually set new values using the pre defined closures.
New is the last line: I dont use a >x = „“< instruction for suppressing the output. I use a more meaningful message: the number of jobs.

Links for 2010-07-20

According to Entwickler.COM Microsoft has published a free ebook about „Cloud Computing“ by Bob Muglia.

On Wakaleo the development of an open source book about Hudson: „Continuous Integration with Hudson„. First chapters are online …

Golem.DE has found a free German video workshop about Gimp 2.6.

On DZone Hudson creator Kohsuke Kawaguchi introduced his new startup, InfraDNA, which provides support and consulting for the Hudson Continuous Integration system.

Again on DZone there is a nice introduction into HtmlUnit. It provides a Java based WebClient which you can control via its API. With this you could write JUnit tests. But more easily you could write them with the additional assert-Methods:

@Test
public void testGoogle(){
 WebClient webClient = new WebClient();
 HtmlPage currentPage = webClient.getPage("http://www.google.com/");
 assertEquals("Google", currentPage.getTitleText());
}
@Test public void htmlunitAsserts() {
 // Load a page
 webClient.getPage("http://www.google.com/search?q=htmlunit");

 // JUnit asserts and WebClient API
 assertEquals(200,currentPage.getWebResponse().getStatusCode());
 assertEquals("OK",currentPage.getWebResponse().getStatusMessage());

 // HtmlUnit asserts
 WebAssert.assertTextPresent(currentPage, "htmlunit");
 WebAssert.assertTitleContains(currentPage, "htmlunit");
 WebAssert.assertLinkPresentWithText(currentPage, "Advanced search");

 // XPath Query
 assertTrue(currentPage.getByXPath("//h3").size()>0); //result number

 // Cookies
 assertNotNull(webClient.getCookieManager().getCookie("NID"));
}

According to Entwickler.COM Microsoft has published a bunch of Powerpoint-Templates for demonstrating the new features of PPT 2010.

If you ask yourself what Darth Vader and Yoda are doing after making the movies with George Lucas, GolemDE has found the answer: they are creating TomToms next voices … 😉

If you are updating to Java 1.6_21 and having problems with Eclipse, have a look at this blog entry: it show how to tune the JVM settings …

Also if you write JPA applications you should have a good test suite. So looking at the blog „Patterns for Better Unit Testing with JPA“ is not waste of time 😉

Hudson: Overview of the suggestd timeout settings

In my last post I explained why and how to check the timeout settings for Hudson jobs.

On our mailinglist for Hudson users at Apache there was a suggestion to get an overview of (computed) suggested timeout settings.

So here is the follow up to my earlier code …

hudsonInstance = hudson.model.Hudson.instance</pre>
allItems = hudsonInstance.items
activeJobs = allItems.findAll{job -> job.isBuildable()}
wrappableJobs = activeJobs.findAll{job -> job instanceof hudson.model.BuildableItemWithBuildWrappers}

jobsWithoutTimeout = wrappableJobs.findAll { job ->
 job.getBuildWrappersList().findAll{it instanceof hudson.plugins.build_timeout.BuildTimeoutWrapper }[0] == null
}

println "Suggested timeout values for jobs without any ($jobsWithoutTimeout.size in total):"
jobsWithoutTimeout.each { job ->
 defaultTimeout = Math.round(job.estimatedDuration * 2 / 1000 / 60)
 if (defaultTimeout < 10) defaultTimeout = 10
 String s = defaultTimeout
 s = s.padLeft(4)
 println "$s | $job.name"
}

x = ""

The new stuff is only the creation in the last few lines. Nothing special – apart from the conversion from Long to String for getting padLeft() work 😉

The result is a „table“ like this:

How to check if all Hudson jobs have a timeout?

At Apaches Hudson installation I have sometimes seen the situation where too many builds are stuck and therefore blocking the executors. And sadly for me – the executors my own jobs require …

Having a policy to use the „build timeout plugin“ and kill jobs which are running too long (thinking more of „not running any more“ 😉 is good. But having a program which checks this is better …

So I tried a little bit Groovy’in for the Groovy console:

hudsonInstance = hudson.model.Hudson.instance
allItems = hudsonInstance.items
activeJobs = allItems.findAll{job -> job.isBuildable()}
wrappableJobs = activeJobs.findAll{job -> job instanceof hudson.model.BuildableItemWithBuildWrappers}

jobsWithoutTimeout = wrappableJobs.findAll { job ->
 job.getBuildWrappersList().findAll{it instanceof hudson.plugins.build_timeout.BuildTimeoutWrapper }[0] == null
}

println "There are $jobsWithoutTimeout.size jobs without timeout:"
jobsWithoutTimeout.each { println "- $it.name" }

x = ""

In line 1 we get the reference to the Hudson singleton. Then we get the list of all item in line 2 which we filter in line 3 to get only buildable items, like our jobs. The line 4 contains the first thing special to this requirement: the item must be able to have a BuildWrapper.

But the most thing is done in line 5 which filters again with a closure: get all BuildWrappers for the job, but only if it is our TimeOut-Plugin. Because it can be registered only once, I check the first element of that list. It must be null for being a problem. Otherwise the job has a timeout setting.

After that, the last two lines are simply out … and the last line supresses the result output in the console.

Update:

Antoine Tulme had consulted Kohsuke Kawaguchi and he sees three possibilities of forcing the timeout setting:

1. We cannot make the timeout field mandatory.
2. We can create a plugin that presets the timeout field.
3. We can iterate over the projects and set a value for the timeouts en masse.

Good, so I evaluate my „iteration solution“ a little more.

We have a list of all jobs without settings and so we only have to iterate over this list, instantiate and initialize the BuildTimeoutWrapper and add it to the jobs wrapper-list:

jobsWithoutTimeout.each { job ->
 defaultTimeout = 180
 defaultFailBuild = false
 plugin = new hudson.plugins.build_timeout.BuildTimeoutWrapper(defaultTimeout, defaultFailBuild)
 job.getBuildWrappersList().add(plugin)
}

BTW – If you want to work with a plugin, you could start with the Create Job Advances Plugin – maybe this requires code enhancement … and it will only for future jobs, not for existing one.

Update:

The last update of the script for setting the timeout value is this:

hudsonInstance = hudson.model.Hudson.instance
allItems = hudsonInstance.items
activeJobs = allItems.findAll{job -> job.isBuildable()}
defaultFailBuild = true

println "Cur   |  Est  | Name"
activeJobs.each { job ->
 // Get the Timeout-PlugIn
 wrapper = job.getBuildWrappersList().findAll{it instanceof hudson.plugins.build_timeout.BuildTimeoutWrapper }[0]

 // Get the current Timeout, if any
 currentTimeout = (wrapper != null) ? wrapper.timeoutMinutes : ""

 // Calculate a new timeout with a min-value
 defaultTimeout = Math.round(job.estimatedDuration * 2 / 1000 / 60)
 if (defaultTimeout < 10) defaultTimeout = 10

 // Update the timeout, maybe requires instantiation
 action = (wrapper != null) ? "updated" : "established"
 if (wrapper == null) {
 plugin = new hudson.plugins.build_timeout.BuildTimeoutWrapper(defaultTimeout, defaultFailBuild)
 job.getBuildWrappersList().add(plugin)
 } else {
 wrapper.timeoutMinutes = defaultTimeout
 }

 // String preparation for table output
 String defaultTimeoutStr = defaultTimeout
 defaultTimeoutStr = defaultTimeoutStr.padLeft(5)
 String currentTimeoutStr = currentTimeout
 currentTimeoutStr = currentTimeoutStr.padLeft(5)
 String jobname = job.name.padRight(40)

 // Table output
 println "$currentTimeoutStr | $defaultTimeoutStr | $jobname | $action "
}

x = ""

This updates all timeout settings and reports this like here:

Links for 2009-11-13

Today I found two tools for testing: Byteman and YouDebug.

While I haven’t have a deeper look at Byteman I realized that YouDebug is writte by Kohsuke Kawaguchi. And it is very funny to recognizing him after Args4J and Hudson with another project. The world is small and you’ll see another every twice … or more 😉

Hudson: start a list of jobs using Groovy console

Recently I wrote how to get a list of failed jobs in Hudson.

Rob Whitlock asked how to restart that list.

Here is the code:

joblist = hudson.model.Hudson.instance.items.findAll{job -> job.isBuildable()}  

startServer = "admin computer"
startNote   = "bulk start"
cause = new hudson.model.Cause.RemoteCause(startServer, startNote)
joblist.each{run -> run.scheduleBuild(cause)}

In the first line I just get a list of jobs from somewhere.

The interesting part is line 6: here I start the build or more precise – reschedule it. Hudson starts it somewhere in the future.

There is a scheduleBuild() method without argument, but it is deprecated. That’s why I create a ‚cause‘ first. So the build knows why it is run. Usually you have a „started by user XYZ“ or „started by upstream project“ here.

Links for 2009-07-28

GrepCode is a search index for open source available java code. Searchable codebases are OpenJDK, Maven-Central, JBoss.com, Jetty.Mortbay.org and Eclipse 3.4.2. If you miss ‚your‘ project you could send them a message.

Zend Technologies published seven screencast about PHP development with Eclipse PDT 2.1.

• Code Navigation
• Code Analysis and Auto-fix
• PHP 5.3 Development
• PHP Code Refactoring
• RAD Tools
• Root Cause Analysis
• Jump Starting Web Application Development
• Using Zend Studio & Server Integration

Lawers found newsletter about open source topics – in their focus. (Sorry, too much special words for me to translate the German article 😉

Netbeans 6.7 Maven and Hudson Demo

eBook „Mastering PowerShell“ available for free.

I found an overview article …

On JavaWorld John Ferguson Smart wrote a nice article about some of the most interesting Java evolutions in 2008. He discovers tools for Build automatisation (Hudson, Bamboo, Maven, Ant, Grant, Gradle, Subversion), Testing tools (JBehave, easyb, Selenium, WebTest) and IDEs (Eclipse, NetBeans). Conclusion: a very nice read.