How to check if all Hudson jobs have a timeout?

At Apaches Hudson installation I have sometimes seen the situation where too many builds are stuck and therefore blocking the executors. And sadly for me – the executors my own jobs require …

Having a policy to use the “build timeout plugin” and kill jobs which are running too long (thinking more of “not running any more” is good. But having a program which checks this is better …

So I tried a little bit Groovy’in for the Groovy console:

hudsonInstance = hudson.model.Hudson.instance
allItems = hudsonInstance.items
activeJobs = allItems.findAll{job -> job.isBuildable()}
wrappableJobs = activeJobs.findAll{job -> job instanceof hudson.model.BuildableItemWithBuildWrappers}

jobsWithoutTimeout = wrappableJobs.findAll { job ->
 job.getBuildWrappersList().findAll{it instanceof hudson.plugins.build_timeout.BuildTimeoutWrapper }[0] == null
}

println "There are $jobsWithoutTimeout.size jobs without timeout:"
jobsWithoutTimeout.each { println "- $it.name" }

x = ""

In line 1 we get the reference to the Hudson singleton. Then we get the list of all item in line 2 which we filter in line 3 to get only buildable items, like our jobs. The line 4 contains the first thing special to this requirement: the item must be able to have a BuildWrapper.

But the most thing is done in line 5 which filters again with a closure: get all BuildWrappers for the job, but only if it is our TimeOut-Plugin. Because it can be registered only once, I check the first element of that list. It must be null for being a problem. Otherwise the job has a timeout setting.

After that, the last two lines are simply out … and the last line supresses the result output in the console.

Update:

Antoine Tulme had consulted Kohsuke Kawaguchi and he sees three possibilities of forcing the timeout setting:

1. We cannot make the timeout field mandatory.
2. We can create a plugin that presets the timeout field.
3. We can iterate over the projects and set a value for the timeouts en masse.

Good, so I evaluate my “iteration solution” a little more.

We have a list of all jobs without settings and so we only have to iterate over this list, instantiate and initialize the BuildTimeoutWrapper and add it to the jobs wrapper-list:

jobsWithoutTimeout.each { job ->
 defaultTimeout = 180
 defaultFailBuild = false
 plugin = new hudson.plugins.build_timeout.BuildTimeoutWrapper(defaultTimeout, defaultFailBuild)
 job.getBuildWrappersList().add(plugin)
}

BTW – If you want to work with a plugin, you could start with the Create Job Advances Plugin – maybe this requires code enhancement … and it will only for future jobs, not for existing one.

Update:

The last update of the script for setting the timeout value is this:

hudsonInstance = hudson.model.Hudson.instance
allItems = hudsonInstance.items
activeJobs = allItems.findAll{job -> job.isBuildable()}
defaultFailBuild = true

println "Cur   |  Est  | Name"
activeJobs.each { job ->
 // Get the Timeout-PlugIn
 wrapper = job.getBuildWrappersList().findAll{it instanceof hudson.plugins.build_timeout.BuildTimeoutWrapper }[0]

 // Get the current Timeout, if any
 currentTimeout = (wrapper != null) ? wrapper.timeoutMinutes : ""

 // Calculate a new timeout with a min-value
 defaultTimeout = Math.round(job.estimatedDuration * 2 / 1000 / 60)
 if (defaultTimeout < 10) defaultTimeout = 10

 // Update the timeout, maybe requires instantiation
 action = (wrapper != null) ? "updated" : "established"
 if (wrapper == null) {
 plugin = new hudson.plugins.build_timeout.BuildTimeoutWrapper(defaultTimeout, defaultFailBuild)
 job.getBuildWrappersList().add(plugin)
 } else {
 wrapper.timeoutMinutes = defaultTimeout
 }

 // String preparation for table output
 String defaultTimeoutStr = defaultTimeout
 defaultTimeoutStr = defaultTimeoutStr.padLeft(5)
 String currentTimeoutStr = currentTimeout
 currentTimeoutStr = currentTimeoutStr.padLeft(5)
 String jobname = job.name.padRight(40)

 // Table output
 println "$currentTimeoutStr | $defaultTimeoutStr | $jobname | $action "
}

x = ""

This updates all timeout settings and reports this like here:

Explore posts in the same categories: en, Hudson

Tags: build timeout, hudson, policy

You can comment below, or link to this permanent URL from your own site.