How to check if all Hudson jobs have a timeout?

At Apaches Hudson installation I have sometimes seen the situation where too many builds are stuck and therefore blocking the executors. And sadly for me – the executors my own jobs require …

Having a policy to use the „build timeout plugin“ and kill jobs which are running too long (thinking more of „not running any more“😉 is good. But having a program which checks this is better …

So I tried a little bit Groovy’in for the Groovy console:

hudsonInstance = hudson.model.Hudson.instance
allItems = hudsonInstance.items
activeJobs = allItems.findAll{job -> job.isBuildable()}
wrappableJobs = activeJobs.findAll{job -> job instanceof hudson.model.BuildableItemWithBuildWrappers}

jobsWithoutTimeout = wrappableJobs.findAll { job ->
 job.getBuildWrappersList().findAll{it instanceof hudson.plugins.build_timeout.BuildTimeoutWrapper }[0] == null
}

println "There are $jobsWithoutTimeout.size jobs without timeout:"
jobsWithoutTimeout.each { println "- $it.name" }

x = ""

In line 1 we get the reference to the Hudson singleton. Then we get the list of all item in line 2 which we filter in line 3 to get only buildable items, like our jobs. The line 4 contains the first thing special to this requirement: the item must be able to have a BuildWrapper.

But the most thing is done in line 5 which filters again with a closure: get all BuildWrappers for the job, but only if it is our TimeOut-Plugin. Because it can be registered only once, I check the first element of that list. It must be null for being a problem. Otherwise the job has a timeout setting.

After that, the last two lines are simply out … and the last line supresses the result output in the console.

Update:

Antoine Tulme had consulted Kohsuke Kawaguchi and he sees three possibilities of forcing the timeout setting:

  1. We cannot make the timeout field mandatory.
  2. We can create a plugin that presets the timeout field.
  3. We can iterate over the projects and set a value for the timeouts en masse.

Good, so I evaluate my „iteration solution“ a little more.

We have a list of all jobs without settings and so we only have to iterate over this list, instantiate and initialize the BuildTimeoutWrapper and add it to the jobs wrapper-list:

jobsWithoutTimeout.each { job ->
 defaultTimeout = 180
 defaultFailBuild = false
 plugin = new hudson.plugins.build_timeout.BuildTimeoutWrapper(defaultTimeout, defaultFailBuild)
 job.getBuildWrappersList().add(plugin)
}

BTW – If you want to work with a plugin, you could start with the Create Job Advances Plugin – maybe this requires code enhancement … and it will only for future jobs, not for existing one.

Update:

The last update of the script for setting the timeout value is this:


hudsonInstance = hudson.model.Hudson.instance
allItems = hudsonInstance.items
activeJobs = allItems.findAll{job -> job.isBuildable()}
defaultFailBuild = true

println "Cur   |  Est  | Name"
activeJobs.each { job ->
 // Get the Timeout-PlugIn
 wrapper = job.getBuildWrappersList().findAll{it instanceof hudson.plugins.build_timeout.BuildTimeoutWrapper }[0]

 // Get the current Timeout, if any
 currentTimeout = (wrapper != null) ? wrapper.timeoutMinutes : ""

 // Calculate a new timeout with a min-value
 defaultTimeout = Math.round(job.estimatedDuration * 2 / 1000 / 60)
 if (defaultTimeout < 10) defaultTimeout = 10

 // Update the timeout, maybe requires instantiation
 action = (wrapper != null) ? "updated" : "established"
 if (wrapper == null) {
 plugin = new hudson.plugins.build_timeout.BuildTimeoutWrapper(defaultTimeout, defaultFailBuild)
 job.getBuildWrappersList().add(plugin)
 } else {
 wrapper.timeoutMinutes = defaultTimeout
 }

 // String preparation for table output
 String defaultTimeoutStr = defaultTimeout
 defaultTimeoutStr = defaultTimeoutStr.padLeft(5)
 String currentTimeoutStr = currentTimeout
 currentTimeoutStr = currentTimeoutStr.padLeft(5)
 String jobname = job.name.padRight(40)

 // Table output
 println "$currentTimeoutStr | $defaultTimeoutStr | $jobname | $action "
}

x = ""

This updates all timeout settings and reports this like here:

Report of Timout-Setter

Explore posts in the same categories: en, Hudson

Schlagwörter: , ,

You can comment below, or link to this permanent URL from your own site.

7 Kommentare - “How to check if all Hudson jobs have a timeout?”

  1. Manolo Says:

    I think 180 minutes could be a very large period which could make the build system stuck. May be a better solution is configure each project with a timeout closest to its normal duration, I think 2 or 3 times the last duration, like this:

    jobsWithoutTimeout.each { job ->
    defaultTimeout = job.lastBuild.duration * 2 / 1000
    defaultFailBuild = false
    plugin = new hudson.plugins.build_timeout.BuildTimeoutWrapper(defaultTimeout, defaultFailBuild)
    job.getBuildWrappersList().add(plugin)
    }

  2. Manolo Says:

    Instead of lastBuild.duration, the estimate duration should be a more accurate value.

    defaultTimeout = Math.round(job.estimatedDuration * 2 / 1000)

  3. janmaterne Says:

    Yes I thought about iterating through all runs and calculating a project specific timeout value. But using job.estimatedDuration is a nice way. Thanks.


  4. […] Jan’s Blog My personal blog about all and nothing … « How to check if all Hudson jobs have a timeout? […]


  5. […] closures which update a given Hudson job (line 010-018). The basic structure is the one I used in earlier scripts … The work here is in lines 053-088. But that’s pretty easy: check the given values and […]

  6. BuggedTech Says:

    I’ve been trying to searc for a solution as to how to configure the timeout plugin and found your site since the wiki doesn’t say how to do it.

    Perhaps groovy is the best solution but I don’t know how to code in Java. Can someone add a configuration(for a java-noob like me) to the plugin please?🙂

  7. janmaterne Says:

    The timeout plugin is only configured with two values „activate the plugin“ and „how long a job should run“.

    This Groovy code sets these two values.

    The advantage of Groovy is, that you can run this code inside the running Hudson/Jenkins instance and „reconfigure“ the timeout plugin. And it can use values from earlier builds (job.estimatedDuration).

    With Java this is not possible. … without writing a plugin …


Schreibe einen Kommentar

Trage deine Daten unten ein oder klicke ein Icon um dich einzuloggen:

WordPress.com-Logo

Du kommentierst mit Deinem WordPress.com-Konto. Abmelden / Ändern )

Twitter-Bild

Du kommentierst mit Deinem Twitter-Konto. Abmelden / Ändern )

Facebook-Foto

Du kommentierst mit Deinem Facebook-Konto. Abmelden / Ändern )

Google+ Foto

Du kommentierst mit Deinem Google+-Konto. Abmelden / Ändern )

Verbinde mit %s


%d Bloggern gefällt das: