Posted tagged ‘Hudson’

Dynamic Continuous Integration

October 9, 2009
I have an idea for continuous integration that is a bit beyond what I’m aware of as far as the capabilities of yhudson are concerned.
We work as a team on a project. This project is stored in SVN (like all good projects should be) and is continuously checked out by yhudson to be built on a CI server. However, yhudson is set up to just check out the latest version of trunk on a commit, checking every 5 minutes. The problem is that most developers do their work on development branches, which don’t take part in the CI cycle. These development changes do not gain the benefit of continuous integration until the changes are merged into trunk which results in a lot of loss in developer potential.
In our development system, people create branches for each bug or feature set that they are working on. Now if a bug is trivially small or easy to see and fix, it might not warrant its own branch and will be just fixed on trunk. But for larger changes, a branch is cut so that development that one person is performing won’t adversely effect other developers. It also allows us to postpone features from our monthly release cycle. If you’re working on a feature that is only halfway complete by the time our release branch is cut and code freeze begins, you can just not merge your change into trunk and you won’t dirty up the release branch with half-finished code. Once you finish your change, you merge into trunk and solve any conflicts that might have come up. It’s not a perfect process, but I don’t know if anyone’s will classify (git excluded).
Because of these separate branches, developers are not able to harness the utility of yhudson and our CI environments. yHudson is set up with a specific location in SVN in mind when you create a project, usually trunk. If you want to build your branch, you have to set up a new project and go through all of the configuration of your trunk project but with the new location. There’s no real way to just duplicate a project so you can change a setting, and even if there were the manual steps involved would be enough that most developers won’t bother. There should be a better way.
I think that there should be a different style of project able to be used in yHudson. The project should define a standard build like we did with trunk on our project, but also allow it to monitor all of the folders under branches and treat each one as their own project. Whenever a change is made to one of those branches, a build will be performed. If a new branch is created, a new project should dynamically be created for that branch. When the build is made, it should be deployed according to the Staging instructions of the original project, but to a dynamically generated VM or from a pool of VMs with modifiable settings for things like ports and server names. This will allow us to do continuous deployment on each of our development branches so that if a QA tester finds a problem with a new feature a developer is working on in a branch, the developer can quickly provide a fix and have it automatically go through a full set of continuous integration tests and be deployed to the staging environment for the tester to try out again.
This fast turn around time between bug reporting, bug fixing, and fix deployment could greatly enhance productivity and let us try out different solutions, faster. With a fast feedback loop, we can work directly with the QA department or our customers to solve the issue they’re having and let them try it out right away in such a fashion that doesn’t effect anyone else’s bugs or problems that they are working on.
In essence, dynamic continuous integration would broaden the usage of our continuous integration servers to cover all development, even on separate branches, without the need to manually create (or forget to create) projects for them. I think that if we worked on creating such a functionality in yHudson, along with some helper tools for utilizing large pools of servers for fast staging environment, we can create stronger bonds between developers and testers, along with developers and their customers by having a shorter feedback loop and faster turnaround of problems.

I have an idea for continuous integration that is a bit beyond what I’m aware of as far as the capabilities of hudson are concerned.

We work as a team on a project. This project is stored in SVN (like all good projects should be) and is continuously checked out by yhudson to be built on a CI server. However, hudson is set up to just check out the latest version of trunk on a commit, checking every 5 minutes. The problem is that most developers do their work on development branches, which don’t take part in the CI cycle. These development changes do not gain the benefit of continuous integration until the changes are merged into trunk which results in a lot of loss in developer potential.

In our development system, people create branches for each bug or feature set that they are working on. Now if a bug is trivially small or easy to see and fix, it might not warrant its own branch and will be just fixed on trunk. But for larger changes, a branch is cut so that development that one person is performing won’t adversely effect other developers. It also allows us to postpone features from our monthly release cycle. If you’re working on a feature that is only halfway complete by the time our release branch is cut and code freeze begins, you can just not merge your change into trunk and you won’t dirty up the release branch with half-finished code. Once you finish your change, you merge into trunk and solve any conflicts that might have come up. It’s not a perfect process, but I don’t know if anyone’s will classify (git excluded).

Because of these separate branches, developers are not able to harness the utility of hudson and our CI environments. Hudson is set up with a specific location in SVN in mind when you create a project, usually trunk. If you want to build your branch, you have to set up a new project and go through all of the configuration of your trunk project but with the new location. There’s no real way to just duplicate a project so you can change a setting, and even if there were the manual steps involved would be enough that most developers won’t bother. There should be a better way.

I think that there should be a different style of project able to be used in Hudson. The project should define a standard build like we did with trunk on our project, but also allow it to monitor all of the folders under branches and treat each one as their own project. Whenever a change is made to one of those branches, a build will be performed. If a new branch is created, a new project should dynamically be created for that branch. When the build is made, it should be deployed according to the Staging instructions of the original project, but to a dynamically generated VM or from a pool of VMs with modifiable settings for things like ports and server names. This will allow us to do continuous deployment on each of our development branches so that if a QA tester finds a problem with a new feature a developer is working on in a branch, the developer can quickly provide a fix and have it automatically go through a full set of continuous integration tests and be deployed to the staging environment for the tester to try out again.

This fast turn around time between bug reporting, bug fixing, and fix deployment could greatly enhance productivity and let us try out different solutions, faster. With a fast feedback loop, we can work directly with the QA department or our customers to solve the issue they’re having and let them try it out right away in such a fashion that doesn’t effect anyone else’s bugs or problems that they are working on.

In essence, dynamic continuous integration would broaden the usage of our continuous integration servers to cover all development, even on separate branches, without the need to manually create (or forget to create) projects for them. I think that if we worked on creating such a functionality in Hudson, along with some helper tools for utilizing large pools of servers for fast staging environment, we can create stronger bonds between developers and testers, along with developers and their customers by having a shorter feedback loop and faster turnaround of problems.


Follow

Get every new post delivered to your Inbox.