Removing Unversioned Files and Modified Files as Part of a Continuous Integration Build

As part of my continuous integration builds using CruiseControl, I’ve fallen into the habit of the following pattern:

  1. Perform an SVN Update (get the latest release)
  2. Overwrite the updated project directory with a set of static files. For example, if my project lives in ${cc.home}/projects/${}, I’ll have another directory under ${cc.home}/nonvcsfiles/${} in which I store unversioned content.
  3. Perform the build
  4. Copy out the build artifacts.

Why do I follow such a pattern, you ask? The reasons are twofold:

First, I deal with some rather complex deployments to multiple servers, etc. and this is an easy way of overwriting properties files, web.xml files (Java), and web.config files (.NET) with server specific values. Since I also use CruiseControl to deploy to these servers as part of a larger, integration test scheme, I might have the same build doing a lot of different things. This way, I can store a directory of new or modified files that is simply xcopied over the build directory AFTER the SVN update. When producing artifacts such as WAR files, it allows me to pre-tailor the web.xml file prior to deployment.

Second, it allows me to copy in large quantities of binary files that might be linked in. A number of my projects are now making use of Maven and Artifactory to maintain version relationships to Jar files, etc., and I’m aware that you can use svn:externals to store external libraries or code. In some cases, however, I’m using the same CruiseControl instances to build .NET, Java (ant) and Java (maven) projects all at once, and the simplest thing to do is copy external library binaries into the requisite lib directories without putting them explicitly under version control. Maybe you’ve got a problem with this, and I can understand that, but sometimes, the quick solution is the easy solution. We keep good records of our library dependencies, and store them under a directory structure that makes versioning evident; additionally, we document the dependencies and versions in a repeatable way. Enough about that, that’s another holy war…

The net result of this is that once I’m done with a build in CruiseControl, my build directory is littered with modified configuration files, and newly copied binary libraries and some other crud. The question is, what do you do to revert your build directory back to it’s pristine, trunk versioned goodness?

The answer is twofold: revert any modified files using svn revert, and then delete unversioned files and directories.

The second part of that turned out to be a little trickier than I would have thought. If you want to delete unversioned files automatically, you’ve got a bit of problem: you need some external scripting or a quick macro / command line argument, which required some research.

After a little looking, I found a nice summary of methods that you can use at this link: Automatically Remove Subversion Unversioned Files. It shows a number of methods and number of scripting languages.

For my use, I need to do this on a Windows 2003 machine with the subversion command line tools. Unfortunately, in the link above, I did find a solution for a command line script on windows, but it doesn’t work for directories, and it doesn’t work for files with spaces in the name. However, with a few tweaks, here’s the three lines of script needed.

svn revert -R .
for /f "tokens=1*" %i in ('svn status ^| find "?"') do del /q "%j"
for /f "tokens=1*" %i in ('svn status ^| find "?"') do rd /s /q "%j"

The first line reverts any of the files that are modified. The second line deletes any unversioned files, and the third line deletes any unversioned directories (and their contents, recursively).

There are two GREAT things about this process:

  • If there are certain build files or directories that you don’t want deleted or cleaned up, you can add these to the svnignore list, and these will be ignored by version control and NOT deleted via this process.
  • This ensures that any additional non-versioned files that make it onto your build server over time are accounted for, either appearing in version control, or appearing the special nonvcsfiles directories.

The final step is to drop these items into an ANT task that can be called either before a new build, or at the end of a build to clean up. Just use the exec method for each line, like this:

<!-- clean out nonversioned files -->
<target name="clean">
<!-- revert all SVN version controlled files -->
<exec executable="svn">
<arg value="revert"/>
<arg value="-R"/>
<arg value="."/>

<!-- delete all unversioned files not explicit ignored by SVN-->
<exec executable="cmd">
<arg value="/c"/>
<arg value="for /f &quot;tokens=1*&quot; %i in ('svn status ^| find &quot;?&quot;') do del /q &quot;%j&quot;"/>

<!-- delete all unversioned directories not explicit ignored by SVN-->
<exec executable="cmd">
<arg value="/c"/>
<arg value="for /f &quot;tokens=1*&quot; %i in ('svn status ^| find &quot;?&quot;') do rd /s /q &quot;%j&quot;"/>