Using subversion to manage code

I have finally come to terms with the fact that I need some kind of version control for the projects I am working on and the best bet these days is Subversion (svn). I have been using svn for some time now via a GUI client (Linux: kdesvn, Windows: tortoisesvn); however, it turns out that working with svn from the command line is pretty easy and far more versatile. For a complete treatment of this subject, check out the online documentation here and an excellent cheat sheet here. What follows is a quick primer on the very basics of setting up and managing your svn repositories on a local machine or server.

For ease of use I will describe the creation of a single repository for a single project. This means a little more overhead; however, it makes the repository more portable and flexible in the long run.

1) First we set up the repository structure in a temporary folder (either on the server or locally):

$ mkdir ~/tmp
$ mkdir ~/tmp/project1
$ mkdir ~/tmp/project1/trunk
$ mkdir ~/tmp/project1/branches
$ mkdir ~/tmp/project1/tags

2) Now, make a folder to hold your repositories and create an empty repository for your project.

$ mkdir ~/svnroot
$ svnadmin create ~/svnroot/project1

3) Import the folder structure into the empty repository. After this import, the folders in tmp can be removed -- they are only there to make the creation of the folder structure easier.

$ svn import ~/tmp/project1 file:///home/myusername/svnroot/project1 -m "Initial import"

4) Finally, make your current project folder a "working copy" of the repository. Checkout the trunk (or head) of the repository to the folder where project1 currently resides (in this example, the existing project files are located at ~/working/project1).
If you created your repository on a local folder:

$ svn checkout file:///home/myusername/svnroot/project1/trunk ~/working/project1

Alternatively, if you created your repository on a remote server:

$ svn checkout svn+ssh://remote.server.name/home/myusername/svnroot/project1/trunk ~/working/project1

Because the repository is empty at this stage, all the above commands do is create a .svn folder in the ~/working/project1 directory. The following command will show that there are folders and files in the project directory that are not currently part of the repository:

$ svn st
?      somefolder
?      someotherfolder
?      somefile.txt

Now you need to add all of the files and folders in this directory to the repository. This is easily accomplished using a bit of awk code (modified from a post here):

$ svn status | grep "^\?" | awk -F "      " '{print $2}' | tr "\n" "\0" | xargs -0 svn add
$ svn st
A      somefolder
A      somefolder/file1.txt
A      somefolder/file2.txt
A      someotherfolder
A      someotherfolder/file3.txt
A      somefile.txt

Now you just need to commit these changes and your working directory is up to date:

$ svn commit -m "Adding original files to repository."

5) When you are ready to commit new changes to the repository, make sure that all new files/folders are added and all deleted files/folders are removed:

$ svn status | grep "^\?" | awk -F "      " '{print $2}' | tr "\n" "\0" | xargs -0 svn add
$ svn status | grep "^\!" | awk -F "      " '{print $2}' | tr "\n" "\0" | xargs -0 svn remove
$ svn commit -m "Some comment to remind you why you are committing changes..."

6) Finally, if for some reason you want to remove a working directory from versioning (i.e., get rid of the .svn folders that are placed in every folder subfolder), use the following:

$ cd ~/working/project1
$ rm -rf 'find . -name .svn'

Update: If your svn repository changes to a new server name, use the following syntax to update your working directory:

$ cd ~/working/project1
$svn info
[the current URL and other info are printed to the screen]
$svn switch --relocate svn+ssh://OLD.URL/path/to/svnrepo svn+ssh://NEW.URL/path/to/svnrepo .
$svn commit -m "new update"

Note: If you mis-type the old URL, "svn switch" will fail silently. So make sure to check that it has updated by using the "svn info" command.

Installing Ubuntu Linux on existing LVM partition.

I wanted to install Ubuntu on a computer that already housed Fedora Linux. It seems that Fedora defaults to installing /root and /swap on an LVM partition -- something that Ubuntu does not recognize out of the box. After poking about online, here is the process I followed (compiled from here, here, and here.):

Warning: As always, any time you are messing about with drive partitions, you risk something tragic happening to your data. Backup critical bits before continuing.

First deal with reducing the existing logical volume:

# Boot Ubuntu from live-CD.

# Install lvm2:
$ sudo apt-get install lvm2

# Load the necessary module(s):
$ sudo modprobe dm-mod

# Scan your system for LVM volumes and identify in the output the volume group name that has your Fedora volume (mine proved to be VolGroup00):
$ sudo vgscan

# Activate the volume:
$ sudo vgchange -ay VolGroup00

# Find the logical volume that has your Fedora root filesystem (mine proved to be LogVol00):
$ sudo lvs

# Determine how much space is used on the volume:
$ sudo vgdisplay

# Check the filesystem
sudo e2fsck -f /dev/VolGroup00/LogVol00

# Resize the filesystem to desired size (CRITICAL before proceeding to next step!)
$ resize2fs -f /dev/VolGroup00/LogVol00 16G

# Reduce the size of the logical volume
$ lvreduce -L10G /dev/VolGroup00/LogVol00

Create new logical volumes for /root and /home (in my case /swap already existed):

# First make sure how much free space remains in the volume group
$ sudo vgdisplay

# Create \UbuntuRoot and \home
$ sudo lvcreate -n UbuntuRoot -L10G VolGroup00
$ sudo lvcreate -n home -L18.28G VolGroup00

# Now format the new logical volumes:
$ sudo mkfs -j /dev/VolGroup00/UbuntuRoot -L root
$ sudo mkfs -j /dev/VolGroup00/home -L home

Then run the installer, choosing the appropriate volumes. Note that I had already created a hard partition to serve as /boot --- this is the only way a lvm install will work.

Finally, make sure that the newly installed system has lvm2 included (otherwise it will not start). The following is adapted from here.

# Create a mountpoint for the newly installed system:
$ sudo mkdir /target
$ sudo mkdir /target

# Mount the newly installed system:
$ sudo mount /dev/VolGroup00/UbuntuRoot /target
$ sudo mount /dev/sda6 /target/boot

# Activate those directories and install lvm2
$ sudo chroot /target
$ sudo apt-get update
$ sudo apt-get install lvm2

Now you can restart into the new system!

Error capture

In a recent post to r-sig-ecology, Mike Colvin suggested the following to capture errors within a loop:

for (i in 1:1000){
fit<-try(lm(y~x,dataset))
results<- ifelse(class(fit)=="try-error", NA, fit$coefficients)
}

Plotting contours

Plenty of packages allow you to plot contours of a "z" value; however, I wanted to be able to plot a specific density contour of a sample from a bivariate distribution over a plot that was a function of the x and y parameters. The example only plots the density of x and y; however, if you set plot.dens=FALSE, the contours will be added to the active display device (i.e., over an image(x,y,function(x,y){...})).

This function also lets you specify the line widths and types for each of the contours.

###########################################
## drawcontour.R
## Written by J.D. Forester, 17 March 2008
###########################################

##This function draws an approximate density contour based on
##empirical, bivariate data.


##change testit to FALSE if sourcing the file
testit=TRUE


draw.contour<-function(a,alpha=0.95,plot.dens=FALSE, line.width=2, line.type=1, limits=NULL, density.res=300,spline.smooth=-1,...){
  ##a is a list or matrix of x and y coordinates (e.g., a=list("x"=rnorm(100),"y"=rnorm(100)))
  ## if a is a list or dataframe, the components must be labeled "x" and "y"
  ## if a is a matrix, the first column is assumed to be x, the second y
  ##alpha is the contour level desired
  ##if plot.dens==TRUE, then the joint density of x and y are plotted,
  ##   otherwise the contour is added to the current plot.
  ##density.res controls the resolution of the density plot

  ##A key assumption of this function is that very little probability mass lies outside the limits of
  ## the x and y values in "a". This is likely reasonable if the number of observations in a is large.
  
  require(MASS)
  require(ks)
  if(length(line.width)!=length(alpha)){
    line.width <- rep(line.width[1],length(alpha))
  }

  if(length(line.type)!=length(alpha)){
    line.type <- rep(line.type[1],length(alpha))
  }
  
  if(is.matrix(a)){
    a=list("x"=a[,1],"y"=a[,2])
  }
  ##generate approximate density values
  if(is.null(limits)){
    limits=c(range(a$x),range(a$y))
  }
  f1<-kde2d(a$x,a$y,n=density.res,lims=limits)

  ##plot empirical density
  if(plot.dens) image(f1,...)
  
  if(is.null(dev.list())){
    ##ensure that there is a window in which to draw the contour
    plot(a,type="n",xlim=limits[1:2],ylim=limits[3:4],...)
  }
    
  ##estimate critical contour value
  ## assume that density outside of plot is very small

  zdens <- rev(sort(f1$z))
  Czdens <- cumsum(zdens)
  Czdens <- (Czdens/Czdens[length(zdens)])
  for(cont.level in 1:length(alpha)){
    ##This loop allows for multiple contour levels
    crit.val <- zdens[max(which(Czdens<=alpha[cont.level]))]
   
    ##determine coordinates of critical contour
    b.full=contourLines(f1,levels=crit.val)
    for(c in 1:length(b.full)){
      ##This loop is used in case the density is multimodal or if the desired contour
      ##  extends outside the plotting region
      b=list("x"=as.vector(unlist(b.full[[c]][2])),"y"=as.vector(unlist(b.full[[c]][3])))
      
      ##plot desired contour
      line.dat<-xspline(b,shape=spline.smooth,open=TRUE,draw=FALSE)
      lines(line.dat,lty=line.type[cont.level],lwd=line.width[cont.level])
    }
  }
}

##############################
##Test the function
##############################

##generate data
if(testit){
  n=10000
  a<-list("x"=rnorm(n,400,100),"y"=rweibull(n,2,100))
  draw.contour(a=a,alpha=c(0.95,0.5,0.05),line.width=c(2,1,2),line.type=c(1,2,1),plot.dens=TRUE, xlab="X", ylab="Y")
}

Drop unused factor levels

When creating a subset of a dataframe, I often exclude rows based on the level of a factor. However, the "levels" of the factor remain intact. This is the intended behavior of R, but it can cause problems in some cases. I finally discovered how to clean up levels in this post to R-Help. Here is an example:

> a <- factor(letters)
> a
 [1] a b c d e f g h i j k l m n o p q r s t u v w x y z
Levels: a b c d e f g h i j k l m n o p q r s t u v w x y z

## Now, even though b only includes five letters, 
## all 26 are listed in the levels
> b <- a[1:5]
> b
[1] a b c d e
Levels: a b c d e f g h i j k l m n o p q r s t u v w x y z

## This behavior can be changed using the following syntax:
> b <- a[1:5,drop = TRUE]
> b
[1] a b c d e
Levels: a b c d e

Another way to deal with this is to use the dropUnusedLevels() command in the Hmisc library. The only issue here is that behavior is changed globally which may have undesired consequences (see the post listed above).

****UPDATE****
As Jeff Hollister mentions in the comments, there is another way to do this:

a<-factor(letters)
b<-factor(a[1:5])

Yet another way, if you are working with data frames that by default convert characters into factors, was suggested on r-sig-ecology by Hadley Wickham:

options(stringsAsFactors = FALSE)
a <-data.frame("alpha"=letters)
b<-a[1:5]

Convert factors to numbers

If you have a vector of factors it is easy to get the factor level; however, I always forget how to extract the factor value. I ran into the answer here.

> x<-factor(c(round(rnorm(10),2),"A","B",NA))
> x
 [1] 1.61  1.12  1.26  0.09  -0.13 0.16  -0.03 -0.1  0.09  -0.47 
 [11] A     B    
Levels: -0.03 0.09 -0.1 -0.13 0.16 -0.47 1.12 1.26 1.61 A B
> as.numeric(x)
 [1]  9  7  8  2  4  5  1  3  2  6 10 11 NA
> as.numeric(levels(x)[x])
 [1]  1.61  1.12  1.26  0.09 -0.13  0.16 -0.03 -0.10  0.09 -0.47    
 [11]   NA    NA   NA
Warning message:
NAs introduced by coercion 

**UPDATE**
And another method mentioned in the comments (that I like better):

> as.numeric(as.character(x))

Preparing plots for publication

The plotting capabilities of R are excellent; however, when I am preparing a figure for publication, I often need to combine multiple plots or add objects (e.g., arrows or text) to an existing plot. While this can be accomplished in R, my patience for tweaking layout parameters tends to run out quickly. I searched around and found a nice solution using open source software (an example with Matlab plots is described here).

1) Create the desired figure in R and export as an eps file:

library(ggplot2)
postscript("myfig.eps",horizontal=FALSE, paper="special",height=10,width=10)
qplot(x,y,data=mydat)
dev.off()

2) Install the necessary software on your computer, and then convert the image file to one more easily edited.

#install software
sudo aptitude install pstoedit tgif

#clean up the eps file so it is more easily read
eps2eps myfig.eps myfig2.eps

#convert the eps file to a tgif object
pstoedit -f tgif myfig2.eps myfig2.obj

3) Edit the new file myfig2.obj in tgif.