File Synchronisation with Unison

As a web developer, one of my code tasks is to deploy a website to a live environment. This boils down to copying lots of files in lots folders from my development environment (a Windows 7 computer) to a live environment (could be Linux or Windows). To give you an idea of what "lots" can be, a typical Magento installation without any customisations, themes or 3rd party components weighs in at around 10,000 file across 3,500 folders.

I also like to propagate changes from a live environment back to my dev environment (or at the very least be aware of such changes so they don't sneak up on me in a dark alley).

Sounds like a bi-directional file synchronisation problem across different operating systems! Easy!

 

The most simple solution for this in the Windows world is robocopy. It can do all the above, as long as there's an SMB/CIF share for the site. Unfortunately, this isn't always true of our Linux hosts. And it can be pretty slow, even if not much has changed, because robocopy has to at least enumerate every file and folder with its last modified date and time to see if anything has changed (slow in this case is perhaps 10-15 seconds). And, robocopy can be destructive (I've lost data using the "/MIR" option in the past); as in it can delete files / folders without warning, backup or recourse.

The classic Linux solution is rsync. But, that involves installing cygwin and significant installation overhead on Windows.

I wanted something faster, more cross platform and that would show me what its about to do before it does it. Faster as in instant. Cross platform as in... err... supporting Windows to Linux sync with only SSH shell access. And non-destructive as in a GUI which summarises changes before I hit a "go" button.

Enter Unison.

Unison does cross platform file sync (any combination of Linux and Windows).

Unison is very fast. It tells me all changes in that 10k Magento tree in 2 seconds (and I configure it to only look at the two folders I regularly modify for instant feedback). It's so fast because it uses its own protocol to transmit changes between two sites; and the unison binary runs locally on each server in parallel.

Unison (at least its GUI version) tells me what its going to do (deletions, new files, modified) before I do it.

Unison detects conflicts bi-directionally (it keeps a database of each side to detect if a file has been changed ).

Unison runs entirely as a user mode application. Which means you can install it in your home directory and launch it as any other application without getting permission from IT departments (please, do get permission from your IT department though, it really will save you a lot of bother in the long run).

Unison washes your dishes..... OK, Unison doesn't wash your dishes, but it seems to do pretty much everything else I want it to!

 

Three things Unison isn't good at though 1) encrypting traffic between sites; you need an encrypted VPN or tunnel over SSH and 2) it's user manual is hard to access in the application itself (it's basically a Unix style man page in a tiny text only window). Fortunately, the manual is available in HTML (unfortunately, I only realised that when I was writing this post). 3) The GUI version is based on GTK+ which isn't included with the Windows binaries; you have to download and install GTK yourself from an unrelated 3rd party project (not even the GTK+ website). I'm sure developers can handle #3, but it is annoying in this age of all-in-one installers.

Enough sales and marketing gumpf.

Installation

To install in Windows, download the Unison binaries and the GTK binaries (if those links are broken, navigate via the main Unison site). Unzip Unison somewhere (on my dev PC I put it in "c:\users\murray.grant\AppData\Local\Unison", and on Windows servers in "c:\program files (x86)\Unison"). Install GTK. You'll probably want to add the GTK folder to your path environment variable (right click Computer -> Properties -> Advanced System Properties -> Environment Variables -> Variable: Path -> Edit -> add ";C:\Program Files\Common Files\GTK\2.0\bin").

To install in Linux there are Debian (apt-get) and Redhat (rpm) binaries available (and others).

Configuration

Unison configuration files live in "%USERPROFILE%\.unison" in Windows and "~/.unison" in Unix. You use them to configure the locations you want to synchronise and files to include or ignore. Here are some examples.

For an old ASP website:

# Unison preferences
label = Company-Website-Prod-All
root = C:\projects\Company-Website\website
root = socket://server.company.local:21234/f:/shares/public/intranet.company.com.au

ignore = Path RCT/Conference
ignore = Path RCT/ProgramMaterials/CD
ignore = Path Test%20Project
ignore = Name thumbs.db

fastcheck = yes

For the full Magento sync:

# Unison preferences
label = Company-Project-Prod-All
root = C:\projects\Company-Project\trunk\magento
root = socket://localhost:41234//home/webhosts/companysite.com.au

ignore = Name .ssh
ignore = Name .svn
ignore = Name .settings
ignore = Name .buildpath
ignore = Name .project
ignore = Path var
ignore = Path media

# Production specific config / cache files
ignore = Name app/etc/local.xml
ignore = Path includes/config.php
ignore = Path includes/src

fastcheck = yes

The minimal Magento sync just adds some "path" options to specify specific folders to sync:

path = app/etc
path = app/code/local
path = app/design/frontend/default/sitetheme
path = skin/frontend/default/sitetheme

Running Unison

You need to start Unison on the server before you can sync. On Linux:

webserver$ unison -socket 41234

Or on Windows:

"%PROGRAMFILES(x86)%\Unison\Unison-2.40.61 Text.exe"

I prefer to keep my unison preferences files with the projects, so I have a batch file to copy the preferences file and invoke Unison:

copy Company-Project-Prod-All.prf "%USERPROFILE%\.unison\Company-Project-Prod-All.prf"
"%LOCALAPPDATA%\Unison\Unison-2.40.61 Gtk+.exe" "Company-Project-Prod-All"

And when it's all in action:

Gotchas

Your local and the server Unison version must match the first 2 numbers (eg: 2.27.157 will connect to 2.27.57, but 2.40.61 will not). Windows binaries are available for several recent versions so you can match whatever you use server side.

And watch out for files which only differ by case. In Linux "file.php" and "File.php" are two different files, but they are the same in Windows (well, NTFS knows they're different, but Win32 doesn't). This hasn't been a big problem for me, and it's pretty bad practise to rely on this behaviour in Linux anyway.

And please, don't make a Unison server public. Traffic is unencrypted, and totally unauthorised (no password required). If this is proprietary or secret source code you're copying around, a public Unison server is about the same as posting it all on GitHub. Tunnel through SSH. Create a VPN. Do anything except make a Unison server public.

0
Implementing a workflow in Magento
Meet the two newest members of our team

Comments

 
No comments made yet. Be the first to submit a comment
Mobile Version | Desktop Version