Deploying Updates from Windows Dev Box to Linux Server
I develop on my local Windows 7 (or XP netbook when out of town) and to test/deploy projects I need to send them to my Gentoo Linux server. While manual copying of files quickly becomes a pain in the arse a fully fledged continuous integration environment is needlessly complex for a single programmer project so I created the following batch script that once run checks my windows folders for new & updated files since the last time the script was run then sends them across the ether using scp1.
The script has been tested on XP & 7 successfully.
For the linux side of things I run a python paster instance with the —reload flag that means paster will pickup changes and restart automatically, making deployment from my windows box a single click action.
Initially I started out a little rusty in my command line knowledge and had set about getting timestamps from first principles – lots of find, substring comparisons and messy embedded if control statements. Then I found command extensions which gave convenient access to file properties. Forfile and robocopy were potentially very useful shortcuts but neither are found in the base WinXP system so I eventually arrived at the following solution. The key part where datetime comparisons are made is in the main loop:
dir /b /tw /od !folder!%timestampfile% !file! | more +1 > %tmpfile% set /p latest= < %tmpfile% del %tmpfile%
Here the [tmp] timestamp file is compared with the current file in the recursive loop and ranked in ascending order of modification datetime (/b cuts out unwanted output from dir, /tw tells it to use file modification time for sorting and /od tells it to order by oldest first). This is piped to more which takes the bottom line (the newest file) and throws it into %tmpfile%. The variable !latest! then gets set to the value (/p switch tells set to expect a single line of input) and the temporary file is deleted. Afterwards we test to ensure !latest! is actually a file we want to transfer and not the timestamp file (or the batch file for that matter) and if so we copy it across to our transfer folder. Anyways:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86
@echo OFF REM IMPORTANT: REM 1. after you checkout your project into the working directory, create this batch file AND a blank txt file named %timestampfile% in your top level project directory. REM 1. make sure timestampfile IS unique (no other FILES with this NAME) within all subfolders of your working directory REM 2. make sure tmpfile IS unique within pwd REM 3. make sure tmp IS unique directory REM 4. set your pscp, conn & pw strings TO appropriate values REM REM original post: http://aleatory.clientsideweb.net/2012/01/18/deploying-windows-dev-to-linux-server set tmp=C:\tmp\ set tmpfile=tmp.txt set timestampfile=timestamp.txt set pscp="C:\path\to\pscp.exe" set conn=user@server:/path/TO/directory set pw=yourpasswdhere REM ensure w2k command extensions are enabled AND delayed expansion FOR the control LOOP setlocal enableextensions enabledelayedexpansion REM tmp directory FOR copying modified FILES across MKDIR %tmp% CALL :strlen prelen %~dp0 set /a LEN=2+prelen REM need TO copy timestampfile temporarily TO each subfolder FOR datetime comparison in main LOOP. FOR /d /r %%j in (*) DO ( copy %timestampfile% %%j ) REM substring %var:~start,length% - 0-based REM LOOP through all FILES in directory & SUB directories REM IF last modified time greater than %timestampfile% - copy TO set TO ssh across FOR /r %%i in (*) DO ( set file=%%~dpnxi set file=!file:~%len%! set folder=%%~dpi set folder=!folder:~%len%! REM below will find last modified time FOR each AND RETURN latest file REM always returns 2nd file IF both were modified in the same minute, so we should PUT source file 2nd. REM t the moment /tw doesn't work as expected when comparing timestampfile with files inside different folders; REM so the only immediate solution IS TO have a temp copy of timestampfile in the same directory ON each LOOP... dir /b /tw /od !folder!%timestampfile% !file! | more +1 > %tmpfile% set /p latest= < %tmpfile% del %tmpfile% REM IF latest isn't this batch file or the timestamp file then copy across, maintaining relative folder structure IF NOT !latest!==%~nx0 IF NOT !latest!==%timestampfile% ( echo %tmp%!folder!!file! MKDIR %tmp%!folder! & copy !file! %tmp%!folder! ) ) REM timestampfile cleanup FOR /d /r %%j in (*) DO ( del %%j\%timestampfile% ) %pscp% -batch -pw %pw% -r %tmp%* %conn% RMDIR /s /q %tmp% REM update timestamp file echo %date%-%time%>%timestampfile% GOTO :EOF REM FUNCTION TO count 'string' length :strlen <resultVar> <stringVar> ( setlocal EnableDelayedExpansion set "s=!%~2!#" set "len=0" FOR %%P in (4096 2048 1024 512 256 128 64 32 16 8 4 2 1) DO ( IF "!s:~%%P,1!" NEQ "" ( set /a "len+=%%P" set "s=!s:~%%P!" ) ) ) ( endlocal set "%~1=%len%" EXIT /b )
(apologies for the non-batch file syntax highlighting – there’s a bug in GeSHi syntax highlighting engine that inserts HTML cruft into the code.)
You’ll note the intermediate step it takes of creating a temporary folder to place the files to be copied. This means I can copy the whole thing across as a oner rather than a long-winded new ssh connection for each one. The ideal windows solution would be to create a network folder using SAMBA on the Linux server where I could simply copy the files across in one go. If the file sizes were massive I might do this but I already use SSH for sysadmin so to cut down on one less potential security hole I’ll keep it simple.
Notably I had to use pukka REM statements and not the faster double colon to avoid the comments within the FOR loop being interpreted as commands and returning the following error:
The filename, directory name, or volume label syntax is incorrect
Also the one caveat with the above batch code is it won’t copy anything that has the same name as the script or the timestampfile value – so name them accordingly. If there is a better way of doing any of the steps I took then you are welcome to comment below.
Overall, yeah batch file processing is hard.
1. pscp – a command line scp utility.