I'm in a team of FPGA developers creating designs on Windows machines but we have a Linux build server. In order to provide a common build scripts between the Linux Build Server and our local Windows PCs we use Cygwin to execute the build scripts locally so that the same build scripts can be used in both environments. We now need a means to ensure that each person using the build scripts has the same installation of Cygwin, even if they need to reinstall their Windows PC. The problems I aim to solve here are the evolving nature of the Cygwin mirrors which means any two installations a few days apart can be different, and scripting the selection of packages to install.
- Scripting the Installation Alone
- Scripting the Dated Mirror and Installation
- Scripting the Dated Mirror
- Scripting the Installation
- Conclusions
- References
Scripting the Installation Alone
The installation script will work perfectly well without an internal mirror. But pause for a moment. As each day passes, more Cygwin packages get updated. This means an installation today will be different to one a few days ago, and mean that you cannot ensure that multiple installations across different machines are identical. This might be fine for personal use, but for a team of developers this could prove to be a problem if a few Cygwin tools have different versions across each machine. For this reason I will deal with creating a Cygwin mirror first.
Scripting the Dated Mirror and Installation
Recall that Cygwin mirrors 'evolve' over time, keeping up to date with all the latest versions of each installation package. The aim here is to snapshot the mirror at a specific time, so that there is a stable non-evolving installation reference. We can then create further snapshots of the mirror over time, and refer to each snapshot by a date, i.e. we have a sequence of dated mirrors. This will allow us to roll the team forward to new versions of Cygwin that are approved for use (tested to work with the build script) in a controlled manner, without forcing each team member to update on the same day, which might not be convenient to them.
With the dated mirrors approach, it is possible that each incremental date need only store the differences between each dated mirror such that only the first dated mirror is a full mirror to ensure an efficient use of storage. This is achieved using rsync's out of the box behaviour to create a mirror. Another practical issue is that any wisened IT department will ensure their IT network(s) are protected by a firewall. The firewall will probably prevent users reaching out directly to the Internet-based mirror to install directly. Therefore the IT department will need to create the appropriate waivers for the dated mirrors to work (which can take months in a large corporation!), and provide the infrastructure to host the dated mirrors too. Our aim here is to ensure that users can install from the internal server containing the dated mirror without impediment, leaving the IT department with a one-off work package to create the initial internal Cygwin dated mirror setup.
Scripting the Dated Mirror
Cygwin provides a basic description of how to use rsync to create a mirror, see Creating a local Cygwin mirror with rsync. I expand on this further by making a sequence of dated mirrors as shown in the Bash script below.
#!/bin/bash
#
# Create a Cygwin mirror for offline installation
# Ref: https://www.cygwin.com/package-server.html
#
# CAREFUL! Delete mess with:
# cd ${MIRRORLOC}
# find * -maxdepth 0 -type d | xargs rm -rf
#
# NB. Configure your Apache webserver to allow directory listings on ${MIRRORLOC}. Either
# through .htaccess or the webservers's site configuration in /etc/apache2/sites-available.
#
# Options +Indexes
#
# Cygwin mirror site chosen from the list at https://cygwin.com/mirrors.html
MIRRORSITE="rsync://cygwin.mirror.constant.com/cygwin-ftp"
# Next Does not work
#MIRRORSITE="rsync://cygwin.mirrors.hoobly.com"
#MIRRORSITE="rsync://mirrors.kernel.org/sourceware/cygwin"
# Local directory to mirror to
MIRRORLOC="/data/website/cygwin-mirror"
# Number of dated mirrors to retain, anything older will be deleted
KEEP=3
THIS=$(realpath ${0})
SCRIPT=$(basename ${THIS})
SCRIPTDIR=$(dirname ${THIS})
# Usage: join ' , ' a b c
# => "a , b , c"
#
# Ref: https://stackoverflow.com/questions/1527049/how-can-i-join-elements-of-an-array-in-bash
#
function join() {
local d=${1}
shift
local f=${1}
shift
printf %s "${f}" "${@/#/$d}"
}
DIR=$(date '+%Y-%m-%d-%H-%M-%S')
cd ${MIRRORLOC}
echo "Mirroring to: ${MIRRORLOC}/${DIR}"
declare -a RSYNCOPTS=(
"-azv"
"--info=progress2"
)
NEWEST=$(find ${MIRRORLOC}/* -type d -prune -exec ls -d {} \; | tail -1)
mkdir -p ${MIRRORLOC}/${DIR}
cd ${MIRRORLOC}/${DIR}
# Download the installer
wget "https://cygwin.com/setup-x86_64.exe"
chmod +x setup-x86_64.exe
# Mirror the packages for installation
time for arch in noarch x86_64; do
mkdir -p ${arch}
# Get size estimate
if [[ (! -z ${NEWEST}) && (-d ${NEWEST}) ]]; then
echo "Hard linking to : ${NEWEST}/${arch}"
# Get estimates for the transfer
rsync --dry-run --stats \
$(join " " ${RSYNCOPTS[@]}) \
--link-dest=${NEWEST} \
${MIRRORSITE}/${arch} \
${MIRRORLOC}/${DIR} | sed -n '/Total transferred file size/ p'
rsync $(join " " ${RSYNCOPTS[@]}) \
--link-dest=${NEWEST} \
${MIRRORSITE}/${arch} \
${MIRRORLOC}/${DIR}
else
rsync --dry-run --stats \
$(join " " ${RSYNCOPTS[@]}) \
${MIRRORSITE}/${arch} \
${MIRRORLOC}/${DIR} | sed -n '/Total transferred file size/ p'
rsync $(join " " ${RSYNCOPTS[@]}) \
${MIRRORSITE}/${arch} \
${MIRRORLOC}/${DIR}
fi
done
# Automate the deletion of old mirrors
FORDELETION=$(find ${MIRRORLOC}/* -type d -prune | xargs ls -d1t | tail -n +$((${KEEP} + 1)))
for m in ${FORDELETION}; do
echo "Deleting mirror at: ${m}"
rm -r ${m}
done
# Find out how much disc space we're taking up
du -sh ${MIRRORLOC}/*
rsync's --link-dest options is great for this implementation because if the file already exists unaltered from the copy in the specified directory, a UNIX hard link is created to it instead of making a second copy. The first time rsync is run, the destination files are created and the full sources are copied to destination. Thereafter, only changes in source are copied to the destination. The second dated mirror appears to contain the full Cygwin mirror, but it does not. This means the subsequent dated mirrors can be used as if they were full mirrors for installing Cygwin. The use of hard links makes the whole process space efficient.
And there is more. When the original or oldest dated mirror is deleted, the hard links simply have one link to them removed, and only when the link count reaches zero does the file get deleted. This means when you delete the oldest dated mirror, the new oldest dated mirror now becomes the full copy, no longer just a difference copy. That is neat and entirely what we want here. It means that if you want, you can have a rolling programme of dated mirrors by deleting all those mirrors older than a certain date, or just keeping the last n. The script above implements this using the KEEP variable to set how many dated mirrors to retain when invoking the mirror script.
Scripting the Installation
As mentioned before, this can be used to install from either the local dated mirror or directly from an Internet-based official mirror. As we are installing on Windows, we now switch to using a batch file. This batch file has two installation modes, one for using a mirror access by HTTP and one for using a mirror that can be accessed locally by SMB/CIFS. The LOCAL variable allows you to switch between the two modes. A LOCAL installation via SMB/CIFS will not need to download the packages to be installed, but read them in place from their SMB/CIFS destination as specified by the --local-package-dir parameter. An HTTP installation will require the packages to be downloaded before being installed. In this case the --local-package-dir switch determines where the files are downloaded to before being read for installation.
@echo off
rem
rem Reference: https://superuser.com/questions/40545/upgrading-and-installing-packages-through-the-cygwin-command-line/301026#301026
rem Setup command-line arguments: https://cygwin.com/faq/faq.html#faq.setup.cli
rem
rem Full mirror created by this script is 96GB and took ~2 hours to rsync.
rem Subsequent updates took seconds and after 2 days < 100 MB changes packages.
rem
rem Check the script setup:
rem
rem ***************************************************************************
rem Choose the dated mirror to install from
set DATEDMIRROR=2021-07-08-14-03-58
rem Choose the location on your local disk to install to
set INSTALLDIR=D:\cygwin64
rem Need one of the following two, either install over HTTP(s) or via local path names
rem Could not get a mapped drive letter to work here, it's better to use the SMB/CIFS share name.
rem This is probably because once you elevate to Administrator, the drive mappings can be different.
rem i.e. NOT T:\cygwin-mirror but...
set MIRRORPATH=\\cygwin.local\website\cygwin-mirror
set MIRRORURL=https://www.cygwin.local/cygwin-mirror/%DATEDMIRROR%/
rem Choose installation method: local or mirror?
rem 0 for install from mirror over HTTP(s)
rem 1 for install using local path names and mapped drives
rem If the Cygwin mirror has no webserver, you must use local.
rem If you use the "install from mirror" option, the package files get downloaded locally and take additional space.
set LOCAL=1
rem ***************************************************************************
rem derived variables
set SETUPPATH=%MIRRORPATH%\%DATEDMIRROR%
rem common variable between the two installation types
set CATEGORIES=Base,Devel
rem rhash includes crc32
set PACKAGES=bc,cygutils-extra,nc,procps,psmisc,rhash,zip
rem Cygwin setup 2.905
rem `
rem Command Line Options:
rem
rem --allow-unsupported-windows Allow old, unsupported Windows versions
rem -a --arch Architecture to install (x86_64 or x86)
rem -C --categories Specify entire categories to install
rem -o --delete-orphans Remove orphaned packages
rem -A --disable-buggy-antivirus Disable known or suspected buggy anti virus
rem software packages during execution.
rem -D --download Download packages from internet only
rem -f --force-current Select the current version for all packages
rem -h --help Print help
rem -I --include-source Automatically install source for every
rem package installed
rem -i --ini-basename Use a different basename, e.g. "foo",
rem instead of "setup"
rem -U --keep-untrusted-keys Use untrusted keys and retain all
rem -L --local-install Install packages from local directory only
rem -l --local-package-dir Local package directory
rem -m --mirror-mode Skip package availability check when
rem installing from local directory (requires
rem local directory to be clean mirror!)
rem -B --no-admin Do not check for and enforce running as
rem Administrator
rem -d --no-desktop Disable creation of desktop shortcut
rem -r --no-replaceonreboot Disable replacing in-use files on next
rem reboot.
rem -n --no-shortcuts Disable creation of desktop and start menu
rem shortcuts
rem -N --no-startmenu Disable creation of start menu shortcut
rem -X --no-verify Don't verify setup.ini signatures
rem --no-version-check Suppress checking if a newer version of
rem setup is available
rem --enable-old-keys Enable old cygwin.com keys
rem -O --only-site Do not download mirror list. Only use sites
rem specified with -s.
rem -M --package-manager Semi-attended chooser-only mode
rem -P --packages Specify packages to install
rem -p --proxy HTTP/FTP proxy (host:port)
rem -Y --prune-install Prune the installation to only the requested
rem packages
rem -K --pubkey URL or absolute path of extra public key
rem file (RFC4880 format)
rem -q --quiet-mode Unattended setup mode
rem Comment: You still get the setup progress dialogue box
rem -c --remove-categories Specify categories to uninstall
rem -x --remove-packages Specify packages to uninstall
rem -R --root Root installation directory
rem -S --sexpr-pubkey Extra DSA public key in s-expr format
rem -s --site Download site URL
rem -u --untrusted-keys Use untrusted saved extra keys
rem -g --upgrade-also Also upgrade installed packages
rem --user-agent User agent string for HTTP requests
rem -v --verbose Verbose output
rem -V --version Show version
rem -W --wait When elevating, wait for elevated child
rem process
rem
rem Do the work
rem
if exist %SETUPPATH%\setup-x86_64.exe (
echo Executing: %SETUPPATH%\setup-x86_64.exe
if %LOCAL% EQU 1 (
echo List of available dated mirrors
dir %MIRRORPATH%
echo.
echo You have chosen: %DATEDMIRROR%
echo.
echo Ignore "Cygwin Setup (Not Responding)" dialogue box, it's just busy!
echo.
%SETUPPATH%\setup-x86_64.exe ^
--no-desktop ^
--quiet-mode ^
--wait ^
--arch x86_64 ^
--root %INSTALLDIR% ^
--local-package-dir %MIRRORPATH%\%DATEDMIRROR% ^
--local-install ^
--categories %CATEGORIES% ^
--packages %PACKAGES% ^
--quiet-mode
) else (
%SETUPPATH%\setup-x86_64.exe ^
--no-desktop ^
--quiet-mode ^
--wait ^
--arch x86_64 ^
--root %INSTALLDIR% ^
--only-site ^
--local-package-dir %USERPROFILE%\Downloads ^
--site %MIRRORURL% ^
--categories %CATEGORIES% ^
--packages %PACKAGES% ^
--quiet-mode
)
) else (
echo Cannot find '%SETUPPATH%\setup-x86_64.exe', check the DATEDMIRROR variable
)
pause
You specify what to install via the CATEGORIES and PACKAGES variables. When you amend those variables to change what should be installed, it should only be necessary to re-run the batch file to effect the changes.
Conclusions
I have described how to implement a controlled and well defined installation of Cygwin from a snapshot of a mirror site through to making sure that each team member has the same packages installed. This reduces the variation in Cygwin installs so that build scripts can be qualified against a dated mirror of Cygwin, and the team can roll forward to new versions of Cygwin in a controlled way too.