Local Ubuntu Mirror


We were getting tired of our Ubuntu servers reaching out to the internet every day/week to fetch updates and patches. Man if you thought Windoze was bad, these Ubuntu servers know their way around consuming bandwidth as well. So in the Windoze world we had something called WSUS (if it is still called that?) so I wanted something similar to keep roughly 150+ servers from going out to fetch their bits.

The APT-MIRROR package to the rescue. Setup is quite easy, but i did not find any ‘cook book’ that fitted all my needs.

First off I need to store these mirrors on a nice de-duplicated NFS server (Nexenta/ZFS) to keep it from consuming to many GB’s. So what we did is create a NFS export on one of our SAN’s with root access for the Mirror server, simple enough.

On the server side

On the server we install APT-MIRROR with a simple apt-get command;

apt-get install apt-mirror

This installs the package and creates the structure in /var/spool/apt-mirror

Now we mount our NFS mount-point to the /var/spool/apt-mirror/mirror from the FSTAB in the usual way

<hostname>:/volumes/pool1/mirror /var/spool/apt-mirror/mirror nfs auto,noatime,nolock,bg,nfsvers=3,intr,tcp,actimeo=1800 0 0

(yes, our NFS is still V3, since it serves some ESXI hosts as well)

Now it is time to configure our /etc/apt/mirror.list, we ended up with

############# config ##################
#
set base_path /var/spool/apt-mirror
#
set mirror_path $base_path/mirror
set skel_path $base_path/skel
set var_path $base_path/var
set cleanscript $var_path/clean.sh
set postmirror_script $var_path/postmirror.sh
set run_postmirror 1
set nthreads 10
set _tilde 5
set defaultarch amd64

############## end config ##############

deb http://tw.archive.ubuntu.com/ubuntu precise main restricted universe multiverse
deb http://tw.archive.ubuntu.com/ubuntu precise-security main restricted universe multiverse
deb http://tw.archive.ubuntu.com/ubuntu precise-updates main restricted universe multiverse
deb http://tw.archive.ubuntu.com/ubuntu precise-proposed main restricted universe multiverse
deb http://tw.archive.ubuntu.com/ubuntu precise-backports main restricted universe multiverse
deb http://tw.archive.ubuntu.com/ubuntu precise main main/debian-installer restricted restricted/debian-installer universe universe/debian-installer multiverse multiverse/debian-installer

clean http://tw.archive.ubuntu.com/ubuntu

Now how did we decide to go for http://tw.archive.ubuntu.com/ubuntu to fetch our mirror, for us this was the best performing (not necessarily closest) mirror that is up-to-date. A way to determine your fastest mirror is with the handy netselect tool. how to install and use;

First fetch the latest binary (at time of writing 0.3.ds1-25

wget http://ftp.us.debian.org/debian/pool/main/n/netselect/netselect_0.3.ds1-25_amd64.deb

Then throw it against the mirror lists from launchpad, and add some grep magic to make the output readable

netselect -v -s10 -t20 `wget -q -O- https://launchpad.net/ubuntu/+archivemirrors | grep -P -B8 “statusUP|statusSIX” | grep -o -P “(f|ht)tp.*\”” | tr ‘”\n’ ‘ ‘`

For us it came back with the Hong Kong Chinese University and some others, the Taiwan repo was at the 3rd place, We decided to use the Taiwan one since the Hong Kong archives tend not up to date, maybe we should open up ours to have an up-to-date repo down one here 🙂 . Anyway, you should verify your results against the mirror list on https://launchpad.net/ubuntu/+archivemirrors to get your close and up to date repo.

It is now time to do a first time population of the repository by manually executing

apt-mirror -c apt-mirror

This can take some time as we will pull down

Now the newly downloaded repo can be made available through a web server of choice (Apache for us). We just linked the ubuntu folder to the /vat/www, we could also share the folder out as an RSYNC repo on the Nexenta storage server, maybe at a later time, not today.

ln -s /var/spool/apt-mirror/mirror/tw.archive.ubuntu.com/ubuntu /var/www/ubuntu

Now all that is left to do on the server side is to add the update script to your cron tab for scheduled execution, this is made easy as the package provides this schedule file in /etc/cron.d/apt-mirror, the only thing you need to do there is to uncomment the line

On the Client Side

On the client side we need to edit the /etc/apt/sources.list. Here we commented every deb repo and left the deb-src pointing to the us repositories. Then on the top of the file we added the deb repe’s for our own mirror.

deb http://<domain>/ubuntu precise main restricted universe multiverse
deb http://<domain>/ubuntu precise-security main restricted universe multiverse
deb http://<domain>/ubuntu precise-updates main restricted universe multiverse
deb http://<domain>/ubuntu precise-proposed main restricted universe multiverse
deb http://<domain>/ubuntu precise-backports main restricted universe multiverse

#deb http://us.archive.ubuntu.com/ubuntu/ precise main restricted
deb-src http://us.archive.ubuntu.com/ubuntu/ precise main restricted
#deb http://us.archive.ubuntu.com/ubuntu/ precise-updates main restricted
deb-src http://us.archive.ubuntu.com/ubuntu/ precise-updates main restricted
#deb http://us.archive.ubuntu.com/ubuntu/ precise universe
deb-src http://us.archive.ubuntu.com/ubuntu/ precise universe
#deb http://us.archive.ubuntu.com/ubuntu/ precise-updates universe
deb-src http://us.archive.ubuntu.com/ubuntu/ precise-updates universe
#deb http://us.archive.ubuntu.com/ubuntu/ precise multiverse
deb-src http://us.archive.ubuntu.com/ubuntu/ precise multiverse
#deb http://us.archive.ubuntu.com/ubuntu/ precise-updates multiverse
deb-src http://us.archive.ubuntu.com/ubuntu/ precise-updates multiverse
deb-src http://us.archive.ubuntu.com/ubuntu/ precise-backports main restricted universe multiverse
#deb http://us.archive.ubuntu.com/ubuntu/ precise-backports main restricted universe multiverse
#deb http://security.ubuntu.com/ubuntu precise-security main restricted
deb-src http://security.ubuntu.com/ubuntu precise-security main restricted
#deb http://security.ubuntu.com/ubuntu precise-security universe
deb-src http://security.ubuntu.com/ubuntu precise-security universe
#deb http://security.ubuntu.com/ubuntu precise-security multiverse
deb-src http://security.ubuntu.com/ubuntu precise-security multiverse

now run an apt-get update / apt-get upgrade, and all should be coming from your own repo

As always, hope this helps someone save time, if it does not work out for you, pleaee  leave a comment and we’ll try to help where possible

– Fault