how to setup a local mirror to build iso

General questions
1

Greetings,

I have been building the iso using http://packages.vyatta.com mirror.
Looks like the Vyatta site may fold soon. Hence I am trying to archive all
the packages in a local mirror and build with it.
Appreciate if someone can give a few pointers how to do this.

What I have done so far:

  1. build iso once and found 380 deb packages in livecd/cache/packages_chroot and packages_bootstrap.
  2. use apt-ftparchive to create a mirror with the 380 .deb.
  3. modify livecd/mk.livecd.functions to have LB_MIRROR_BINARY,
    LB_MIRROR_BOOTSTRAP, dev_repo_root point at my local.

But when I run make iso, it stops at this:
Failure trying to run: chroot /root/build-iso/livecd/chroot dpkg --force-depends --install /var/cache/apt/archives/base-passwd_3.5.22_i386.deb /var/cache/apt/archives/base-files_6.0squeeze5_i386.deb

I couldn’t find any log/status file to tell me why chroot fails.

Thanks
sial

2

The problem with my first trial was that I hit this bug:

The “Packages” file in the Debian mirror needs to be sorted,
else the postinstall script of base-files pkg fails because /etc/passwd
does not exist. I use apt-ftparchive to create my mirror and it does not
sort. May be other tools do. Anyway the solution I chose is to add a
script to sort the packages alphabetically.

Show my procedures below. Usual disclaimer: use at your own risk.
No gaurantee it will work in your environment.

Based on:

Main changes:

  • Create two repos instead of one.
  • The Packages file for Debain packages is sorted alphabetically to avoid
    base-passwd and base-files conflict during core packages install.
  • apt-ftparchive conf files and update scripts in current work dir, not $HOME.

Requirements:

apt-utils
apache2 web server
basic terminal knowledge

In this tutorial I will use i386 architecture. If you are using any other, be sure to change i386 to your architecture code below.

First add a separate user. I used user ‘apt’.

Run these commands as root user

adduser apt --disabled-password (You can set password later if needed, or you can su to that user via terminal or console)

export APTUSER=/home/apt

mkdir $APTUSER/logs

mkdir -p $APTUSER/public_html/debian-oxnard/pool/main (directory to place debian packages)

mkdir -p $APTUSER/public_html/debian-oxnard/dists/squeeze/main/binary-i386

mkdir -p $APTUSER/public_html/vyatta-dev/oxnard/pool/main (directory to place Vyatta built packages)

mkdir -p $APTUSER/public_html/vyatta-dev/oxnard/dists/oxnard/main/binary-i386

touch $APTUSER/logs/access.log $APTUSER/logs/error.log (These two are used by apache web server)

Let’s create some files. Use your favourite editor

File $APTUSER/public_html/debian-oxnard/aptftp.conf

APT::FTPArchive::Release {
  Origin "Debian";
  Label "Debian";
  Suite "stable";
  Codename "squeeze";
  Architectures "i386";
  Components "main";
  Description "Debian packages for oxnard build";
};

File $APTUSER/public_html/debian-oxnard/aptgenerate.conf

Dir::ArchiveDir ".";
Dir::CacheDir ".";
TreeDefault::Directory "pool/";
TreeDefault::SrcDirectory "pool/";
Default::Packages::Extensions ".deb";
Default::Packages::Compress ". gzip";
Default::Sources::Compress "gzip";
Default::Contents::Compress "gzip";

BinDirectory "dists/squeeze/main/binary-i386" {
  Packages "dists/squeeze/main/binary-i386/Packages";
  Contents "dists/squeeze/Contents-i386";
  SrcPackages "dists/squeeze/main/source/Sources";
};

Tree "dists/squeeze" {
  Sections "main";
  Architectures "i386";
};

File $APTUSER/public_html/debian-oxnard/update (script to update debian packages)

#!/bin/bash
INDEXFILE=`sed -n -e 's/^ *Packages "\(.*\)";/\1/p' aptgenerate.conf`
apt-ftparchive generate -c=aptftp.conf aptgenerate.conf
mv $INDEXFILE /tmp/Packages.$$
$HOME/sort_packages.pl /tmp/Packages.$$ > $INDEXFILE
gzip -c $INDEXFILE > ${INDEXFILE}.gz
rm -f /tmp/Packages.$$
apt-ftparchive release -c=aptftp.conf dists/squeeze > dists/squeeze/Release
rm -f dists/squeeze/Release.gpg
gpg -u <yourgpgkeyID> -bo dists/squeeze/Release.gpg dists/squeeze/Release
rm packages-i386.db

File $APTUSER/public_html/vyatta-dev/oxnard/aptftp_dev.conf

APT::FTPArchive::Release {
  Origin "Vyatta";
  Label "Vyatta";
  Suite "unstable";
  Codename "oxnard";
  Architectures "i386";
  Components "main";
  Description "Vyatta built packages for oxnard";
};

File $APTUSER/public_html/vyatta-dev/oxnard/aptgenerate_dev.conf

Dir::ArchiveDir ".";
Dir::CacheDir ".";
TreeDefault::Directory "pool/";
TreeDefault::SrcDirectory "pool/";
Default::Packages::Extensions ".deb";
Default::Packages::Compress ". gzip";
Default::Sources::Compress "gzip";
Default::Contents::Compress "gzip";

BinDirectory "dists/oxnard/main/binary-i386" {
  Packages "dists/oxnard/main/binary-i386/Packages";
  Contents "dists/oxnard/Contents-i386";
  SrcPackages "dists/oxnard/main/source/Sources";
};

Tree "dists/oxnard" {
  Sections "main";
  Architectures "i386";
};

File $APTUSER/public_html/vyatta-dev/oxnard/update_dev (The packages update script)
NB: the Packages file in this repo does not need to be sorted.

#!/bin/bash

apt-ftparchive generate -c=aptftp_dev.conf aptgenerate_dev.conf
apt-ftparchive release -c=aptftp_dev.conf dists/oxnard > dists/oxnard/Release
rm -f dists/oxnard/Release.gpg
gpg -u <yourgpgkeyID> -bo dists/oxnard/Release.gpg dists/oxnard/Release
rm packages-i386.db

File $APTUSER/sort_packages.pl

#!/usr/bin/perl
 
use strict;
my $infile = $ARGV[0];
my $fh;
open($fh, "<", $infile) or die "open $infile fail $!\n";
my @lines = <$fh>;
close($fh);

my %h;
my $i = 0;
foreach (@lines) {
  if (/^Package: (.*)/) {
    $h{$1} = $i;
  }
  $i++;
}

my @pkglist = sort keys %h;
foreach my $name (@pkglist) {
  printpkg($name);
}

sub printpkg
{
  my $name = shift;
  my $l;
  my $j = $h{$name};
  do {
	$l = $lines[$j++];
    print "$l";
  } while (length($l) > 1);
}

Make the scripts executable:

chmod 755 $APTUSER/public_html/debian-oxnard/update

chmod 755 $APTUSER/public_html/vyatta-dev/oxnard/update_dev

chmod 755 $APTUSER/sort_packages.pl

And now change ownership of all files and directories created now to user apt

chown -R apt:apt $APTUSER

Note: These commands could also be run as normal user, but to avoid constant user switching I done it this way.

Now, let’s configure apache

ifconfig eth0 | grep ‘inet addr’

inet addr:192.168.1.253 Bcast:192.168.1.255 Mask:255.255.255.0

Since eth0 is my primary nic and 192.168.1.253 is my internal ip address, I will use it for configuring apache web server

Again, use your favourite text editor to edit/create these files

File /etc/apache2/ports.conf

NameVirtualHost 192.168.1.253 :80
Listen 80

File /etc/apache2/sites-available/apt

<VirtualHost 192.168.1.253 :80>
        ServerAdmin yourname@email.com

        DocumentRoot /home/apt/public_html
        <Directory />
            Options FollowSymLinks
            AllowOverride None
        </Directory>

        <Directory /home/apt/public_html>
            Options Indexes FollowSymLinks MultiViews
            AllowOverride All
            Order allow,deny
            Allow from all
        </Directory>

        CustomLog /home/apt/logs/access.log combined
        ErrorLog /home/apt/logs/error.log

        LogLevel warn

</VirtualHost>

Now run

a2ensite apt

invoke-rc.d apache2 restart

And try to access http://yourip/, in my case http://192.168.1.253 / You should see debian-oxnard and vyatta-dev.

Since there are no more commands to be run as root, let’s switch to apt user

su - apt

Generate GPG key for archive signing

gpg --gen-key (Use values you desire and be sure not to forget the passphrase)

gpg -K should give you output like this

gpg -K

/home/apt/.gnupg/secring.gpg

sec 2048R/yyyyyyyy 2014-04-02
uid Your Name your@email.com
ssb 2048R/F1B2F1B2 2014-04-02

Run

gpg --export yyyyyyyy > yourkey.gpg (This will be the public archive key)

At build time copy yourkey.gpg to build-iso/livecd/config.vyatta/chroot_sources/vyatta.chroot.gpg. Otherwise you will be asked this question:

Next put the real gpg key ID in $HOME/public_html/debian-oxnard/update, i.e. change

gpg -u -bao dists/squeeze/Release.gpg dists/squeeze/Release to

gpg -u yyyyyyyy -bao dists/squeeze/Release.gpg dists/squeeze/Release

And $HOME/public_html/vyatta-dev/oxnard/update_dev:

gpg -u -bao dists/oxnard/Release.gpg dists/oxnard/Release to

gpg -u yyyyyyyy -bao dists/oxnard/Release.gpg dists/oxnard/Release

Now, it is time to put your packages into $HOME/public_html/debian-oxnard/pool/main and $HOME/public_html/vyatta-dev/oxnard/pool/main.

After you done this, cd to ~/public_html/debian-oxnard and run

./update

Now repeat with the other repository, cd to ~/public_html/vyatta-dev/oxnard and run

./update_dev

Next, modify build-iso to use the local mirror.
You may want to check-in yourkey.gpg as livecd/config.vyatta/chroot_sources/vyatta.chroot.gpg permanently.
Then search livecd/mk.livecd.function for the 4 lines involving mirror layout. Change them as follows:
line 291:

LB_MIRROR_BINARY=http://192.168.1.253/debian-$branch

line 292:

LB_MIRROR_BOOTSTRAP=http://192.168.1.253/debian-$branch

line 295:

local dev_repo_root=http://192.168.1.253/vyatta-dev

line 314:

if [ ! -f vyatta.chroot ]; then
  echo "deb $dev_repo_root/$branch $branch main" >vyatta.chroot
fi