Next Previous Contents

8. Yummifying your servers: yum-arch

8.1 Non technical explanation.

Basically all yum-arch does is digest the rpms that you have in a directory and create a header file for each rpm. That statement is VERY HIGH LEVEL and is not meant to diminish the importance of the yum-arch program. yum-arch is the cornerstone of yum. Without it you may as well stick with rpm.

8.2 Creating headers for an ftp server.

You will need to run yum-arch as root.

It is important that you create the yum headers in the correct directory. You could, if you wanted to, run yum-arch at the beginning of your repository and create one large header directory of all the rpm headers. You could, but you shouldn't.

The headers directory that you create will have a direct impact on what you will put in your yum.conf file so you will need to think about this carefully. You will have to point yum (via yum.conf) at the directory that has the headers directory in it or yum will not find the headers and not find the packages.

You must make sure that you set out your directory structure clearly and correctly. This just makes thing easier when something goes wrong. It also makes it easier to rsync your local mirror if you replicate the remote servers directory structure.

An example tree in /var/ftp/pub/ could look like this:

        /9/
          /base/
               /athlon
               /i386
               /i586
               /i686
          /freshrpms
          /fedora
          /kde/
              /3.1.2
              /3.1.3
          /rawhide
          /updates/
                  /athlon
                  /i386
                  /i586
                  /i686
So if you ran
root@cartman>yum-arch /var/ftp/pub/9
yum-arch would create a 'header' directory in /var/ftp/pub/9 and place all the header files for the entire tree below /9/ in that directory.

The directory structure would look as follows:

        /9/
          /base/
               /athlon
               /i386
               /i586
               /i686
          /freshrpms
          /fedora
          /headers              # yum-arch creates the headers here.
          /kde/
              /3.1.2
              /3.1.3
          /rawhide
          /updates/
                  /athlon
                  /i386
                  /i586
                  /i686

That would work fine but it is not the cleanest way to use yum-arch. There are also issues with some of the newer functionality in yum like groups when all your headers are dumped in one directory.

So a solution would be to run yum-arch against a distinct sub-repository in the repository tree. An example of this is:

root@cartman>yum-arch /var/ftp/pub/9/base
root@cartman>yum-arch /var/ftp/pub/9/freshrpms
root@cartman>yum-arch /var/ftp/pub/9/fedora
               etc...
After yum-arch has digested all the rpms you will have a header directory in each sub-repository and your directory structure will look like this:
        /9/
          /base/
               /athlon
               /headers
               /i386
               /i586
               /i686
          /freshrpms/
                    /headers
          /fedora/
                 /headers
          /kde/
              /3.1.2
              /3.1.3
              /headers
          /rawhide/
                  /headers
          /updates/
                  /athlon
                  /headers
                  /i386
                  /i586
                  /i686

Now that you have created the headers manually you will need to maintain your yum.conf file to point the client at the headers so you can start using yum.

8.3 Advanced Commands

yum-arch has a number of options that can be used to affect the way that the way that the headers are created.

root@cartman>yum-arch [options] directory

check dependencies and conflicts in tree (-d)

root@cartman>yum-arch -d directory
This command is really useful if you have a large number of rpms in a repository and you just want to create headers for the most recent rpms. yum-arch -d is very verbose, you will get a large amount of information of what yum-arch is doing so it might be a good idea to output this to a file as there can be a large number of conflicts and dependancies.

Here is an example of a conflict:

Checking deps 91 % completewarning: package samba-client = 3.0.0-5rc1 was already added, replacing with samba-client <= 3.0.0-5rc1
And here is an example of a failed dependancy:
depcheck: package xmms-devel needs xmms = 1.2.7
These messages are very useful and point to where problems might be occuring in your repository. Lot of people ignore these messages but then that is their choice. Messages are there for a reason.

Potential solutions for conflicts.

This is pretty simple. Delete the oldest rpms from your repository. This should really be done using rsync (as noted above). rsync, used to replicate a remote mirror locally, can be configured to delete local files that do not exist on the remote server.

Potential solutions for dependancy issues.

This may or may not be simple.

If the dependency problem is caused by the required package just not being available on any of the repositories you are mirroring, you will either have to look for the package at rpmfind or (if you have sources or the data that needs to be packaged) build the rpm yourself. Building rpm's is discussed below.

A word of warning: Any packages that are not on the mirrors may be deleted from your local repository by rsync (if you use --delete to maintain a perfect mirror) so you will need to keep these site-local packages in a "local" repository. You have been warned.

If the dependency problem is more complex (and it may be very complex if you have two or more packages from different repositories whose builders used different naming/numbering conventions, or if you have "legacy", possibly proprietary or closed source packages that require old versions of libraries while the rest of your applications require modern libraries) then you will have to work much harder to resolve them. Indeed, in some cases no resolution will be possible -- there are plenty of sites out there that are running "obsolete" linux distributions because one of their mission-critical applications requires the libc or other libraries intertwined with that particular distribution and all its binaries.

Fortunately, just as yum permits one to easily upgrade (so why would one ever need a complex repository tree?) it permits one to easily maintain a complex repository tree, so the solution in the worst cases may well be to maintain separate repository trees for non-conflicted distributions. Sure, it's ugly, but in the open source world (or for that matter the closed source world) one cannot make writers of important packages comply with a common set of rules, one can only hope that they do and deal with it when they don't.

make output verbose. (-v and -vv)

root@cartman>yum-arch -v directory
This command is very useful if you want to see what yum-arch is doing with every package in your repository. You will need to output this data to a file to be able to use it as there is a large amount of information produced. Here is an example of the verbose output:
ignoring older pkg: redhat/9/i386/openssh-askpass-gnome-3.5p1-6.9.i386.rpm
Digesting rpm - openssh-askpass-3.5p1-6.9.i386.rpm - 95/3202
Already found tuple: openssh-askpass i386

ignoring older pkg: redhat/9/i386/openssh-askpass-3.5p1-6.9.i386.rpm
Digesting rpm - openssh-3.5p1-6.9.i386.rpm - 96/3202
Already found tuple: openssh i386

ignoring older pkg: redhat/9/i386/openssh-3.5p1-6.9.i386.rpm
Digesting rpm - openssl096-0.9.6-17.i386.rpm - 97/3202
Digesting rpm - openssl-0.9.7a-5.i386.rpm - 98/3202
Digesting rpm - pam_smb-1.1.6-9.9.i386.rpm - 99/3202
Digesting rpm - php-4.2.2-17.2.i386.rpm - 100/3202
Digesting rpm - php-devel-4.2.2-17.2.i386.rpm - 101/3202
Digesting rpm - php-imap-4.2.2-17.2.i386.rpm - 102/3202
Digesting rpm - php-ldap-4.2.2-17.2.i386.rpm - 103/3202
Digesting rpm - php-manual-4.2.2-17.2.i386.rpm - 104/3202

How much info do you need? Once you are happy that yum-arch is working correctly and generating the correct headers you do not need the output to be verbose. yum-arch is usually run as a cron job so you are not really aware of the output generated. It is good to know that it is there and has helped me to explain some anomalies in the past.

do not generate headers. (-n)

root@cartman>yum-arch -n directory

When checking dependencies, one may not want to actually generate headers. Perhaps one has just added a new package and wants to see if it conflicts, but doesn't want there to be a window where the headers set is "broken" if there are indeed major conflicts. The -n flag permits the automatic generation of headers to be surpressed.

make output more quiet. (-q)

root@cartman>yum-arch -q directory
Sssshhhhhhh.

The -q flag does reduce the amount of information that yum-arch generates but yum-arch will never run totally silent. Any conflicts or redundant packages will be reported to the screen. Once you are happy that yum-arch is doing what you want it to do (ie. the second time you run it) you hould create a cron job that will run yum-arch daily. The -q flag is the best flag to use as yum-arch creates the headers and only tells you about the warnings.

check pkgs with gpg and md5 checksums (-c) cannot be used with -n (obviously)

root@cartman>yum-arch -c directory

Hmmmmmm. You are just looking for hassles with this command. I can see its uses but not all packages have an MD5 checksum nor do they have a gpg signature. So if you are security conscious and you are not sure about the packages that you have downloaded then this flag is for you.

[THIS FLAG DOES NOT WORK FOR ME -- jp]

compress headers using gzip. (-z)

root@cartman>yum-arch -z directory
....

pay attention to symlinks (Default is to ignore symlinks). (-l)

root@cartman>yum-arch -l directory
.....

...all you really need to do

root@cartman>yum-arch directory

When you use yum-arch to create the headers for your repository, yum-arch will always produce these two lines at the end of the screen report (except -q):

   Total: 3202                  #Total packages read
   Used: 1521                   #Total headers created

That is all you really need to know. What are thee two lines telling me? Basically I need to clean up my repository as more than half of my packages are not having headers created for them. This is normally due to th fact that some of my packages are duplicated in the repository or there are conflicts. This is probably caused by the fact that I have the rawhide packages in my repository. Bleeding edge and all that.


Next Previous Contents