How to remove duplicate packages from a failed yum transaction

Environment

  • CentOS 8 - 7 + Old versions
  • Red Hat Enterprise Linux 8 - 7 + Old RHE

Issue

  • How to remove the older versions of a package which has multiple versions installed on CentOS?
  • After a failed yum transaction, yum-complete-transaction fails to complete and the duplicates are still present.
  • yum fails with <package_name-version1> is a duplicate with <package_name-version2>
  • package-cleanup --cleandupes fails with Error: Depsolving loop limit reached

Resolution

  1. Try to complete the transaction
  2. Manually remove duplicates
  3. Identify remaining rpm corruption

Try to complete the transaction

Note: This option only applies to CentOS 5,6,7

  • Before manually removing the duplicates, confirm if yum-complete-transaction will resolve the issue.
 yum-complete-transaction
  • If this completes successfully, so long as there are no further duplicates, you do not need to run the remaining steps.
  • If there are no duplicates, we can clear this message with the --cleanup-only option
 yum-complete-transaction --cleanup-only

Manually remove duplicates

The first step is to backup the rpm database. The remaining steps create a list of one oft he duplicate package and removes them from the database only. With the --justdb flag we ensure no necessary files are removed until after we update. If there is any concern that the files are incorrect we can verify with rpm -Va .

CentOS 5

  • CentOS 5 does not have yum check so we must use an alternate method.
tar cjf /tmp/rpm_db.tar.bz2 /var/lib/{rpm,yum}
package-cleanup --dupes &> /tmp/duplicate     

# Note: Check the /tmp/duplicate file and remove the line(s) which is not a package name.

awk 'NR%2==0' /tmp/duplicate > /tmp/remove
cat /tmp/remove | egrep -v "\:" > /tmp/remove1
cat /tmp/remove |  egrep ":" | awk -F':' '{ print $NF }' >> /tmp/remove1 
for i in `cat /tmp/remove1`; do rpm -e --justdb --nodeps $i; done
yum update 

CentOS 6,7

  • yum check can take quite a while but the output is necessary for these steps
tar cjf /tmp/rpm_db.tar.bz2 /var/lib/{rpm,yum}
yum check &> /tmp/yumcheck
grep "duplicate" /tmp/yumcheck | awk '{ print $NF }' | egrep -v "\:" > /tmp/duplicaterpms
grep "duplicate" /tmp/yumcheck | awk '{ print $NF }' | egrep ":" | awk -F':' '{ print $NF }' >> /tmp/duplicaterpms
for i in $(cat /tmp/duplicaterpms); do rpm -e --justdb --nodeps $i; done
yum update

CentOS 8

  • For RHEL 8, we want to backup /var/lib/dnf and /etc/dnf*
tar cjf /tmp/rpm_db.tar.bz2 /var/lib/{rpm,dnf} /etc/dnf*
repoquery --duplicated --latest-limit=-1 -q &> /tmp/duplicateolder
for i in `cat /tmp/duplicateolder`; do rpm -ev --justdb --nodeps $i; done
yum update

After removing the Duplicates

  • If a new kernel was part of the transaction that failed, you may have to reinstall that kernel as well to generate the correct grub entries and initrd/initramfs
yum reinstall kernel-<version>-<release>.<arch>

Identify remaining rpm corruption

  • To identify any remaining corruption we can use rpm -Va which verifies all files on the system that are provided by rpms, against the rpms you have installed.
    • If there are duplicates on the system, an rpm -Va will be extremely misleading suggesting that most of the binaries are wrong when in fact it’s comparing 1 binary against 2 packages and telling you that it doesn’t match one of the 2 it checked. By removing the duplicates first, we can then use rpm -Va to identify any remaining rpm vs file mismatch.
  • If you have prelinking enabled, we want to clear that cache first before running rpm -Va or it obscures the output.
prelink -ua; rpm -Va &> /tmp/rpm_Va.out
  • Now ruling out configuration files which we expect to show up in this output, we can identify a list of packages to reinstall.
awk '$2 !~/^[cg]$/ {print $NF}' /tmp/rpm_Va.out | xargs rpm -qf 2>/dev/null | sort -u |& tee /tmp/pkgs.out
  • Reinstall the packages to replace any files that don’t match the rpms installed.
yum reinstall $(cat /tmp/pkgs.out)

Root Cause

When yum is interrupted during an update, it can leave the old versions installed in addition to the new packages it was updating to leaving duplicates. The next time yum is used, the duplicate packages cause confusion in yum’s logic and it can fail with “protected multilib” errors or misleading dependency errors.

Note: There is a tool called package-cleanup which older versions will remove dependencies of duplicates while newer versions simply won’t remove a duplicate that something depends on. Since, after we remove the duplicates, we update and confirm no further issues are present, we can safely remove the packages with the steps provided here.

This issue most commonly occurs when yum is interrupted. A few common reasons are below:

  • If you run yum update over ssh and connectivity is interrupted during the update
  • If you send SIGINT , kill -9 , or issue a CTRL+c to the yum process while it’s running
  • If the system reboots during the update

There are many possible reasons this could occur. If this occurs repetitively, it could be caused by automation software interrupting the updates when it decides it’s taking too long or the command has timed out when often it has not.

How to debug recurring rpm database corruption

Diagnostic Steps

  • Identify any duplicates on the system
package-cleanup --dupes
yum check
  • With CentOS 6 and newer, you can use yum history to look for recent failed yum transactions marked by a *
yum history
  • For reviewing rpm -Va output, you can refer to rpm’s man page.
       The format of the output is a string of 9 characters, a  possible  attribute
       marker:

       c %config configuration file.
       d %doc documentation file.
       g %ghost file (i.e. the file contents are not included in the package payload).
       l %license license file.
       r %readme readme file.

       from  the  package header, followed by the file name.  Each of the 9 charac‐
       ters denotes the result of a comparison of attribute(s) of the file  to  the
       value of those attribute(s) recorded in the database.  A single "." (period)
       means the test passed, while a single "?" (question mark) indicates the test
       could  not  be performed (e.g. file permissions prevent reading). Otherwise,
       the (mnemonically emBoldened) character denotes failure of the corresponding
       --verify test:

       S file Size differs
       M Mode differs (includes permissions and file type)
       5 digest (formerly MD5 sum) differs
       D Device major/minor number mismatch
       L readLink(2) path mismatch
       U User ownership differs
       G Group ownership differs
       T mTime differs
       P caPabilities differ