This makes the obsoletes security filters consistent with upgrade
security filters.
This API is used from check-update and from Info and List commands.
- For check-update we don't want to include resolved advisories to have
identical result to the actual update. That is bz2101421 use case.
- For Info and List commands the --obsoletes switch: "List packages
installed on the system that are obsoleted by packages in any known
repository." Given this specification in makes sense not to
consider resolved advisories when we also use security filters.
There is still a general case when someone uses the API or any potential
future use and I think it is best to have the behavior unified for
"upgrades" and "obsoletes".
= changelog =
msg: Don't include resolved advisories for obsoletes filtering with security filters
type: bugfix
resolves: https://bugzilla.redhat.com/show_bug.cgi?id=2101421
Tests: https://github.com/rpm-software-management/ci-dnf-stack/pull/1134
`installed_query` could be missing packages. If we specify we want to
upgrade a specific nevra that is not yet installed, then `installed_query`
is empty because it is based on user input, but there could be other
versions of the pkg installed.
Eg: if kernel-1 and kernel-3 are installed and we specify we want to
upgrade kernel-2, nothing should be done because we already have higher
version, but now `installed_query` would be empty and kernel-2 would be
installed.
Therefore, we need to use `installed_all`.
https://bugzilla.redhat.com/show_bug.cgi?id=2097757
Without this patch dnf can create the following transaction during dnf upgrade --security when there is an advisory for B-2-2:
```
repo @System 0 testtags <inline>
#>=Pkg: A 1 1 x86_64
#>=Pkg: B 1 1 x86_64
#>=Req: A = 1-1
repo available 0 testtags <inline>
#>=Pkg: A 2 2 x86_64
#>=Pkg: B 2 2 x86_64
#>=Req: A = 2-2
system x86_64 rpm @System
job update oneof A-1-1.x86_64@@System B-2-2.x86_64@available [targeted,setevr,setarch]
result transaction,problems
```
Problem is that without forcebest nothing gets upgraded despite the available advisory and --security switch.
This can also be seen in CI test case: rpm-software-management/ci-dnf-stack#1130
Consider the following call paths (mixed Python and C), extending from
livecd-creator down to libsolv:
main [livecd-tools/tools/livecd-creator]
install() [livecd-tools/imgcreate/creator.py]
fill_sack() [dnf/dnf/base.py]
_add_repo_to_sack() [dnf/dnf/base.py]
load_repo() [libdnf/python/hawkey/sack-py.cpp]
dnf_sack_load_repo() [libdnf/libdnf/dnf-sack.cpp]
write_main() [libdnf/libdnf/dnf-sack.cpp]
repo_add_solv() [libsolv/src/repo_solv.c]
repopagestore_read_or_setup_pages() [libsolv/src/repopage.c]
dup()
write_ext() [libdnf/libdnf/dnf-sack.cpp]
repo_add_solv() [libsolv/src/repo_solv.c]
repopagestore_read_or_setup_pages() [libsolv/src/repopage.c]
dup()
The dup() calls create the following file descriptors (output from
"lsof"):
> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
> python3 6500 root 7r REG 8,1 25320727 395438 /var/tmp/imgcreate-mytcghah/install_root/var/cache/dnf/fedora.solv (deleted)
> python3 6500 root 8r REG 8,1 52531426 395450 /var/tmp/imgcreate-mytcghah/install_root/var/cache/dnf/fedora-filenames.solvx
These file descriptors are *owned* by the DnfSack object (which is derived
from GObject), as follows:
sack->priv->pool->repos[1]->repodata[1]->store.pagefd = 7
sack->priv->pool->repos[1]->repodata[2]->store.pagefd = 8
^ ^ ^ ^ ^ ^ ^
| | | | | | |
| | | | | | int
| | | | | Repopagestore [libsolv/src/repopage.h]
| | | | Repodata [libsolv/src/repodata.h]
| | | struct s_Repo [libsolv/src/repo.h]
| | struct s_Pool (aka Pool) [libsolv/src/pool.h]
| DnfSackPrivate [libdnf/libdnf/dnf-sack.cpp]
DnfSack [libdnf/libdnf/dnf-sack.h]
The file descriptors are *supposed* to be closed on the following call
path:
main [livecd-tools/tools/livecd-creator]
install() [livecd-tools/imgcreate/creator.py]
close() [livecd-tools/imgcreate/dnfinst.py]
close() [dnf/dnf/base.py]
reset() [dnf/dnf/base.py]
_sack = None
_goal = None
_transaction = None
...
dnf_sack_finalize() [libdnf/libdnf/dnf-sack.cpp]
pool_free() [libsolv/src/pool.c]
pool_freeallrepos() [libsolv/src/pool.c]
repo_freedata() [libsolv/src/repo.c]
repodata_freedata() [libsolv/src/repodata.c]
repopagestore_free() [libsolv/src/repopage.c]
close()
Namely, when dnf.Base.reset() [dnf/dnf/base.py] is called with (sack=True,
goal=True), the reference counts of the objects pointed to by the "_sack",
"_goal" and "_transaction" fields are supposed to reach zero, and then, as
part of the DnfSack object's finalization, the libsolv file descriptors
are supposed to be closed.
Now, while this *may* happen immediately in dnf.Base.reset(), it may as
well not. The reason is that there is a multitude of *circular references*
between DnfSack and the packages that it contains. When dnf.Base.reset()
is entered, we have the following picture:
_sack _goal
| |
v v
+----------------+ +-------------+
| DnfSack object | <--- | Goal object |
+----------------+ +-------------+
|^ |^ |^
|| || ||
|| || ||
+--||----||----||---+
| v| v| v| | <-- _transaction
| Pkg1 Pkg2 PkgN |
| |
| Transaction oject |
+-------------------+
That is, the reference count of the DnfSack object is (1 + 1 + N), where N
is the number of packages in the transaction. Details:
(a) The first reference comes from the "_sack" field, established like
this:
main [livecd-tools/tools/livecd-creator]
install() [livecd-tools/imgcreate/creator.py]
fill_sack() [dnf/dnf/base.py]
_build_sack() [dnf/dnf/sack.py]
Sack()
sack_init() [libdnf/python/hawkey/sack-py.cpp]
dnf_sack_new() [libdnf/libdnf/dnf-sack.cpp]
(b) The second reference on the DnfSack object comes from "_goal":
main [livecd-tools/tools/livecd-creator]
install() [livecd-tools/imgcreate/creator.py]
fill_sack() [dnf/dnf/base.py]
_goal = Goal(_sack)
goal_init() [libdnf/python/hawkey/goal-py.cpp]
Py_INCREF(_sack)
(c) Then there is one reference to "_sack" *per package* in the
transaction:
main [livecd-tools/tools/livecd-creator]
install() [livecd-tools/imgcreate/creator.py]
runInstall() [livecd-tools/imgcreate/dnfinst.py]
resolve() [dnf/dnf/base.py]
_goal2transaction() [dnf/dnf/base.py]
list_installs() [libdnf/python/hawkey/goal-py.cpp]
list_generic() [libdnf/python/hawkey/goal-py.cpp]
packagelist_to_pylist() [libdnf/python/hawkey/iutil-py.cpp]
new_package() [libdnf/python/hawkey/sack-py.cpp]
Py_BuildValue()
ts.add_install()
list_installs() creates a list of packages that need to be installed
by DNF. Inside the loop in packagelist_to_pylist(), which constructs
the elements of that list, Py_BuildValue() is called with the "O"
format specifier, and that increases the reference count on "_sack".
Subsequently, in the _goal2transaction() method, we iterate over the
package list created by list_installs(), and add each package to the
transaction (ts.add_install()). After _goal2transaction() returns,
this transaction is assigned to "self._transaction" in resolve(). This
is where the last N (back-)references on the DnfSack object come from.
(d) Now, to quote the defintion of the DnfSack object
("libdnf/docs/hawkey/tutorial-py.rst"):
> *Sack* is an abstraction for a collection of packages.
That's why the DnfSack object references all the Pkg1 through PkgN
packages.
So, when the dnf.Base.reset() method completes, the picture changes like
this:
_sack _goal
| |
-- [CUT] -- -- [CUT] --
| |
v | v
+----------------+ [C] +-------------+
| DnfSack object | <-[U]- | Goal object |
+----------------+ [T] +-------------+
|^ |^ |^ |
|| || ||
|| || || |
+--||----||----||---+ [C]
| v| v| v| | <--[U]-- _transaction
| Pkg1 Pkg2 PkgN | [T]
| | |
| Transaction oject |
+-------------------+
and we are left with N reference cycles (one between each package and the
same DnfSack object).
This set of cycles can only be cleaned up by Python's generational garbage
collector <https://stackify.com/python-garbage-collection/>. The GC will
collect the DnfSack object, and consequently close the libsolv page file
descriptors via dnf_sack_finalize() -- but garbage collection will happen
*only eventually*, unpredictably.
This means that the dnf.Base.reset() method breaks its interface contract:
> Make the Base object forget about various things.
because the libsolv file descriptors can (and frequently do, in practice)
survive dnf.Base.reset().
In general, as long as the garbage collector only tracks process-private
memory blocks, there's nothing wrong; however, file descriptors are
visible to the kernel. When dnf.Base.reset() *temporarily* leaks file
descriptors as explained above, then immediately subsequent operations
that depend on those file descriptors having been closed, can fail.
An example is livecd-creator's unmounting of:
/var/tmp/imgcreate-mytcghah/install_root/var/cache/dnf
which the kernel refuses, due to libsolv's still open file descriptors
pointing into that filesystem:
> umount: /var/tmp/imgcreate-mytcghah/install_root/var/cache/dnf: target
> is busy.
> Unable to unmount /var/tmp/imgcreate-mytcghah/install_root/var/cache/dnf
> normally, using lazy unmount
(Unfortunately, the whole lazy umount idea is misguided in livecd-tools;
it's a misfeature that should be removed, as it permits the corruption of
the loop-backed filesystem. Now that the real bug is being fixed in DNF,
lazy umount is not needed as a (broken) workaround in livecd-tools. But
that's a separate patch for livecd-tools:
<https://github.com/livecd-tools/livecd-tools/pull/227>.)
Plug the fd leak by forcing a garbage collection in dnf.Base.reset()
whenever we cut the "_sack", "_goal" and "_transaction" links -- that is,
when the "sack" and "goal" parameters are True.
Note that precisely due to the unpredictable behavior of the garbage
collector, reproducing the bug may prove elusive. In order to reproduce it
deterministically, through usage with livecd-creator, disabling automatic
garbage collection with the following patch (for livecd-tools) is
sufficient:
> diff --git a/tools/livecd-creator b/tools/livecd-creator
> index 291de10cbbf9..8d2c740c238b 100755
> --- a/tools/livecd-creator
> +++ b/tools/livecd-creator
> @@ -31,6 +31,8 @@ from dnf.exceptions import Error as DnfBaseError
> import imgcreate
> from imgcreate.errors import KickstartError
>
> +import gc
> +
> class Usage(Exception):
> def __init__(self, msg = None, no_error = False):
> Exception.__init__(self, msg, no_error)
> @@ -261,5 +263,6 @@ def do_nss_libs_hack():
> return hack
>
> if __name__ == "__main__":
> + gc.disable()
> hack = do_nss_libs_hack()
> sys.exit(main())
Also note that you need to use livecd-tools at git commit 4afde9352e82 or
later, for this fix to make any difference: said commit fixes a different
(independent) bug in livecd-tools that produces identical symptoms, but
from a different origin. In other words, if you don't have commit
4afde9352e82 in your livecd-tools install, then said bug in livecd-tools
will mask this DNF fix.
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Users with different than english locale are not able to update their
systems in case that some of updates are already downloaded in the dnf
cache (e.g. using dnf-automatic).
The error string is taken from librepo target where it is stored
untranslated. Therefore we need to compare untranslated versions of the
string.
= changelog =
msg: Fix download errors handling in non-english locales
type: bugfix
resolves: https://bugzilla.redhat.com/show_bug.cgi?id=2024527
See https://bugzilla.redhat.com/show_bug.cgi?id=2057340
and https://github.com/benjaminp/six/issues/359
dnf should never import Python modules from /usr/bin but users can
have files in there that look like Python modules and Python will
try to import them and fail.
Consider a tool that is *not* written in Python and is called "copy.pyc".
Naturally, it resides in /usr/bin/copy.pyc and dnf fails:
Traceback (most recent call last):
File "/usr/bin/dnf", line 57, in <module>
from dnf.cli import main
File "/usr/lib/python3.10/site-packages/dnf/__init__.py", line 30, in <module>
import dnf.base
File "/usr/lib/python3.10/site-packages/dnf/base.py", line 31, in <module>
from copy import deepcopy
ImportError: bad magic number in 'copy': b'...'
Similarly, a tool actually written in Python, called "copy.py"
might as well own /usr/bin/copy.py and dnf fails as well:
Traceback (most recent call last):
File "/usr/bin/dnf", line 57, in <module>
from dnf.cli import main
File "/usr/lib/python3.10/site-packages/dnf/__init__.py", line 30, in <module>
import dnf.base
File "/usr/lib/python3.10/site-packages/dnf/base.py", line 31, in <module>
from copy import deepcopy
ImportError: cannot import name 'deepcopy' from 'copy' (/usr/bin/copy.py)
Either problem can happen for a variety of names.
We better not let that happen.
A more general solution that would prevent Python doing this entirely
does not exists yet, see https://discuss.python.org/t/4235
Hence, proposing this to dnf, which is a critical piece of the system.
Fixes permission denied issue when dnf command is run by an unprivileged
user with `-C` option (i.e. using root cache) and uncompressed
groups.xml has not yet been generated by root:
$ dnf -C groups list
[Errno 13] Permission denied: '/var/cache/dnf/fedora-37d3cae0527b6391/repodata/gen'
Now the uncompressed file is generated in a temporary location in such
case.
= changelog =
msg: Fix decompression of groups.xml
type: bugfix
resolves: https://bugzilla.redhat.com/show_bug.cgi?id=2030255
DNF was using private method `hawkey.Sack._rpmdb_version()` from libdnf.
The method computes SHA1 hash from sorted list of hashes stored in
the headers of the instaled packages. And it adds prefix of the number
of installed packages to the computed hash. The result was stored
to the history database and used to detect changes in the rpm database.
The patch uses new oficial librpm API function
`rpm.TransactionSet.dbCookie()`. This is a cleaner solution.
It is also a step to remove the `._rpmdb_version()` method from libdnf.
It is an attempt to remove SHA1 calculations from libdnf.
Troubleshooting FIPS compatibility.
= changelog =
msg: Use rpm.TransactionSet.dbCookie() to determining if rpmdb has changed
type: bugfix
resolves: https://bugzilla.redhat.com/show_bug.cgi?id=2043476
This command is currently a bit confusing to use because it requires
either a full file path, or else a glob. Simply searching for a
filename alone does not work unless you add a glob, but this requirement
is implicit and never explicitly mentioned in the command's help text.
This commit makes that explicit by mentioning it in the command usage,
and also by adding a hint when no matches are returned.
Resolves https://bugzilla.redhat.com/show_bug.cgi?id=1963704
On ELN, the %{fedora} macro is not present. Instead, it has a
%{rhel} macro equal to the next major release of RHEL.
Signed-off-by: Stephen Gallagher <sgallagh@redhat.com>
The rpm-plugin-systemd-inhibit can only work on a system with systemd.
At the time of adding the dependency, rich dependencies were not yet
in Fedora, so adding it now.
= changelog =
msg: Recommend rpm-plugin-systemd-inhibit only if systemd
type: bugfix
resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1947924
The python3-unbound is used in dnssec.py for gpgkey_dns_verification,
which is an optional feature disabled by default and it depends on
DNSSEC that is not used by redhat.com. Therefore, we don't want to have
this dependency by default on RHEL.
= changelog =
msg: Don't recommend python3-unbound on RHEL
type: bugfix
resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1947924
In case the install spec refers to a local rpm file the swap command
would fail with this error:
Error: Cannot add local packages, because transaction job already exists
Changing the order in which the installation and removal parts are
performed fixes the issue.
= changelog =
msg: Fix swap command to work with local rpm files correctly
type: bugfix
resolves: https://bugzilla.redhat.com/show_bug.cgi?id=2036434
These options are currently rather confusing because they do not make it
clear that they operate only temporarily on the currently-used dnf
command. This commit clarifies them by mentioning that.
Resolves https://bugzilla.redhat.com/show_bug.cgi?id=2031414
`Repo` object has always been constructed with default synchronization
strategy. The configuration option `cacheonly` was ignored. DNF
application set synchronization strategy later in the `Cli` object
during processing demands.
The fix takes into account the `cacheonly` option during the construction
of the `Repo` object. Synchronization strategy may still be overriden
during demand processing.
Jan Kolarik