folkslinux
/
mamba
mirror of https://github.com/mamba-org/mamba.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

MRepo refactor (#3118) 

* Rename Enum members in MRepo

* use solv::Pool in MRepo

* Free MRepo add_pip_as_python_dependency

* Free MRepo repodata reading

* Free MRepo read_solv

* Free MRepo write_solv

* Remove MRepo constructor

* Remove MPool capture from MRepo

* Add missing pip dependency

* Refactor solv file serialization

* No lock on non-existant files

* Cleanup dead code

* write_solve create-directories

* Clean MRepo member functions

* Decouple MRepo and Pool

* Finetune Clang-format

* Add filename location to error

* Do not fail on solv file write

* Load new pool loader for repodata and .solv files

* Remove repo name in favor of URL

* Refactor MRepo index loading

* Fix SubDirData.writable_solv_cache

* Fix solv file loading

* Lock repodata.json while reading metadata

* Refactor expected in load_subdir_in_pool

* use expected in solver helpers

* Refactor helpers functions and error messages in Repo

* Add MPool::create_repo_from_packages

* Use load_installed_packages_in_pool

* Remove MRepo ctor

* Fix rebase

* Refactor MRepo::priority

* Update Taskfile realpath to work on MacOS

* Fix virtual packages being removed

* Move MRepo::priority to MPool

* Rename MRepo > solver::libsolv::RepoInfo

* Move RepoInfo parameters for its own library

* Move make_package_info to helpers

* Add libmambapy.solver.libsolv

* Add solver::libsolv::Priorities op

* Bind libsolv parameters

* Bind RepodataOrigin

* Merge libsolv/serialization and libsolv/parameters

* Bind new MPool methods

* Add RepoInfo op

* Add RepoInfo changes to log

* Rename attribute

* Remove RepoInfo::repo
This commit is contained in:
Antoine Prouvost 2024-01-17 16:44:47 +01:00 committed by GitHub
parent cc95098a57
commit 7d6856e9b5
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
40 changed files with 1901 additions and 1130 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
.clang-format
View File

@ -15,6 +15,7 @@ AllowShortLoopsOnASingleLine: 'false'
AlwaysBreakTemplateDeclarations: 'true'
BinPackArguments: false
BinPackParameters: false
BreakAfterAttributes: Leave
BreakBeforeBinaryOperators: All
BreakBeforeBraces: Allman
BreakBeforeTernaryOperators: 'true'

6
Taskfile.dist.yml
View File

@ -11,11 +11,9 @@ vars:
CMAKE_PRESET: 'mamba-unix-shared-debug-dev'
CACHE_DIR: '{{.BUILD_DIR}}/pkgs'
DOCS_DIR: '{{.BUILD_DIR}}/docs'
DOCS_DOXYGEN_XML_DIR:
sh: realpath -m '{{.DOCS_DIR}}/doxygen-xml'
DOCS_DOXYGEN_XML_DIR: '{{.PWD}}/{{.DOCS_DIR}}/doxygen-xml'
MAMBA_NAME: 'mamba' # Depend on preset...
TEST_MAMBA_EXE:
sh: 'realpath -m {{.CMAKE_BUILD_DIR}}/micromamba/{{.MAMBA_NAME}}'
TEST_MAMBA_EXE: '{{.PWD}}/{{.CMAKE_BUILD_DIR}}/micromamba/{{.MAMBA_NAME}}'
CPU_PERCENTAGE: 75
CPU_TOTAL:
sh: >-

13
docs/source/developer_zone/changes-2.0.rst
View File

@ -71,9 +71,17 @@ Changes inlcude:
- ``PackageInfo`` has been moved to ``libmambapy.specs``.
Some attributes have been given a more explicit name ``fn`` > ``filename``,
``url`` > ``package_url``.
- ``Repo`` has been redesigned into a lightweight ``RepoInfo`` and moved to
``libmambapy.solver.libsolv``.
The creation and modification of repos happens through the ``Pool``, with methods such as
``Pool.add_repo_from_repodata_json`` and ``Pool.add_repo_from_packages``, but also high-level
free functions such as ``load_subdir_in_pool`` and ``load_installed_packages_in_pool``.
.. TODO include final decision for Channels as URLs.
For many changes, an exception throwing placeholder has ben kept to advise developpers on the new
direction to take.
Libmamba (C++)
**************
The C++ library ``libmamba`` has received significant changes.
@ -89,6 +97,11 @@ The main changes are:
- A cleanup of ``ChannelContext`` for be a light proxy and parameter holder wrapping the
``specs::Channel``.
- A new ``repodata.json`` parser using `simdjson <https://simdjson.org/>`_.
- [WIP] Creation of the ``solver::libsolv`` sub-namespace for full isolation of libsolv, and a
solver API without ``Context``.
It currently contains:
- A refactoring and thinning of ``MRepo`` as a new ``RepoInfo``.
- Improved downloaders.
.. TODO OCI registry

25
libmamba/CMakeLists.txt
View File

@ -151,6 +151,7 @@ set(
# Filesystem library
${LIBMAMBA_SOURCE_DIR}/fs/filesystem.cpp
# C++ utility library
${LIBMAMBA_SOURCE_DIR}/util/cfile.cpp
${LIBMAMBA_SOURCE_DIR}/util/cryptography.cpp
${LIBMAMBA_SOURCE_DIR}/util/encoding.cpp
${LIBMAMBA_SOURCE_DIR}/util/environment.cpp
@ -161,13 +162,6 @@ set(
${LIBMAMBA_SOURCE_DIR}/util/string.cpp
${LIBMAMBA_SOURCE_DIR}/util/url.cpp
${LIBMAMBA_SOURCE_DIR}/util/url_manip.cpp
# C++ wrapping of libsolv
${LIBMAMBA_SOURCE_DIR}/solv-cpp/pool.cpp
${LIBMAMBA_SOURCE_DIR}/solv-cpp/queue.cpp
${LIBMAMBA_SOURCE_DIR}/solv-cpp/repo.cpp
${LIBMAMBA_SOURCE_DIR}/solv-cpp/solvable.cpp
${LIBMAMBA_SOURCE_DIR}/solv-cpp/solver.cpp
${LIBMAMBA_SOURCE_DIR}/solv-cpp/transaction.cpp
# Implementation of version and matching specs
${LIBMAMBA_SOURCE_DIR}/specs/archive.cpp
${LIBMAMBA_SOURCE_DIR}/specs/authentication_info.cpp
@ -182,6 +176,17 @@ set(
${LIBMAMBA_SOURCE_DIR}/specs/unresolved_channel.cpp
${LIBMAMBA_SOURCE_DIR}/specs/version.cpp
${LIBMAMBA_SOURCE_DIR}/specs/version_spec.cpp
# C++ wrapping of libsolv
${LIBMAMBA_SOURCE_DIR}/solv-cpp/pool.cpp
${LIBMAMBA_SOURCE_DIR}/solv-cpp/queue.cpp
${LIBMAMBA_SOURCE_DIR}/solv-cpp/repo.cpp
${LIBMAMBA_SOURCE_DIR}/solv-cpp/solvable.cpp
${LIBMAMBA_SOURCE_DIR}/solv-cpp/solver.cpp
${LIBMAMBA_SOURCE_DIR}/solv-cpp/transaction.cpp
# Solver interface
${LIBMAMBA_SOURCE_DIR}/solver/libsolv/helpers.cpp
${LIBMAMBA_SOURCE_DIR}/solver/libsolv/parameters.cpp
${LIBMAMBA_SOURCE_DIR}/solver/libsolv/repo_info.cpp
# Artifacts validation
${LIBMAMBA_SOURCE_DIR}/validation/tools.cpp
${LIBMAMBA_SOURCE_DIR}/validation/errors.cpp
@ -220,7 +225,6 @@ set(
${LIBMAMBA_SOURCE_DIR}/core/package_fetcher.cpp
${LIBMAMBA_SOURCE_DIR}/core/package_paths.cpp
${LIBMAMBA_SOURCE_DIR}/core/query.cpp
${LIBMAMBA_SOURCE_DIR}/core/repo.cpp
${LIBMAMBA_SOURCE_DIR}/core/run.cpp
${LIBMAMBA_SOURCE_DIR}/core/shell_init.cpp
${LIBMAMBA_SOURCE_DIR}/core/solver.cpp
@ -262,6 +266,7 @@ set(
# Utility library
${LIBMAMBA_INCLUDE_DIR}/mamba/util/build.hpp
${LIBMAMBA_INCLUDE_DIR}/mamba/util/cast.hpp
${LIBMAMBA_INCLUDE_DIR}/mamba/util/cfile.hpp
${LIBMAMBA_INCLUDE_DIR}/mamba/util/compare.hpp
${LIBMAMBA_INCLUDE_DIR}/mamba/util/conditional.hpp
${LIBMAMBA_INCLUDE_DIR}/mamba/util/cryptography.hpp
@ -300,6 +305,9 @@ set(
${LIBMAMBA_INCLUDE_DIR}/mamba/specs/unresolved_channel.hpp
${LIBMAMBA_INCLUDE_DIR}/mamba/specs/version.hpp
${LIBMAMBA_INCLUDE_DIR}/mamba/specs/version_spec.hpp
# Solver Interface
${LIBMAMBA_INCLUDE_DIR}/mamba/solver/libsolv/parameters.hpp
${LIBMAMBA_INCLUDE_DIR}/mamba/solver/libsolv/repo_info.hpp
# Artifacts validation
${LIBMAMBA_INCLUDE_DIR}/mamba/validation/tools.hpp
${LIBMAMBA_INCLUDE_DIR}/mamba/validation/errors.hpp
@ -332,7 +340,6 @@ set(
${LIBMAMBA_INCLUDE_DIR}/mamba/core/progress_bar.hpp
${LIBMAMBA_INCLUDE_DIR}/mamba/core/pinning.hpp
${LIBMAMBA_INCLUDE_DIR}/mamba/core/query.hpp
${LIBMAMBA_INCLUDE_DIR}/mamba/core/repo.hpp
${LIBMAMBA_INCLUDE_DIR}/mamba/core/run.hpp
${LIBMAMBA_INCLUDE_DIR}/mamba/core/shell_init.hpp
${LIBMAMBA_INCLUDE_DIR}/mamba/core/solution.hpp

2
libmamba/include/mamba/api/install.hpp
View File

@ -16,8 +16,8 @@
#include "mamba/core/package_cache.hpp"
#include "mamba/core/pool.hpp"
#include "mamba/core/repo.hpp"
#include "mamba/fs/filesystem.hpp"
#include "mamba/solver/libsolv/repo_info.hpp"
#include "mamba/specs/match_spec.hpp"
#include "mamba/specs/package_info.hpp"

99
libmamba/include/mamba/core/pool.hpp
View File

@ -12,12 +12,22 @@
#include <solv/pooltypes.h>
#include "mamba/core/error_handling.hpp"
#include "mamba/solver/libsolv/parameters.hpp"
#include "mamba/solver/libsolv/repo_info.hpp"
#include "mamba/specs/package_info.hpp"
namespace mamba
{
class ChannelContext;
class Context;
class PrefixData;
class MSubdirData;
namespace fs
{
class u8path;
}
namespace solv
{
@ -61,6 +71,45 @@ namespace mamba
solv::ObjPool& pool();
const solv::ObjPool& pool() const;
auto add_repo_from_repodata_json(
const fs::u8path& path,
std::string_view url,
solver::libsolv::PipAsPythonDependency add = solver::libsolv::PipAsPythonDependency::No,
solver::libsolv::RepodataParser parser = solver::libsolv::RepodataParser::Mamba
) -> expected_t<solver::libsolv::RepoInfo>;
auto add_repo_from_native_serialization(
const fs::u8path& path,
const solver::libsolv::RepodataOrigin& expected,
solver::libsolv::PipAsPythonDependency add = solver::libsolv::PipAsPythonDependency::No
) -> expected_t<solver::libsolv::RepoInfo>;
template <typename Iter>
auto add_repo_from_packages(
Iter first_package,
Iter last_package,
std::string_view name = "",
solver::libsolv::PipAsPythonDependency add = solver::libsolv::PipAsPythonDependency::No
) -> solver::libsolv::RepoInfo;
template <typename Range>
auto add_repo_from_packages(
const Range& packages,
std::string_view name = "",
solver::libsolv::PipAsPythonDependency add = solver::libsolv::PipAsPythonDependency::No
) -> solver::libsolv::RepoInfo;
auto native_serialize_repo(
const solver::libsolv::RepoInfo& repo,
const fs::u8path& path,
const solver::libsolv::RepodataOrigin& metadata
) -> expected_t<solver::libsolv::RepoInfo>;
void set_installed_repo(const solver::libsolv::RepoInfo& repo);
void
set_repo_priority(const solver::libsolv::RepoInfo& repo, solver::libsolv::Priorities priorities);
void remove_repo(::Id repo_id, bool reuse_ids);
ChannelContext& channel_context() const;
@ -84,7 +133,55 @@ namespace mamba
* - Facilitate (potential) future investigation of parallel solves.
*/
std::shared_ptr<MPoolData> m_data;
auto add_repo_from_packages_impl_pre(std::string_view name) -> solver::libsolv::RepoInfo;
void add_repo_from_packages_impl_loop(
const solver::libsolv::RepoInfo& repo,
const specs::PackageInfo& pkg
);
void add_repo_from_packages_impl_post(
const solver::libsolv::RepoInfo& repo,
solver::libsolv::PipAsPythonDependency add
);
};
} // namespace mamba
// TODO machinery functions in separate files
auto load_subdir_in_pool(const Context& ctx, MPool& pool, const MSubdirData& subdir)
-> expected_t<solver::libsolv::RepoInfo>;
auto load_installed_packages_in_pool(const Context& ctx, MPool& pool, const PrefixData& prefix)
-> solver::libsolv::RepoInfo;
/********************
* Implementation *
********************/
template <typename Iter>
auto MPool::add_repo_from_packages(
Iter first_package,
Iter last_package,
std::string_view name,
solver::libsolv::PipAsPythonDependency add
) -> solver::libsolv::RepoInfo
{
auto repo = add_repo_from_packages_impl_pre(name);
for (; first_package != last_package; ++first_package)
{
add_repo_from_packages_impl_loop(repo, *first_package);
}
add_repo_from_packages_impl_post(repo, add);
return repo;
}
template <typename Range>
auto MPool::add_repo_from_packages(
const Range& packages,
std::string_view name,
solver::libsolv::PipAsPythonDependency add
) -> solver::libsolv::RepoInfo
{
return add_repo_from_packages(packages.begin(), packages.end(), name, add);
}
}
#endif // MAMBA_POOL_HPP

138
libmamba/include/mamba/core/repo.hpp
View File

@ -1,138 +0,0 @@
// Copyright (c) 2019, QuantStack and Mamba Contributors
//
// Distributed under the terms of the BSD 3-Clause License.
//
// The full license is in the file LICENSE, distributed with this software.
#ifndef MAMBA_CORE_REPO_HPP
#define MAMBA_CORE_REPO_HPP
#include <string>
#include <string_view>
#include <tuple>
#include <vector>
#include <nlohmann/json_fwd.hpp>
#include <solv/pooltypes.h>
#include "mamba/core/pool.hpp"
extern "C"
{
typedef struct s_Repo Repo;
typedef struct s_Repodata Repodata;
}
namespace mamba
{
namespace fs
{
class u8path;
}
namespace specs
{
class PackageInfo;
}
class PrefixData;
/**
* Represents a channel subdirectory
* index.
*/
struct RepoMetadata
{
std::string url = {};
std::string etag = {};
std::string mod = {};
bool pip_added = false;
};
auto operator==(const RepoMetadata& lhs, const RepoMetadata& rhs) -> bool;
auto operator!=(const RepoMetadata& lhs, const RepoMetadata& rhs) -> bool;
void to_json(nlohmann::json& j, const RepoMetadata& m);
void from_json(const nlohmann::json& j, RepoMetadata& p);
/**
* A wrapper class of libsolv Repo.
* Represents a channel subdirectory and
* is built using a ready-to-use index/metadata
* file (see ``MSubdirData``).
*/
class MRepo
{
public:
enum class RepodataParser
{
automatic,
mamba,
libsolv,
};
enum class LibsolvCache
{
yes,
no,
};
MRepo(
MPool& pool,
const std::string& name,
const fs::u8path& filename,
const RepoMetadata& meta,
RepodataParser parser = RepodataParser::automatic,
LibsolvCache use_cache = LibsolvCache::yes
);
MRepo(MPool& pool, const PrefixData& prefix_data);
MRepo(MPool& pool, const std::string& name, const std::vector<specs::PackageInfo>& uris);
MRepo(const MRepo&) = delete;
MRepo(MRepo&&) = default;
MRepo& operator=(const MRepo&) = delete;
MRepo& operator=(MRepo&&) = default;
void set_installed();
void set_priority(int priority, int subpriority);
Id id() const;
Repo* repo() const;
struct [[deprecated]] PyExtraPkgInfo
{
std::string noarch;
std::string repo_url;
};
[[deprecated]] auto py_name() const -> std::string_view;
[[deprecated]] auto py_priority() const -> std::tuple<int, int>;
[[deprecated]] auto py_clear(bool reuse_ids) -> bool;
[[deprecated]] auto py_size() const -> std::size_t;
[[deprecated]] void
py_add_extra_pkg_info(const std::map<std::string, PyExtraPkgInfo>& additional_info);
private:
auto name() const -> std::string_view;
void add_pip_as_python_dependency();
void clear(bool reuse_ids = true);
void load_file(const fs::u8path& filename, RepodataParser parser, LibsolvCache use_cache);
void libsolv_read_json(const fs::u8path& filename);
void mamba_read_json(const fs::u8path& filename);
bool read_solv(const fs::u8path& filename);
void write_solv(fs::u8path path);
MPool m_pool;
RepoMetadata m_metadata = {};
Repo* m_repo = nullptr; // This is a view managed by libsolv pool
};
} // namespace mamba
#endif // MAMBA_REPO_HPP

11
libmamba/include/mamba/core/subdirdata.hpp
View File

@ -15,7 +15,6 @@
#include "mamba/core/download.hpp"
#include "mamba/core/error_handling.hpp"
#include "mamba/core/package_cache.hpp"
#include "mamba/core/repo.hpp"
#include "mamba/core/util.hpp"
#include "mamba/fs/filesystem.hpp"
@ -124,6 +123,14 @@ namespace mamba
void clear_cache();
const std::string& name() const;
const MSubdirMetadata& metadata() const;
expected_t<fs::u8path> valid_solv_cache() const;
fs::u8path writable_solv_cache() const;
expected_t<fs::u8path> valid_json_cache() const;
[[deprecated("since version 2.0 use ``valid_solv_cache`` or ``valid_json_cache`` instead")]]
expected_t<std::string> cache_path() const;
static expected_t<void> download_indexes(
@ -133,8 +140,6 @@ namespace mamba
DownloadMonitor* download_monitor = nullptr
);
expected_t<MRepo> create_repo(MPool& pool) const;
private:
MSubdirData(

58
libmamba/include/mamba/solver/libsolv/parameters.hpp Normal file
View File

@ -0,0 +1,58 @@
// Copyright (c) 2023, QuantStack and Mamba Contributors
//
// Distributed under the terms of the BSD 3-Clause License.
//
// The full license is in the file LICENSE, distributed with this software.
#ifndef MAMBA_SOLVER_LIBSOLV_PARAMETERS_HPP
#define MAMBA_SOLVER_LIBSOLV_PARAMETERS_HPP
#include <string>
#include <nlohmann/json_fwd.hpp>
namespace mamba::solver::libsolv
{
enum class RepodataParser
{
Mamba,
Libsolv,
};
enum class PipAsPythonDependency : bool
{
No = false,
Yes = true,
};
struct Priorities
{
using value_type = int;
value_type priority = 0;
value_type subpriority = 0;
};
[[nodiscard]] auto operator==(const Priorities& lhs, const Priorities& rhs) -> bool;
[[nodiscard]] auto operator!=(const Priorities& lhs, const Priorities& rhs) -> bool;
/**
* Metadata serialized with a repository index.
*
* This is used to identify if the binary serialization is out of date with the expected
* index.
*/
struct RepodataOrigin
{
std::string url = {};
std::string etag = {};
std::string mod = {};
};
auto operator==(const RepodataOrigin& lhs, const RepodataOrigin& rhs) -> bool;
auto operator!=(const RepodataOrigin& lhs, const RepodataOrigin& rhs) -> bool;
void to_json(nlohmann::json& j, const RepodataOrigin& m);
void from_json(const nlohmann::json& j, RepodataOrigin& p);
}
#endif

72
libmamba/include/mamba/solver/libsolv/repo_info.hpp Normal file
View File

@ -0,0 +1,72 @@
// Copyright (c) 2019, QuantStack and Mamba Contributors
//
// Distributed under the terms of the BSD 3-Clause License.
//
// The full license is in the file LICENSE, distributed with this software.
#ifndef MAMBA_SOLVER_LIBSOLV_REPO_INFO_HPP
#define MAMBA_SOLVER_LIBSOLV_REPO_INFO_HPP
#include <string_view>
#include "mamba/solver/libsolv/parameters.hpp"
extern "C"
{
using Repo = struct s_Repo;
}
namespace mamba
{
class MPool;
class MTransaction;
}
namespace mamba::solver::libsolv
{
/**
* A libsolv repository descriptor.
*
* In libsolv, most of the data is help in the Pool, and repo are tightly coupled with them.
* This repository class is a lightwight description of a repository returned when creating
* a new repository in the Pool.
* Some modifications to the repo are possible throught the Pool.
* @see MPool::add_repo_from_repodata_json
* @see MPool::add_repo_from_packages
* @see MPool::remove_repo
*/
class RepoInfo
{
public:
using RepoId = int;
RepoInfo(const RepoInfo&) = default;
RepoInfo(RepoInfo&&) = default;
auto operator=(const RepoInfo&) -> RepoInfo& = default;
auto operator=(RepoInfo&&) -> RepoInfo& = default;
[[nodiscard]] auto id() const -> RepoId;
[[nodiscard]] auto name() const -> std::string_view;
[[nodiscard]] auto package_count() const -> std::size_t;
[[nodiscard]] auto priority() const -> Priorities;
private:
::Repo* m_ptr = nullptr; // This is a view managed by libsolv pool
explicit RepoInfo(::Repo* repo);
friend class ::mamba::MPool;
friend class ::mamba::MTransaction; // As long as MTransaction leaks libsolv impl
friend auto operator==(RepoInfo lhs, RepoInfo rhs) -> bool;
};
auto operator==(RepoInfo lhs, RepoInfo rhs) -> bool;
auto operator!=(RepoInfo lhs, RepoInfo rhs) -> bool;
}
#endif

70
libmamba/include/mamba/util/cfile.hpp Normal file
View File

@ -0,0 +1,70 @@
// Copyright (c) 2023, QuantStack and Mamba Contributors
//
// Distributed under the terms of the BSD 3-Clause License.
//
// The full license is in the file LICENSE, distributed with this software.
#ifndef MAMBA_UTIL_CFILE_HPP
#define MAMBA_UTIL_CFILE_HPP
#include <cstdio>
#include <memory>
#include <system_error>
#include <tl/expected.hpp>
#include "mamba/fs/filesystem.hpp"
namespace mamba::util
{
class CFile
{
public:
/**
* Open a file with C API.
*
* In case of error, set the error code @p ec.
*
* @param path must hae filesystem default encoding.
*/
static auto try_open( //
const fs::u8path& path,
const char* mode,
std::error_code& ec
) -> CFile;
static auto try_open( //
const fs::u8path& path,
const char* mode
) -> tl::expected<CFile, std::error_code>;
CFile(CFile&&) = default;
auto operator=(CFile&&) -> CFile& = default;
/**
* The destructor will flush and close the file descriptor.
*
* Like ``std::fstream``, exceptions are ignored.
* Explicitly call @ref close to get the exception.
*/
~CFile();
void try_close(std::error_code& ec) noexcept;
[[nodiscard]] auto try_close() noexcept -> tl::expected<void, std::error_code>;
auto raw() noexcept -> std::FILE*;
private:
struct FileClose
{
void operator()(std::FILE* ptr);
};
std::unique_ptr<std::FILE, FileClose> m_ptr = nullptr;
explicit CFile(std::FILE* ptr);
};
}
#endif

69
libmamba/src/api/channel_loader.cpp
View File

@ -6,19 +6,19 @@
#include "mamba/api/channel_loader.hpp"
#include "mamba/core/channel_context.hpp"
#include "mamba/core/download.hpp"
#include "mamba/core/download_progress_bar.hpp"
#include "mamba/core/output.hpp"
#include "mamba/core/pool.hpp"
#include "mamba/core/prefix_data.hpp"
#include "mamba/core/repo.hpp"
#include "mamba/core/subdirdata.hpp"
#include "mamba/util/string.hpp"
#include "mamba/solver/libsolv/repo_info.hpp"
namespace mamba
{
namespace
{
MRepo create_repo_from_pkgs_dir(MPool& pool, const fs::u8path& pkgs_dir)
solver::libsolv::RepoInfo
create_repo_from_pkgs_dir(const Context& ctx, MPool& pool, const fs::u8path& pkgs_dir)
{
if (!fs::exists(pkgs_dir))
{
@ -40,7 +40,7 @@ namespace mamba
}
prefix_data.load_single_record(repodata_record_json);
}
return MRepo(pool, prefix_data);
return load_installed_packages_in_pool(ctx, pool, prefix_data);
}
void create_subdirs(
@ -50,7 +50,7 @@ namespace mamba
MultiPackageCache& package_caches,
std::vector<MSubdirData>& subdirs,
std::vector<mamba_error>& error_list,
std::vector<std::pair<int, int>>& priorities,
std::vector<solver::libsolv::Priorities>& priorities,
int& max_prio,
specs::CondaURL& prev_channel_url
)
@ -75,7 +75,7 @@ namespace mamba
subdirs.push_back(std::move(sdir));
if (ctx.channel_priority == ChannelPriority::Disabled)
{
priorities.push_back(std::make_pair(0, 0));
priorities.push_back({ /* .priority= */ 0, /* .subpriority= */ 0 });
}
else
{
@ -85,7 +85,7 @@ namespace mamba
max_prio--;
prev_channel_url = channel.url();
}
priorities.push_back(std::make_pair(max_prio, 0));
priorities.push_back({ /* .priority= */ max_prio, /* .subpriority= */ 0 });
}
}
}
@ -98,7 +98,7 @@ namespace mamba
std::vector<MSubdirData> subdirs;
std::vector<std::pair<int, int>> priorities;
std::vector<solver::libsolv::Priorities> priorities;
int max_prio = static_cast<int>(ctx.channels.size());
auto prev_channel_url = specs::CondaURL();
@ -174,7 +174,7 @@ namespace mamba
LOG_INFO << "Creating repo from pkgs_dir for offline";
for (const auto& c : ctx.pkgs_dirs)
{
create_repo_from_pkgs_dir(pool, c);
create_repo_from_pkgs_dir(ctx, pool, c);
}
}
std::string prev_channel;
@ -194,29 +194,31 @@ namespace mamba
continue;
}
auto repo = subdir.create_repo(pool);
if (repo)
{
auto& prio = priorities[i];
repo.value().set_priority(prio.first, prio.second);
}
else
{
if (is_retry & RETRY_SUBDIR_FETCH)
{
std::stringstream ss;
ss << "Could not load repodata.json for " << subdir.name() << " after retry."
<< "Please check repodata source. Exiting." << std::endl;
error_list.push_back(mamba_error(ss.str(), mamba_error_code::repodata_not_loaded));
}
else
{
LOG_WARNING << "Could not load repodata.json for " << subdir.name()
<< ". Deleting cache, and retrying.";
subdir.clear_cache();
loading_failed = true;
}
}
load_subdir_in_pool(ctx, pool, subdir)
.transform([&](solver::libsolv::RepoInfo&& repo)
{ pool.set_repo_priority(repo, priorities[i]); })
.or_else(
[&](const auto& error)
{
if (is_retry & RETRY_SUBDIR_FETCH)
{
std::stringstream ss;
ss << "Could not load repodata.json for " << subdir.name()
<< " after retry."
<< "Please check repodata source. Exiting." << std::endl;
error_list.push_back(
mamba_error(ss.str(), mamba_error_code::repodata_not_loaded)
);
}
else
{
LOG_WARNING << "Could not load repodata.json for " << subdir.name()
<< ". Deleting cache, and retrying.";
subdir.clear_cache();
loading_failed = true;
}
}
);
}
if (loading_failed)
@ -235,5 +237,4 @@ namespace mamba
return error_list.empty() ? return_type()
: return_type(make_unexpected(std::move(error_list)));
}
}

9
libmamba/src/api/install.cpp
View File

@ -517,9 +517,7 @@ namespace mamba
prefix_pkgs.push_back(it.first);
}
prefix_data.add_packages(get_virtual_packages(ctx));
MRepo(pool, prefix_data);
load_installed_packages_in_pool(ctx, pool, prefix_data);
MSolver solver(
pool,
@ -670,9 +668,8 @@ namespace mamba
PrefixData& prefix_data = exp_prefix_data.value();
MultiPackageCache pkg_caches(ctx.pkgs_dirs, ctx.validation_params);
prefix_data.add_packages(get_virtual_packages(ctx));
MRepo(pool, prefix_data); // Potentially re-alloc (moves in memory) Solvables
// in the pool
load_installed_packages_in_pool(ctx, pool, prefix_data);
std::vector<detail::other_pkg_mgr_spec> others;
// Note that the Transaction will gather the Solvables,

5
libmamba/src/api/remove.cpp
View File

@ -11,9 +11,9 @@
#include "mamba/core/package_cache.hpp"
#include "mamba/core/pool.hpp"
#include "mamba/core/prefix_data.hpp"
#include "mamba/core/repo.hpp"
#include "mamba/core/solver.hpp"
#include "mamba/core/transaction.hpp"
#include "mamba/solver/libsolv/repo_info.hpp"
namespace mamba
{
@ -87,7 +87,8 @@ namespace mamba
PrefixData& prefix_data = exp_prefix_data.value();
MPool pool{ ctx, channel_context };
MRepo(pool, prefix_data);
load_installed_packages_in_pool(ctx, pool, prefix_data);
const fs::u8path pkgs_dirs(ctx.prefix_params.root_prefix / "pkgs");
MultiPackageCache package_caches({ pkgs_dirs }, ctx.validation_params);

6
libmamba/src/api/repoquery.cpp
View File

@ -14,7 +14,7 @@
#include "mamba/core/channel_context.hpp"
#include "mamba/core/package_cache.hpp"
#include "mamba/core/prefix_data.hpp"
#include "mamba/core/repo.hpp"
#include "mamba/solver/libsolv/repo_info.hpp"
#include "mamba/util/string.hpp"
namespace mamba
@ -50,7 +50,9 @@ namespace mamba
throw std::runtime_error(exp_prefix_data.error().what());
}
PrefixData& prefix_data = exp_prefix_data.value();
MRepo(pool, prefix_data);
load_installed_packages_in_pool(ctx, pool, prefix_data);
if (format != QueryResultFormat::Json)
{
Console::stream()

4
libmamba/src/api/update.cpp
View File

@ -69,9 +69,7 @@ namespace mamba
prefix_pkgs.push_back(it.first);
}
prefix_data.add_packages(get_virtual_packages(ctx));
MRepo(pool, prefix_data);
load_installed_packages_in_pool(ctx, pool, prefix_data);
MSolver solver(
pool,

296
libmamba/src/core/pool.cpp
View File

@ -11,6 +11,15 @@
#include <solv/pool.h>
#include <solv/selection.h>
#include <solv/solver.h>
#include "mamba/core/prefix_data.hpp"
#include "mamba/core/subdirdata.hpp"
#include "mamba/core/virtual_packages.hpp"
#include "mamba/util/build.hpp"
#include "mamba/util/random.hpp"
#include "mamba/util/string.hpp"
#include "solver/libsolv/helpers.hpp"
extern "C" // Incomplete header
{
#include <solv/conda.h>
@ -21,6 +30,7 @@ extern "C" // Incomplete header
#include "mamba/core/context.hpp"
#include "mamba/core/output.hpp"
#include "mamba/core/pool.hpp"
#include "mamba/solver/libsolv/repo_info.hpp"
#include "mamba/specs/match_spec.hpp"
#include "solv-cpp/pool.hpp"
#include "solv-cpp/queue.hpp"
@ -282,62 +292,11 @@ namespace mamba
return id;
}
namespace
{
auto make_package_info(const solv::ObjPool& pool, solv::ObjSolvableViewConst s)
-> specs::PackageInfo
{
specs::PackageInfo out = {};
out.name = s.name();
out.version = s.version();
out.build_string = s.build_string();
out.noarch = specs::noarch_parse(s.noarch()).value_or(specs::NoArchType::No);
out.build_number = s.build_number();
out.channel = s.channel();
out.package_url = s.url();
out.subdir = s.subdir();
out.filename = s.file_name();
out.license = s.license();
out.size = s.size();
out.timestamp = s.timestamp();
out.md5 = s.md5();
out.sha256 = s.sha256();
const auto dep_to_str = [&pool](solv::DependencyId id)
{ return pool.dependency_to_string(id); };
{
const auto deps = s.dependencies();
out.depends.reserve(deps.size());
std::transform(deps.cbegin(), deps.cend(), std::back_inserter(out.depends), dep_to_str);
}
{
const auto cons = s.constraints();
out.constrains.reserve(cons.size());
std::transform(cons.cbegin(), cons.cend(), std::back_inserter(out.constrains), dep_to_str);
}
{
const auto id_to_str = [&pool](solv::StringId id)
{ return std::string(pool.get_string(id)); };
auto feats = s.track_features();
out.track_features.reserve(feats.size());
std::transform(
feats.begin(),
feats.end(),
std::back_inserter(out.track_features),
id_to_str
);
}
return out;
}
}
std::optional<specs::PackageInfo> MPool::id2pkginfo(Id solv_id) const
{
if (const auto solv = pool().get_solvable(solv_id))
{
return { make_package_info(pool(), solv.value()) };
return { solver::libsolv::make_package_info(pool(), solv.value()) };
}
return std::nullopt;
}
@ -352,8 +311,239 @@ namespace mamba
return pool_dep2str(const_cast<::Pool*>(pool().raw()), dep_id);
}
auto MPool::add_repo_from_repodata_json(
const fs::u8path& path,
std::string_view url,
solver::libsolv::PipAsPythonDependency add,
solver::libsolv::RepodataParser parser
) -> expected_t<solver::libsolv::RepoInfo>
{
if (!fs::exists(path))
{
return make_unexpected(
fmt::format(R"(File "{}" does not exist)", path),
mamba_error_code::repodata_not_loaded
);
}
auto repo = pool().add_repo(url).second;
repo.set_url(std::string(url));
auto make_repo = [&]() -> expected_t<solv::ObjRepoView>
{
if (parser == solver::libsolv::RepodataParser::Mamba)
{
return solver::libsolv::mamba_read_json(
pool(),
repo,
path,
std::string(url),
context().use_only_tar_bz2
);
}
return solver::libsolv::libsolv_read_json(repo, path, context().use_only_tar_bz2)
.transform(
[&url](solv::ObjRepoView repo)
{
solver::libsolv::set_solvables_url(repo, std::string(url));
return repo;
}
);
};
return make_repo()
.transform(
[&](solv::ObjRepoView repo) -> solver::libsolv::RepoInfo
{
if (add == solver::libsolv::PipAsPythonDependency::Yes)
{
solver::libsolv::add_pip_as_python_dependency(pool(), repo);
}
repo.internalize();
return solver::libsolv::RepoInfo{ repo.raw() };
}
)
.or_else([&](const auto&) { pool().remove_repo(repo.id(), /* reuse_ids= */ true); });
}
auto MPool::add_repo_from_native_serialization(
const fs::u8path& path,
const solver::libsolv::RepodataOrigin& expected,
solver::libsolv::PipAsPythonDependency add
) -> expected_t<solver::libsolv::RepoInfo>
{
auto repo = pool().add_repo(expected.url).second;
return solver::libsolv::read_solv(pool(), repo, path, expected, static_cast<bool>(add))
.transform(
[&](solv::ObjRepoView repo) -> solver::libsolv::RepoInfo
{
repo.set_url(expected.url);
solver::libsolv::set_solvables_url(repo, expected.url);
if (add == solver::libsolv::PipAsPythonDependency::Yes)
{
solver::libsolv::add_pip_as_python_dependency(pool(), repo);
}
repo.internalize();
return solver::libsolv::RepoInfo(repo.raw());
}
)
.or_else([&](const auto&) { pool().remove_repo(repo.id(), /* reuse_ids= */ true); });
}
auto MPool::add_repo_from_packages_impl_pre(std::string_view name) -> solver::libsolv::RepoInfo
{
if (name.empty())
{
return solver::libsolv::RepoInfo(
pool().add_repo(util::generate_random_alphanumeric_string(20)).second.raw()
);
}
return solver::libsolv::RepoInfo(pool().add_repo(name).second.raw());
}
void MPool::add_repo_from_packages_impl_loop(
const solver::libsolv::RepoInfo& repo,
const specs::PackageInfo& pkg
)
{
auto s_repo = solv::ObjRepoView(*repo.m_ptr);
auto [id, solv] = s_repo.add_solvable();
solver::libsolv::set_solvable(pool(), solv, pkg);
}
void MPool::add_repo_from_packages_impl_post(
const solver::libsolv::RepoInfo& repo,
solver::libsolv::PipAsPythonDependency add
)
{
auto s_repo = solv::ObjRepoView(*repo.m_ptr);
if (add == solver::libsolv::PipAsPythonDependency::Yes)
{
solver::libsolv::add_pip_as_python_dependency(pool(), s_repo);
}
s_repo.internalize();
}
auto MPool::native_serialize_repo(
const solver::libsolv::RepoInfo& repo,
const fs::u8path& path,
const solver::libsolv::RepodataOrigin& metadata
) -> expected_t<solver::libsolv::RepoInfo>
{
assert(repo.m_ptr != nullptr);
return solver::libsolv::write_solv(solv::ObjRepoView(*repo.m_ptr), path, metadata)
.transform([](solv::ObjRepoView repo) { return solver::libsolv::RepoInfo(repo.raw()); });
}
void MPool::remove_repo(::Id repo_id, bool reuse_ids)
{
pool().remove_repo(repo_id, reuse_ids);
}
} // namespace mamba
void MPool::set_installed_repo(const solver::libsolv::RepoInfo& repo)
{
pool().set_installed_repo(repo.id());
}
void
MPool::set_repo_priority(const solver::libsolv::RepoInfo& repo, solver::libsolv::Priorities priorities)
{
// NOTE: The Pool is not involved directly in this operations, but since it is needed
// in so many repo operations, this setter was put here to keep the Repo class
// immutable.
static_assert(std::is_same_v<decltype(repo.m_ptr->priority), solver::libsolv::Priorities::value_type>);
repo.m_ptr->priority = priorities.priority;
repo.m_ptr->subpriority = priorities.subpriority;
}
// TODO machinery functions in separate files
auto load_subdir_in_pool(const Context& ctx, MPool& pool, const MSubdirData& subdir)
-> expected_t<solver::libsolv::RepoInfo>
{
const auto expected_cache_origin = solver::libsolv::RepodataOrigin{
/* .url= */ util::rsplit(subdir.metadata().url(), "/", 1).front(),
/* .etag= */ subdir.metadata().etag(),
/* .mod= */ subdir.metadata().last_modified(),
};
const auto add_pip = static_cast<solver::libsolv::PipAsPythonDependency>(
ctx.add_pip_as_python_dependency
);
const auto json_parser = ctx.experimental_repodata_parsing
? solver::libsolv::RepodataParser::Mamba
: solver::libsolv::RepodataParser::Libsolv;
// Solv files are too slow on Windows.
if (!util::on_win)
{
auto maybe_repo = subdir.valid_solv_cache().and_then(
[&](fs::u8path&& solv_file) {
return pool.add_repo_from_native_serialization(
solv_file,
expected_cache_origin,
add_pip
);
}
);
if (maybe_repo)
{
return maybe_repo;
}
}
return subdir.valid_json_cache()
.and_then(
[&](fs::u8path&& repodata_json)
{
LOG_INFO << "Trying to load repo from json file " << repodata_json;
return pool.add_repo_from_repodata_json(
repodata_json,
util::rsplit(subdir.metadata().url(), "/", 1).front(),
add_pip,
json_parser
);
}
)
.transform(
[&](solver::libsolv::RepoInfo&& repo) -> solver::libsolv::RepoInfo
{
if (!util::on_win)
{
pool.native_serialize_repo(repo, subdir.writable_solv_cache(), expected_cache_origin)
.or_else(
[&](const auto& err)
{
LOG_WARNING << R"(Fail to write native serialization to file ")"
<< subdir.writable_solv_cache() << R"(" for repo ")"
<< subdir.name() << ": " << err.what();
;
}
);
}
return std::move(repo);
}
);
}
auto load_installed_packages_in_pool(const Context& ctx, MPool& pool, const PrefixData& prefix)
-> solver::libsolv::RepoInfo
{
// TODO(C++20): We could do a PrefixData range that returns packages without storing thems.
auto pkgs = prefix.sorted_records();
// TODO(C++20): We only need a range that concatenate both
for (auto&& pkg : get_virtual_packages(ctx))
{
pkgs.push_back(std::move(pkg));
}
// Not adding Pip dependency since it might needlessly make the installed/active environment
// broken if pip is not already installed (debatable).
auto repo = pool.add_repo_from_packages(
pkgs,
"installed",
solver::libsolv::PipAsPythonDependency::No
);
pool.set_installed_repo(repo);
return repo;
}
}

670
libmamba/src/core/repo.cpp
View File

@ -1,670 +0,0 @@
// Copyright (c) 2019, QuantStack and Mamba Contributors
//
// Distributed under the terms of the BSD 3-Clause License.
//
// The full license is in the file LICENSE, distributed with this software.
#include <array>
#include <string_view>
#include <tuple>
#include <simdjson.h>
#include <solv/repo.h>
#include <solv/repo_solv.h>
#include <solv/repo_write.h>
#include <solv/solvable.h>
extern "C" // Incomplete header
{
#include <solv/conda.h>
#include <solv/repo_conda.h>
}
#include "mamba/core/context.hpp"
#include "mamba/core/output.hpp"
#include "mamba/core/pool.hpp"
#include "mamba/core/prefix_data.hpp"
#include "mamba/core/repo.hpp"
#include "mamba/core/util.hpp"
#include "mamba/fs/filesystem.hpp"
#include "mamba/specs/conda_url.hpp"
#include "mamba/specs/package_info.hpp"
#include "mamba/util/build.hpp"
#include "mamba/util/string.hpp"
#include "solv-cpp/pool.hpp"
#include "solv-cpp/repo.hpp"
#define MAMBA_TOOL_VERSION "2.0"
#define MAMBA_SOLV_VERSION MAMBA_TOOL_VERSION "_" LIBSOLV_VERSION_STRING
namespace mamba
{
namespace nl = nlohmann;
namespace
{
auto attrs(const RepoMetadata& m)
{
return std::tie(m.url, m.etag, m.mod, m.pip_added);
}
}
auto operator==(const RepoMetadata& lhs, const RepoMetadata& rhs) -> bool
{
return attrs(lhs) == attrs(rhs);
}
auto operator!=(const RepoMetadata& lhs, const RepoMetadata& rhs) -> bool
{
return !(lhs == rhs);
}
void to_json(nlohmann::json& j, const RepoMetadata& m)
{
j["url"] = m.url;
j["etag"] = m.etag;
j["mod"] = m.mod;
j["pip_added"] = m.pip_added;
}
void from_json(const nlohmann::json& j, RepoMetadata& m)
{
m.url = j.value("url", m.url);
m.etag = j.value("etag", m.etag);
m.mod = j.value("mod", m.mod);
m.pip_added = j.value("pip_added", m.pip_added);
}
namespace
{
// Keeping the solv-cpp header private
auto srepo(const MRepo& r) -> solv::ObjRepoViewConst
{
return solv::ObjRepoViewConst{ *const_cast<const ::Repo*>(r.repo()) };
}
auto srepo(MRepo& r) -> solv::ObjRepoView
{
return solv::ObjRepoView{ *r.repo() };
}
void set_solvable(MPool& pool, solv::ObjSolvableView solv, const specs::PackageInfo& pkg)
{
solv.set_name(pkg.name);
solv.set_version(pkg.version);
solv.set_build_string(pkg.build_string);
if (pkg.noarch != specs::NoArchType::No)
{
auto noarch = std::string(specs::noarch_name(pkg.noarch)); // SSO
solv.set_noarch(noarch);
}
solv.set_build_number(pkg.build_number);
solv.set_channel(pkg.channel);
solv.set_url(pkg.package_url);
solv.set_subdir(pkg.subdir);
solv.set_file_name(pkg.filename);
solv.set_license(pkg.license);
solv.set_size(pkg.size);
// TODO conda timestamp are not Unix timestamp.
// Libsolv normalize them this way, we need to do the same here otherwise the current
// package may get arbitrary priority.
solv.set_timestamp(
(pkg.timestamp > 253402300799ULL) ? (pkg.timestamp / 1000) : pkg.timestamp
);
solv.set_md5(pkg.md5);
solv.set_sha256(pkg.sha256);
for (const auto& dep : pkg.depends)
{
// TODO pool's matchspec2id
const solv::DependencyId dep_id = pool_conda_matchspec(pool, dep.c_str());
assert(dep_id);
solv.add_dependency(dep_id);
}
for (const auto& cons : pkg.constrains)
{
// TODO pool's matchspec2id
const solv::DependencyId dep_id = pool_conda_matchspec(pool, cons.c_str());
assert(dep_id);
solv.add_constraint(dep_id);
}
solv.add_track_features(pkg.track_features);
solv.add_self_provide();
}
auto lsplit_track_features(std::string_view features)
{
constexpr auto is_sep = [](char c) -> bool { return (c == ',') || util::is_space(c); };
auto [_, tail] = util::lstrip_if_parts(features, is_sep);
return util::lstrip_if_parts(tail, [&](char c) { return !is_sep(c); });
}
[[nodiscard]] auto set_solvable(
MPool& pool,
const std::string& repo_url_str,
const specs::CondaURL& repo_url,
solv::ObjSolvableView solv,
const std::string& filename,
const simdjson::dom::element& pkg,
const std::string& default_subdir
) -> bool
{
// Not available from RepoDataPackage
solv.set_file_name(filename);
solv.set_url((repo_url / filename).str(specs::CondaURL::Credentials::Show));
solv.set_channel(repo_url_str);
if (auto name = pkg["name"].get_string(); !name.error())
{
solv.set_name(name.value_unsafe());
}
else
{
LOG_WARNING << R"(Found invalid name in ")" << filename << R"(")";
return false;
}
if (auto version = pkg["version"].get_string(); !version.error())
{
solv.set_version(version.value_unsafe());
}
else
{
LOG_WARNING << R"(Found invalid version in ")" << filename << R"(")";
return false;
}
if (auto build_string = pkg["build"].get_string(); !build_string.error())
{
solv.set_build_string(build_string.value_unsafe());
}
else
{
LOG_WARNING << R"(Found invalid build in ")" << filename << R"(")";
return false;
}
if (auto build_number = pkg["build_number"].get_uint64(); !build_number.error())
{
solv.set_build_number(build_number.value_unsafe());
}
else
{
LOG_WARNING << R"(Found invalid build_number in ")" << filename << R"(")";
return false;
}
if (auto subdir = pkg["subdir"].get_c_str(); !subdir.error())
{
solv.set_subdir(subdir.value_unsafe());
}
else
{
solv.set_subdir(default_subdir);
}
if (auto size = pkg["size"].get_uint64(); !size.error())
{
solv.set_size(size.value_unsafe());
}
if (auto md5 = pkg["md5"].get_c_str(); !md5.error())
{
solv.set_md5(md5.value_unsafe());
}
if (auto sha256 = pkg["sha256"].get_c_str(); !sha256.error())
{
solv.set_sha256(sha256.value_unsafe());
}
if (auto elem = pkg["noarch"]; !elem.error())
{
if (auto val = elem.get_bool(); !val.error() && val.value_unsafe())
{
solv.set_noarch("generic");
}
else if (auto noarch = elem.get_c_str(); !noarch.error())
{
solv.set_noarch(noarch.value_unsafe());
}
}
if (auto license = pkg["license"].get_c_str(); !license.error())
{
solv.set_license(license.value_unsafe());
}
// TODO conda timestamp are not Unix timestamp.
// Libsolv normalize them this way, we need to do the same here otherwise the current
// package may get arbitrary priority.
if (auto timestamp = pkg["timestamp"].get_uint64(); !timestamp.error())
{
const auto time = timestamp.value_unsafe();
solv.set_timestamp((time > 253402300799ULL) ? (time / 1000) : time);
}
if (auto depends = pkg["depends"].get_array(); !depends.error())
{
for (auto elem : depends)
{
if (auto dep = elem.get_c_str(); !dep.error())
{
if (const auto dep_id = pool_conda_matchspec(pool, dep.value_unsafe()))
{
solv.add_dependency(dep_id);
}
}
}
}
if (auto constrains = pkg["constrains"].get_array(); !constrains.error())
{
for (auto elem : constrains)
{
if (auto cons = elem.get_c_str(); !cons.error())
{
if (const auto dep_id = pool_conda_matchspec(pool, cons.value_unsafe()))
{
solv.add_constraint(dep_id);
}
}
}
}
if (auto obj = pkg["track_features"]; !obj.error())
{
if (auto track_features_arr = obj.get_array(); !track_features_arr.error())
{
for (auto elem : track_features_arr)
{
if (auto feat = elem.get_string(); !feat.error())
{
solv.add_track_feature(feat.value_unsafe());
}
}
}
else if (auto track_features_str = obj.get_string(); !track_features_str.error())
{
auto splits = lsplit_track_features(track_features_str.value_unsafe());
while (!splits[0].empty())
{
solv.add_track_feature(splits[0]);
splits = lsplit_track_features(splits[1]);
}
}
}
solv.add_self_provide();
return true;
}
void set_repo_solvables(
MPool& pool,
solv::ObjRepoView repo,
const std::string& repo_url_str,
const specs::CondaURL& repo_url,
const std::string& default_subdir,
const simdjson::dom::object& packages
)
{
std::string filename = {};
for (const auto& [fn, pkg] : packages)
{
auto [id, solv] = repo.add_solvable();
filename = fn;
const bool parsed = set_solvable(
pool,
repo_url_str,
repo_url,
solv,
filename,
pkg,
default_subdir
);
if (parsed)
{
LOG_DEBUG << "Adding package record to repo " << fn;
}
else
{
repo.remove_solvable(id, /* reuse_id= */ true);
LOG_WARNING << "Failed to parse from repodata " << fn;
}
}
}
void set_solvables_url(solv::ObjRepoView repo, const std::string& repo_url)
{
// WARNING cannot call ``url()`` at this point because it has not been internalized.
// Setting the channel url on where the solvable so that we can retrace
// where it came from
const auto url = specs::CondaURL::parse(repo_url);
repo.for_each_solvable(
[&](solv::ObjSolvableView s)
{
// The solvable url, this is not set in libsolv parsing so we set it manually
// while we still rely on libsolv for parsing
// TODO
s.set_url((url / s.file_name()).str(specs::CondaURL::Credentials::Show));
// The name of the channel where it came from, may be different from repo name
// for instance with the installed repo
s.set_channel(repo_url);
}
);
}
}
MRepo::MRepo(
MPool& pool,
const std::string& name,
const fs::u8path& index,
const RepoMetadata& metadata,
RepodataParser parser,
LibsolvCache use_cache
)
: m_pool(pool)
, m_metadata(metadata)
{
auto [_, repo] = pool.pool().add_repo(name);
m_repo = repo.raw();
repo.set_url(m_metadata.url);
load_file(index, parser, use_cache);
repo.internalize();
}
MRepo::MRepo(MPool& pool, const std::string& name, const std::vector<specs::PackageInfo>& package_infos)
: m_pool(pool)
{
auto [_, repo] = pool.pool().add_repo(name);
m_repo = repo.raw();
for (auto& info : package_infos)
{
LOG_INFO << "Adding package record to repo " << info.name;
auto [id, solv] = srepo(*this).add_solvable();
set_solvable(m_pool, solv, info);
}
repo.internalize();
}
MRepo::MRepo(MPool& pool, const PrefixData& prefix_data)
: m_pool(pool)
{
auto [repo_id, repo] = pool.pool().add_repo("installed");
m_repo = repo.raw();
for (auto& [name, record] : prefix_data.records())
{
LOG_INFO << "Adding package record to repo " << record.name;
auto [id, solv] = srepo(*this).add_solvable();
set_solvable(m_pool, solv, record);
}
if (pool.context().add_pip_as_python_dependency)
{
add_pip_as_python_dependency();
}
repo.internalize();
pool.pool().set_installed_repo(repo_id);
}
void MRepo::set_installed()
{
m_pool.pool().set_installed_repo(srepo(*this).id());
}
void MRepo::set_priority(int priority, int subpriority)
{
m_repo->priority = priority;
m_repo->subpriority = subpriority;
}
auto MRepo::name() const -> std::string_view
{
return srepo(*this).name();
}
Id MRepo::id() const
{
return srepo(*this).id();
}
Repo* MRepo::repo() const
{
return m_repo;
}
void MRepo::add_pip_as_python_dependency()
{
const solv::DependencyId python_id = pool_conda_matchspec(m_pool, "python");
const solv::DependencyId pip_id = pool_conda_matchspec(m_pool, "pip");
srepo(*this).for_each_solvable(
[&](solv::ObjSolvableView s)
{
if ((s.name() == "python") && !s.version().empty() && (s.version()[0] >= '2'))
{
s.add_dependency(pip_id);
}
if (s.name() == "pip")
{
s.add_dependency(python_id, SOLVABLE_PREREQMARKER);
}
}
);
}
void MRepo::libsolv_read_json(const fs::u8path& filename)
{
LOG_INFO << "Reading repodata.json file " << filename << " for repo " << name()
<< " using libsolv";
// TODO make this as part of options of the repo/pool
const int flags = m_pool.context().use_only_tar_bz2 ? CONDA_ADD_USE_ONLY_TAR_BZ2 : 0;
srepo(*this).legacy_read_conda_repodata(filename, flags);
}
void MRepo::mamba_read_json(const fs::u8path& filename)
{
LOG_INFO << "Reading repodata.json file " << filename << " for repo " << name()
<< " using mamba";
auto parser = simdjson::dom::parser();
const auto repodata = parser.load(filename);
// An override for missing package subdir is found in at the top level
auto default_subdir = std::string();
if (auto subdir = repodata.at_pointer("/info/subdir").get_string(); subdir.error())
{
default_subdir = std::string(subdir.value_unsafe());
}
const auto repo_url = specs::CondaURL::parse(m_metadata.url);
if (auto pkgs = repodata["packages"].get_object(); !pkgs.error())
{
set_repo_solvables(
m_pool,
srepo(*this),
m_metadata.url,
repo_url,
default_subdir,
pkgs.value()
);
}
if (auto pkgs = repodata["packages.conda"].get_object();
!pkgs.error() && !m_pool.context().use_only_tar_bz2)
{
set_repo_solvables(
m_pool,
srepo(*this),
m_metadata.url,
repo_url,
default_subdir,
pkgs.value()
);
}
}
bool MRepo::read_solv(const fs::u8path& filename)
{
LOG_INFO << "Attempting to read libsolv solv file " << filename << " for repo " << name();
auto repo = srepo(*this);
auto lock = LockFile(filename);
repo.read(filename);
const auto read_metadata = RepoMetadata{
/* .url= */ std::string(repo.url()),
/* .etag= */ std::string(repo.etag()),
/* .mod= */ std::string(repo.mod()),
/* .pip_added= */ repo.pip_added(),
};
const auto tool_version = repo.tool_version();
{
auto j = nlohmann::json(m_metadata);
j["tool_version"] = tool_version;
LOG_INFO << "Expecting solv metadata : " << j.dump();
}
{
auto j = nlohmann::json(read_metadata);
j["tool_version"] = tool_version;
LOG_INFO << "Loaded solv metadata : " << j.dump();
}
if ((tool_version != std::string_view(MAMBA_SOLV_VERSION))
|| (read_metadata == RepoMetadata{}) || (read_metadata != m_metadata))
{
LOG_INFO << "Metadata from solv are NOT valid, canceling solv file load";
repo.clear(/* reuse_ids= */ false);
return false;
}
LOG_INFO << "Metadata from solv are valid, loading successful";
return true;
}
void MRepo::load_file(const fs::u8path& filename, RepodataParser parser, LibsolvCache use_cache)
{
auto repo = srepo(*this);
bool is_solv = filename.extension() == ".solv";
fs::u8path solv_file = filename;
solv_file.replace_extension("solv");
fs::u8path json_file = filename;
json_file.replace_extension("json");
LOG_INFO << "Reading cache files '" << (filename.parent_path() / filename).string()
<< ".*' for repo index '" << name() << "'";
// .solv files are slower to load than simdjson on Windows
if (!util::on_win && is_solv && (use_cache == LibsolvCache::yes))
{
const auto lock = LockFile(solv_file);
const bool read = read_solv(solv_file);
if (read)
{
set_solvables_url(repo, m_metadata.url);
return;
}
}
const auto& ctx = m_pool.context();
{
const auto lock = LockFile(json_file);
if ((parser == RepodataParser::mamba)
|| (parser == RepodataParser::automatic && ctx.experimental_repodata_parsing))
{
mamba_read_json(json_file);
}
else
{
libsolv_read_json(json_file);
set_solvables_url(repo, m_metadata.url);
}
}
// TODO move this to a more structured approach for repodata patching?
if (ctx.add_pip_as_python_dependency)
{
add_pip_as_python_dependency();
}
if (!util::on_win && (name() != "installed"))
{
write_solv(solv_file);
}
}
void MRepo::write_solv(fs::u8path filename)
{
LOG_INFO << "Writing libsolv solv file " << filename << " for repo " << name();
auto repo = srepo(*this);
repo.set_url(m_metadata.url);
repo.set_etag(m_metadata.etag);
repo.set_mod(m_metadata.mod);
repo.set_pip_added(m_metadata.pip_added);
repo.set_tool_version(MAMBA_SOLV_VERSION);
repo.internalize();
repo.write(filename);
}
void MRepo::clear(bool reuse_ids)
{
m_pool.remove_repo(id(), reuse_ids);
}
auto MRepo::py_name() const -> std::string_view
{
return name();
}
auto MRepo::py_priority() const -> std::tuple<int, int>
{
return std::make_tuple(m_repo->priority, m_repo->subpriority);
}
auto MRepo::py_clear(bool reuse_ids) -> bool
{
clear(reuse_ids);
return true;
}
auto MRepo::py_size() const -> std::size_t
{
return srepo(*this).solvable_count();
}
void MRepo::py_add_extra_pkg_info(const std::map<std::string, PyExtraPkgInfo>& extra)
{
auto repo = srepo(*this);
repo.for_each_solvable(
[&](solv::ObjSolvableView s)
{
if (auto const it = extra.find(std::string(s.name())); it != extra.cend())
{
if (auto const& noarch = it->second.noarch; !noarch.empty())
{
s.set_noarch(noarch);
}
if (auto const& repo_url = it->second.repo_url; !repo_url.empty())
{
s.set_channel(repo_url);
}
}
}
);
repo.internalize();
}
} // namespace mamba

57
libmamba/src/core/subdirdata.cpp
View File

@ -184,9 +184,10 @@ namespace mamba
bool MSubdirMetadata::check_valid_metadata(const fs::u8path& file)
{
if (m_stored_file_size != fs::file_size(file))
if (const auto new_size = fs::file_size(file); new_size != m_stored_file_size)
{
LOG_INFO << "File size changed, invalidating metadata";
LOG_INFO << "File size changed, expected " << m_stored_file_size << " but got "
<< new_size << "; invalidating metadata";
return false;
}
#ifndef _WIN32
@ -287,8 +288,12 @@ namespace mamba
auto MSubdirMetadata::from_repodata_file(const fs::u8path& repodata_file)
-> expected_subdir_metadata
{
std::ifstream in_file = open_ifstream(repodata_file);
const std::string json = extract_subjson(in_file);
const std::string json = [](const fs::u8path& file) -> std::string
{
auto lock = LockFile(file);
std::ifstream in_file = open_ifstream(file);
return extract_subjson(in_file);
}(repodata_file);
try
{
@ -449,6 +454,34 @@ namespace mamba
return m_name;
}
const MSubdirMetadata& MSubdirData::metadata() const
{
return m_metadata;
}
expected_t<fs::u8path> MSubdirData::valid_solv_cache() const
{
if (m_json_cache_valid && m_solv_cache_valid)
{
return (get_cache_dir(m_valid_cache_path) / m_solv_fn).string();
}
return make_unexpected("Cache not loaded", mamba_error_code::cache_not_loaded);
}
fs::u8path MSubdirData::writable_solv_cache() const
{
return m_writable_pkgs_dir / "cache" / m_solv_fn;
}
expected_t<fs::u8path> MSubdirData::valid_json_cache() const
{
if (m_json_cache_valid)
{
return (get_cache_dir(m_valid_cache_path) / m_json_fn).string();
}
return make_unexpected("Cache not loaded", mamba_error_code::cache_not_loaded);
}
expected_t<std::string> MSubdirData::cache_path() const
{
// TODO invalidate solv cache on version updates!!
@ -511,22 +544,6 @@ namespace mamba
return expected_t<void>();
}
expected_t<MRepo> MSubdirData::create_repo(MPool& pool) const
{
assert(&pool.context() == p_context);
using return_type = expected_t<MRepo>;
RepoMetadata meta{
/* .url= */ util::rsplit(m_metadata.url(), "/", 1).front(),
/* .etag= */ m_metadata.etag(),
/* .mod= */ m_metadata.last_modified(),
/* .pip_added= */ p_context->add_pip_as_python_dependency,
};
const auto cache = cache_path();
return cache ? return_type(MRepo(pool, m_name, *cache, meta))
: return_type(forward_error(cache));
}
MSubdirData::MSubdirData(
Context& ctx,
ChannelContext& channel_context,

29
libmamba/src/core/transaction.cpp
View File

@ -156,6 +156,17 @@ namespace mamba
{
auto pkginfo = get_pkginfo(id);
// In libsolv, system dependencies are provided as a special dependency,
// while in Conda it is implemented as a virtual package.
// Maybe there is a way to tell libsolv to never try to install or remove these
// solvables (SOLVER_LOCK or SOLVER_USERINSTALLED?).
// In the meantime (and probably later for safety) we filter all virtual
// packages out.
if (util::starts_with(pkginfo.name, "__")) // i.e. is_virtual_package
{
return;
}
// Artificial packages are packages that were added to implement a feature
// (e.g. a pin) but do not represent a Conda package.
// They can appear in the transaction depending on libsolv flags.
@ -300,9 +311,11 @@ namespace mamba
)
: MTransaction(pool, caches)
{
MRepo mrepo{ m_pool,
"__explicit_specs__",
make_pkg_info_from_explicit_match_specs(specs_to_install) };
auto mrepo = m_pool.add_repo_from_packages(
make_pkg_info_from_explicit_match_specs(specs_to_install),
"__explicit_specs__",
solver::libsolv::PipAsPythonDependency::No
);
m_pool.create_whatprovides();
@ -344,7 +357,7 @@ namespace mamba
Console::instance().json_write({ { "success", remove_success } });
// find repo __explicit_specs__ and install all packages from it
auto repo = solv::ObjRepoView(*mrepo.repo());
auto repo = solv::ObjRepoView(*mrepo.m_ptr);
repo.for_each_solvable_id([&](solv::SolvableId id) { decision.push_back(id); });
auto trans = solv::ObjTransaction::from_solvables(m_pool.pool(), decision);
@ -563,12 +576,16 @@ namespace mamba
: MTransaction(pool, caches)
{
LOG_INFO << "MTransaction::MTransaction - packages already resolved (lockfile)";
MRepo mrepo = MRepo(m_pool, "__explicit_specs__", packages);
auto mrepo = m_pool.add_repo_from_packages(
packages,
"__explicit_specs__",
solver::libsolv::PipAsPythonDependency::No
);
m_pool.create_whatprovides();
solv::ObjQueue decision = {};
// find repo __explicit_specs__ and install all packages from it
auto repo = solv::ObjRepoView(*mrepo.repo());
auto repo = solv::ObjRepoView(*mrepo.m_ptr);
repo.for_each_solvable_id([&](solv::SolvableId id) { decision.push_back(id); });
auto trans = solv::ObjTransaction::from_solvables(m_pool.pool(), decision);

160
libmamba/src/solv-cpp/repo.cpp
View File

@ -5,15 +5,7 @@
// The full license is in the file LICENSE, distributed with this software.
#include <cassert>
#include <cerrno>
#include <cstdio>
#include <exception>
#include <iostream>
#include <sstream>
#ifdef _WIN32
#include <windows.h>
#endif
#include <solv/pool.h>
#include <solv/repo.h>
@ -26,7 +18,6 @@ extern "C" // Incomplete header in libsolv 0.7.23
#include <solv/repo_conda.h>
}
#include "mamba/fs/filesystem.hpp"
#include "solv-cpp/repo.hpp"
namespace mamba::solv
@ -99,111 +90,18 @@ namespace mamba::solv
return std::nullopt;
}
namespace
auto ObjRepoViewConst::write(std::FILE* solv_file) const -> tl::expected<void, std::string>
{
class CFile
{
public:
/**
* Open a file with C API.
*
* @param path must hae filesystem default encoding.
*/
static auto open(const mamba::fs::u8path& path, const char* mode) -> CFile;
/**
* The destructor will flush and close the file descriptor.
*
* Like ``std::fstream``, exceptions are ignored.
* Explicitly call @ref close to get the exception.
*/
~CFile();
void close();
auto raw() noexcept -> std::FILE*;
private:
CFile(std::FILE* ptr, std::string name);
std::FILE* m_ptr = nullptr;
std::string m_name = {};
};
CFile::CFile(std::FILE* ptr, std::string name)
: m_ptr{ ptr }
, m_name{ name }
const auto write_res = ::repo_write(const_cast<::Repo*>(raw()), solv_file);
if (write_res == 0)
{
return {};
}
CFile::~CFile()
if (const char* str = ::pool_errstr(raw()->pool))
{
try
{
close();
}
catch (const std::exception& e)
{
std::cerr << "Developer error: "
"uncaught exception in CFile::~CFile, "
"explicitly call CFile::close to handle exception.\n"
<< e.what();
}
}
auto CFile::open(const mamba::fs::u8path& path, const char* mode) -> CFile
{
#ifdef _WIN32
// Mode MUST be an ASCII string
const auto wmode = std::wstring(mode, mode + std::strlen(mode));
auto ptr = ::_wfsopen(path.wstring().c_str(), wmode.c_str(), _SH_DENYNO);
if (ptr == nullptr)
{
throw std::system_error(GetLastError(), std::generic_category());
}
return { ptr, path.string() };
#else
std::string name = path.string();
std::FILE* ptr = std::fopen(name.c_str(), mode);
if (ptr == nullptr)
{
throw std::system_error(errno, std::generic_category());
}
return { ptr, std::move(name) };
#endif
}
void CFile::close()
{
const auto close_res = std::fclose(m_ptr); // This flush too
if (close_res != 0)
{
// TODO(C++20) fmt::format
auto ss = std::stringstream();
ss << "Unable to close file " << m_name;
throw std::runtime_error(ss.str());
}
}
auto CFile::raw() noexcept -> std::FILE*
{
return m_ptr;
}
}
void ObjRepoViewConst::write(const mamba::fs::u8path& solv_file) const
{
auto file = CFile::open(solv_file, "wb");
const auto write_res = ::repo_write(const_cast<::Repo*>(raw()), file.raw());
if (write_res != 0)
{
// TODO(C++20) fmt::format
auto ss = std::stringstream();
ss << "Unable to write repo '" << name() << "' to file";
throw std::runtime_error(ss.str());
return tl::unexpected(std::string(str));
}
return tl::unexpected("Unknow error");
}
/***********************************
@ -226,39 +124,33 @@ namespace mamba::solv
::repo_empty(raw(), static_cast<int>(reuse_ids));
}
void ObjRepoView::read(const mamba::fs::u8path& solv_file) const
auto ObjRepoView::read(std::FILE* solv_file) const -> tl::expected<void, std::string>
{
auto file = CFile::open(solv_file, "rb");
const auto read_res = ::repo_add_solv(raw(), file.raw(), 0);
if (read_res != 0)
const auto read_res = ::repo_add_solv(raw(), solv_file, 0);
if (read_res == 0)
{
// TODO(C++20) fmt::format
auto ss = std::stringstream();
ss << "Unable to read repo solv file '" << name() << '\'';
if (const char* str = ::pool_errstr(raw()->pool))
{
ss << ", error was: " << str;
}
throw std::runtime_error(ss.str());
return {};
}
if (const char* str = ::pool_errstr(raw()->pool))
{
return tl::unexpected(std::string(str));
}
return tl::unexpected("Unknow error");
}
void
ObjRepoView::legacy_read_conda_repodata(const mamba::fs::u8path& repodata_file, int flags) const
auto ObjRepoView::legacy_read_conda_repodata(std::FILE* repodata_file, int flags) const
-> tl::expected<void, std::string>
{
auto file = CFile::open(repodata_file, "rb");
const auto res = ::repo_add_conda(raw(), file.raw(), flags);
if (res != 0)
const auto res = ::repo_add_conda(raw(), repodata_file, flags);
if (res == 0)
{
// TODO(C++20) fmt::format
auto ss = std::stringstream();
ss << "Unable to read repodata JSON file '" << name() << '\'';
if (const char* str = ::pool_errstr(raw()->pool))
{
ss << ", error was: " << str;
}
throw std::runtime_error(ss.str());
return {};
}
if (const char* str = ::pool_errstr(raw()->pool))
{
return tl::unexpected(std::string(str));
}
return tl::unexpected("Unknow error");
}
auto ObjRepoView::add_solvable() const -> std::pair<SolvableId, ObjSolvableView>

14
libmamba/src/solv-cpp/repo.hpp
View File

@ -13,6 +13,7 @@
#include <utility>
#include <solv/repo.h>
#include <tl/expected.hpp>
#include "solv-cpp/ids.hpp"
#include "solv-cpp/solvable.hpp"
@ -110,11 +111,11 @@ namespace mamba::solv
/**
* Write repository information to file.
*
* @param solv_file A standard path with system encoding.
* @param solv_file A file pointer opened in binary mode.
* @warning This is a binary file that is not portable and may not even remain valid among
* different libsolv build, let alone versions.
*/
void write(const mamba::fs::u8path& solv_file) const;
[[nodiscard]] auto write(std::FILE* solv_file) const -> tl::expected<void, std::string>;
private:
@ -221,10 +222,10 @@ namespace mamba::solv
/**
* Read repository information from file.
*
* @param solv_file A standard path with system encoding.
* @param solv_file A file pointer opened in binary mode.
* @see ObjRepoViewConst::write
*/
void read(const mamba::fs::u8path& solv_file) const;
[[nodiscard]] auto read(std::FILE* solv_file) const -> tl::expected<void, std::string>;
/**
* Read repository information from a conda repodata.json.
@ -232,9 +233,10 @@ namespace mamba::solv
* This function is part of libsolv and does not read all attributes of the repodata.
* It is meant to be replaced. Parsing should be done by the user and solvables
* added through the API.
* @param repodata_file A standard path with system encoding.
* @param repodata_file A file pointer opened in binary mode.
*/
void legacy_read_conda_repodata(const mamba::fs::u8path& repodata_file, int flags = 0) const;
auto legacy_read_conda_repodata(std::FILE* repodata_file, int flags = 0) const
-> tl::expected<void, std::string>;
/** Add an empty solvable to the repository. */
auto add_solvable() const -> std::pair<SolvableId, ObjSolvableView>;

568
libmamba/src/solver/libsolv/helpers.cpp Normal file
View File

@ -0,0 +1,568 @@
// Copyright (c) 2023, QuantStack and Mamba Contributors
//
// Distributed under the terms of the BSD 3-Clause License.
//
// The full license is in the file LICENSE, distributed with this software.
#include <string>
#include <string_view>
#include <simdjson.h>
#include <solv/conda.h>
#include <solv/repo.h>
#include <solv/repo_conda.h>
#include <solv/repo_solv.h>
#include <solv/repo_write.h>
#include <solv/solvable.h>
#include "mamba/core/output.hpp"
#include "mamba/core/util.hpp"
#include "mamba/specs/conda_url.hpp"
#include "mamba/util/cfile.hpp"
#include "mamba/util/string.hpp"
#include "solver/libsolv/helpers.hpp"
#define MAMBA_TOOL_VERSION "2.0"
#define MAMBA_SOLV_VERSION MAMBA_TOOL_VERSION "_" LIBSOLV_VERSION_STRING
namespace mamba::solver::libsolv
{
void set_solvable(solv::ObjPool& pool, solv::ObjSolvableView solv, const specs::PackageInfo& pkg)
{
solv.set_name(pkg.name);
solv.set_version(pkg.version);
solv.set_build_string(pkg.build_string);
if (pkg.noarch != specs::NoArchType::No)
{
auto noarch = std::string(specs::noarch_name(pkg.noarch)); // SSO
solv.set_noarch(noarch);
}
solv.set_build_number(pkg.build_number);
solv.set_channel(pkg.channel);
solv.set_url(pkg.package_url);
solv.set_subdir(pkg.subdir);
solv.set_file_name(pkg.filename);
solv.set_license(pkg.license);
solv.set_size(pkg.size);
// TODO conda timestamp are not Unix timestamp.
// Libsolv normalize them this way, we need to do the same here otherwise the current
// package may get arbitrary priority.
solv.set_timestamp((pkg.timestamp > 253402300799ULL) ? (pkg.timestamp / 1000) : pkg.timestamp);
solv.set_md5(pkg.md5);
solv.set_sha256(pkg.sha256);
for (const auto& dep : pkg.depends)
{
// TODO pool's matchspec2id
const solv::DependencyId dep_id = pool.add_conda_dependency(dep);
assert(dep_id);
solv.add_dependency(dep_id);
}
for (const auto& cons : pkg.constrains)
{
// TODO pool's matchspec2id
const solv::DependencyId dep_id = pool.add_conda_dependency(cons);
assert(dep_id);
solv.add_constraint(dep_id);
}
solv.add_track_features(pkg.track_features);
solv.add_self_provide();
}
auto make_package_info(const solv::ObjPool& pool, solv::ObjSolvableViewConst s)
-> specs::PackageInfo
{
specs::PackageInfo out = {};
out.name = s.name();
out.version = s.version();
out.build_string = s.build_string();
out.noarch = specs::noarch_parse(s.noarch()).value_or(specs::NoArchType::No);
out.build_number = s.build_number();
out.channel = s.channel();
out.package_url = s.url();
out.subdir = s.subdir();
out.filename = s.file_name();
out.license = s.license();
out.size = s.size();
out.timestamp = s.timestamp();
out.md5 = s.md5();
out.sha256 = s.sha256();
const auto dep_to_str = [&pool](solv::DependencyId id)
{ return pool.dependency_to_string(id); };
{
const auto deps = s.dependencies();
out.depends.reserve(deps.size());
std::transform(deps.cbegin(), deps.cend(), std::back_inserter(out.depends), dep_to_str);
}
{
const auto cons = s.constraints();
out.constrains.reserve(cons.size());
std::transform(cons.cbegin(), cons.cend(), std::back_inserter(out.constrains), dep_to_str);
}
{
const auto id_to_str = [&pool](solv::StringId id)
{ return std::string(pool.get_string(id)); };
auto feats = s.track_features();
out.track_features.reserve(feats.size());
std::transform(feats.begin(), feats.end(), std::back_inserter(out.track_features), id_to_str);
}
return out;
}
namespace
{
auto lsplit_track_features(std::string_view features)
{
constexpr auto is_sep = [](char c) -> bool { return (c == ',') || util::is_space(c); };
auto [_, tail] = util::lstrip_if_parts(features, is_sep);
return util::lstrip_if_parts(tail, [&](char c) { return !is_sep(c); });
}
[[nodiscard]] auto set_solvable(
solv::ObjPool& pool,
const std::string& repo_url_str,
const specs::CondaURL& repo_url,
solv::ObjSolvableView solv,
const std::string& filename,
const simdjson::dom::element& pkg,
const std::string& default_subdir
) -> bool
{
// Not available from RepoDataPackage
solv.set_file_name(filename);
solv.set_url((repo_url / filename).str(specs::CondaURL::Credentials::Show));
solv.set_channel(repo_url_str);
if (auto name = pkg["name"].get_string(); !name.error())
{
solv.set_name(name.value_unsafe());
}
else
{
LOG_WARNING << R"(Found invalid name in ")" << filename << R"(")";
return false;
}
if (auto version = pkg["version"].get_string(); !version.error())
{
solv.set_version(version.value_unsafe());
}
else
{
LOG_WARNING << R"(Found invalid version in ")" << filename << R"(")";
return false;
}
if (auto build_string = pkg["build"].get_string(); !build_string.error())
{
solv.set_build_string(build_string.value_unsafe());
}
else
{
LOG_WARNING << R"(Found invalid build in ")" << filename << R"(")";
return false;
}
if (auto build_number = pkg["build_number"].get_uint64(); !build_number.error())
{
solv.set_build_number(build_number.value_unsafe());
}
else
{
LOG_WARNING << R"(Found invalid build_number in ")" << filename << R"(")";
return false;
}
if (auto subdir = pkg["subdir"].get_c_str(); !subdir.error())
{
solv.set_subdir(subdir.value_unsafe());
}
else
{
solv.set_subdir(default_subdir);
}
if (auto size = pkg["size"].get_uint64(); !size.error())
{
solv.set_size(size.value_unsafe());
}
if (auto md5 = pkg["md5"].get_c_str(); !md5.error())
{
solv.set_md5(md5.value_unsafe());
}
if (auto sha256 = pkg["sha256"].get_c_str(); !sha256.error())
{
solv.set_sha256(sha256.value_unsafe());
}
if (auto elem = pkg["noarch"]; !elem.error())
{
if (auto val = elem.get_bool(); !val.error() && val.value_unsafe())
{
solv.set_noarch("generic");
}
else if (auto noarch = elem.get_c_str(); !noarch.error())
{
solv.set_noarch(noarch.value_unsafe());
}
}
if (auto license = pkg["license"].get_c_str(); !license.error())
{
solv.set_license(license.value_unsafe());
}
// TODO conda timestamp are not Unix timestamp.
// Libsolv normalize them this way, we need to do the same here otherwise the current
// package may get arbitrary priority.
if (auto timestamp = pkg["timestamp"].get_uint64(); !timestamp.error())
{
const auto time = timestamp.value_unsafe();
solv.set_timestamp((time > 253402300799ULL) ? (time / 1000) : time);
}
if (auto depends = pkg["depends"].get_array(); !depends.error())
{
for (auto elem : depends)
{
if (auto dep = elem.get_c_str(); !dep.error())
{
if (const auto dep_id = pool.add_conda_dependency(dep.value_unsafe()))
{
solv.add_dependency(dep_id);
}
}
}
}
if (auto constrains = pkg["constrains"].get_array(); !constrains.error())
{
for (auto elem : constrains)
{
if (auto cons = elem.get_c_str(); !cons.error())
{
if (const auto dep_id = pool.add_conda_dependency(cons.value_unsafe()))
{
solv.add_constraint(dep_id);
}
}
}
}
if (auto obj = pkg["track_features"]; !obj.error())
{
if (auto track_features_arr = obj.get_array(); !track_features_arr.error())
{
for (auto elem : track_features_arr)
{
if (auto feat = elem.get_string(); !feat.error())
{
solv.add_track_feature(feat.value_unsafe());
}
}
}
else if (auto track_features_str = obj.get_string(); !track_features_str.error())
{
auto splits = lsplit_track_features(track_features_str.value_unsafe());
while (!splits[0].empty())
{
solv.add_track_feature(splits[0]);
splits = lsplit_track_features(splits[1]);
}
}
}
solv.add_self_provide();
return true;
}
void set_repo_solvables(
solv::ObjPool& pool,
solv::ObjRepoView repo,
const std::string& repo_url_str,
const specs::CondaURL& repo_url,
const std::string& default_subdir,
const simdjson::dom::object& packages
)
{
std::string filename = {};
for (const auto& [fn, pkg] : packages)
{
auto [id, solv] = repo.add_solvable();
filename = fn;
const bool parsed = set_solvable(
pool,
repo_url_str,
repo_url,
solv,
filename,
pkg,
default_subdir
);
if (parsed)
{
LOG_DEBUG << "Adding package record to repo " << fn;
}
else
{
repo.remove_solvable(id, /* reuse_id= */ true);
LOG_WARNING << "Failed to parse from repodata " << fn;
}
}
}
}
auto libsolv_read_json(solv::ObjRepoView repo, const fs::u8path& filename, bool only_tar_bz2)
-> expected_t<solv::ObjRepoView>
{
LOG_INFO << "Reading repodata.json file " << filename << " for repo " << repo.name()
<< " using libsolv";
const int flags = only_tar_bz2 ? CONDA_ADD_USE_ONLY_TAR_BZ2 : 0;
const auto lock = LockFile(filename);
return util::CFile::try_open(filename, "rb")
.transform_error([](std::error_code&& ec) { return ec.message(); })
.and_then(
[&](util::CFile&& file_ptr) -> tl::expected<void, std::string>
{
auto out = repo.legacy_read_conda_repodata(file_ptr.raw(), flags);
file_ptr.try_close().or_else([&](auto const& err) { //
LOG_WARNING << R"(Fail to close file ")" << filename << R"(": )" << err;
});
return out;
}
)
.transform([&]() { return repo; })
.transform_error(
[](std::string&& str)
{ return mamba_error(std::move(str), mamba_error_code::repodata_not_loaded); }
);
}
auto mamba_read_json(
solv::ObjPool& pool,
solv::ObjRepoView repo,
const fs::u8path& filename,
const std::string& repo_url,
bool only_tar_bz2
) -> expected_t<solv::ObjRepoView>
{
LOG_INFO << "Reading repodata.json file " << filename << " for repo " << repo.name()
<< " using mamba";
auto parser = simdjson::dom::parser();
const auto lock = LockFile(filename);
const auto repodata = parser.load(filename);
// An override for missing package subdir is found in at the top level
auto default_subdir = std::string();
if (auto subdir = repodata.at_pointer("/info/subdir").get_string(); subdir.error())
{
default_subdir = std::string(subdir.value_unsafe());
}
const auto parsed_url = specs::CondaURL::parse(repo_url);
if (auto pkgs = repodata["packages"].get_object(); !pkgs.error())
{
set_repo_solvables(pool, repo, repo_url, parsed_url, default_subdir, pkgs.value());
}
if (auto pkgs = repodata["packages.conda"].get_object(); !pkgs.error() && !only_tar_bz2)
{
set_repo_solvables(pool, repo, repo_url, parsed_url, default_subdir, pkgs.value());
}
return { repo };
}
[[nodiscard]] auto read_solv(
solv::ObjPool& pool,
solv::ObjRepoView repo,
const fs::u8path& filename,
const solver::libsolv::RepodataOrigin& expected,
bool expected_pip_added
) -> expected_t<solv::ObjRepoView>
{
using RepodataOrigin = solver::libsolv::RepodataOrigin;
static constexpr auto expected_binary_version = std::string_view(MAMBA_SOLV_VERSION);
LOG_INFO << "Attempting to read libsolv solv file " << filename << " for repo "
<< repo.name();
if (!fs::exists(filename))
{
return make_unexpected(
fmt::format(R"(File "{}" does not exist)", filename),
mamba_error_code::repodata_not_loaded
);
}
{
auto j = nlohmann::json(expected);
j["tool_version"] = expected_binary_version;
LOG_INFO << "Expecting solv metadata : " << j.dump();
}
auto lock = LockFile(filename);
return util::CFile::try_open(filename, "rb")
.transform_error([](std::error_code&& ec) { return ec.message(); })
.and_then(
[&](util::CFile&& file_ptr) -> tl::expected<void, std::string>
{
auto out = repo.read(file_ptr.raw());
file_ptr.try_close().or_else([&](auto const& err) { //
LOG_WARNING << R"(Fail to close file ")" << filename << R"(": )" << err;
});
return out;
}
)
.transform_error(
[](std::string&& str)
{ return mamba_error(std::move(str), mamba_error_code::repodata_not_loaded); }
)
.and_then(
[&]() -> expected_t<solv::ObjRepoView>
{
auto read_binary_version = repo.tool_version();
if (read_binary_version != expected_binary_version)
{
repo.clear(/* reuse_ids= */ true);
return make_unexpected(
"Metadata from solv are binary incompatible",
mamba_error_code::repodata_not_loaded
);
}
const auto read_metadata = RepodataOrigin{
/* .url= */ std::string(repo.url()),
/* .etag= */ std::string(repo.etag()),
/* .mod= */ std::string(repo.mod()),
};
{
auto j = nlohmann::json(read_metadata);
j["tool_version"] = read_binary_version;
LOG_INFO << "Loaded solv metadata : " << j.dump();
}
if ((read_metadata == RepodataOrigin{}) || (read_metadata != expected))
{
repo.clear(/* reuse_ids= */ true);
return make_unexpected(
"Metadata from solv are outdated",
mamba_error_code::repodata_not_loaded
);
}
const bool read_pip_added = repo.pip_added();
if (expected_pip_added != read_pip_added)
{
if (expected_pip_added)
{
add_pip_as_python_dependency(pool, repo);
LOG_INFO << "Added missing pip dependencies";
}
else
{
repo.clear(/* reuse_ids= */ true);
return make_unexpected(
"Metadata from solv contain extra pip dependencies",
mamba_error_code::repodata_not_loaded
);
}
}
LOG_INFO << "Metadata from solv are valid, loading successful";
return { repo };
}
);
}
auto
write_solv(solv::ObjRepoView repo, fs::u8path filename, const solver::libsolv::RepodataOrigin& metadata)
-> expected_t<solv::ObjRepoView>
{
LOG_INFO << "Writing libsolv solv file " << filename << " for repo " << repo.name();
repo.set_url(metadata.url);
repo.set_etag(metadata.etag);
repo.set_mod(metadata.mod);
repo.set_tool_version(MAMBA_SOLV_VERSION);
repo.internalize();
fs::create_directories(filename.parent_path());
const auto lock = LockFile(fs::exists(filename) ? filename : filename.parent_path());
return util::CFile::try_open(filename, "wb")
.transform_error([](std::error_code&& ec) { return ec.message(); })
.and_then(
[&](util::CFile&& file_ptr) -> tl::expected<void, std::string>
{
auto out = repo.write(file_ptr.raw());
file_ptr.try_close().or_else([&](auto const& err) { //
LOG_WARNING << R"(Fail to close file ")" << filename << R"(": )" << err;
});
return out;
}
)
.transform([&]() { return repo; })
.transform_error(
[](std::string&& str)
{ return mamba_error(std::move(str), mamba_error_code::repodata_not_loaded); }
);
}
void set_solvables_url(solv::ObjRepoView repo, const std::string& repo_url)
{
// WARNING cannot call ``url()`` at this point because it has not been internalized.
// Setting the channel url on where the solvable so that we can retrace
// where it came from
const auto url = specs::CondaURL::parse(repo_url);
repo.for_each_solvable(
[&](solv::ObjSolvableView s)
{
// The solvable url, this is not set in libsolv parsing so we set it manually
// while we still rely on libsolv for parsing
// TODO
s.set_url((url / s.file_name()).str(specs::CondaURL::Credentials::Show));
// The name of the channel where it came from, may be different from repo name
// for instance with the installed repo
s.set_channel(repo_url);
}
);
}
void add_pip_as_python_dependency(solv::ObjPool& pool, solv::ObjRepoView repo)
{
const solv::DependencyId python_id = pool.add_conda_dependency("python");
const solv::DependencyId pip_id = pool.add_conda_dependency("pip");
repo.for_each_solvable(
[&](solv::ObjSolvableView s)
{
if ((s.name() == "python") && !s.version().empty() && (s.version()[0] >= '2'))
{
s.add_dependency(pip_id);
}
if (s.name() == "pip")
{
s.add_dependency(python_id, SOLVABLE_PREREQMARKER);
}
}
);
repo.set_pip_added(true);
}
}

61
libmamba/src/solver/libsolv/helpers.hpp Normal file
View File

@ -0,0 +1,61 @@
// Copyright (c) 2023, QuantStack and Mamba Contributors
//
// Distributed under the terms of the BSD 3-Clause License.
//
// The full license is in the file LICENSE, distributed with this software.
#ifndef MAMBA_SOLVER_LIBSOLV_HERLPERS
#define MAMBA_SOLVER_LIBSOLV_HERLPERS
#include "mamba/core/error_handling.hpp"
#include "mamba/solver/libsolv/parameters.hpp"
#include "mamba/specs/package_info.hpp"
#include "solv-cpp/pool.hpp"
#include "solv-cpp/repo.hpp"
#include "solv-cpp/solvable.hpp"
namespace mamba::fs
{
class u8path;
}
namespace mamba::solver::libsolv
{
void set_solvable(solv::ObjPool& pool, solv::ObjSolvableView solv, const specs::PackageInfo& pkg);
auto make_package_info(const solv::ObjPool& pool, solv::ObjSolvableViewConst s)
-> specs::PackageInfo;
[[nodiscard]] auto libsolv_read_json( //
solv::ObjRepoView repo,
const fs::u8path& filename,
bool only_tar_bz2
) -> expected_t<solv::ObjRepoView>;
[[nodiscard]] auto mamba_read_json(
solv::ObjPool& pool,
solv::ObjRepoView repo,
const fs::u8path& filename,
const std::string& repo_url,
bool only_tar_bz2
) -> expected_t<solv::ObjRepoView>;
[[nodiscard]] auto read_solv(
solv::ObjPool& pool,
solv::ObjRepoView repo,
const fs::u8path& filename,
const RepodataOrigin& expected,
bool expected_pip_added
) -> expected_t<solv::ObjRepoView>;
[[nodiscard]] auto write_solv( //
solv::ObjRepoView repo,
fs::u8path filename,
const RepodataOrigin& metadata
) -> expected_t<solv::ObjRepoView>;
void set_solvables_url(solv::ObjRepoView repo, const std::string& repo_url);
void add_pip_as_python_dependency(solv::ObjPool& pool, solv::ObjRepoView repo);
}
#endif

58
libmamba/src/solver/libsolv/parameters.cpp Normal file
View File

@ -0,0 +1,58 @@
// Copyright (c) 2023, QuantStack and Mamba Contributors
//
// Distributed under the terms of the BSD 3-Clause License.
//
// The full license is in the file LICENSE, distributed with this software.
#include <tuple>
#include <nlohmann/json.hpp>
#include "mamba/solver/libsolv/parameters.hpp"
#include "mamba/util/string.hpp"
namespace mamba::solver::libsolv
{
namespace
{
auto attrs(const Priorities& p)
{
return std::tie(p.priority, p.subpriority);
}
}
auto operator==(const Priorities& lhs, const Priorities& rhs) -> bool
{
return attrs(lhs) == attrs(rhs);
}
auto operator!=(const Priorities& lhs, const Priorities& rhs) -> bool
{
return !(lhs == rhs);
}
auto operator==(const RepodataOrigin& lhs, const RepodataOrigin& rhs) -> bool
{
return (util::rstrip(lhs.url, '/') == util::rstrip(rhs.url, '/')) && (lhs.etag == rhs.etag)
&& (lhs.mod == rhs.mod);
}
auto operator!=(const RepodataOrigin& lhs, const RepodataOrigin& rhs) -> bool
{
return !(lhs == rhs);
}
void to_json(nlohmann::json& j, const RepodataOrigin& m)
{
j["url"] = m.url;
j["etag"] = m.etag;
j["mod"] = m.mod;
}
void from_json(const nlohmann::json& j, RepodataOrigin& p)
{
p.url = j.value("url", "");
p.etag = j.value("etag", "");
p.mod = j.value("mod", "");
}
}

51
libmamba/src/solver/libsolv/repo_info.cpp Normal file
View File

@ -0,0 +1,51 @@
// Copyright (c) 2019, QuantStack and Mamba Contributors
//
// Distributed under the terms of the BSD 3-Clause License.
//
// The full license is in the file LICENSE, distributed with this software.
#include <string_view>
#include <type_traits>
#include "mamba/solver/libsolv/repo_info.hpp"
#include "solv-cpp/repo.hpp"
namespace mamba::solver::libsolv
{
RepoInfo::RepoInfo(::Repo* repo)
: m_ptr(repo)
{
}
auto RepoInfo::name() const -> std::string_view
{
return solv::ObjRepoViewConst(*m_ptr).name();
}
auto RepoInfo::priority() const -> Priorities
{
static_assert(std::is_same_v<decltype(m_ptr->priority), Priorities::value_type>);
return { /* .priority= */ m_ptr->priority, /* .subpriority= */ m_ptr->subpriority };
}
auto RepoInfo::package_count() const -> std::size_t
{
return solv::ObjRepoViewConst(*m_ptr).solvable_count();
}
auto RepoInfo::id() const -> RepoId
{
static_assert(std::is_same_v<RepoId, ::Id>);
return solv::ObjRepoViewConst(*m_ptr).id();
}
auto operator==(RepoInfo lhs, RepoInfo rhs) -> bool
{
return lhs.m_ptr == rhs.m_ptr;
}
auto operator!=(RepoInfo lhs, RepoInfo rhs) -> bool
{
return !(rhs == lhs);
}
} // namespace mamba

105
libmamba/src/util/cfile.cpp Normal file
View File

@ -0,0 +1,105 @@
// Copyright (c) 2023, QuantStack and Mamba Contributors
//
// Distributed under the terms of the BSD 3-Clause License.
//
// The full license is in the file LICENSE, distributed with this software.
#include <iostream>
#ifdef _WIN32
#include <windows.h>
#endif
#include "mamba/util/cfile.hpp"
namespace mamba::util
{
namespace
{
void try_close_impl(std::FILE* ptr, std::error_code& ec) noexcept
{
if (ptr)
{
const auto close_res = std::fclose(ptr); // This flush too
if (close_res != 0)
{
ec = std::make_error_code(std::errc::io_error);
}
}
}
}
void CFile::FileClose::operator()(std::FILE* ptr)
{
auto ec = std::error_code();
try_close_impl(ptr, ec);
if (ec)
{
std::cerr << "Developer error: error closing file in CFile::~CFile, "
"explicitly call CFile::try_close to handle error.\n";
}
}
CFile::CFile(std::FILE* ptr)
: m_ptr{ ptr }
{
}
CFile::~CFile() = default;
auto CFile::try_open(const mamba::fs::u8path& path, const char* mode, std::error_code& ec) -> CFile
{
#ifdef _WIN32
// Mode MUST be an ASCII string
const auto wmode = std::wstring(mode, mode + std::strlen(mode));
auto ptr = ::_wfsopen(path.wstring().c_str(), wmode.c_str(), _SH_DENYNO);
if (ptr == nullptr)
{
ec = std::error_code(GetLastError(), std::generic_category());
}
return CFile{ ptr };
#else
std::string name = path.string();
std::FILE* ptr = std::fopen(name.c_str(), mode);
if (ptr == nullptr)
{
ec = std::error_code(errno, std::generic_category());
}
return CFile{ ptr };
#endif
}
auto CFile::try_open(const mamba::fs::u8path& path, const char* mode)
-> tl::expected<CFile, std::error_code>
{
auto io_error = std::error_code();
auto file_ptr = util::CFile::try_open(path, mode, io_error);
if (io_error)
{
return tl::unexpected(io_error);
}
return { std::move(file_ptr) };
}
void CFile::try_close(std::error_code& ec) noexcept
{
try_close_impl(m_ptr.get(), ec);
[[maybe_unused]] auto raw = m_ptr.release(); // No need to call dtor anymore
}
auto CFile::try_close() noexcept -> tl::expected<void, std::error_code>
{
auto io_error = std::error_code();
try_close(io_error);
if (io_error)
{
return tl::unexpected(io_error);
}
return {};
}
auto CFile::raw() noexcept -> std::FILE*
{
return m_ptr.get();
}
}

10
libmamba/tests/src/core/test_satisfiability_error.cpp
View File

@ -16,12 +16,12 @@
#include "mamba/core/channel_context.hpp"
#include "mamba/core/pool.hpp"
#include "mamba/core/prefix_data.hpp"
#include "mamba/core/repo.hpp"
#include "mamba/core/satisfiability_error.hpp"
#include "mamba/core/solver.hpp"
#include "mamba/core/subdirdata.hpp"
#include "mamba/core/util.hpp"
#include "mamba/fs/filesystem.hpp"
#include "mamba/solver/libsolv/repo_info.hpp"
#include "mamba/specs/package_info.hpp"
#include "mamba/util/random.hpp"
#include "mamba/util/string.hpp"
@ -154,7 +154,7 @@ namespace
const auto repodata_f = create_repodata_json(tmp_dir.path, packages);
auto pool = MPool{ ctx, channel_context };
MRepo(pool, "some-name", repodata_f, RepoMetadata{ /* .url= */ "some-url" });
pool.add_repo_from_repodata_json(repodata_f, "some-url");
auto solver = MSolver(
std::move(pool),
std::vector{ std::pair{ SOLVER_FLAG_ALLOW_DOWNGRADE, 1 } }
@ -361,7 +361,7 @@ namespace
for (auto& sub_dir : sub_dirs)
{
sub_dir.create_repo(pool);
auto repo = load_subdir_in_pool(ctx, pool, sub_dir);
}
}
@ -387,8 +387,8 @@ namespace
);
prefix_data.add_packages(virtual_packages);
auto pool = MPool{ ctx, channel_context };
auto repo = MRepo{ pool, prefix_data };
repo.set_installed();
load_installed_packages_in_pool(ctx, pool, prefix_data);
auto cache = MultiPackageCache({ tmp_dir.path / "cache" }, ctx.validation_params);
create_cache_dir(cache.first_writable_path());

16
libmamba/tests/src/solv-cpp/test_repo.cpp
View File

@ -10,10 +10,12 @@
#include <doctest/doctest.h>
#include "mamba/core/util.hpp"
#include "mamba/util/cfile.hpp"
#include "solv-cpp/pool.hpp"
#include "solv-cpp/repo.hpp"
using namespace mamba::solv;
using namespace mamba;
TEST_SUITE("solv::ObjRepo")
{
@ -158,7 +160,13 @@ TEST_SUITE("solv::ObjRepo")
{
auto dir = mamba::TemporaryDirectory();
const auto solv_file = dir.path() / "test-forge.hpp";
repo.write(solv_file);
{
auto fptr = util::CFile::try_open(solv_file, "wb");
REQUIRE(fptr);
const auto written = repo.write(fptr->raw());
REQUIRE(written);
REQUIRE(fptr->try_close());
}
SUBCASE("Read repo from file")
{
@ -168,7 +176,11 @@ TEST_SUITE("solv::ObjRepo")
// Create new repo from file
auto [repo_id2, repo2] = pool.add_repo("test-forge");
repo2.read(solv_file);
auto fptr = util::CFile::try_open(solv_file, "rb");
REQUIRE(fptr);
const auto read = repo2.read(fptr->raw());
REQUIRE(read);
REQUIRE(fptr->try_close());
CHECK_EQ(repo2.solvable_count(), n_solvables);
// True because we reused ids

6
libmambapy/CMakeLists.txt
View File

@ -26,9 +26,13 @@ find_package(pybind11 REQUIRED)
pybind11_add_module(
bindings
src/libmambapy/bindings/longpath.manifest
# Entry point to all submodules
src/libmambapy/bindings/bindings.cpp
src/libmambapy/bindings/specs.cpp
# All bindings used to live in a global module
src/libmambapy/bindings/legacy.cpp
# Submodules
src/libmambapy/bindings/specs.cpp
src/libmambapy/bindings/solver_libsolv.cpp
)
target_include_directories(bindings PRIVATE src/libmambapy/bindings)

2
libmambapy/setup.py
View File

@ -25,7 +25,7 @@ def libmambapy_version():
skbuild.setup(
version=libmambapy_version(),
packages=["libmambapy", "libmambapy.bindings"],
packages=["libmambapy", "libmambapy.bindings", "libmambapy.solver"],
package_dir={"": "src"},
package_data={"libmambapy": ["py.typed", "__init__.pyi"]},
cmake_languages=["CXX"],

1
libmambapy/src/libmambapy/__init__.py
View File

@ -1,6 +1,7 @@
# Import all submodules so that one can use them directly with `import libmambapy`
import libmambapy.version
import libmambapy.specs
import libmambapy.solver
# Legacy which used to combine everything
from libmambapy.bindings.legacy import * # noqa: F403

1
libmambapy/src/libmambapy/bindings/bindings.cpp
View File

@ -9,5 +9,6 @@
PYBIND11_MODULE(bindings, m)
{
mambapy::bind_submodule_specs(m.def_submodule("specs"));
mambapy::bind_submodule_solver_libsolv(m.def_submodule("solver").def_submodule("libsolv"));
mambapy::bind_submodule_legacy(m.def_submodule("legacy"));
}

1
libmambapy/src/libmambapy/bindings/bindings.hpp
View File

@ -12,6 +12,7 @@
namespace mambapy
{
void bind_submodule_specs(pybind11::module_ m);
void bind_submodule_solver_libsolv(pybind11::module_ m);
void bind_submodule_legacy(pybind11::module_ m);
}
#endif

157
libmambapy/src/libmambapy/bindings/legacy.cpp
View File

@ -27,13 +27,13 @@
#include "mamba/core/pool.hpp"
#include "mamba/core/prefix_data.hpp"
#include "mamba/core/query.hpp"
#include "mamba/core/repo.hpp"
#include "mamba/core/satisfiability_error.hpp"
#include "mamba/core/solver.hpp"
#include "mamba/core/subdirdata.hpp"
#include "mamba/core/transaction.hpp"
#include "mamba/core/util_os.hpp"
#include "mamba/core/virtual_packages.hpp"
#include "mamba/solver/libsolv/repo_info.hpp"
#include "mamba/util/string.hpp"
#include "mamba/validation/tools.hpp"
#include "mamba/validation/update_framework_v0_6.hpp"
@ -57,11 +57,12 @@ namespace query
}
void
deprecated(const char* message)
deprecated(std::string_view message, std::string_view since_version = "1.5")
{
const auto warnings = py::module_::import("warnings");
const auto builtins = py::module_::import("builtins");
warnings.attr("warn")(message, builtins.attr("DeprecationWarning"), py::arg("stacklevel") = 2);
auto total_message = fmt::format("Deprecated since version {}: {}", since_version, message);
warnings.attr("warn")(total_message, builtins.attr("DeprecationWarning"), py::arg("stacklevel") = 2);
}
template <typename PyClass>
@ -263,6 +264,10 @@ bind_submodule_impl(pybind11::module_ m)
{
using namespace mamba;
/***************
* Migrators *
***************/
struct PackageInfoV2Migrator
{
};
@ -283,13 +288,37 @@ bind_submodule_impl(pybind11::module_ m)
py::class_<MatchSpecV2Migrator>(m, "MatchSpec")
.def(py::init(
[](py::args, py::kwargs) -> MatchSpecV2Migrator {
[](py::args, py::kwargs) -> MatchSpecV2Migrator
{
// V2 migration
throw std::runtime_error(
"libmambapy.MatchSpec has been moved to libmambapy.specs.MatchSpec"
);
}
));
struct RepoV2Migrator
{
};
py::class_<RepoV2Migrator>(m, "Repo").def(py::init(
[](py::args, py::kwargs) -> RepoV2Migrator
{
throw std::runtime_error( //
"Use Pool.add_repo_from_repodata_json or Pool.add_repo_from_native_serialization"
" instead and cache with Pool.native_serialize_repo."
" Also consider load_subdir_in_pool for a high_level function to load"
" subdir index and manage cache, and load_installed_packages_in_pool for loading"
" prefix packages."
"The Repo class itself has been moved to libmambapy.solver.libsolv.RepoInfo."
);
}
));
/**************
* Bindings *
**************/
// declare earlier to avoid C++ types in docstrings
auto pyPrefixData = py::class_<PrefixData>(m, "PrefixData");
auto pySolver = py::class_<MSolver>(m, "Solver");
@ -313,7 +342,6 @@ bind_submodule_impl(pybind11::module_ m)
py::add_ostream_redirect(m, "ostream_redirect");
py::class_<MPool>(m, "Pool")
.def(
py::init<>(
@ -327,7 +355,66 @@ bind_submodule_impl(pybind11::module_ m)
.def("create_whatprovides", &MPool::create_whatprovides)
.def("select_solvables", &MPool::select_solvables, py::arg("id"), py::arg("sorted") = false)
.def("matchspec2id", &MPool::matchspec2id, py::arg("spec"))
.def("id2pkginfo", &MPool::id2pkginfo, py::arg("id"));
.def("id2pkginfo", &MPool::id2pkginfo, py::arg("id"))
.def(
"add_repo_from_repodata_json",
&MPool::add_repo_from_repodata_json,
py::arg("path"),
py::arg("url"),
py::arg("add_pip_as_python_dependency") = solver::libsolv::PipAsPythonDependency::No,
py::arg("repodata_parsers") = solver::libsolv::RepodataParser::Mamba
)
.def(
"add_repo_from_native_serialization",
&MPool::add_repo_from_native_serialization,
py::arg("path"),
py::arg("expected"),
py::arg("add_pip_as_python_dependency") = solver::libsolv::PipAsPythonDependency::No
)
.def(
"add_repo_from_packages",
[](MPool& pool,
py::iterable packages,
std::string_view name,
solver::libsolv::PipAsPythonDependency add)
{
// TODO(C++20): No need to copy in a vector, simply transform the input range.
auto pkg_infos = std::vector<specs::PackageInfo>();
for (py::handle pkg : packages)
{
pkg_infos.push_back(pkg.cast<specs::PackageInfo>());
}
return pool.add_repo_from_packages(pkg_infos, name, add);
},
py::arg("packages"),
py::arg("name") = "",
py::arg("add_pip_as_python_dependency") = solver::libsolv::PipAsPythonDependency::No
)
.def(
"native_serialize_repo",
&MPool::native_serialize_repo,
py::arg("repo"),
py::arg("path"),
py::arg("metadata")
)
.def("set_installed_repo", &MPool::set_installed_repo, py::arg("repo"))
.def("set_repo_priority", &MPool::set_repo_priority, py::arg("repo"), py::arg("priorities"));
m.def(
"load_subdir_in_pool",
&load_subdir_in_pool,
py::arg("context"),
py::arg("pool"),
py::arg("subdir")
);
m.def(
"load_installed_packages_in_pool",
&load_installed_packages_in_pool,
py::arg("context"),
py::arg("pool"),
py::arg("prefix_data")
);
py::class_<MultiPackageCache>(m, "MultiPackageCache")
.def(py::init<>(
@ -342,25 +429,6 @@ bind_submodule_impl(pybind11::module_ m)
.def("get_tarball_path", &MultiPackageCache::get_tarball_path)
.def_property_readonly("first_writable_path", &MultiPackageCache::first_writable_path);
py::class_<MRepo::PyExtraPkgInfo>(m, "ExtraPkgInfo")
.def(py::init<>())
.def_readwrite("noarch", &MRepo::PyExtraPkgInfo::noarch)
.def_readwrite("repo_url", &MRepo::PyExtraPkgInfo::repo_url);
py::class_<MRepo>(m, "Repo")
.def(py::init(
[](MPool& pool, const std::string& name, const std::string& filename, const std::string& url
) { return MRepo(pool, name, filename, RepoMetadata{ /* .url=*/url }); }
))
.def(py::init<MPool&, const PrefixData&>())
.def("add_extra_pkg_info", &MRepo::py_add_extra_pkg_info)
.def("set_installed", &MRepo::set_installed)
.def("set_priority", &MRepo::set_priority)
.def("name", &MRepo::py_name)
.def("priority", &MRepo::py_priority)
.def("size", &MRepo::py_size)
.def("clear", &MRepo::py_clear);
py::class_<MTransaction>(m, "Transaction")
.def(py::init<>(
[](MSolver& solver, MultiPackageCache& mpc)
@ -632,13 +700,46 @@ bind_submodule_impl(pybind11::module_ m)
py::class_<MSubdirData>(m, "SubdirData")
.def(
"create_repo",
[](MSubdirData& subdir, MPool& pool) -> MRepo
{ return extract(subdir.create_repo(pool)); }
[](MSubdirData& subdir, MPool& pool) -> solver::libsolv::RepoInfo
{
deprecated("Use `load_subdir_in_pool` instead", "2.0");
return extract(load_subdir_in_pool(mambapy::singletons.context(), pool, subdir));
}
)
.def("loaded", &MSubdirData::is_loaded)
.def(
"valid_solv_cache",
// TODO make a proper well tested type caster for expected types.
[](const MSubdirData& self) -> std::optional<fs::u8path>
{
if (auto f = self.valid_solv_cache())
{
return { *std::move(f) };
}
return std::nullopt;
}
)
.def(
"valid_json_cache",
[](const MSubdirData& self) -> std::optional<fs::u8path>
{
if (auto f = self.valid_json_cache())
{
return { *std::move(f) };
}
return std::nullopt;
}
)
.def(
"cache_path",
[](const MSubdirData& self) -> std::string { return extract(self.cache_path()); }
[](const MSubdirData& self) -> std::string
{
deprecated(
"Use `SubdirData.valid_solv_path` or `SubdirData.valid_json` path instead",
"2.0"
);
return extract(self.cache_path());
}
);
using mambapy::SubdirIndex;

90
libmambapy/src/libmambapy/bindings/solver_libsolv.cpp Normal file
View File

@ -0,0 +1,90 @@
// Copyright (c) 2023, QuantStack and Mamba Contributors
//
// Distributed under the terms of the BSD 3-Clause License.
//
// The full license is in the file LICENSE, distributed with this software.
#include <pybind11/operators.h>
#include <pybind11/pybind11.h>
#include "mamba/solver/libsolv/parameters.hpp"
#include "mamba/solver/libsolv/repo_info.hpp"
#include "bindings.hpp"
#include "utils.hpp"
namespace mambapy
{
void bind_submodule_solver_libsolv(pybind11::module_ m)
{
namespace py = pybind11;
using namespace mamba::solver::libsolv;
py::enum_<RepodataParser>(m, "RepodataParser")
.value("Mamba", RepodataParser::Mamba)
.value("Libsolv", RepodataParser::Libsolv)
.def(py::init(&enum_from_str<RepodataParser>));
py::implicitly_convertible<py::str, RepodataParser>();
py::enum_<PipAsPythonDependency>(m, "PipAsPythonDependency")
.value("No", PipAsPythonDependency::No)
.value("Yes", PipAsPythonDependency::Yes)
.def(py::init([](bool val) { return static_cast<PipAsPythonDependency>(val); }));
py::implicitly_convertible<py::bool_, PipAsPythonDependency>();
py::class_<Priorities>(m, "Priorities")
.def(
py::init(
[](int priority, int subpriority) -> Priorities
{
return {
/* .priority= */ priority,
/* .subpriority= */ subpriority,
};
}
),
py::arg("priority") = 0,
py::arg("subpriority") = 0
)
.def_readwrite("priority", &Priorities::priority)
.def_readwrite("subpriority", &Priorities::subpriority)
.def(py::self == py::self)
.def(py::self != py::self)
.def("__copy__", &copy<Priorities>)
.def("__deepcopy__", &deepcopy<Priorities>, py::arg("memo"));
py::class_<RepodataOrigin>(m, "RepodataOrigin")
.def(
py::init(
[](std::string_view url, std::string_view etag, std::string_view mod) -> RepodataOrigin
{
return {
/* .url= */ std::string(url),
/* .etag= */ std::string(etag),
/* .mod= */ std::string(mod),
};
}
),
py::arg("url") = "",
py::arg("etag") = "",
py::arg("mod") = ""
)
.def_readwrite("url", &RepodataOrigin::url)
.def_readwrite("etag", &RepodataOrigin::etag)
.def_readwrite("mod", &RepodataOrigin::mod)
.def(py::self == py::self)
.def(py::self != py::self)
.def("__copy__", &copy<RepodataOrigin>)
.def("__deepcopy__", &deepcopy<RepodataOrigin>, py::arg("memo"));
py::class_<RepoInfo>(m, "RepoInfo")
.def("id", &RepoInfo::id)
.def("name", &RepoInfo::name)
.def("priority", &RepoInfo::priority)
.def("package_count", &RepoInfo::package_count)
.def(py::self == py::self)
.def(py::self != py::self)
.def("__copy__", &copy<RepoInfo>)
.def("__deepcopy__", &deepcopy<RepoInfo>, py::arg("memo"));
}
}

1
libmambapy/src/libmambapy/solver/__init__.py Normal file
View File

@ -0,0 +1 @@
import libmambapy.solver.libsolv # noqa: F401

2
libmambapy/src/libmambapy/solver/libsolv.py Normal file
View File

@ -0,0 +1,2 @@
# This file exists on its own rather than in `__init__.py` to make `import libmambapy.solver.libsolv` work.
from libmambapy.bindings.solver.libsolv import * # noqa: F403

85
libmambapy/tests/test_solver_libsolv.py Normal file
View File

@ -0,0 +1,85 @@
import copy
import pytest
import libmambapy.solver.libsolv as libsolv
def test_import_submodule():
import libmambapy.solver.libsolv as solv
# Dummy execution
_p = solv.Priorities
def test_import_recursive():
import libmambapy as mamba
# Dummy execution
_p = mamba.solver.libsolv.Priorities
def test_RepodataParser():
assert libsolv.RepodataParser.Mamba.name == "Mamba"
assert libsolv.RepodataParser.Libsolv.name == "Libsolv"
assert libsolv.RepodataParser("Libsolv") == libsolv.RepodataParser.Libsolv
with pytest.raises(KeyError):
libsolv.RepodataParser("NoParser")
def test_PipASPythonDependency():
assert libsolv.PipAsPythonDependency.No.name == "No"
assert libsolv.PipAsPythonDependency.Yes.name == "Yes"
assert libsolv.PipAsPythonDependency(True) == libsolv.PipAsPythonDependency.Yes
def test_Priorities():
p = libsolv.Priorities(priority=-1, subpriority=-2)
assert p.priority == -1
assert p.subpriority == -2
# Setters
p.priority = 33
p.subpriority = 0
assert p.priority == 33
assert p.subpriority == 0
# Operators
assert p == p
assert p != libsolv.Priorities()
# Copy
other = copy.deepcopy(p)
assert other is not p
assert other == p
def test_RepodataOrigin():
orig = libsolv.RepodataOrigin(
url="https://conda.anaconda.org/conda-forge", mod="the-mod", etag="the-etag"
)
assert orig.url == "https://conda.anaconda.org/conda-forge"
assert orig.etag == "the-etag"
assert orig.mod == "the-mod"
# Setters
orig.url = "https://repo.mamba.pm"
orig.etag = "other-etag"
orig.mod = "other-mod"
assert orig.url == "https://repo.mamba.pm"
assert orig.etag == "other-etag"
assert orig.mod == "other-mod"
# Operators
assert orig == orig
assert orig != libsolv.RepodataOrigin()
# Copy
other = copy.deepcopy(orig)
assert other is not orig
assert other == orig

2
micromamba/tests/test_remove.py
View File

@ -110,7 +110,7 @@ def test_remove_in_use(tmp_home, tmp_root_prefix, tmp_xtensor_env, tmp_env_name)
)
time.sleep(1)
helpers.remove("python", "-v", "-p", tmp_xtensor_env, no_dry_run=True)
helpers.remove("python", "-v", "-p", str(tmp_xtensor_env), no_dry_run=True)
if platform.system() == "Windows":
pyexe_trash = Path(str(pyexe) + ".mamba_trash")