798 lines
35 KiB
ReStructuredText
798 lines
35 KiB
ReStructuredText
|
|
====================
|
|
``<atomic>`` Design
|
|
====================
|
|
|
|
There were originally 3 designs under consideration. They differ in where most
|
|
of the implementation work is done. The functionality exposed to the customer
|
|
should be identical (and conforming) for all three designs.
|
|
|
|
|
|
Design A: Minimal work for the library
|
|
======================================
|
|
The compiler supplies all of the intrinsics as described below. This list of
|
|
intrinsics roughly parallels the requirements of the C and C++ atomics proposals.
|
|
The C and C++ library implementations simply drop through to these intrinsics.
|
|
Anything the platform does not support in hardware, the compiler
|
|
arranges for a (compiler-rt) library call to be made which will do the job with
|
|
a mutex, and in this case ignoring the memory ordering parameter (effectively
|
|
implementing ``memory_order_seq_cst``).
|
|
|
|
Ultimate efficiency is preferred over run time error checking. Undefined
|
|
behavior is acceptable when the inputs do not conform as defined below.
|
|
|
|
.. code-block:: cpp
|
|
|
|
// In every intrinsic signature below, type* atomic_obj may be a pointer to a
|
|
// volatile-qualified type. Memory ordering values map to the following meanings:
|
|
// memory_order_relaxed == 0
|
|
// memory_order_consume == 1
|
|
// memory_order_acquire == 2
|
|
// memory_order_release == 3
|
|
// memory_order_acq_rel == 4
|
|
// memory_order_seq_cst == 5
|
|
|
|
// type must be trivially copyable
|
|
// type represents a "type argument"
|
|
bool __atomic_is_lock_free(type);
|
|
|
|
// type must be trivially copyable
|
|
// Behavior is defined for mem_ord = 0, 1, 2, 5
|
|
type __atomic_load(const type* atomic_obj, int mem_ord);
|
|
|
|
// type must be trivially copyable
|
|
// Behavior is defined for mem_ord = 0, 3, 5
|
|
void __atomic_store(type* atomic_obj, type desired, int mem_ord);
|
|
|
|
// type must be trivially copyable
|
|
// Behavior is defined for mem_ord = [0 ... 5]
|
|
type __atomic_exchange(type* atomic_obj, type desired, int mem_ord);
|
|
|
|
// type must be trivially copyable
|
|
// Behavior is defined for mem_success = [0 ... 5],
|
|
// mem_failure <= mem_success
|
|
// mem_failure != 3
|
|
// mem_failure != 4
|
|
bool __atomic_compare_exchange_strong(type* atomic_obj,
|
|
type* expected, type desired,
|
|
int mem_success, int mem_failure);
|
|
|
|
// type must be trivially copyable
|
|
// Behavior is defined for mem_success = [0 ... 5],
|
|
// mem_failure <= mem_success
|
|
// mem_failure != 3
|
|
// mem_failure != 4
|
|
bool __atomic_compare_exchange_weak(type* atomic_obj,
|
|
type* expected, type desired,
|
|
int mem_success, int mem_failure);
|
|
|
|
// type is one of: char, signed char, unsigned char, short, unsigned short, int,
|
|
// unsigned int, long, unsigned long, long long, unsigned long long,
|
|
// char16_t, char32_t, wchar_t
|
|
// Behavior is defined for mem_ord = [0 ... 5]
|
|
type __atomic_fetch_add(type* atomic_obj, type operand, int mem_ord);
|
|
|
|
// type is one of: char, signed char, unsigned char, short, unsigned short, int,
|
|
// unsigned int, long, unsigned long, long long, unsigned long long,
|
|
// char16_t, char32_t, wchar_t
|
|
// Behavior is defined for mem_ord = [0 ... 5]
|
|
type __atomic_fetch_sub(type* atomic_obj, type operand, int mem_ord);
|
|
|
|
// type is one of: char, signed char, unsigned char, short, unsigned short, int,
|
|
// unsigned int, long, unsigned long, long long, unsigned long long,
|
|
// char16_t, char32_t, wchar_t
|
|
// Behavior is defined for mem_ord = [0 ... 5]
|
|
type __atomic_fetch_and(type* atomic_obj, type operand, int mem_ord);
|
|
|
|
// type is one of: char, signed char, unsigned char, short, unsigned short, int,
|
|
// unsigned int, long, unsigned long, long long, unsigned long long,
|
|
// char16_t, char32_t, wchar_t
|
|
// Behavior is defined for mem_ord = [0 ... 5]
|
|
type __atomic_fetch_or(type* atomic_obj, type operand, int mem_ord);
|
|
|
|
// type is one of: char, signed char, unsigned char, short, unsigned short, int,
|
|
// unsigned int, long, unsigned long, long long, unsigned long long,
|
|
// char16_t, char32_t, wchar_t
|
|
// Behavior is defined for mem_ord = [0 ... 5]
|
|
type __atomic_fetch_xor(type* atomic_obj, type operand, int mem_ord);
|
|
|
|
// Behavior is defined for mem_ord = [0 ... 5]
|
|
void* __atomic_fetch_add(void** atomic_obj, ptrdiff_t operand, int mem_ord);
|
|
void* __atomic_fetch_sub(void** atomic_obj, ptrdiff_t operand, int mem_ord);
|
|
|
|
// Behavior is defined for mem_ord = [0 ... 5]
|
|
void __atomic_thread_fence(int mem_ord);
|
|
void __atomic_signal_fence(int mem_ord);
|
|
|
|
If desired the intrinsics taking a single ``mem_ord`` parameter can default
|
|
this argument to 5.
|
|
|
|
If desired the intrinsics taking two ordering parameters can default ``mem_success``
|
|
to 5, and ``mem_failure`` to ``translate_memory_order(mem_success)`` where
|
|
``translate_memory_order(mem_success)`` is defined as:
|
|
|
|
.. code-block:: cpp
|
|
|
|
int translate_memory_order(int o) {
|
|
switch (o) {
|
|
case 4:
|
|
return 2;
|
|
case 3:
|
|
return 0;
|
|
}
|
|
return o;
|
|
}
|
|
|
|
Below are representative C++ implementations of all of the operations. Their
|
|
purpose is to document the desired semantics of each operation, assuming
|
|
``memory_order_seq_cst``. This is essentially the code that will be called
|
|
if the front end calls out to compiler-rt.
|
|
|
|
.. code-block:: cpp
|
|
|
|
template <class T>
|
|
T __atomic_load(T const volatile* obj) {
|
|
unique_lock<mutex> _(some_mutex);
|
|
return *obj;
|
|
}
|
|
|
|
template <class T>
|
|
void __atomic_store(T volatile* obj, T desr) {
|
|
unique_lock<mutex> _(some_mutex);
|
|
*obj = desr;
|
|
}
|
|
|
|
template <class T>
|
|
T __atomic_exchange(T volatile* obj, T desr) {
|
|
unique_lock<mutex> _(some_mutex);
|
|
T r = *obj;
|
|
*obj = desr;
|
|
return r;
|
|
}
|
|
|
|
template <class T>
|
|
bool __atomic_compare_exchange_strong(T volatile* obj, T* exp, T desr) {
|
|
unique_lock<mutex> _(some_mutex);
|
|
if (std::memcmp(const_cast<T*>(obj), exp, sizeof(T)) == 0) // if (*obj == *exp)
|
|
{
|
|
std::memcpy(const_cast<T*>(obj), &desr, sizeof(T)); // *obj = desr;
|
|
return true;
|
|
}
|
|
std::memcpy(exp, const_cast<T*>(obj), sizeof(T)); // *exp = *obj;
|
|
return false;
|
|
}
|
|
|
|
// May spuriously return false (even if *obj == *exp)
|
|
template <class T>
|
|
bool __atomic_compare_exchange_weak(T volatile* obj, T* exp, T desr) {
|
|
unique_lock<mutex> _(some_mutex);
|
|
if (std::memcmp(const_cast<T*>(obj), exp, sizeof(T)) == 0) // if (*obj == *exp)
|
|
{
|
|
std::memcpy(const_cast<T*>(obj), &desr, sizeof(T)); // *obj = desr;
|
|
return true;
|
|
}
|
|
std::memcpy(exp, const_cast<T*>(obj), sizeof(T)); // *exp = *obj;
|
|
return false;
|
|
}
|
|
|
|
template <class T>
|
|
T __atomic_fetch_add(T volatile* obj, T operand) {
|
|
unique_lock<mutex> _(some_mutex);
|
|
T r = *obj;
|
|
*obj += operand;
|
|
return r;
|
|
}
|
|
|
|
template <class T>
|
|
T __atomic_fetch_sub(T volatile* obj, T operand) {
|
|
unique_lock<mutex> _(some_mutex);
|
|
T r = *obj;
|
|
*obj -= operand;
|
|
return r;
|
|
}
|
|
|
|
template <class T>
|
|
T __atomic_fetch_and(T volatile* obj, T operand) {
|
|
unique_lock<mutex> _(some_mutex);
|
|
T r = *obj;
|
|
*obj &= operand;
|
|
return r;
|
|
}
|
|
|
|
template <class T>
|
|
T __atomic_fetch_or(T volatile* obj, T operand) {
|
|
unique_lock<mutex> _(some_mutex);
|
|
T r = *obj;
|
|
*obj |= operand;
|
|
return r;
|
|
}
|
|
|
|
template <class T>
|
|
T __atomic_fetch_xor(T volatile* obj, T operand) {
|
|
unique_lock<mutex> _(some_mutex);
|
|
T r = *obj;
|
|
*obj ^= operand;
|
|
return r;
|
|
}
|
|
|
|
void* __atomic_fetch_add(void* volatile* obj, ptrdiff_t operand) {
|
|
unique_lock<mutex> _(some_mutex);
|
|
void* r = *obj;
|
|
(char*&)(*obj) += operand;
|
|
return r;
|
|
}
|
|
|
|
void* __atomic_fetch_sub(void* volatile* obj, ptrdiff_t operand) {
|
|
unique_lock<mutex> _(some_mutex);
|
|
void* r = *obj;
|
|
(char*&)(*obj) -= operand;
|
|
return r;
|
|
}
|
|
|
|
void __atomic_thread_fence() {
|
|
unique_lock<mutex> _(some_mutex);
|
|
}
|
|
|
|
void __atomic_signal_fence() {
|
|
unique_lock<mutex> _(some_mutex);
|
|
}
|
|
|
|
|
|
Design B: Something in between
|
|
==============================
|
|
This is a variation of design A which puts the burden on the library to arrange
|
|
for the correct manipulation of the run time memory ordering arguments, and only
|
|
calls the compiler for well-defined memory orderings. I think of this design as
|
|
the worst of A and C, instead of the best of A and C. But I offer it as an
|
|
option in the spirit of completeness.
|
|
|
|
.. code-block:: cpp
|
|
|
|
// type must be trivially copyable
|
|
bool __atomic_is_lock_free(const type* atomic_obj);
|
|
|
|
// type must be trivially copyable
|
|
type __atomic_load_relaxed(const volatile type* atomic_obj);
|
|
type __atomic_load_consume(const volatile type* atomic_obj);
|
|
type __atomic_load_acquire(const volatile type* atomic_obj);
|
|
type __atomic_load_seq_cst(const volatile type* atomic_obj);
|
|
|
|
// type must be trivially copyable
|
|
type __atomic_store_relaxed(volatile type* atomic_obj, type desired);
|
|
type __atomic_store_release(volatile type* atomic_obj, type desired);
|
|
type __atomic_store_seq_cst(volatile type* atomic_obj, type desired);
|
|
|
|
// type must be trivially copyable
|
|
type __atomic_exchange_relaxed(volatile type* atomic_obj, type desired);
|
|
type __atomic_exchange_consume(volatile type* atomic_obj, type desired);
|
|
type __atomic_exchange_acquire(volatile type* atomic_obj, type desired);
|
|
type __atomic_exchange_release(volatile type* atomic_obj, type desired);
|
|
type __atomic_exchange_acq_rel(volatile type* atomic_obj, type desired);
|
|
type __atomic_exchange_seq_cst(volatile type* atomic_obj, type desired);
|
|
|
|
// type must be trivially copyable
|
|
bool __atomic_compare_exchange_strong_relaxed_relaxed(volatile type* atomic_obj,
|
|
type* expected,
|
|
type desired);
|
|
bool __atomic_compare_exchange_strong_consume_relaxed(volatile type* atomic_obj,
|
|
type* expected,
|
|
type desired);
|
|
bool __atomic_compare_exchange_strong_consume_consume(volatile type* atomic_obj,
|
|
type* expected,
|
|
type desired);
|
|
bool __atomic_compare_exchange_strong_acquire_relaxed(volatile type* atomic_obj,
|
|
type* expected,
|
|
type desired);
|
|
bool __atomic_compare_exchange_strong_acquire_consume(volatile type* atomic_obj,
|
|
type* expected,
|
|
type desired);
|
|
bool __atomic_compare_exchange_strong_acquire_acquire(volatile type* atomic_obj,
|
|
type* expected,
|
|
type desired);
|
|
bool __atomic_compare_exchange_strong_release_relaxed(volatile type* atomic_obj,
|
|
type* expected,
|
|
type desired);
|
|
bool __atomic_compare_exchange_strong_release_consume(volatile type* atomic_obj,
|
|
type* expected,
|
|
type desired);
|
|
bool __atomic_compare_exchange_strong_release_acquire(volatile type* atomic_obj,
|
|
type* expected,
|
|
type desired);
|
|
bool __atomic_compare_exchange_strong_acq_rel_relaxed(volatile type* atomic_obj,
|
|
type* expected,
|
|
type desired);
|
|
bool __atomic_compare_exchange_strong_acq_rel_consume(volatile type* atomic_obj,
|
|
type* expected,
|
|
type desired);
|
|
bool __atomic_compare_exchange_strong_acq_rel_acquire(volatile type* atomic_obj,
|
|
type* expected,
|
|
type desired);
|
|
bool __atomic_compare_exchange_strong_seq_cst_relaxed(volatile type* atomic_obj,
|
|
type* expected,
|
|
type desired);
|
|
bool __atomic_compare_exchange_strong_seq_cst_consume(volatile type* atomic_obj,
|
|
type* expected,
|
|
type desired);
|
|
bool __atomic_compare_exchange_strong_seq_cst_acquire(volatile type* atomic_obj,
|
|
type* expected,
|
|
type desired);
|
|
bool __atomic_compare_exchange_strong_seq_cst_seq_cst(volatile type* atomic_obj,
|
|
type* expected,
|
|
type desired);
|
|
|
|
// type must be trivially copyable
|
|
bool __atomic_compare_exchange_weak_relaxed_relaxed(volatile type* atomic_obj,
|
|
type* expected,
|
|
type desired);
|
|
bool __atomic_compare_exchange_weak_consume_relaxed(volatile type* atomic_obj,
|
|
type* expected,
|
|
type desired);
|
|
bool __atomic_compare_exchange_weak_consume_consume(volatile type* atomic_obj,
|
|
type* expected,
|
|
type desired);
|
|
bool __atomic_compare_exchange_weak_acquire_relaxed(volatile type* atomic_obj,
|
|
type* expected,
|
|
type desired);
|
|
bool __atomic_compare_exchange_weak_acquire_consume(volatile type* atomic_obj,
|
|
type* expected,
|
|
type desired);
|
|
bool __atomic_compare_exchange_weak_acquire_acquire(volatile type* atomic_obj,
|
|
type* expected,
|
|
type desired);
|
|
bool __atomic_compare_exchange_weak_release_relaxed(volatile type* atomic_obj,
|
|
type* expected,
|
|
type desired);
|
|
bool __atomic_compare_exchange_weak_release_consume(volatile type* atomic_obj,
|
|
type* expected,
|
|
type desired);
|
|
bool __atomic_compare_exchange_weak_release_acquire(volatile type* atomic_obj,
|
|
type* expected,
|
|
type desired);
|
|
bool __atomic_compare_exchange_weak_acq_rel_relaxed(volatile type* atomic_obj,
|
|
type* expected,
|
|
type desired);
|
|
bool __atomic_compare_exchange_weak_acq_rel_consume(volatile type* atomic_obj,
|
|
type* expected,
|
|
type desired);
|
|
bool __atomic_compare_exchange_weak_acq_rel_acquire(volatile type* atomic_obj,
|
|
type* expected,
|
|
type desired);
|
|
bool __atomic_compare_exchange_weak_seq_cst_relaxed(volatile type* atomic_obj,
|
|
type* expected,
|
|
type desired);
|
|
bool __atomic_compare_exchange_weak_seq_cst_consume(volatile type* atomic_obj,
|
|
type* expected,
|
|
type desired);
|
|
bool __atomic_compare_exchange_weak_seq_cst_acquire(volatile type* atomic_obj,
|
|
type* expected,
|
|
type desired);
|
|
bool __atomic_compare_exchange_weak_seq_cst_seq_cst(volatile type* atomic_obj,
|
|
type* expected,
|
|
type desired);
|
|
|
|
// type is one of: char, signed char, unsigned char, short, unsigned short, int,
|
|
// unsigned int, long, unsigned long, long long, unsigned long long,
|
|
// char16_t, char32_t, wchar_t
|
|
type __atomic_fetch_add_relaxed(volatile type* atomic_obj, type operand);
|
|
type __atomic_fetch_add_consume(volatile type* atomic_obj, type operand);
|
|
type __atomic_fetch_add_acquire(volatile type* atomic_obj, type operand);
|
|
type __atomic_fetch_add_release(volatile type* atomic_obj, type operand);
|
|
type __atomic_fetch_add_acq_rel(volatile type* atomic_obj, type operand);
|
|
type __atomic_fetch_add_seq_cst(volatile type* atomic_obj, type operand);
|
|
|
|
// type is one of: char, signed char, unsigned char, short, unsigned short, int,
|
|
// unsigned int, long, unsigned long, long long, unsigned long long,
|
|
// char16_t, char32_t, wchar_t
|
|
type __atomic_fetch_sub_relaxed(volatile type* atomic_obj, type operand);
|
|
type __atomic_fetch_sub_consume(volatile type* atomic_obj, type operand);
|
|
type __atomic_fetch_sub_acquire(volatile type* atomic_obj, type operand);
|
|
type __atomic_fetch_sub_release(volatile type* atomic_obj, type operand);
|
|
type __atomic_fetch_sub_acq_rel(volatile type* atomic_obj, type operand);
|
|
type __atomic_fetch_sub_seq_cst(volatile type* atomic_obj, type operand);
|
|
|
|
// type is one of: char, signed char, unsigned char, short, unsigned short, int,
|
|
// unsigned int, long, unsigned long, long long, unsigned long long,
|
|
// char16_t, char32_t, wchar_t
|
|
type __atomic_fetch_and_relaxed(volatile type* atomic_obj, type operand);
|
|
type __atomic_fetch_and_consume(volatile type* atomic_obj, type operand);
|
|
type __atomic_fetch_and_acquire(volatile type* atomic_obj, type operand);
|
|
type __atomic_fetch_and_release(volatile type* atomic_obj, type operand);
|
|
type __atomic_fetch_and_acq_rel(volatile type* atomic_obj, type operand);
|
|
type __atomic_fetch_and_seq_cst(volatile type* atomic_obj, type operand);
|
|
|
|
// type is one of: char, signed char, unsigned char, short, unsigned short, int,
|
|
// unsigned int, long, unsigned long, long long, unsigned long long,
|
|
// char16_t, char32_t, wchar_t
|
|
type __atomic_fetch_or_relaxed(volatile type* atomic_obj, type operand);
|
|
type __atomic_fetch_or_consume(volatile type* atomic_obj, type operand);
|
|
type __atomic_fetch_or_acquire(volatile type* atomic_obj, type operand);
|
|
type __atomic_fetch_or_release(volatile type* atomic_obj, type operand);
|
|
type __atomic_fetch_or_acq_rel(volatile type* atomic_obj, type operand);
|
|
type __atomic_fetch_or_seq_cst(volatile type* atomic_obj, type operand);
|
|
|
|
// type is one of: char, signed char, unsigned char, short, unsigned short, int,
|
|
// unsigned int, long, unsigned long, long long, unsigned long long,
|
|
// char16_t, char32_t, wchar_t
|
|
type __atomic_fetch_xor_relaxed(volatile type* atomic_obj, type operand);
|
|
type __atomic_fetch_xor_consume(volatile type* atomic_obj, type operand);
|
|
type __atomic_fetch_xor_acquire(volatile type* atomic_obj, type operand);
|
|
type __atomic_fetch_xor_release(volatile type* atomic_obj, type operand);
|
|
type __atomic_fetch_xor_acq_rel(volatile type* atomic_obj, type operand);
|
|
type __atomic_fetch_xor_seq_cst(volatile type* atomic_obj, type operand);
|
|
|
|
void* __atomic_fetch_add_relaxed(void* volatile* atomic_obj, ptrdiff_t operand);
|
|
void* __atomic_fetch_add_consume(void* volatile* atomic_obj, ptrdiff_t operand);
|
|
void* __atomic_fetch_add_acquire(void* volatile* atomic_obj, ptrdiff_t operand);
|
|
void* __atomic_fetch_add_release(void* volatile* atomic_obj, ptrdiff_t operand);
|
|
void* __atomic_fetch_add_acq_rel(void* volatile* atomic_obj, ptrdiff_t operand);
|
|
void* __atomic_fetch_add_seq_cst(void* volatile* atomic_obj, ptrdiff_t operand);
|
|
|
|
void* __atomic_fetch_sub_relaxed(void* volatile* atomic_obj, ptrdiff_t operand);
|
|
void* __atomic_fetch_sub_consume(void* volatile* atomic_obj, ptrdiff_t operand);
|
|
void* __atomic_fetch_sub_acquire(void* volatile* atomic_obj, ptrdiff_t operand);
|
|
void* __atomic_fetch_sub_release(void* volatile* atomic_obj, ptrdiff_t operand);
|
|
void* __atomic_fetch_sub_acq_rel(void* volatile* atomic_obj, ptrdiff_t operand);
|
|
void* __atomic_fetch_sub_seq_cst(void* volatile* atomic_obj, ptrdiff_t operand);
|
|
|
|
void __atomic_thread_fence_relaxed();
|
|
void __atomic_thread_fence_consume();
|
|
void __atomic_thread_fence_acquire();
|
|
void __atomic_thread_fence_release();
|
|
void __atomic_thread_fence_acq_rel();
|
|
void __atomic_thread_fence_seq_cst();
|
|
|
|
void __atomic_signal_fence_relaxed();
|
|
void __atomic_signal_fence_consume();
|
|
void __atomic_signal_fence_acquire();
|
|
void __atomic_signal_fence_release();
|
|
void __atomic_signal_fence_acq_rel();
|
|
void __atomic_signal_fence_seq_cst();
|
|
|
|
Design C: Minimal work for the front end
|
|
========================================
|
|
The ``<atomic>`` header is one of the most closely coupled headers to the compiler.
|
|
Ideally when you invoke any function from ``<atomic>``, it should result in highly
|
|
optimized assembly being inserted directly into your application -- assembly that
|
|
is not otherwise representable by higher level C or C++ expressions. The design of
|
|
the libc++ ``<atomic>`` header started with this goal in mind. A secondary, but
|
|
still very important goal is that the compiler should have to do minimal work to
|
|
facilitate the implementation of ``<atomic>``. Without this second goal, then
|
|
practically speaking, the libc++ ``<atomic>`` header would be doomed to be a
|
|
barely supported, second class citizen on almost every platform.
|
|
|
|
Goals:
|
|
|
|
- Optimal code generation for atomic operations
|
|
- Minimal effort for the compiler to achieve goal 1 on any given platform
|
|
- Conformance to the C++0X draft standard
|
|
|
|
The purpose of this document is to inform compiler writers what they need to do
|
|
to enable a high performance libc++ ``<atomic>`` with minimal effort.
|
|
|
|
The minimal work that must be done for a conforming ``<atomic>``
|
|
----------------------------------------------------------------
|
|
The only "atomic" operations that must actually be lock free in
|
|
``<atomic>`` are represented by the following compiler intrinsics:
|
|
|
|
.. code-block:: cpp
|
|
|
|
__atomic_flag__ __atomic_exchange_seq_cst(__atomic_flag__ volatile* obj, __atomic_flag__ desr) {
|
|
unique_lock<mutex> _(some_mutex);
|
|
__atomic_flag__ result = *obj;
|
|
*obj = desr;
|
|
return result;
|
|
}
|
|
|
|
void __atomic_store_seq_cst(__atomic_flag__ volatile* obj, __atomic_flag__ desr) {
|
|
unique_lock<mutex> _(some_mutex);
|
|
*obj = desr;
|
|
}
|
|
|
|
Where:
|
|
|
|
- If ``__has_feature(__atomic_flag)`` evaluates to 1 in the preprocessor then
|
|
the compiler must define ``__atomic_flag__`` (e.g. as a typedef to ``int``).
|
|
- If ``__has_feature(__atomic_flag)`` evaluates to 0 in the preprocessor then
|
|
the library defines ``__atomic_flag__`` as a typedef to ``bool``.
|
|
- To communicate that the above intrinsics are available, the compiler must
|
|
arrange for ``__has_feature`` to return 1 when fed the intrinsic name
|
|
appended with an '_' and the mangled type name of ``__atomic_flag__``.
|
|
|
|
For example if ``__atomic_flag__`` is ``unsigned int``:
|
|
|
|
.. code-block:: cpp
|
|
|
|
// __has_feature(__atomic_flag) == 1
|
|
// __has_feature(__atomic_exchange_seq_cst_j) == 1
|
|
// __has_feature(__atomic_store_seq_cst_j) == 1
|
|
|
|
typedef unsigned int __atomic_flag__;
|
|
|
|
unsigned int __atomic_exchange_seq_cst(unsigned int volatile*, unsigned int) {
|
|
// ...
|
|
}
|
|
|
|
void __atomic_store_seq_cst(unsigned int volatile*, unsigned int) {
|
|
// ...
|
|
}
|
|
|
|
That's it! Compiler writers do the above and you've got a fully conforming
|
|
(though sub-par performance) ``<atomic>`` header!
|
|
|
|
|
|
Recommended work for a higher performance ``<atomic>``
|
|
------------------------------------------------------
|
|
It would be good if the above intrinsics worked with all integral types plus
|
|
``void*``. Because this may not be possible to do in a lock-free manner for
|
|
all integral types on all platforms, a compiler must communicate each type that
|
|
an intrinsic works with. For example, if ``__atomic_exchange_seq_cst`` works
|
|
for all types except for ``long long`` and ``unsigned long long`` then:
|
|
|
|
.. code-block:: cpp
|
|
|
|
__has_feature(__atomic_exchange_seq_cst_b) == 1 // bool
|
|
__has_feature(__atomic_exchange_seq_cst_c) == 1 // char
|
|
__has_feature(__atomic_exchange_seq_cst_a) == 1 // signed char
|
|
__has_feature(__atomic_exchange_seq_cst_h) == 1 // unsigned char
|
|
__has_feature(__atomic_exchange_seq_cst_Ds) == 1 // char16_t
|
|
__has_feature(__atomic_exchange_seq_cst_Di) == 1 // char32_t
|
|
__has_feature(__atomic_exchange_seq_cst_w) == 1 // wchar_t
|
|
__has_feature(__atomic_exchange_seq_cst_s) == 1 // short
|
|
__has_feature(__atomic_exchange_seq_cst_t) == 1 // unsigned short
|
|
__has_feature(__atomic_exchange_seq_cst_i) == 1 // int
|
|
__has_feature(__atomic_exchange_seq_cst_j) == 1 // unsigned int
|
|
__has_feature(__atomic_exchange_seq_cst_l) == 1 // long
|
|
__has_feature(__atomic_exchange_seq_cst_m) == 1 // unsigned long
|
|
__has_feature(__atomic_exchange_seq_cst_Pv) == 1 // void*
|
|
|
|
Note that only the ``__has_feature`` flag is decorated with the argument
|
|
type. The name of the compiler intrinsic is not decorated, but instead works
|
|
like a C++ overloaded function.
|
|
|
|
Additionally, there are other intrinsics besides ``__atomic_exchange_seq_cst``
|
|
and ``__atomic_store_seq_cst``. They are optional. But if the compiler can
|
|
generate faster code than provided by the library, then clients will benefit
|
|
from the compiler writer's expertise and knowledge of the targeted platform.
|
|
|
|
Below is the complete list of *sequentially consistent* intrinsics, and
|
|
their library implementations. Template syntax is used to indicate the desired
|
|
overloading for integral and ``void*`` types. The template does not represent a
|
|
requirement that the intrinsic operate on **any** type!
|
|
|
|
.. code-block:: cpp
|
|
|
|
// T is one of:
|
|
// bool, char, signed char, unsigned char, short, unsigned short,
|
|
// int, unsigned int, long, unsigned long,
|
|
// long long, unsigned long long, char16_t, char32_t, wchar_t, void*
|
|
|
|
template <class T>
|
|
T __atomic_load_seq_cst(T const volatile* obj) {
|
|
unique_lock<mutex> _(some_mutex);
|
|
return *obj;
|
|
}
|
|
|
|
template <class T>
|
|
void __atomic_store_seq_cst(T volatile* obj, T desr) {
|
|
unique_lock<mutex> _(some_mutex);
|
|
*obj = desr;
|
|
}
|
|
|
|
template <class T>
|
|
T __atomic_exchange_seq_cst(T volatile* obj, T desr) {
|
|
unique_lock<mutex> _(some_mutex);
|
|
T r = *obj;
|
|
*obj = desr;
|
|
return r;
|
|
}
|
|
|
|
template <class T>
|
|
bool __atomic_compare_exchange_strong_seq_cst_seq_cst(T volatile* obj, T* exp, T desr) {
|
|
unique_lock<mutex> _(some_mutex);
|
|
if (std::memcmp(const_cast<T*>(obj), exp, sizeof(T)) == 0) {
|
|
std::memcpy(const_cast<T*>(obj), &desr, sizeof(T));
|
|
return true;
|
|
}
|
|
std::memcpy(exp, const_cast<T*>(obj), sizeof(T));
|
|
return false;
|
|
}
|
|
|
|
template <class T>
|
|
bool __atomic_compare_exchange_weak_seq_cst_seq_cst(T volatile* obj, T* exp, T desr) {
|
|
unique_lock<mutex> _(some_mutex);
|
|
if (std::memcmp(const_cast<T*>(obj), exp, sizeof(T)) == 0)
|
|
{
|
|
std::memcpy(const_cast<T*>(obj), &desr, sizeof(T));
|
|
return true;
|
|
}
|
|
std::memcpy(exp, const_cast<T*>(obj), sizeof(T));
|
|
return false;
|
|
}
|
|
|
|
// T is one of:
|
|
// char, signed char, unsigned char, short, unsigned short,
|
|
// int, unsigned int, long, unsigned long,
|
|
// long long, unsigned long long, char16_t, char32_t, wchar_t
|
|
|
|
template <class T>
|
|
T __atomic_fetch_add_seq_cst(T volatile* obj, T operand) {
|
|
unique_lock<mutex> _(some_mutex);
|
|
T r = *obj;
|
|
*obj += operand;
|
|
return r;
|
|
}
|
|
|
|
template <class T>
|
|
T __atomic_fetch_sub_seq_cst(T volatile* obj, T operand) {
|
|
unique_lock<mutex> _(some_mutex);
|
|
T r = *obj;
|
|
*obj -= operand;
|
|
return r;
|
|
}
|
|
|
|
template <class T>
|
|
T __atomic_fetch_and_seq_cst(T volatile* obj, T operand) {
|
|
unique_lock<mutex> _(some_mutex);
|
|
T r = *obj;
|
|
*obj &= operand;
|
|
return r;
|
|
}
|
|
|
|
template <class T>
|
|
T __atomic_fetch_or_seq_cst(T volatile* obj, T operand) {
|
|
unique_lock<mutex> _(some_mutex);
|
|
T r = *obj;
|
|
*obj |= operand;
|
|
return r;
|
|
}
|
|
|
|
template <class T>
|
|
T __atomic_fetch_xor_seq_cst(T volatile* obj, T operand) {
|
|
unique_lock<mutex> _(some_mutex);
|
|
T r = *obj;
|
|
*obj ^= operand;
|
|
return r;
|
|
}
|
|
|
|
void* __atomic_fetch_add_seq_cst(void* volatile* obj, ptrdiff_t operand) {
|
|
unique_lock<mutex> _(some_mutex);
|
|
void* r = *obj;
|
|
(char*&)(*obj) += operand;
|
|
return r;
|
|
}
|
|
|
|
void* __atomic_fetch_sub_seq_cst(void* volatile* obj, ptrdiff_t operand) {
|
|
unique_lock<mutex> _(some_mutex);
|
|
void* r = *obj;
|
|
(char*&)(*obj) -= operand;
|
|
return r;
|
|
}
|
|
|
|
void __atomic_thread_fence_seq_cst() {
|
|
unique_lock<mutex> _(some_mutex);
|
|
}
|
|
|
|
void __atomic_signal_fence_seq_cst() {
|
|
unique_lock<mutex> _(some_mutex);
|
|
}
|
|
|
|
One should consult the (currently draft) `C++ Standard <https://wg21.link/n3126>`_
|
|
for the details of the definitions for these operations. For example,
|
|
``__atomic_compare_exchange_weak_seq_cst_seq_cst`` is allowed to fail
|
|
spuriously while ``__atomic_compare_exchange_strong_seq_cst_seq_cst`` is not.
|
|
|
|
If on your platform the lock-free definition of ``__atomic_compare_exchange_weak_seq_cst_seq_cst``
|
|
would be the same as ``__atomic_compare_exchange_strong_seq_cst_seq_cst``, you may omit the
|
|
``__atomic_compare_exchange_weak_seq_cst_seq_cst`` intrinsic without a performance cost. The
|
|
library will prefer your implementation of ``__atomic_compare_exchange_strong_seq_cst_seq_cst``
|
|
over its own definition for implementing ``__atomic_compare_exchange_weak_seq_cst_seq_cst``.
|
|
That is, the library will arrange for ``__atomic_compare_exchange_weak_seq_cst_seq_cst`` to call
|
|
``__atomic_compare_exchange_strong_seq_cst_seq_cst`` if you supply an intrinsic for the strong
|
|
version but not the weak.
|
|
|
|
Taking advantage of weaker memory synchronization
|
|
-------------------------------------------------
|
|
So far, all of the intrinsics presented require a **sequentially consistent** memory ordering.
|
|
That is, no loads or stores can move across the operation (just as if the library had locked
|
|
that internal mutex). But ``<atomic>`` supports weaker memory ordering operations. In all,
|
|
there are six memory orderings (listed here from strongest to weakest):
|
|
|
|
.. code-block:: cpp
|
|
|
|
memory_order_seq_cst
|
|
memory_order_acq_rel
|
|
memory_order_release
|
|
memory_order_acquire
|
|
memory_order_consume
|
|
memory_order_relaxed
|
|
|
|
(See the `C++ Standard <https://wg21.link/n3126>`_ for the detailed definitions of each of these orderings).
|
|
|
|
On some platforms, the compiler vendor can offer some or even all of the above
|
|
intrinsics at one or more weaker levels of memory synchronization. This might
|
|
lead for example to not issuing an ``mfence`` instruction on the x86.
|
|
|
|
If the compiler does not offer any given operation, at any given memory ordering
|
|
level, the library will automatically attempt to call the next highest memory
|
|
ordering operation. This continues up to ``seq_cst``, and if that doesn't
|
|
exist, then the library takes over and does the job with a ``mutex``. This
|
|
is a compile-time search and selection operation. At run time, the application
|
|
will only see the few inlined assembly instructions for the selected intrinsic.
|
|
|
|
Each intrinsic is appended with the 7-letter name of the memory ordering it
|
|
addresses. For example a ``load`` with ``relaxed`` ordering is defined by:
|
|
|
|
.. code-block:: cpp
|
|
|
|
T __atomic_load_relaxed(const volatile T* obj);
|
|
|
|
And announced with:
|
|
|
|
.. code-block:: cpp
|
|
|
|
__has_feature(__atomic_load_relaxed_b) == 1 // bool
|
|
__has_feature(__atomic_load_relaxed_c) == 1 // char
|
|
__has_feature(__atomic_load_relaxed_a) == 1 // signed char
|
|
...
|
|
|
|
The ``__atomic_compare_exchange_strong(weak)`` intrinsics are parameterized
|
|
on two memory orderings. The first ordering applies when the operation returns
|
|
``true`` and the second ordering applies when the operation returns ``false``.
|
|
|
|
Not every memory ordering is appropriate for every operation. ``exchange``
|
|
and the ``fetch_XXX`` operations support all 6. But ``load`` only supports
|
|
``relaxed``, ``consume``, ``acquire`` and ``seq_cst``. ``store`` only supports
|
|
``relaxed``, ``release``, and ``seq_cst``. The ``compare_exchange`` operations
|
|
support the following 16 combinations out of the possible 36:
|
|
|
|
.. code-block:: cpp
|
|
|
|
relaxed_relaxed
|
|
consume_relaxed
|
|
consume_consume
|
|
acquire_relaxed
|
|
acquire_consume
|
|
acquire_acquire
|
|
release_relaxed
|
|
release_consume
|
|
release_acquire
|
|
acq_rel_relaxed
|
|
acq_rel_consume
|
|
acq_rel_acquire
|
|
seq_cst_relaxed
|
|
seq_cst_consume
|
|
seq_cst_acquire
|
|
seq_cst_seq_cst
|
|
|
|
Again, the compiler supplies intrinsics only for the strongest orderings where
|
|
it can make a difference. The library takes care of calling the weakest
|
|
supplied intrinsic that is as strong or stronger than the customer asked for.
|
|
|
|
Note about ABI
|
|
==============
|
|
With any design, the (back end) compiler writer should note that the decision to
|
|
implement lock-free operations on any given type (or not) is an ABI-binding decision.
|
|
One can not change from treating a type as not lock free, to lock free (or vice-versa)
|
|
without breaking your ABI.
|
|
|
|
For example:
|
|
|
|
**TU1.cpp**:
|
|
|
|
.. code-block:: cpp
|
|
|
|
extern atomic<long long> A;
|
|
int foo() { return A.compare_exchange_strong(w, x); }
|
|
|
|
|
|
**TU2.cpp**:
|
|
|
|
.. code-block:: cpp
|
|
|
|
extern atomic<long long> A;
|
|
void bar() { return A.compare_exchange_strong(y, z); }
|
|
|
|
If only **one** of these calls to ``compare_exchange_strong`` is implemented with
|
|
mutex-locked code, then that mutex-locked code will not be executed mutually
|
|
exclusively of the one implemented in a lock-free manner.
|