terapines
/
circt
mirror of https://github.com/llvm/circt.git
2
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

[ESI] Remove last references to capnp (#7315) 

The cosim document was 90% out of date. Added a note regarding
documentation status.
This commit is contained in:
John Demme 2024-07-12 06:36:10 -07:00 committed by GitHub
parent b68c01def2
commit 462f6b5934
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
16 changed files with 22 additions and 227 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
.github/workflows/pycdePublish.yml vendored
View File

@ -111,7 +111,7 @@ jobs:
- name: Build additional c++ deps
shell: pwsh
run: |
& "${env:VCPKG_INSTALLATION_ROOT}/vcpkg" --triplet x64-windows install zlib capnproto
& "${env:VCPKG_INSTALLATION_ROOT}/vcpkg" --triplet x64-windows install zlib
- name: Get CIRCT
uses: actions/checkout@v4

3
docs/Dialects/ESI/_index.md
View File

@ -2,6 +2,9 @@
The Elastic Silicon Interconnect dialect aims to aid in accelerator system construction.
**WARNING**: The ESI dialect has evolved significantly since its inception while
these documents have not. As such, large parts are significantly out-of-date.
[TOC]
## Application channels

166
docs/Dialects/ESI/cosim.md
View File

@ -1,166 +0,0 @@
# ESI cosimulation model
Elastic Silicon Interfaces provides a feature called cosimulation. Cosim in
general allows communication between the simulation and software. In the ESI
case, it is typed and can be used to build an application and language
specific API which is nearly identical to how the real hardware would
interface. This allows users to simulate against the actual target software
(or some simplification of it), enabling easier co-design.
ESI cosim uses [Cap'nProto](https://capnproto.org/) as a message format and
RPC client/server. Capnp was chosen due to its relatively low-overhead
encoding/decoding (as compared to Protocol Buffers and ilk) is a good fit for
hardware. Using a standard messaging protocol allows users to work with a
variety of languages: the Cap'nProto website lists C++, C#, Erlang, Go,
Haskell, JavaScript, Ocaml, Python, and Rust as languages which support
messages and RPC!
Status: **production**
Documentation status: **out of date** needs to be updated to reflect the new RPC
interface and library.
## Usage
To interface with RTL simulators, the [DPI
interface](https://en.wikipedia.org/wiki/SystemVerilog_DPI) is used. ESI
cosim builds and provides a set of SystemVerilog sources and a shared library
which implement both DPI sides. The shared library (C++) starts a capnp RPC
server for client(s) to connect to and interface with the simulations.
### Generating a system-specific schema
ESI has the capability to generate a Cap'nProto schema customized to an ESI
system. Run command below on an MLIR assembly file with `esi.cosim` ops. It
will find all of the cosim ops and output a capnp schema struct for each
input and output type.
`circt-translate <esi_system.mlir> -export-esi-capnp`
Comments in the generated file indicate the type converted from. In cases
where the ESI type is smaller than the capnp type (e.g. `i5` vs `UInt8`), the
ESI-generated conversion gasket will simply ignore the extra bits.
The struct IDs (`0x<hexStructID>`) will match the `TypeID`s in the
`EsiDpiInterfaceDesc` which dynamically describes each endpoint, described
below.
### Endpoints
ESI cosim works through a notion of *endpoints* -- typed, bi-directional
cosim bridges which are exposed over RPC. Endpoints are registered with the
RPC interface.
On the RTL side, we provide a SystemVerilog module (`Cosim_Endpoint`) which
provides a simple interface to the client. The modules instances take care of
registering themselves. `DataOut` and `DataIn` carry the raw capnp messages
with corresponding control signals. At present, we only support fixed-size
messages.
```systemverilog
module Cosim_Endpoint
#(
parameter int ENDPOINT_ID = -1,
parameter longint ESI_TYPE_ID = -1,
parameter int TYPE_SIZE_BITS = -1
)
(
input logic clk,
input logic rst,
output logic DataOutValid,
input logic DataOutReady,
output logic[TYPE_SIZE_BITS-1:0] DataOut,
input logic DataInValid,
output logic DataInReady,
input logic [TYPE_SIZE_BITS-1:0] DataIn
);
```
The RPC interface allows clients to query all the registered endpoints, grab
a reference to one, and send/receive messages and/or raw data. Once one
client opens an Endpoint, it is locked until said client closes it.
```capnp
interface CosimDpiServer {
list @0 () -> (ifaces :List(EsiDpiInterfaceDesc));
open @1 [S, T] (iface :EsiDpiInterfaceDesc) -> (iface :EsiDpiEndpoint(S, T));
}
struct EsiDpiInterfaceDesc {
sendTypeID @0 :UInt64;
recvTypeID @1 :UInt64;
endpointID @2 :Int32;
}
interface EsiDpiEndpoint(SendMsgType, RecvMsgType) {
send @0 (msg :SendMsgType);
recv @1 (block :Bool = true) -> (hasData :Bool, resp :RecvMsgType); # If 'resp' null, no data
close @2 ();
}
struct UntypedData {
data @0 :Data;
}
```
This RPC interface can be used from any supported language. Here's an example for Python:
```py
import capnp
class LoopbackTester:
def __init__(self, schemaPath):
self.dpi = capnp.load(schemaPath)
hostname = os.uname()[1]
self.rpc_client = capnp.TwoPartyClient(f"{hostname}:1111")
self.cosim = self.rpc_client.bootstrap().cast_as(self.dpi.CosimDpiServer)
def openEP(self):
ifaces = self.cosim.list().wait().ifaces
openResp = self.cosim.open(ifaces[0]).wait()
assert openResp.iface is not None
return openResp.iface
def write(self, ep):
r = random.randrange(0, 2**24)
data = r.to_bytes(3, 'big')
print(f'Sending: {binascii.hexlify(data)}')
ep.send(self.dpi.UntypedData.new_message(data=data)).wait()
return data
def read(self, ep):
while True:
recvResp = ep.recv(False).wait()
if recvResp.hasData:
break
else:
time.sleep(0.1)
assert recvResp.resp is not None
dataMsg = recvResp.resp.as_struct(self.dpi.UntypedData)
data = dataMsg.data
print(binascii.hexlify(data))
return data
def write_read(self):
ep = self.openEP()
print("Testing writes")
dataSent = self.write(ep)
print()
print("Testing reads")
dataRecv = self.read(ep)
ep.close().wait()
assert dataSent == dataRecv
```
## Implementation of the RPC server DPI plugin
In short, an instance of `Cosim_Endpoint` registers itself. The first
registration starts the RPC server (or it can be started via a direct dpi
call). Starting the RPC server involves spining up a thread in which the RPC
server runs. Communication between the simulator thread(s) and the RPC server
thread is through per-endpoint, thread-safe queues. The DPI functions poll
for incoming data or push outgoing data to/from said queues. There is no flow
control yet so it is currently very easy to bloat the infinitely-sized
queues. For the time being, flow-contol has be handled at a higher level.

15
docs/GettingStarted.md
View File

@ -148,16 +148,13 @@ dependencies are installed on your system. They are:
- libfl2 # Ubuntu only (ignore if gives error)
- libfl-dev # Ubuntu only (ignore if gives error)
7) **Install Cap'nProto** (optional, affects ESI dialect only)
7) **Install GRPC** (optional, affects ESI runtime only)
The ESI runtime requires GRPC for cosimulation. The `utils/get-grpc.sh` script
installs a known good version of GRPC to a directory within the CIRCT source
code. Alternatively, you can install GRPC using your package manager, though the
version may not be compatible with the ESI runtime so results may vary.
Some of the ESI dialect code requires [libcapnp](https://capnproto.org/), 0.9.1 or newer.
(Specifically, the [cosimulation](Dialects/ESI/cosim.md) component.) Most of
the ESI cosim integration tests also require the python bindings: pycapnp.
The `utils/get-capnp.sh` script downloads, compiles, and installs a known
good version to a directory within the circt source code. It optionally
installs pycapnp via 'pip3'. The capnp compile requires libtool.
Alternatively, you can use a docker image we provide via
`utils/run-docker.sh`.
8) **Install OR-Tools** (optional, enables additional schedulers)

5
docs/PyCDE/compiling.md
View File

@ -66,12 +66,11 @@ python -m pip install -r frontends/PyCDE/python/requirements.txt
Although not scrictly needed for PyCDE develoment, scripts for some tools you
might want to install are located in utils/
(Cap'n Proto, Verilator, OR-Tools):
(Verilator, OR-Tools):
```bash
utils/get-capnp.sh
utils/get-verilator.sh
utils/get-or-tools
utils/get-or-tools.sh
```
Install PyCDE with CMake. PyCDE requires cmake version >= 3.21:

28
frontends/PyCDE/auditwheel.sh
View File

@ -1,28 +0,0 @@
#!/usr/bin/env bash
##===- Run auditwheels with special options ------------------*- Script -*-===##
#
# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
# See https://llvm.org/LICENSE.txt for license information.
# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
#
##===----------------------------------------------------------------------===##
#
# Run auditwheels, but filter out all the shared objects in the 'collateral'
# directory. Those aren't run by pycde, just copied to build packages. Called by
# cibuildwheel.
#
##===----------------------------------------------------------------------===##
set -e
EXCLUDES="
--exclude libcapnp-0.9.1.so
--exclude libcapnp-rpc-0.9.1.so
--exclude libkj-0.9.1.so
--exclude libkj-async-0.9.1.so
--exclude libkj-gzip-0.9.1.so
--exclude libMtiPli.so
--exclude libEsiCosimDpiServer.so"
echo auditwheel repair -w $1 $2 $EXCLUDES
auditwheel repair -w $1 $2 $EXCLUDES

1
frontends/PyCDE/integration_test/lit.site.cfg.py.in
View File

@ -44,7 +44,6 @@ config.yosys_path = "@YOSYS_PATH@"
config.quartus_path = "@QUARTUS_PATH@"
config.vivado_path = "@VIVADO_PATH@"
config.questa_path = "@QUESTA_PATH@"
config.esi_capnp = "@ESI_CAPNP@"
config.esi_runtime = "@ESI_RUNTIME@"
config.esi_runtime_path = "@ESIRuntimePath@"
config.iverilog_path = "@IVERILOG_PATH@"

3
frontends/PyCDE/pyproject.toml
View File

@ -30,7 +30,6 @@ manylinux-x86_64-image = "ghcr.io/circt/images/pycde-build"
[tool.cibuildwheel.linux]
# Use our internal auditwheel script so as to not mess up the collateral.
repair-wheel-command = "frontends/PyCDE/auditwheel.sh {dest_dir} {wheel}"
environment-pass = [
"SCCACHE_GHA_ENABLED",
"ACTIONS_CACHE_URL",
@ -39,7 +38,7 @@ environment-pass = [
"SETUPTOOLS_SCM_DEBUG",
"BUILD_TYPE",
"RUN_TESTS",
"COMPILER_LAUNCHER"
"COMPILER_LAUNCHER",
]
[project]

6
frontends/PyCDE/setup.py
View File

@ -67,8 +67,12 @@ class CMakeBuild(build_py):
"-DCIRCT_ENABLE_FRONTENDS=PyCDE",
"-DLLVM_EXTERNAL_PROJECTS=circt",
"-DLLVM_EXTERNAL_CIRCT_SOURCE_DIR={}".format(circt_dir),
"-DESI_RUNTIME=ON",
]
# ESI runtime not currently supported on Windows.
if os.name == "nt":
cmake_args += ["-DESI_RUNTIME=OFF"]
else:
cmake_args += ["-DESI_RUNTIME=ON"]
if "COMPILER_LAUNCHER" in os.environ:
cmake_args += [
f"-DCMAKE_C_COMPILER_LAUNCHER={os.environ['COMPILER_LAUNCHER']}",

6
frontends/PyCDE/src/CMakeLists.txt
View File

@ -107,9 +107,3 @@ install(IMPORTED_RUNTIME_ARTIFACTS PyCDE_CIRCTPythonCAPI
DESTINATION python_packages/pycde/circt/_mlir_libs
COMPONENT PyCDE
)
install(RUNTIME_DEPENDENCY_SET PyCDE_RUNTIME_DEPS
DESTINATION python_packages/pycde/circt/_mlir_libs
PRE_EXCLUDE_REGEXES .*
PRE_INCLUDE_REGEXES capnp kj
COMPONENT PyCDE
)

1
frontends/PyCDE/test/lit.site.cfg.py.in
View File

@ -44,7 +44,6 @@ config.yosys_path = "@YOSYS_PATH@"
config.quartus_path = "@QUARTUS_PATH@"
config.vivado_path = "@VIVADO_PATH@"
config.questa_path = "@QUESTA_PATH@"
config.esi_capnp = "@ESI_CAPNP@"
config.iverilog_path = "@IVERILOG_PATH@"
config.bindings_python_enabled = @CIRCT_BINDINGS_PYTHON_ENABLED@

4
lib/Dialect/ESI/runtime/cosim_dpi_server/Cosim_DpiPkg.sv
View File

@ -32,11 +32,11 @@ import "DPI-C" sv2cCosimserverEpRegister =
function int cosim_ep_register(
// The endpoint ID.
input string endpoint_id,
// The capnp type id which the _RPC client_ is sending us.
// The ESI type id which the _RPC client_ is sending us.
input string from_host_type_id,
// The send types max size, in bytes.
input int from_host_type_size,
// The capnp type id which we are sending to the _RPC client_.
// The ESI type id which we are sending to the _RPC client_.
input string to_host_type_id,
// The recv types max size, in bytes.
input int to_host_type_size);

2
lib/Dialect/ESI/runtime/cosim_dpi_server/DpiEntryPoints.cpp
View File

@ -48,7 +48,7 @@ static void log(char *epId, bool toClient, const MessageData &msg) {
// Separate 32-bit words.
if (i % 4 == 0 && i > 0)
fprintf(logFile, " ");
// Separate 64-bit words (capnp word size)
// Separate 64-bit words
if (i % 8 == 0 && i > 0)
fprintf(logFile, " ");
fprintf(logFile, " %02x", b);

2
lib/Dialect/ESI/runtime/cpp/lib/backends/Cosim.cpp
View File

@ -1,4 +1,4 @@
//===- Cosim.cpp - Connection to ESI simulation via capnp RPC -------------===//
//===- Cosim.cpp - Connection to ESI simulation via GRPC ------------------===//
//
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.

4
test/lit.cfg.py
View File

@ -68,10 +68,6 @@ if config.verilator_path != "":
tools.append('verilator')
config.available_features.add('verilator')
# Enable ESI's Capnp tests if they're supported.
if config.esi_capnp != "":
config.available_features.add('capnp')
if config.zlib == "1":
config.available_features.add('zlib')

1
test/lit.site.cfg.py.in
View File

@ -36,7 +36,6 @@ config.circt_obj_root = "@CIRCT_BINARY_DIR@"
config.circt_tools_dir = "@CIRCT_TOOLS_DIR@"
config.circt_shlib_dir = "@LLVM_LIBRARY_OUTPUT_INTDIR@"
config.verilator_path = "@VERILATOR_PATH@"
config.esi_capnp = "@ESI_CAPNP@"
config.zlib = "@HAVE_ZLIB@"
config.scheduling_or_tools = "@SCHEDULING_OR_TOOLS@"
config.llhd_sim_enabled = @CIRCT_LLHD_SIM_ENABLED@