更新readme
This commit is contained in:
parent
2b57e7ace6
commit
bc16d0b3c9
76
README_en.md
76
README_en.md
|
|
@ -1,6 +1,6 @@
|
|||
**Read in: [English](README_en.md) | [中文](README.md)**
|
||||
|
||||
# JCS-pub（JointCloud Storage Public Infrastructure + Out-of-the-Box Storage Client）
|
||||
# JCS-pub (JointCloud Storage Public Infrastructure + Ready-to-Use Storage Client)
|
||||
|
||||
## Project Overview
|
||||
JointCloud Storage is a storage service model built on a peer-to-peer intercloud collaboration mechanism. It manages storage resources across multiple clouds and provides users with a unified data storage service. The core idea emphasizes the independent and equal status of each cloud, connecting them in a non-intrusive manner. It also promotes intercloud collaboration by integrating computing, storage, and networking resources across clouds to deliver high-quality storage services.
|
||||
|
|
@ -16,7 +16,7 @@ This project aims to turn JointCloud Storage into a public infrastructure that i
|
|||
|
||||
## Features
|
||||
|
||||
### 1、Data Migration
|
||||
### 1. Data Migration
|
||||
|
||||
- **Cross-Cloud Migration Support**: Enables users to migrate data across multiple cloud storage providers.
|
||||
- **Policy-Driven Migration Engine**
|
||||
|
|
@ -25,7 +25,7 @@ This project aims to turn JointCloud Storage into a public infrastructure that i
|
|||
- Post-Migration Actions: Choose whether to retain or delete the original data after migration.
|
||||
- **Migration Efficiency Optimization**: Improve migration performance and reduce bandwidth costs by leveraging the **JointCloud Storage Public Infrastructure**.
|
||||
|
||||
### 2、Cross-Cloud Data Storage
|
||||
### 2. Cross-Cloud Data Storage
|
||||
|
||||
- **Unified Multi-Cloud View**: Store data across multiple cloud platforms while presenting a unified view to users, abstracting away multi-cloud complexity.
|
||||
- **Multi-Level Disaster Recovery**: Provides cloud-level resilience through various redundancy strategies, including erasure coding, replication, and hybrid redundancy (erasure coding + replication).
|
||||
|
|
@ -39,7 +39,7 @@ This project aims to turn JointCloud Storage into a public infrastructure that i
|
|||
- FUSE file system
|
||||
- **Access Efficiency Optimization**: Boost cross-cloud data access speed and reduce traffic costs via the **JointCloud Storage Public Infrastructure**.
|
||||
|
||||
### 3、Hybrid Local + Multi-Cloud Storage
|
||||
### 3. Hybrid Local + Multi-Cloud Storage
|
||||
|
||||
- **Unified Hybrid Storage View**: Store data across both **local file systems** and **multiple remote cloud platforms**, while presenting a unified view and hiding underlying complexities.
|
||||
- **Intelligent Data Collaboration Strategy**: Offers flexible policies for syncing data between local and remote storage.
|
||||
|
|
@ -52,16 +52,16 @@ This project aims to turn JointCloud Storage into a public infrastructure that i
|
|||
- Command-line interface
|
||||
- FUSE file system
|
||||
|
||||
### 4、Ready to Use
|
||||
### 4. Ready-to-Use Deployment
|
||||
|
||||
- Simply download and install the client — no additional components or configuration needed.
|
||||
|
||||
### 5、JointCloud Storage Public Infrastructure
|
||||
### 5. JointCloud Storage Public Infrastructure
|
||||
|
||||
- **Unified Hybrid Storage View**: Supports unified access across **local file systems** and **multiple remote cloud platforms**, abstracting the complexity behind a consistent interface.
|
||||
- **Dual Operation Modes**:
|
||||
- Standalone Mode: Fully offline usage without any external service dependencies.
|
||||
- Infrastructure-Connected Mode: Operates with public infrastructure support(**enhances performance and reduces bandwidth cost**)
|
||||
- Infrastructure-Connected Mode: Operates with public infrastructure support(**enhances performance and reduces bandwidth cost**).
|
||||
- **Open Source Public Infrastructure**:
|
||||
- Users can self-deploy or connect to existing public infrastructure.
|
||||
- Free Public Infrastructure Access:
|
||||
|
|
@ -76,14 +76,14 @@ This project aims to turn JointCloud Storage into a public infrastructure that i
|
|||
<center>
|
||||
<img src="docs/figs/architecture.png" width=50% /></center>
|
||||
|
||||
### 1、Cloud Storage Space
|
||||
### 1. Cloud Storage Space
|
||||
- Before using the JointCloud Storage client, users must prepare their own cloud storage space. This typically refers to object storage buckets or directories in file storage services. Both public cloud storage services and private cloud deployments are supported.
|
||||
- For tutorials on registering mainstream public cloud storage services, see: [Guide](docs/公有云注册及使用教程.md)
|
||||
|
||||
### 2、JointCloud Storage Public Infrastructure
|
||||
### 2. JointCloud Storage Public Infrastructure
|
||||
- The public infrastructure consists of multiple proxy nodes. These nodes collaborate with clients to perform cross-cloud operations such as data migration, upload, and download. They also optimize data transfer routes and support concurrent access, improving performance and reducing bandwidth costs. This design prevents the client from becoming a performance bottleneck.
|
||||
|
||||
### 3、JointCloud Storage Client
|
||||
### 3. JointCloud Storage Client
|
||||
- The JointCloud Storage Client is deployed on the user's server and serves as both a data service gateway and metadata management node.
|
||||
- Users manage their cloud storage data through various interfaces provided by the client.
|
||||
- All metadata and cloud storage credentials remain on the client managed by the user. When a proxy node needs to access a user’s cloud storage, the client temporarily grants access permissions as needed.
|
||||
|
|
@ -91,12 +91,12 @@ This project aims to turn JointCloud Storage into a public infrastructure that i
|
|||
|
||||
## Installation
|
||||
|
||||
### 1、Third-Party Dependencies
|
||||
### 1. Third-Party Dependencies
|
||||
|
||||
The following components must be installed manually:
|
||||
- `MySQL`: Version 8.0 or above. Please create a database and user account for the JCS client to use.
|
||||
|
||||
### 2、Docker Installation (Recommended)
|
||||
### 2. Docker Installation (Recommended)
|
||||
|
||||
Currently, only the JCS client is available as a Docker image. The jcsctl tool must be used as a precompiled executable — [click here to download]().
|
||||
|
||||
|
|
@ -115,7 +115,7 @@ docker run -v /etc/jcsconfs:/opt/confs \
|
|||
jcs serve -c /opt/confs/config.json # Note: the config path is inside the container
|
||||
```
|
||||
|
||||
### 3、Build from Source
|
||||
### 3. Build from Source
|
||||
|
||||
Before compiling, make sure you have the following installed:
|
||||
- `Go`: Version 1.23 or above
|
||||
|
|
@ -131,14 +131,14 @@ After successful execution, a build directory will be created in the project roo
|
|||
- `coordinator`: Coordination node for the JCS system (can be ignored if using the public JCS infrastructure)
|
||||
- `hub`: Central node of the JCS system (can also be ignored if using the public infrastructure)
|
||||
|
||||
### 4、Install Precompiled Executables
|
||||
### 4. Install Precompiled Executables
|
||||
|
||||
Download and extract the binaries that match your system environment — [click here to download]().
|
||||
|
||||
|
||||
## Usage Guide
|
||||
|
||||
### 1、Generate Configuration File
|
||||
### 1. Generate Configuration File
|
||||
|
||||
Use the `init` command of the JCS client to start the configuration process and follow the prompts to fill in the required information.
|
||||
|
||||
|
|
@ -148,8 +148,8 @@ After completing the command, the following files will be generated:
|
|||
- `config.json`: The full configuration file for the client.
|
||||
- `ca_cert.pem`: Root certificate for the HTTP service.
|
||||
- `ca_key.pem`: Root private key for the HTTP service(**must be securely stored by the user**).
|
||||
- `server_cert.pem`、`server_key.pem`: Server certificate and key signed by the root key.
|
||||
- `client_cert.pem`、`client_key.pem`: Client certificate and key signed by the root key, used by jcsctl or third-party programs.
|
||||
- `server_cert.pem`, `server_key.pem`: Server certificate and key signed by the root key.
|
||||
- `client_cert.pem`, `client_key.pem`: Client certificate and key signed by the root key, used by jcsctl or third-party programs.
|
||||
|
||||
All files except `ca_key.pem` will be used during client operation.
|
||||
|
||||
|
|
@ -221,17 +221,17 @@ The configuration file fields are explained as follows:
|
|||
|
||||
**Note**: If manually editing this JSON file, remove all comments as JSON does not support comments.
|
||||
|
||||
### 2、Command Line
|
||||
### 2. Command Line
|
||||
|
||||
`jcsctl` is the command-line tool for managing the JCS client. It requires certificates for authentication, generated during client initialization. The required files are `ca_cert.pem`, `client_cert.pem`, and `client_key.pem`.
|
||||
|
||||
When starting, `jcsctl` searches for these files in the following order:
|
||||
- Paths specified via command-line options `--ca`, `--cert`, and `--key`.
|
||||
- The directory where the `jcsctl` executable resides
|
||||
- The directory where the `jcsctl` executable resides.
|
||||
|
||||
By default, `jcsctl` attempts to connect to the client at `https://127.0.0.1:7890`. Use the `--endpoint` option to specify a different client address.
|
||||
|
||||
### 3、API
|
||||
### 3. API
|
||||
|
||||
See documentation: [Link](docs/JCS_pub_API.md)
|
||||
|
||||
|
|
@ -245,7 +245,7 @@ Redundancy strategies have a wide-ranging impact on the read/write process and a
|
|||
|
||||
The following explanation is written in an order that prioritizes understanding — it may not reflect the most optimal implementation flow. Please read through the entire section before deciding where to begin.
|
||||
|
||||
### 1、Redundancy Transformation
|
||||
### 1. Redundancy Transformation
|
||||
|
||||
When an object is first uploaded, it exists as a complete file in a single storage space, with a redundancy strategy of `None`. The transformation of its redundancy strategy is handled by the `ChangeRedundancy` task in the JCS client, which is triggered daily at midnight.
|
||||
|
||||
|
|
@ -255,24 +255,24 @@ The transformation generally involves two steps:
|
|||
|
||||
See the implementation in the `change_redundancy.go` and `redundancy_recover.go` files under `gitlink.org.cn/cloudream/jcs-pub/client/internal/ticktock`.
|
||||
|
||||
### 2、Redundancy Shrinking
|
||||
### 2. Redundancy Shrinking
|
||||
|
||||
Redundancy shrinking uses a simulated annealing algorithm to optimize redundancy from three dimensions: **resilience**, **redundancy level**, and **access efficiency**.
|
||||
|
||||
Currently, it supports redundancy shrinking for **replication** and **erasure coding** strategies.
|
||||
|
||||
This feature is optional. If needed, refer to the `redundancy_shrink.go` file in: `gitlink.org.cn/cloudream/jcs-pub/client/internal/ticktock`.
|
||||
This feature is optional. If needed, refer to the `redundancy_shrink.go` file in `gitlink.org.cn/cloudream/jcs-pub/client/internal/ticktock`.
|
||||
|
||||
### 3、Object Download
|
||||
### 3. Object Download
|
||||
|
||||
|
||||
Any functionality that involves downloading objects (not just HTTP-based downloads) must be adapted to support the new redundancy strategies. The download process includes:
|
||||
- **Strategy Selection**
|
||||
- **Strategy Execution**
|
||||
|
||||
The code for **strategy selection** is mainly located in `gitlink.org.cn/cloudream/jcs-pub/client/internal/downloader/strategy`
|
||||
The code for **strategy selection** is mainly located in `gitlink.org.cn/cloudream/jcs-pub/client/internal/downloader/strategy` package.
|
||||
|
||||
The code for **strategy execution** is distributed across multiple parts of the project, including but not limited to `gitlink.org.cn/cloudream/jcs-pub/client/internal/downloader`
|
||||
The code for **strategy execution** is distributed across multiple parts of the project, including but not limited to `gitlink.org.cn/cloudream/jcs-pub/client/internal/downloader` package.
|
||||
|
||||
We recommend using your IDE's search functionality to ensure full coverage when modifying related logic.
|
||||
|
||||
|
|
@ -281,7 +281,7 @@ We recommend using your IDE's search functionality to ensure full coverage when
|
|||
|
||||
Most of the currently implemented instructions can be found in the `gitlink.org.cn/cloudream/jcs-pub/common/pkgs/ioswitch2` package and serve as good references during custom instruction development.
|
||||
|
||||
### 1、Writing Instruction Logic
|
||||
### 1. Writing Instruction Logic
|
||||
|
||||
Each instruction must implement the following interface:
|
||||
```go
|
||||
|
|
@ -299,7 +299,6 @@ type Op interface {
|
|||
If `Execute` returns a non-nil error, the entire plan is considered failed and all ongoing instructions will be aborted.
|
||||
|
||||
Typical implementation steps:
|
||||
|
||||
- **Read Parameters**: Use `BindVar` or related functions to read required inputs. The `VarIDs` should be defined when the instruction is created.
|
||||
- **Process**: Implement your desired logic.
|
||||
- **Output Results**: Use `PutVar` or similar to write results back to the variable table for use by subsequent instructions.
|
||||
|
|
@ -322,7 +321,7 @@ If an instruction produces a stream, consider using `WaitGroup` or similar mecha
|
|||
|
||||
Finally, register your new instruction and data types using `UseOp` and `UseVarValue` from the `gitlink.org.cn/cloudream/common/pkgs/ioswitch/exec` package.
|
||||
|
||||
### 2、Writing DAG Nodes for Instructions
|
||||
### 2. Writing DAG Nodes for Instructions
|
||||
|
||||
The `ioswitch` module expresses execution plans as DAGs (Directed Acyclic Graphs), where each node corresponds to an instruction and edges represent data dependencies.
|
||||
|
||||
|
|
@ -347,7 +346,7 @@ Key concepts:
|
|||
|
||||
You may embed the `NodeBase` struct from the `gitlink.org.cn/cloudream/common/pkgs/ioswitch/dag` package, which implements all functions except `GenerateOp`.
|
||||
|
||||
### 3、Using Custom Instructions
|
||||
### 3. Using Custom Instructions
|
||||
|
||||
The lifecycle of an execution plan includes:
|
||||
- Writing `FromTo`
|
||||
|
|
@ -357,15 +356,13 @@ The lifecycle of an execution plan includes:
|
|||
|
||||
You can extend any of the first three stages to integrate your custom instructions.
|
||||
|
||||
The `From` and `To` models describe where data comes from and goes to. You may define your own as long as they implement:
|
||||
|
||||
The `FromTo` model is used to describe the structure of an execution plan.
|
||||
The `FromTo` model is used to describe the structure of an execution plan:
|
||||
- The `From` component defines where the data originates
|
||||
- The `To` component defines where the data is intended to go
|
||||
|
||||
The plan parser uses predefined rules to convert the FromTo description into a DAG (Directed Acyclic Graph), which represents data dependencies and logical flow between operations. This DAG will eventually be translated into a sequence of executable instructions.
|
||||
|
||||
If necessary, you can define your own custom From and To types, as long as they implement the following interfaces:
|
||||
If necessary, you can define your own custom `From` and `To` types, as long as they implement the following interfaces:
|
||||
```go
|
||||
type From interface {
|
||||
GetStreamIndex() StreamIndex
|
||||
|
|
@ -373,7 +370,7 @@ type From interface {
|
|||
|
||||
type To interface {
|
||||
// The range of the file stream required by this To node.
|
||||
// The specific meaning depends on the value of DataIndex:
|
||||
// The exact meaning depends on the value of DataIndex:
|
||||
// If DataIndex == -1, it refers to the entire file range.
|
||||
// If DataIndex >= 0, it refers to a specific shard (chunk) of the file.
|
||||
GetRange() math2.Range
|
||||
|
|
@ -381,7 +378,7 @@ type To interface {
|
|||
}
|
||||
```
|
||||
|
||||
You’ll also need to modify the DAG parsing logic in `gitlink.org.cn/cloudream/jcs-pub/common/pkgs/ioswitch2/parser/gen`
|
||||
You’ll also need to modify the DAG parsing logic in `gitlink.org.cn/cloudream/jcs-pub/common/pkgs/ioswitch2/parser/gen` package.
|
||||
|
||||
**Note**: Ensure that your custom DAG nodes implement:
|
||||
```go
|
||||
|
|
@ -402,10 +399,7 @@ This ensures compatibility with DAG optimization algorithms that rely on these i
|
|||
|
||||
In the optimization phase, you can add your own steps to match certain patterns and replace or merge nodes to improve efficiency or reduce instruction count.
|
||||
|
||||
After implementing your optimizer, integrate it into the `Parse` function in:
|
||||
`gitlink.org.cn/cloudream/jcs-pub/common/pkgs/ioswitch2/parser`
|
||||
|
||||
Carefully consider the interaction between your logic and other optimization steps.
|
||||
After implementing your optimizer, integrate it into the `Parse` function in `gitlink.org.cn/cloudream/jcs-pub/common/pkgs/ioswitch2/parser` package. Carefully consider the interaction between your logic and other optimization steps.
|
||||
|
||||
|
||||
## License
|
||||
|
|
@ -418,5 +412,5 @@ MulanPSL v2
|
|||
- **Planning & Architecture**: Han Bao
|
||||
- **Design & Implementation**: Xihua Gong, Jiancong Song, Zhishi Ren, Kaixin Zhang, Junhui Kan
|
||||
- **Advisors**: Yijie Wang
|
||||
- **Technical Contact**: Jiancong Song（Email: song-jc@foxmail.com）
|
||||
- **Technical Contact**: Jiancong Song (Email: song-jc@foxmail.com)
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue