更新readme

This commit is contained in:
songjc 2025-07-03 14:44:07 +08:00
parent 853000bdc6
commit 2b57e7ace6
2 changed files with 30 additions and 36 deletions

View File

@ -1,6 +1,6 @@
**Read in: [English](README_en.md) | [中文](README.md)**
# JCS-pubJointCloud Storage Public Infrastructure + Out-of-the-Box Client
# JCS-pubJointCloud Storage Public Infrastructure + Out-of-the-Box Storage Client
## Project Overview
JointCloud Storage is a storage service model built on a peer-to-peer intercloud collaboration mechanism. It manages storage resources across multiple clouds and provides users with a unified data storage service. The core idea emphasizes the independent and equal status of each cloud, connecting them in a non-intrusive manner. It also promotes intercloud collaboration by integrating computing, storage, and networking resources across clouds to deliver high-quality storage services.
@ -30,8 +30,8 @@ This project aims to turn JointCloud Storage into a public infrastructure that i
- **Unified Multi-Cloud View**: Store data across multiple cloud platforms while presenting a unified view to users, abstracting away multi-cloud complexity.
- **Multi-Level Disaster Recovery**: Provides cloud-level resilience through various redundancy strategies, including erasure coding, replication, and hybrid redundancy (erasure coding + replication).
- **Adaptive Redundancy Strategy**
- **Redundancy Scheme**: Customize the number of replicas, EC data blocks, and coding algorithms.
- **Data Placement**Configure where each replica or EC block is stored.
- Redundancy Scheme: Customize the number of replicas, EC data blocks, and coding algorithms.
- Data Placement: Configure where each replica or EC block is stored.
- **Cross-Cloud Command Set**: Supports custom commands such as upload, download, random read, cross-cloud scheduling, and cross-cloud repair.
- **Multiple Data Access Methods**
- REST API
@ -43,9 +43,9 @@ This project aims to turn JointCloud Storage into a public infrastructure that i
- **Unified Hybrid Storage View**: Store data across both **local file systems** and **multiple remote cloud platforms**, while presenting a unified view and hiding underlying complexities.
- **Intelligent Data Collaboration Strategy**: Offers flexible policies for syncing data between local and remote storage.
- **Data Filtering**: Dynamically select files for remote sync based on size, path, or extension.
- **Local Retention**: Configure whether to keep local copies of remote data.
- **Bidirectional Sync**: Independently configure local-to-cloud and cloud-to-local synchronization.
- Data Filtering: Dynamically select files for remote sync based on size, path, or extension.
- Local Retention: Configure whether to keep local copies of remote data.
- Bidirectional Sync: Independently configure local-to-cloud and cloud-to-local synchronization.
- **Collaboration Efficiency Optimization**: Enhance sync performance and reduce traffic costs using the **JointCloud Storage Public Infrastructure**.
- **Multiple Data Access Methods**
- REST API
@ -60,21 +60,21 @@ This project aims to turn JointCloud Storage into a public infrastructure that i
- **Unified Hybrid Storage View**: Supports unified access across **local file systems** and **multiple remote cloud platforms**, abstracting the complexity behind a consistent interface.
- **Dual Operation Modes**:
- **Standalone Mode**: Fully offline usage without any external service dependencies.
- **Infrastructure-Connected Mode**: Operates with public infrastructure support(**enhances performance and reduces bandwidth cost**)
- Standalone Mode: Fully offline usage without any external service dependencies.
- Infrastructure-Connected Mode: Operates with public infrastructure support(**enhances performance and reduces bandwidth cost**)
- **Open Source Public Infrastructure**:
- Users can self-deploy or connect to existing public infrastructure.
- Free Public Infrastructure Access:
- To obtain an account, password, and certificate, please email `song-jc@foxmail.com`. The application process is illustrated below.
<center>
<img src="docs/figs/application.png" width=90% /></center>
<img src="docs/figs/application_en.png" width=70% /></center>
## Architecture Diagram
<center>
<img src="docs/figs/architecture.png" width=70% /></center>
<img src="docs/figs/architecture.png" width=50% /></center>
### 1、Cloud Storage Space
- Before using the JointCloud Storage client, users must prepare their own cloud storage space. This typically refers to object storage buckets or directories in file storage services. Both public cloud storage services and private cloud deployments are supported.
@ -120,7 +120,7 @@ jcs serve -c /opt/confs/config.json # Note: the config path is inside the contai
Before compiling, make sure you have the following installed:
- `Go`: Version 1.23 or above
- `mage`: A Makefile-like build tool. GitHub repo:[click here](https://github.com/magefile/mage)
After installing dependencies, clone this repository and run the following command in the project root:
```powershell
mage bin
@ -142,7 +142,7 @@ Download and extract the binaries that match your system environment — [click
Use the `init` command of the JCS client to start the configuration process and follow the prompts to fill in the required information.
**Note**: If you plan to run the JCS client using the Docker image, all paths in the configuration should refer to locations **inside the container**.
**Note**: If you plan to run the JCS client using the Docker image, all paths in the configuration should refer to locations inside the container.
After completing the command, the following files will be generated:
- `config.json`: The full configuration file for the client.
@ -150,11 +150,10 @@ After completing the command, the following files will be generated:
- `ca_key.pem`: Root private key for the HTTP service(**must be securely stored by the user**).
- `server_cert.pem`、`server_key.pem`: Server certificate and key signed by the root key.
- `client_cert.pem`、`client_key.pem`: Client certificate and key signed by the root key, used by jcsctl or third-party programs.
All files except `ca_key.pem` will be used during client operation.
The configuration file fields are explained as follows:
```json
{
"hubRPC": {
@ -251,11 +250,10 @@ The following explanation is written in an order that prioritizes understanding
When an object is first uploaded, it exists as a complete file in a single storage space, with a redundancy strategy of `None`. The transformation of its redundancy strategy is handled by the `ChangeRedundancy` task in the JCS client, which is triggered daily at midnight.
The transformation generally involves two steps:
- **Strategy Selection**: The system selects a redundancy strategy for the object based on predefined rules (e.g., object size). Note that transformation is not limited to non-redundant objects — an object can switch between different redundancy modes if needed.
- **Execution**: Based on the chosen strategy and the current one, a transformation plan is generated and executed.If you need to customize the execution plan or instructions, refer to the later section on Custom Instructions.
See the implementation in the `change_redundancy.go` and `redundancy_recover.go` files under: `gitlink.org.cn/cloudream/jcs-pub/client/internal/ticktock`.
See the implementation in the `change_redundancy.go` and `redundancy_recover.go` files under `gitlink.org.cn/cloudream/jcs-pub/client/internal/ticktock`.
### 2、Redundancy Shrinking
@ -272,15 +270,11 @@ Any functionality that involves downloading objects (not just HTTP-based downloa
- **Strategy Selection**
- **Strategy Execution**
The code for **strategy selection** is mainly located in:
The code for **strategy selection** is mainly located in `gitlink.org.cn/cloudream/jcs-pub/client/internal/downloader/strategy`
`gitlink.org.cn/cloudream/jcs-pub/client/internal/downloader/strategy`
The code for **strategy execution** is distributed across multiple parts of the project, including but not limited to `gitlink.org.cn/cloudream/jcs-pub/client/internal/downloader`
The code for **strategy execution** is distributed across multiple parts of the project, including but not limited to:
`gitlink.org.cn/cloudream/jcs-pub/client/internal/downloader`
We recommend using your IDEs **search** functionality to ensure full coverage when modifying related logic.
We recommend using your IDE's search functionality to ensure full coverage when modifying related logic.
## Custom Instructions
@ -298,7 +292,7 @@ type Op interface {
```
- `String()`: Used for debugging, returns a description of the instruction.
- `Execute(ctx *ExecContext, e *Executor)error`The core execution logic.
- `Execute(ctx *ExecContext, e *Executor)error`: The core execution logic.
- `ctx`: Execution context, which includes a cancelable context(`Context`) and custom values(`Values`) that can be accessed using functions like `GetValueByType` and `SetValueByType` from the `gitlink.org.cn/cloudream/common/pkgs/ioswitch/exec` package.
- `e`: The executor, which provides access to a variable table through functions like `BindVar`, `PutVar`, or generic variants such as `BindVar` and `BindArray` from the `gitlink.org.cn/cloudream/common/pkgs/ioswitch/exec` package.
@ -310,12 +304,13 @@ Typical implementation steps:
- **Process**: Implement your desired logic.
- **Output Results**: Use `PutVar` or similar to write results back to the variable table for use by subsequent instructions.
If you understand how the module works internally, you are **free to implement any custom logic** as needed.
If you understand how the module works internally, you are free to implement any custom logic as needed.
The data structure used with `PutVar`/`BindVar` must implement:
```go
type VarValue interface {
Clone() VarValue
}
```
@ -346,11 +341,11 @@ type Node interface {
```
Key concepts:
- Each node has four types of **slots**: input/output for streams and input/output for values.
- Each input slot accepts data from **one** node; output slots can fan out to **multiple** nodes.
- Each node has four types of slots: input/output for streams and input/output for values.
- Each input slot accepts data from one node; output slots can fan out to multiple nodes.
- `Env()` specifies the environment where the instruction should run (e.g., Driver, Hub, Any). Most instructions can use Any.
You may embed the `NodeBase` struct from the `gitlink.org.cn/cloudream/common/pkgs/ioswitch/dag` package, which implements all functions **except** `GenerateOp`.
You may embed the `NodeBase` struct from the `gitlink.org.cn/cloudream/common/pkgs/ioswitch/dag` package, which implements all functions except `GenerateOp`.
### 3、Using Custom Instructions
@ -365,8 +360,8 @@ You can extend any of the first three stages to integrate your custom instructio
The `From` and `To` models describe where data comes from and goes to. You may define your own as long as they implement:
The `FromTo` model is used to describe the structure of an execution plan.
- The `From` component defines **where the data originates**
- The `To` component defines **where the data is intended to go**.
- The `From` component defines where the data originates
- The `To` component defines where the data is intended to go
The plan parser uses predefined rules to convert the FromTo description into a DAG (Directed Acyclic Graph), which represents data dependencies and logical flow between operations. This DAG will eventually be translated into a sequence of executable instructions.
@ -386,8 +381,7 @@ type To interface {
}
```
Youll also need to modify the DAG parsing logic in:
`gitlink.org.cn/cloudream/jcs-pub/common/pkgs/ioswitch2/parser/gen`
Youll also need to modify the DAG parsing logic in `gitlink.org.cn/cloudream/jcs-pub/common/pkgs/ioswitch2/parser/gen`
**Note**: Ensure that your custom DAG nodes implement:
```go
@ -421,8 +415,8 @@ MulanPSL v2
## Development Team
- Planning & Architecture: Han Bao
- Design & Implementation: Xihua Gong, Jiancong Song, Zhishi Ren, Kaixin Zhang, Junhui Kan
- Advisors: YiJie Wang
- Technical Contact: Jiancong SongEmail: song-jc@foxmail.com
- **Planning & Architecture**: Han Bao
- **Design & Implementation**: Xihua Gong, Jiancong Song, Zhishi Ren, Kaixin Zhang, Junhui Kan
- **Advisors**: Yijie Wang
- **Technical Contact**: Jiancong SongEmail: song-jc@foxmail.com

Binary file not shown.

After

Width:  |  Height:  |  Size: 26 KiB