forked from JointCloud/pcm-coordinator
zhouqunjie
parent
990c4cf0be
commit
e9359849e3
176
README.md
176
README.md
|
|
@ -1,107 +1,101 @@
|
|||
|
||||
|
||||
|
||||
|
||||
jointcloud云际团队正式发布PCM对等协作框架，框架包含了kubernan核心组件源代码、8个participant协作者组件仓库，以及PCM中定义的超算、智算、数算资源操作及信息收集规格。
|
||||
<p align="center">
|
||||
<a href="https://www.gitlink.org.cn/JointCloud/pcm-coordinator/tree/master/docs">Docs</a> |
|
||||
<a href="./README_zh.md">简体中文</a> |
|
||||
<a href="https://www.gitlink.org.cn/JointCloud/pcm-coordinator/milestones">Milestones</a>
|
||||
</p>
|
||||
|
||||
## 什么是Jointcloud和PCM
|
||||
云际协作环境 (JointCloud cooperation environment, JCCE), JCCE 包括分布云交易、分布云社区、分布云监管以及基于区块链的分布记账系统， JCCE 中与信息交换尤其是价值交换相关的支撑技术, 利于打破云际协作参与方之间的信息不对称， 为参与各方创造多赢机会, 实现对云际计算商业模式创新的强有力支撑。
|
||||
对等协作机制(peer cooperation mechanism, PCM)的愿景是在云际环境中构建一套所有服务提供方和需求方能够自主参与协作，实现对等交易，以非侵入的方式实现各利益主体间便捷高效的连接和调用，并在构建云际生态的同时形成行业标准，吸引更多云际服务商加入贡献。PCM站在云上视角，叠加于云之上，关注云之间协作多于云自身。
|
||||
对等协作机制建立在一个标准的软件定义框架之上，按照这个标准框架，拥有物理云资源的云服务提供者可以自主定义PCM规则框架，其他遵循该规则框架的实体云即可参与协作交易，并进而面向领域应用定制虚拟专用云，满足各类云际参与者如中心侧、边缘侧和终端侧等异构基础设施资源直接自主参与协作和交易的需求。
|
||||
## What is Jointcloud and PCM
|
||||
|
||||
  Jointcloud Cooperation Environment (JCCE) comprises Distributed Cloud Trading, Distributed Cloud Community, Distributed Cloud Supervision, and a Blockchain-based Distributed Accounting System. JCCE, with its supporting technologies related to information and value exchange, facilitates breaking the information asymmetry among participants in cloud collaboration. It creates a win-win opportunity for all involved parties and provides robust support for innovative business models in the cloud collaboration computing environment.
|
||||
|
||||
  The vision of the Peer Cooperation Mechanism (PCM) is to build a set of collaboration mechanisms in the cloud environment where all service providers and demand-side entities can autonomously participate, achieve peer-to-peer transactions, and establish efficient connections and invocations in a non-intrusive manner. PCM focuses more on collaboration among clouds rather than individual clouds themselves.
|
||||
|
||||
  PCM is built upon a standardized software-defined framework. Cloud service providers with physical cloud resources can autonomously define PCM rule frameworks. Other entities adhering to these rule frameworks can then participate in collaborative transactions. Subsequently, tailored virtual private clouds can be created for domain-specific applications, meeting the diverse requirements of various cloud participants, such as central, edge, and terminal infrastructure, enabling them to directly participate in collaboration and transactions.
|
||||
|
||||
## Real-world Issues Addressed by PCM:
|
||||
|
||||
- **Performance Improvement**:
|
||||
|
||||
Developers in the cloud collaboration environment experience performance enhancement upon joining the PCM framework. They can generate code for the internal functionality structure and most specification definitions using descriptive language, allowing them to focus on business logic development without concerning themselves with underlying management functions. The framework offers features such as microservices management, multi-language code generation, and model bridging, reducing coding workload and entry barriers while improving efficiency.
|
||||
|
||||
|
||||
## PCM力求解决的现实问题：
|
||||
- **Platform Lock-in Resolution**:
|
||||
|
||||
- **效能提升**:
|
||||
云际开发者加入PCM框架后，可以通过描述性语言生成框架内部功能结构的代码和大部分规格定义，专注于业务逻辑开发，无需关心底层管理功能。框架提供微服务管理、多语言代码生成和模型桥接功能，可降低编码工作量和入门难度，提高能效。
|
||||
|
||||
- **解决平台锁定问题**:
|
||||
对等协作机制通过多云适配、标准化接口和抽象层等手段，有效解决平台锁定问题，实现跨平台运行、部署和交互。标准化接口简化开发人员工作，降低平台锁定风险。
|
||||
|
||||
- **降低代码开发难度**:
|
||||
对等协作开发框架降低了开发难度，提供了结构化开发模式、现成组件和文档支持，不同级别开发人员能更公平地协同工作。框架可节省约50%的开发时间，初级程序员用约15人天完成原需1人月的任务。自动生成工具、代码模板和组件复用等功能让开发者更专注于业务逻辑实现。无需再培训高级开发人员，可节省时间和人力资源成本，提高投入产出比。
|
||||
|
||||
- **减少代码开发工作量**:
|
||||
PCM框架提供自动化功能和代码生成工具，减少手动编写代码，提高开发速度和代码质量。使用框架的自动生成工具后，整体代码量超过一半由工具生成，低代码率达到约63%。开发者主要工作是编写描述语言文件生成基础结构，然后调整和编写逻辑功能的开发。这使团队能更快速敏捷地交付产品并进行业务优化和用户反馈迭代。
|
||||
The PCM effectively addresses platform lock-in issues through multi-cloud adaptation, standardized interfaces, and abstraction layers. This enables cross-platform operations, deployment, and interaction. Standardized interfaces simplify the work for developers, lowering the risk of platform lock-in.
|
||||
|
||||
|
||||
## 设计架构
|
||||
Kubernan是框架的核心组件，实现了对不同Participant端技术栈的异构屏蔽。框架面向用户侧主要分为资源变更（任务提交）和集约展示两大功能。在任务提交后，PCM通过架设在云算云、智算云、超算云之上的调度中心来实现任务的动态流转。在集约展示部分，PCM主要通过Tracker以及P端主动上报的方式对多个Participant服务的资源、任务等信息进行收集和汇聚，向用户提供标准化的统一接口。用户可以在前端页面上查看PCM提供的信息，也可以通过接口直接获取数据。
|
||||
|
||||
## 服务部署
|
||||
- **Reduced Code Development Complexity**:
|
||||
|
||||
PCM的开发环境需要安装golang 1.18以上版本，参考下文内容将会在本地启动kubernan的相关服务。
|
||||
拉取项目的命令如下
|
||||
|
||||
`git clone https://gitlink.org.cn/jcce-pcm/pcm-coordinator.git`
|
||||
The PCM development framework lowers the complexity of development by providing structured development patterns, ready-made components, and documentation support. Developers of different skill levels can collaborate more fairly. The framework can save approximately 50% of development time, with junior programmers completing tasks that originally required one person-month in about 15 person-days. Features such as automatic generation tools, code templates, and component reuse allow developers to focus more on business logic implementation. There is no need for additional training of advanced developers, saving time, and reducing labor costs while improving the return on investment.
|
||||
|
||||
#### nacos
|
||||
配置管理采用nacos，用户需要在本地或服务器上安装部署nacos服务，对应教程可以参考[nacos安装](https://nacos.io/en-us/docs/quick-start.html)，nacos中添加对应服务的配置文件，具体配置内容参考下文。
|
||||
#### kubernan-rpc
|
||||
rpc服务的配置内容模板如下
|
||||
|
||||
- **Reduced Code Development Workload**:
|
||||
The PCM framework offers automation features and code generation tools, reducing manual code writing and improving development speed and code quality. With the framework's auto-generation tools, over half of the code is generated, achieving a low code rate of approximately 63%. Developers primarily work on writing descriptive language files to generate the basic structure and then adjust and write logic functions. This enables teams to deliver products more quickly, iterate based on business optimization and user feedback, and be more agile.
|
||||
|
||||
|
||||
## Architecture
|
||||
  The Coordinator is the core component of the framework, providing heterogeneous abstraction for different Participant-side technology stacks. The framework, oriented towards the user side, primarily involves two major functionalities: resource changes (task submission) and centralized display. After task submission, PCM achieves dynamic task flow through a scheduling center deployed on cloud computing, intelligent computing, and supercomputing clouds. In the centralized display section, PCM mainly collects and aggregates information about resources and tasks from multiple Participant services through Tracker and actively reported by the Participant side. It provides users with a standardized unified interface. Users can view the information provided by PCM on the frontend page or directly access data through the interface.
|
||||
|
||||
|
||||
## PCM deploy
|
||||
|
||||
The development environment for PCM requires the installation of Go version 1.18 or above. Please refer to the following instructions to locally start the Kubernetes-related services.
|
||||
|
||||
The command to fetch the project is as follows:
|
||||
```
|
||||
git clone https://gitlink.org.cn/jcce-pcm/pcm-coordinator.git
|
||||
```
|
||||
Name: pcm.core.rpc
|
||||
ListenOn: 0.0.0.0:2004
|
||||
|
||||
DB:
|
||||
DataSource: username:password@(x.x.x.x:3306)/pcm?parseTime=true
|
||||
|
||||
  After executing the following command, the Kubernetes RPC service will be launched locally. For its specific functionalities, please refer to the description in the architecture design mentioned earlier.
|
||||
```
|
||||
# get required packages
|
||||
go mod tidy
|
||||
|
||||
SnowflakeConf:
|
||||
MachineId: 1
|
||||
```
|
||||
运行以下命令后将会在本地启动kubernan-rpc服务，其具体功能可参考上文架构设计中的描述。
|
||||
```
|
||||
# get required packages
|
||||
go mod tidy
|
||||
|
||||
# run rpc service
|
||||
go run pcm-coordinator/rpc/pcmcore.go
|
||||
```
|
||||
#### kubernan-api
|
||||
api服务的配置内容模板如下，其中各P端的config信息可以根据实际情况按需配置。
|
||||
```
|
||||
Name: pcm.core.api
|
||||
Host: 0.0.0.0
|
||||
Port: 8999
|
||||
|
||||
DB:
|
||||
DataSource: username:password@(x.x.x.x:3306)/pcm?parseTime=true
|
||||
# run rpc service
|
||||
go run pcm-coordinator/rpc/pcmcore.go
|
||||
```
|
||||
#### coordinator-api
|
||||
  The template for the configuration content of the API service is as follows, where the config information for each Participant (P-side) can be configured as needed based on the actual situation.
|
||||
```
|
||||
Name: pcm.core.api
|
||||
Host: 0.0.0.0
|
||||
Port: 8999
|
||||
|
||||
Redis:
|
||||
Host: x.x.x.x:portnumber
|
||||
Pass: password
|
||||
DB:
|
||||
DataSource: username:password@(x.x.x.x:3306)/pcm?parseTime=true Redis:
|
||||
Host: x.x.x.x:portnumber Pass: password
|
||||
|
||||
# k8s rpc
|
||||
K8sNativeConf:
|
||||
Endpoints: - 127.0.0.1:2003 NonBlock: true
|
||||
#slurm rpc
|
||||
THRpcConf:
|
||||
Endpoints: - 127.0.0.1:2007NonBlock: true
|
||||
|
||||
#modelarts rpc
|
||||
ModelArtsRpcConf:
|
||||
Endpoints: - 127.0.0.1:2002NonBlock: true
|
||||
```
|
||||
  After executing the following command, the Kubernetes API service will be launched locally. Once the service is started, users can make HTTP requests to its interfaces for various functional calls.
|
||||
```
|
||||
# get required packages
|
||||
go mod tidy
|
||||
|
||||
# run rpc service
|
||||
go run pcm-coordinator/api/pcm.go
|
||||
```
|
||||
|
||||
## Upcoming Plans
|
||||
|
||||
# k8s rpc
|
||||
K8sNativeConf:
|
||||
Endpoints:
|
||||
- 127.0.0.1:2003
|
||||
NonBlock: true
|
||||
- Pluginization of Scheduling Algorithms and Definition of Basic Resource Templates
|
||||
- Fundamental Definition of Resource Operation Types
|
||||
- Rapid Development Mode
|
||||
- Implementation of First-level Scheduling
|
||||
|
||||
#slurm rpc
|
||||
THRpcConf:
|
||||
Endpoints:
|
||||
- 127.0.0.1:2007
|
||||
NonBlock: true
|
||||
|
||||
#modelarts rpc
|
||||
ModelArtsRpcConf:
|
||||
Endpoints:
|
||||
- 127.0.0.1:2002
|
||||
NonBlock: true
|
||||
```
|
||||
运行以下命令后将会在本地启动kubernan-api服务，服务启动后用户将可以通过http请求其接口进行各类功能调用。
|
||||
```
|
||||
# get required packages
|
||||
go mod tidy
|
||||
|
||||
# run rpc service
|
||||
go run pcm-coordinator/api/pcm.go
|
||||
```
|
||||
|
||||
## 接下来的计划
|
||||
|
||||
- 调度算法的插件化以及基础资源模板的定义
|
||||
- 资源操作类型的基础定义
|
||||
- 快捷的开发模式
|
||||
- 一级调度的实现
|
||||
|
||||
## 加入＆贡献
|
||||
我们期待您的意见和贡献，欢迎所有的朋友对项目进行指正和改进，共同打造一套高效稳定的云际协作机制。
|
||||
## Join＆Contribute
|
||||
  We look forward to your opinions and contributions. Welcome all friends to provide corrections and improvements to the project, collectively building an efficient and stable cloud collaboration mechanism.
|
||||
|
|
@ -0,0 +1,130 @@
|
|||
|
||||
<p align="center">
|
||||
<a href="https://www.gitlink.org.cn/JointCloud/pcm-coordinator/tree/master/docs">文档</a> |
|
||||
<a href="./README.md">English</a> |
|
||||
<a href="https://www.gitlink.org.cn/JointCloud/pcm-coordinator/milestones">里程碑</a>
|
||||
</p>
|
||||
  jointcloud云际团队正式发布PCM对等协作框架，框架包含了kubernan核心组件源代码、8个participant协作者组件仓库，以及PCM中定义的超算、智算、数算资源操作及信息收集规格。
|
||||
|
||||
## 什么是Jointcloud和PCM
|
||||
  云际协作环境 (JointCloud cooperation environment, JCCE), JCCE 包括分布云交易、分布云社区、分布云监管以及基于区块链的分布记账系统， JCCE 中与信息交换尤其是价值交换相关的支撑技术, 利于打破云际协作参与方之间的信息不对称， 为参与各方创造多赢机会, 实现对云际计算商业模式创新的强有力支撑。
|
||||
|
||||
  对等协作机制(peer cooperation mechanism, PCM)的愿景是在云际环境中构建一套所有服务提供方和需求方能够自主参与协作，实现对等交易，以非侵入的方式实现各利益主体间便捷高效的连接和调用，并在构建云际生态的同时形成行业标准，吸引更多云际服务商加入贡献。PCM站在云上视角，叠加于云之上，关注云之间协作多于云自身。
|
||||
|
||||
  对等协作机制建立在一个标准的软件定义框架之上，按照这个标准框架，拥有物理云资源的云服务提供者可以自主定义PCM规则框架，其他遵循该规则框架的实体云即可参与协作交易，并进而面向领域应用定制虚拟专用云，满足各类云际参与者如中心侧、边缘侧和终端侧等异构基础设施资源直接自主参与协作和交易的需求。
|
||||
|
||||
|
||||
## PCM力求解决的现实问题：
|
||||
|
||||
- **效能提升**:
|
||||
|
||||
  云际开发者加入PCM框架后，可以通过描述性语言生成框架内部功能结构的代码和大部分规格定义，专注于业务逻辑开发，无需关心底层管理功能。框架提供微服务管理、多语言代码生成和模型桥接功能，可降低编码工作量和入门难度，提高能效。
|
||||
|
||||
- **解决平台锁定问题**:
|
||||
|
||||
  对等协作机制通过多云适配、标准化接口和抽象层等手段，有效解决平台锁定问题，实现跨平台运行、部署和交互。标准化接口简化开发人员工作，降低平台锁定风险。
|
||||
|
||||
- **降低代码开发难度**:
|
||||
|
||||
  对等协作开发框架降低了开发难度，提供了结构化开发模式、现成组件和文档支持，不同级别开发人员能更公平地协同工作。框架可节省约50%的开发时间，初级程序员用约15人天完成原需1人月的任务。自动生成工具、代码模板和组件复用等功能让开发者更专注于业务逻辑实现。无需再培训高级开发人员，可节省时间和人力资源成本，提高投入产出比。
|
||||
|
||||
- **减少代码开发工作量**:
|
||||
|
||||
  PCM框架提供自动化功能和代码生成工具，减少手动编写代码，提高开发速度和代码质量。使用框架的自动生成工具后，整体代码量超过一半由工具生成，低代码率达到约63%。开发者主要工作是编写描述语言文件生成基础结构，然后调整和编写逻辑功能的开发。这使团队能更快速敏捷地交付产品并进行业务优化和用户反馈迭代。
|
||||
|
||||
|
||||
## 设计架构
|
||||
|
||||
|
||||
  Coordinator是框架的核心组件，实现了对不同Participant端技术栈的异构屏蔽。框架面向用户侧主要分为资源变更（任务提交）和集约展示两大功能。在任务提交后，PCM通过架设在云算云、智算云、超算云之上的调度中心来实现任务的动态流转。在集约展示部分，PCM主要通过Tracker以及P端主动上报的方式对多个Participant服务的资源、任务等信息进行收集和汇聚，向用户提供标准化的统一接口。用户可以在前端页面上查看PCM提供的信息，也可以通过接口直接获取数据。
|
||||
|
||||
|
||||
## 服务部署
|
||||
|
||||
|
||||
|
||||
  PCM的开发环境需要安装golang 1.18以上版本，参考下文内容将会在本地启动coordinator的相关服务。
|
||||
|
||||
|
||||
  拉取项目的命令如下
|
||||
```
|
||||
git clone https://gitlink.org.cn/jcce-pcm/pcm-coordinator.git
|
||||
```
|
||||
#### coordinator-rpc
|
||||
|
||||
|
||||
  rpc服务的配置内容模板如下
|
||||
|
||||
```
|
||||
Name: pcm.core.rpc
|
||||
ListenOn: 0.0.0.0:2004
|
||||
|
||||
DB:
|
||||
DataSource: username:password@(x.x.x.x:3306)/pcm?parseTime=true
|
||||
|
||||
SnowflakeConf:
|
||||
MachineId: 1
|
||||
```
|
||||
|
||||
  运行以下命令后将会在本地启动kubernan-rpc服务，其具体功能可参考上文架构设计中的描述。
|
||||
|
||||
```
|
||||
# get required packages
|
||||
go mod tidy
|
||||
|
||||
# run rpc service
|
||||
go run pcm-coordinator/rpc/pcmcore.go
|
||||
```
|
||||
|
||||
#### coordinator-api
|
||||
|
||||
  api服务的配置内容模板如下，其中各P端的config信息可以根据实际情况按需配置。
|
||||
```
|
||||
Name: pcm.core.api
|
||||
Host: 0.0.0.0
|
||||
Port: 8999
|
||||
|
||||
DB:
|
||||
DataSource: username:password@(x.x.x.x:3306)/pcm?parseTime=true
|
||||
|
||||
Redis:
|
||||
Host: x.x.x.x:portnumber
|
||||
Pass: password
|
||||
|
||||
|
||||
# k8s rpc
|
||||
K8sNativeConf:
|
||||
Endpoints:
|
||||
- 127.0.0.1:2003
|
||||
NonBlock: true
|
||||
|
||||
#slurm rpc
|
||||
THRpcConf:
|
||||
Endpoints:
|
||||
- 127.0.0.1:2007
|
||||
NonBlock: true
|
||||
|
||||
#modelarts rpc
|
||||
ModelArtsRpcConf:
|
||||
Endpoints:
|
||||
- 127.0.0.1:2002
|
||||
NonBlock: true
|
||||
```
|
||||
  运行以下命令后将会在本地启动coordinator-api服务，服务启动后用户将可以通过http请求其接口进行各类功能调用。
|
||||
```
|
||||
# get required packages
|
||||
go mod tidy
|
||||
|
||||
# run rpc service
|
||||
go run pcm-coordinator/api/pcm.go
|
||||
```
|
||||
|
||||
## 接下来的计划
|
||||
|
||||
- 调度算法的插件化以及基础资源模板的定义
|
||||
- 资源操作类型的基础定义
|
||||
- 快捷的开发模式
|
||||
- 一级调度的实现
|
||||
|
||||
## 加入＆贡献
|
||||
  我们期待您的意见和贡献，欢迎所有的朋友对项目进行指正和改进，共同打造一套高效稳定的云际协作机制。
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 216 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 67 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 54 KiB |
Loading…
Reference in New Issue