Skip to content

Commit

Permalink
release 1.6.6 (#131)
Browse files Browse the repository at this point in the history
* 1.6.6 develop (#125)

* add(sys):add collect api

* fix(fusion):update fusion api

* fix(csv):update csv api

* update db

* add fusion-simple

* update fusion-simple

* add resource field type grpc

* update resource

* update sqllite

* add log

* update sqllite

* update task mpc lr

* fix Collectors.toMap Duplicate key

* delete db project column "secretkey_id"

* Annotate trusted third-party code

* update grpc param

* update fusion

* fix 推理 -- 协作方资源

* add 推理 模板

* update infer

* fix

* update infer

* add 修改合作机构网关和公钥

* update data task resultPath

* update model task

* update change other organInfo

* update NN

* add health

* add health path

* update hfl-nn


* update sync dataset

* update organ resource

* update data_set address

* update change

* update organ change

* update model

* update xml

* update get model data

* update fusion

* update readme

* add yaml

* update base.json

---------

Co-authored-by: fenghaolun <[email protected]>

* Feature/center (#126)

* fix: add logo display control

* fix: field type

* feat: node management refactoring

* fix: remove permission

* fix: icon

* fix: resource size conversion

* fix: document title

* fix: eslint

* feat: connection status

* fix: status text

* fix: cannot convert undefined or null to object

* fix: status

* feature: primihub big model

* refactor: contact icon

* feat: add pagination

* fix: reconnect message

* fix: reconnect

* stash

* feat: change organ info

* fix: draft component

* fix: change other organ info

* fix: offline map

* fix: reapply connect

* Optimizing Dataset Queries (#127)

* update fusion

* update task mpc lr

* fix Collectors.toMap Duplicate key

* delete db project column "secretkey_id"

* Annotate trusted third-party code

* update grpc param

* update fusion

* fix 推理 -- 协作方资源

* add 推理 模板

* update infer

* add 修改合作机构网关和公钥

* update data task resultPath

* update model task

* update change other organInfo

* update NN

* add health

* add health path

* add health path

* update hfl-nn

* update sync dataset

* update organ resource

* update data_set address

* update change

* update organ change

* update fusion

* update xml

* update get model data

* update ftl

* update organ

* update organ

* update resource

---------

Co-authored-by: fenghaolun <[email protected]>

* fix bug (#130)

* fix: add logo display control

* fix: field type

* feat: node management refactoring

* fix: remove permission

* fix: icon

* fix: resource size conversion

* fix: document title

* fix: eslint

* feat: connection status

* fix: status text

* fix: cannot convert undefined or null to object

* fix: status

* feature: primihub big model

* refactor: contact icon

* feat: add pagination

* fix: reconnect message

* fix: reconnect

* stash

* feat: change organ info

* fix: draft component

* fix: change other organ info

* fix: offline map

* fix: reapply connect

* fix: database import error

* feat: resource list add user name

* fix: select auth organs

* fix: select organ auth

* fix: psi task select other organ

* fix: download model task result

* fix: model evaluation score

* fix: model type

---------

Co-authored-by: Like habits <[email protected]>
Co-authored-by: fenghaolun <[email protected]>
Co-authored-by: zjj614 <[email protected]>
  • Loading branch information
4 people authored May 31, 2023
1 parent 7aac926 commit 941b751
Show file tree
Hide file tree
Showing 274 changed files with 5,041 additions and 31,189 deletions.
4 changes: 3 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ and have a clear directory:
├─primihub-platform
├─primihub-fusion
│ ├─fusion-api
│ ├─fusion-grpc
│ ├─fusion-simple
│ └─script
├─primihub-service
│ ├─application
Expand All @@ -29,7 +31,7 @@ Before we start, please refer to [primihub](https://github.com/primihub/primihub

then we can get started from those projects.

- [primihub-fusion](./primihub-fusion/README.md) : connect different primihub-service and have the right data access.
- [primihub-fusion](./primihub-fusion/README.md) : connect primihub-service and have the right data access.
- [primihub-service](./primihub-service/README.md) : provide all most of service capabilities and api.
- [primihub-webconsole](./primihub-webconsole/README.md) : you can operate specific functions and have a clear view.

Expand Down
6 changes: 4 additions & 2 deletions primihub-fusion/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -4,16 +4,18 @@ WORKDIR /opt

ADD . /opt/

RUN mvn clean install -Dmaven.test.skip=true
RUN ARCH=`arch | sed s/arm64/aarch_64/ | sed s/aarch64/aarch_64/ | sed s/amd64/x86_64/` \
&& mvn clean install -Dmaven.test.skip=true -Dos.detected.classifier=linux-${ARCH}

FROM openjdk:8-jre

ENV DEBIAN_FRONTEND=noninteractive

RUN apt update \
&& apt install tzdata \
&& apt -y install tzdata telnet \
&& ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

COPY --from=build /opt/fusion-api/target/*-SNAPSHOT.jar /applications/fusion.jar
COPY --from=build /opt/fusion-simple/target/*-SNAPSHOT.jar /applications/fusion-simple.jar

ENTRYPOINT ["/bin/sh","-c","java -jar -Dfile.encoding=UTF-8 /applications/fusion.jar --spring.profiles.active=test --server.port=8080"]
3 changes: 2 additions & 1 deletion primihub-fusion/Dockerfile.local
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,10 @@ FROM ibmjava:8-jre
ENV DEBIAN_FRONTEND=noninteractive

RUN apt update \
&& apt install tzdata \
&& apt -y install tzdata telnet \
&& ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

ADD fusion-api/target/*-SNAPSHOT.jar /applications/fusion.jar
ADD fusion-simple/target/*-SNAPSHOT.jar /applications/fusion-simple.jar

ENTRYPOINT ["/bin/sh","-c","java -jar -Dfile.encoding=UTF-8 /applications/fusion.jar --spring.profiles.active=test --server.port=8080"]
2 changes: 1 addition & 1 deletion primihub-fusion/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ As long as the finished infos show up, the project have been compiled and packag
## How to run
Before run, make sure that your service dependencies are available and the configuration is correct.

java -jar -Dfile.encoding=UTF-8 ./fusion-api/target/*-SNAPSHOT.jar --server.port=8099
java -jar -Dfile.encoding=UTF-8 ./fusion-api/target/*-SNAPSHOT.jar --server.port=8099 --spring.cloud.nacos.discovery.server-addr=nacos:8848 --spring.cloud.nacos.discovery.namespace=demo --spring.cloud.nacos.config.server-addr=nacos:8848 --spring.cloud.nacos.config.namespace=demo

execute that the command,you can check the url:

Expand Down
29 changes: 29 additions & 0 deletions primihub-fusion/fusion-api/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -79,14 +79,43 @@
<artifactId>fastjson</artifactId>
<version>1.2.70</version>
</dependency>

<dependency>
<groupId>com.alibaba.cloud</groupId>
<artifactId>spring-cloud-starter-alibaba-nacos-config</artifactId>
<version>2.2.7.RELEASE</version>
</dependency>
<dependency>
<groupId>com.alibaba.cloud</groupId>
<artifactId>spring-cloud-starter-alibaba-nacos-discovery</artifactId>
<version>2.2.7.RELEASE</version>
</dependency>
<dependency>
<groupId>com.primihub</groupId>
<artifactId>fusion-grpc</artifactId>
<version>1.0-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-openfeign</artifactId>
<version>2.2.7.RELEASE</version>
</dependency>
</dependencies>

<properties>
<maven.compiler.source>8</maven.compiler.source>
<maven.compiler.target>8</maven.compiler.target>
<os.detected.classifier>osx-x86_64</os.detected.classifier>
</properties>

<build>
<extensions>
<extension>
<groupId>kr.motd.maven</groupId>
<artifactId>os-maven-plugin</artifactId>
<version>1.6.0</version>
</extension>
</extensions>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,14 @@

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.client.discovery.EnableDiscoveryClient;
import org.springframework.cloud.openfeign.EnableFeignClients;
import org.springframework.scheduling.annotation.EnableAsync;

@SpringBootApplication
@EnableDiscoveryClient
@EnableAsync
@EnableFeignClients(basePackages = {"com.primihub"})
public class FusionApplication {
public static void main(String[] args){
SpringApplication.run(FusionApplication.class, args);
Expand Down

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
package com.primihub.constant;

public class SysConstant {
public final static String GRPC_NODE_HASH_KEY = "grpc:node";
public final static String GRPC_DATASET_KEY = "grpc:dataset:<id>";
public final static String GRPC_DATASET_FIELD_EXPAND = "expand";
public final static String GRPC_DATASET_FIELD_ADDRESS = "address";
public final static String GRPC_DATASET_FIELD_DRIVER = "driver";
}
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
import com.primihub.entity.copy.dto.DataFusionCopyDto;
import com.primihub.service.CopyService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

Expand All @@ -16,7 +17,7 @@ public class CopyController {
@Autowired
private CopyService copyService;

@RequestMapping("batchSave")
@PostMapping("batchSave")
public BaseResultEntity batchSave(String globalId,String copyPart){
DataFusionCopyDto copyDto=JSON.parseObject(copyPart, DataFusionCopyDto.class);
if(copyDto.getTableName()==null|| "".equals(copyDto.getTableName())) {
Expand Down
Original file line number Diff line number Diff line change
@@ -1,83 +1,25 @@
package com.primihub.controller;

import com.primihub.entity.base.BaseResultEntity;
import com.primihub.entity.base.BaseResultEnum;
import com.primihub.entity.fusion.param.FusionOrganExtendsParam;
import com.primihub.entity.fusion.param.FusionConnectionParam;
import com.primihub.service.FusionService;
import com.primihub.service.OrganService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.util.StringUtils;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;

@RequestMapping("fusion")
@RestController
public class FusionController {

@Autowired
private FusionService fusionService;
private OrganService organService;

@RequestMapping("healthConnection")
public BaseResultEntity healthConnection(){
return BaseResultEntity.success(System.currentTimeMillis());
}

@RequestMapping("registerConnection")
public BaseResultEntity registerConnection(FusionConnectionParam fusionConnectionParam){
if(fusionConnectionParam.getGlobalId()==null|| "".equals(fusionConnectionParam.getGlobalId())) {
return BaseResultEntity.failure(BaseResultEnum.LACK_OF_PARAM,"globalId");
}
if(fusionConnectionParam.getGlobalName()==null|| "".equals(fusionConnectionParam.getGlobalName().trim())) {
return BaseResultEntity.failure(BaseResultEnum.LACK_OF_PARAM,"globalName");
}
if(fusionConnectionParam.getPinCode()==null|| "".equals(fusionConnectionParam.getPinCode().trim())) {
return BaseResultEntity.failure(BaseResultEnum.LACK_OF_PARAM,"pinCode");
}
if(fusionConnectionParam.getGatewayAddress()==null|| "".equals(fusionConnectionParam.getGatewayAddress().trim())) {
return BaseResultEntity.failure(BaseResultEnum.LACK_OF_PARAM,"gatewayAddress");
}
if ("".equals(fusionConnectionParam.getPublicKey().trim())|| "".equals(fusionConnectionParam.getPrivateKey().trim())) {
return BaseResultEntity.failure(BaseResultEnum.LACK_OF_PARAM,"secret");
}
return fusionService.registerConnection(fusionConnectionParam);
}

@RequestMapping("changeConnection")
public BaseResultEntity changeConnection(FusionConnectionParam fusionConnectionParam){
if(fusionConnectionParam.getGlobalName()==null|| "".equals(fusionConnectionParam.getGlobalName().trim())) {
return BaseResultEntity.failure(BaseResultEnum.LACK_OF_PARAM,"globalName");
}
if(fusionConnectionParam.getGatewayAddress()==null|| "".equals(fusionConnectionParam.getGatewayAddress())) {
return BaseResultEntity.failure(BaseResultEnum.LACK_OF_PARAM,"gatewayAddress");
}
return fusionService.changeConnection(fusionConnectionParam);
}

@RequestMapping("findOrganByGlobalId")
public BaseResultEntity findOrganByGlobalId(String[] globalIdArray){
if(globalIdArray==null||globalIdArray.length==0) {
return BaseResultEntity.failure(BaseResultEnum.LACK_OF_PARAM,"globalIdArray");
}
return fusionService.findOrganByGlobalId(globalIdArray);
@RequestMapping("organData")
public BaseResultEntity organData(String organId, String organName){
return organService.organData(organId,organName);
}

@RequestMapping("changeOrganExtends")
public BaseResultEntity changeOrganExtends(FusionOrganExtendsParam param){
if (StringUtils.isEmpty(param.getIp())) {
return BaseResultEntity.failure(BaseResultEnum.LACK_OF_PARAM,"ip");
}
if (StringUtils.isEmpty(param.getLat())) {
return BaseResultEntity.failure(BaseResultEnum.LACK_OF_PARAM,"lat");
}
if (StringUtils.isEmpty(param.getLon())) {
return BaseResultEntity.failure(BaseResultEnum.LACK_OF_PARAM,"lon");
}
return fusionService.changeOrganExtends(param);
}

@RequestMapping("getOrganExtendsList")
public BaseResultEntity getOrganExtendsList(){
return fusionService.getOrganExtendsList();
}

}
Original file line number Diff line number Diff line change
@@ -1,24 +1,36 @@
package com.primihub.controller;

import com.alibaba.fastjson.JSONArray;
import com.alibaba.fastjson.JSONObject;
import com.alibaba.fastjson.JSONValidator;
import com.primihub.entity.DataSet;
import com.primihub.entity.base.BaseResultEntity;
import com.primihub.entity.base.BaseResultEnum;
import com.primihub.entity.resource.param.OrganResourceParam;
import com.primihub.entity.copy.dto.CopyResourceDto;
import com.primihub.entity.resource.param.ResourceParam;
import com.primihub.service.ResourceService;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.util.StringUtils;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

import java.util.List;
import java.util.Map;
import java.util.Set;

@RequestMapping("fusionResource")
@RestController
@Slf4j
public class FusionResourceController {

@Autowired
private ResourceService resourceService;

@RequestMapping("getResourceList")
public BaseResultEntity getResourceList(ResourceParam resourceParam){
public BaseResultEntity getResourceList(@RequestBody ResourceParam resourceParam){
return resourceService.getResourceList(resourceParam);
}

Expand All @@ -29,6 +41,16 @@ public BaseResultEntity getResourceListById(String[] resourceIdArray,String glob
}
return resourceService.getResourceListById(resourceIdArray,globalId);
}
@RequestMapping("getCopyResource")
public BaseResultEntity getCopyResource(String[] resourceIds){
if(resourceIds==null||resourceIds.length==0) {
return BaseResultEntity.failure(BaseResultEnum.LACK_OF_PARAM,"resourceIds");
}
log.info(JSONObject.toJSONString(resourceIds));
BaseResultEntity copyResource = resourceService.getCopyResource(resourceIds);
log.info(JSONObject.toJSONString(copyResource));
return copyResource;
}

@RequestMapping("getResourceTagList")
public BaseResultEntity getResourceTagList(){
Expand All @@ -42,30 +64,28 @@ public BaseResultEntity getDataResource(String resourceId,String globalId){
}
return resourceService.getDataResource(resourceId,globalId);
}
@RequestMapping("saveOrganResourceAuth")
public BaseResultEntity saveOrganResourceAuth(String organId,String resourceId,String projectId,Integer auditStatus){
if (StringUtils.isEmpty(organId)) {
return BaseResultEntity.failure(BaseResultEnum.LACK_OF_PARAM,"organId");
}
if (StringUtils.isEmpty(resourceId)) {
return BaseResultEntity.failure(BaseResultEnum.LACK_OF_PARAM,"resourceId");
}
if (StringUtils.isEmpty(projectId)) {
return BaseResultEntity.failure(BaseResultEnum.LACK_OF_PARAM,"projectId");
@PostMapping("saveResource")
public BaseResultEntity saveResource(String globalId, @RequestBody List<CopyResourceDto> copyResourceDtoList){
if (StringUtils.isEmpty(globalId)) {
return BaseResultEntity.failure(BaseResultEnum.LACK_OF_PARAM,"globalId");
}
if (auditStatus==null) {
return BaseResultEntity.failure(BaseResultEnum.LACK_OF_PARAM,"auditStatus");
if (copyResourceDtoList==null || copyResourceDtoList.size()==0) {
return BaseResultEntity.failure(BaseResultEnum.LACK_OF_PARAM,"copyResourceDtoList");
}
return resourceService.saveOrganResourceAuth(organId,resourceId,projectId,auditStatus);
return resourceService.batchSaveResource(globalId,copyResourceDtoList);
}

@RequestMapping("getOrganResourceList")
public BaseResultEntity getOrganResourceList(OrganResourceParam param){
if (StringUtils.isEmpty(param.getOrganId())) {
return BaseResultEntity.failure(BaseResultEnum.LACK_OF_PARAM,"organId");
}
return resourceService.getOrganResourceList(param);
@RequestMapping("getTestDataSet")
public BaseResultEntity getTestDataSet(String id){
return resourceService.getTestDataSet(id);
}

@PostMapping("batchSaveTestDataSet")
public BaseResultEntity batchSaveTestDataSet(@RequestBody List<DataSet> dataSets){
if (dataSets == null || dataSets.size()==0) {
return BaseResultEntity.failure(BaseResultEnum.LACK_OF_PARAM,"shareData - dataSets");
}
return resourceService.batchSaveTestDataSet(dataSets);
}

}
Loading

0 comments on commit 941b751

Please sign in to comment.