Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v1.0.0-rc5 #386

Merged
merged 66 commits into from
Jan 29, 2025
Merged
Changes from 1 commit
Commits
Show all changes
66 commits
Select commit Hold shift + click to select a range
0d84746
Minor improvements and fixes
Dec 19, 2024
51001dd
merge-develop
Jan 6, 2025
9da8fe9
updates
Jan 13, 2025
294d90d
merge-develop
Jan 13, 2025
d6ced19
updated node_env for prod deploy
Jan 13, 2025
fc2b5be
added process status in Mapping Packages list view
Jan 13, 2025
ca0be23
added process status in Mapping Packages list view
Jan 13, 2025
ffb4caa
added process status in Mapping Packages list view
Jan 14, 2025
7018d7b
Merge pull request #374 from meaningfy-ws/feature/MWB-897
kaleanych Jan 14, 2025
bc5078d
MWB-952: add result summary for xpath
SilverBerserk Jan 14, 2025
531a309
MWB-952: add charts for shacl and sparql
SilverBerserk Jan 14, 2025
abe6359
MWB-952: update charts style
SilverBerserk Jan 15, 2025
5e4ee18
MWB-952: add chart custom tooltip
SilverBerserk Jan 15, 2025
0331d47
MWB-952: update page layout
SilverBerserk Jan 15, 2025
f00b9ff
MWB-952: add links to pages
SilverBerserk Jan 15, 2025
77dbbd3
MWB-952: update xpath view
SilverBerserk Jan 15, 2025
d247a73
MWB-952: update lower pages for xpath
SilverBerserk Jan 15, 2025
377ba78
MWB-952: update pages
SilverBerserk Jan 15, 2025
18b49c5
MWB-952: fix file selection
SilverBerserk Jan 16, 2025
6210a5a
MWB-952: update colors
SilverBerserk Jan 16, 2025
1b7dbb8
MWB-952: update line graph
SilverBerserk Jan 16, 2025
4113566
MWB-952: update shacl pages
SilverBerserk Jan 16, 2025
0445d46
MWB-952: update shacl and xpath pages file interactions
SilverBerserk Jan 16, 2025
0e2bb9a
MWB-952: fixes for sparql
SilverBerserk Jan 16, 2025
2dfc920
MWB-952: update shacl page
SilverBerserk Jan 16, 2025
0beba2c
MWB-952: fix theme colors
SilverBerserk Jan 16, 2025
374520d
MWB-952: update export button functional
SilverBerserk Jan 16, 2025
3c1e667
added api endpoint for test data transform history
Jan 16, 2025
4cb97e3
MWB-952: colorfull filter for xpath
SilverBerserk Jan 16, 2025
fcebbda
Merge pull request #375 from meaningfy-ws/feature/MWB-893
kaleanych Jan 16, 2025
492eac5
added demo config section
Jan 16, 2025
fdf9af4
MWB-952: update xpath folder table
SilverBerserk Jan 17, 2025
861fdea
MWB-952: update xpath file table
SilverBerserk Jan 17, 2025
972a8dd
Merge pull request #376 from meaningfy-ws/feature/MWB-959
kaleanych Jan 17, 2025
b68aa3a
MWB-952: update sparql result column
SilverBerserk Jan 20, 2025
ec3870a
MWB-952: update sparql tables
SilverBerserk Jan 20, 2025
16719bc
MWB-952: update shacl tables
SilverBerserk Jan 20, 2025
3f81924
MWB-952: fix test
SilverBerserk Jan 21, 2025
d7bb3d4
Merge pull request #377 from meaningfy-ws/feature/MWB-952
kaleanych Jan 21, 2025
679a747
added import logs
Jan 21, 2025
d79f81a
Merge branch 'develop' into feature/MWB-909
Jan 21, 2025
5c941f2
added extra validations and logs on package import
Jan 21, 2025
2a6bc55
added extra validations and logs on package import
Jan 21, 2025
a68ba02
Merge pull request #378 from meaningfy-ws/feature/MWB-960
kaleanych Jan 21, 2025
efe5277
MWB-956: add compare codemirror module
SilverBerserk Jan 23, 2025
1240070
MWB-956: add compare button and select time switch
SilverBerserk Jan 23, 2025
bcf1f7a
MWB-956: change position for compare
SilverBerserk Jan 23, 2025
e0d5e82
Merge pull request #379 from meaningfy-ws/feature/MWB-956
kaleanych Jan 23, 2025
529c6b6
added extra validations and logs on package import
Jan 23, 2025
40a8381
Merge pull request #380 from meaningfy-ws/feature/MWB-961
kaleanych Jan 23, 2025
47572b3
MWB-952: update filter for states
SilverBerserk Jan 24, 2025
c99629f
MWB-952: code clean-up
SilverBerserk Jan 24, 2025
2802373
MWB-952: more code fixes
SilverBerserk Jan 24, 2025
2c11056
MWB-952: add sort by notices
SilverBerserk Jan 24, 2025
4d2dcac
Merge pull request #381 from meaningfy-ws/feature/MWB-952-fix1
kaleanych Jan 24, 2025
8cf9bc8
Fixed the CM import concurency issue
Jan 27, 2025
4b6233d
Fixed the CM import concurency issue
Jan 28, 2025
b511c19
Fixed the CM import concurency issue
Jan 28, 2025
fc29a8e
Updated Sonar github action
Jan 28, 2025
775ce69
Merge pull request #382 from meaningfy-ws/feature/MWB-896
kaleanych Jan 28, 2025
8a7b340
MWB-967: fix search issues
SilverBerserk Jan 28, 2025
8ef8f28
Merge pull request #383 from meaningfy-ws/feature/MWB-967
kaleanych Jan 29, 2025
aa0d18f
MWB-12-33: fix importing formats
SilverBerserk Jan 29, 2025
a673c76
MWB-12-33: update paginations on project
SilverBerserk Jan 29, 2025
747b887
Merge pull request #384 from meaningfy-ws/feature/MWB12-39
kaleanych Jan 29, 2025
1b9755a
Merge pull request #385 from meaningfy-ws/feature/MWB12-33
kaleanych Jan 29, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
added extra validations and logs on package import
Kolea PLESCO authored and Kolea PLESCO committed Jan 21, 2025
commit 5c941f218cd1fb2fe7c8622ce3d4654296ebc7a7
6 changes: 3 additions & 3 deletions mapping_workbench/backend/fields_registry/services/data.py
Original file line number Diff line number Diff line change
@@ -7,14 +7,14 @@


async def get_structural_element_by_unique_fields(
sdk_element_id, bt_id, absolute_xpath, project_id: PydanticObjectId, name: str = None
sdk_element_id, absolute_xpath, project_id: PydanticObjectId, bt_id: str = None, name: str = None
) -> StructuralElement:
project_link = Project.link_from_id(project_id)
return await StructuralElement.find_one(
StructuralElement.project == project_link,
StructuralElement.sdk_element_id == sdk_element_id,
#StructuralElement.name == name,
StructuralElement.bt_id == bt_id,
# StructuralElement.name == name,
# StructuralElement.bt_id == bt_id,
StructuralElement.absolute_xpath == absolute_xpath
)

Original file line number Diff line number Diff line change
@@ -8,7 +8,7 @@
from mapping_workbench.backend.package_importer.adapters.importer_abc import PackageImporterABC
from mapping_workbench.backend.package_importer.models.imported_mapping_suite import ImportedMappingSuite
from mapping_workbench.backend.project.models.entity import Project
from mapping_workbench.backend.tasks.models.task_response import TaskResponse
from mapping_workbench.backend.tasks.models.task_response import TaskResponse, TaskResultWarning
from mapping_workbench.backend.triple_map_fragment.models.entity import GenericTripleMapFragment
from mapping_workbench.backend.user.models.user import User

@@ -71,22 +71,24 @@ async def add_mapping_rules_from_mono(self, mono_package: ImportedMappingSuite):
for mono_rule in mono_package.conceptual_rules:
source_structural_element: StructuralElement = await get_structural_element_by_unique_fields(
sdk_element_id=mono_rule.eforms_sdk_id,
bt_id=mono_rule.bt_id,
absolute_xpath=mono_rule.absolute_xpath,
project_id=self.project.id,
# bt_id=mono_rule.bt_id,
# name=mono_rule.field_name
)

if not source_structural_element:
m = f"CM(sdk_id, bt_id, xpath) not imported: ({mono_rule.eforms_sdk_id}, {mono_rule.bt_id}, {mono_rule.absolute_xpath})"
mwb_logger.log_all_warning(m)
self.warnings.append(m)
continue

if source_structural_element.bt_id != mono_rule.bt_id:
m = f"{mono_rule.eforms_sdk_id}, {source_structural_element.bt_id} <> {mono_rule.bt_id}, {mono_rule.absolute_xpath}"
mwb_logger.log_all_warning(m)
self.warnings.append(TaskResultWarning(message=m, type="CM(sdk_id, sdk_bt_id <> bt_id, xpath) BT ID Mismatch"))

if source_structural_element.name != mono_rule.field_name:
m = f"Field[{source_structural_element.sdk_element_id}] has Imported Name ({mono_rule.field_name}) <> Current Name ({source_structural_element.name})"
mwb_logger.log_all_warning(m)
self.warnings.append(m)
self.warnings.append(TaskResultWarning(message=m, type="Field Name Mismatch"))

# A conceptual mapping rule may have same structural element but different Ontology Fragment
rule: ConceptualMappingRule = await ConceptualMappingRule.find_one(
Original file line number Diff line number Diff line change
@@ -22,7 +22,7 @@
from mapping_workbench.backend.sparql_test_suite.services.data import SPARQL_CM_ASSERTIONS_SUITE_TITLE, \
SPARQL_INTEGRATION_TESTS_SUITE_TITLE
from mapping_workbench.backend.task_manager.adapters.task_progress import TaskProgress
from mapping_workbench.backend.tasks.models.task_response import TaskResponse
from mapping_workbench.backend.tasks.models.task_response import TaskResponse, TaskResultWarning
from mapping_workbench.backend.test_data_suite.models.entity import TestDataSuite, TestDataFileResource, \
TestDataFileResourceFormat
from mapping_workbench.backend.triple_map_fragment.models.entity import TripleMapFragmentFormat, \
@@ -32,7 +32,7 @@

class PackageImporterABC(ABC):
package: MappingPackage
warnings: List[str] = []
warnings: List[TaskResultWarning] = []
task_progress: TaskProgress

def __init__(self, project: Project, user: User, task_response: TaskResponse = None):
Original file line number Diff line number Diff line change
@@ -50,9 +50,9 @@ async def add_mapping_rules_from_mono(self, mono_package: ImportedMappingSuite):
for mono_rule in mono_package.conceptual_rules:
source_structural_element: StructuralElement = await get_structural_element_by_unique_fields(
sdk_element_id=mono_rule.field_name,
bt_id=mono_rule.bt_id,
absolute_xpath=mono_rule.absolute_xpath,
project_id=self.project.id,
# bt_id=mono_rule.bt_id,
# name=mono_rule.field_name
)

5 changes: 3 additions & 2 deletions mapping_workbench/backend/task_manager/adapters/task.py
Original file line number Diff line number Diff line change
@@ -8,7 +8,8 @@
from pebble import ProcessFuture

from mapping_workbench.backend.config import settings
from mapping_workbench.backend.tasks.models.task_response import TaskStatus, TaskProgressData, TaskResponse
from mapping_workbench.backend.tasks.models.task_response import TaskStatus, TaskProgressData, TaskResponse, \
TaskResultWarning
from mapping_workbench.backend.tasks.models.task_result import TaskResult, TaskMetadata, TaskMetadataMeta


@@ -162,7 +163,7 @@ def update_exception_message(self, exception_message: str):
"""
self.task_metadata.exception_message = exception_message

def update_warnings(self, warnings: List[str]):
def update_warnings(self, warnings: List[TaskResultWarning]):
"""
"""
self.task_metadata.warnings = warnings
5 changes: 4 additions & 1 deletion mapping_workbench/backend/tasks/models/task_response.py
Original file line number Diff line number Diff line change
@@ -56,9 +56,12 @@ class TaskProgressData(TaskProgressDataBase):
actions: List[TaskProgressAction] = []
actions_count: int = 0

class TaskResultWarning(BaseModel):
message: str
type: str = None

class TaskResultData(TaskProgressDataBase):
warnings: List[str] = []
warnings: List[TaskResultWarning] = None
data: Any = None


6 changes: 3 additions & 3 deletions mapping_workbench/backend/tasks/models/task_result.py
Original file line number Diff line number Diff line change
@@ -4,7 +4,7 @@
from pydantic import BaseModel

from mapping_workbench.backend.tasks.models.task_entity import TaskEntity
from mapping_workbench.backend.tasks.models.task_response import TaskStatus, TaskProgressData
from mapping_workbench.backend.tasks.models.task_response import TaskStatus, TaskProgressData, TaskResultWarning


class TaskResult:
@@ -14,7 +14,7 @@ class TaskResult:
started_at: datetime = None
finished_at: datetime = None
exception_message: str = None
warnings: List[str] = []
warnings: List[TaskResultWarning] = None
task_status: TaskStatus = TaskStatus.FINISHED


@@ -37,7 +37,7 @@ class TaskMetadata(BaseModel):
started_at: datetime = None
finished_at: datetime = None
exception_message: str = None
warnings: List[str] = []
warnings: List[TaskResultWarning] = None
progress: TaskProgressData = None
meta: TaskMetadataMeta = None,
created_by: Optional[str] = None
51 changes: 43 additions & 8 deletions mapping_workbench/frontend/src/sections/app/tasks/list-table.js
Original file line number Diff line number Diff line change
@@ -30,6 +30,7 @@ import {MenuActionButton, MenuActions} from 'src/components/menu-actions';
import TableSorterHeader from "src/sections/components/table-sorter-header";
import TablePagination from "src/sections/components/table-pagination-pages";
import {mapStatusColor, TaskActions, TaskLine, taskProgressStatus as taskStatuses} from "./task-actions";
import {Box} from "@mui/system";


export const ListTable = (props) => {
@@ -77,6 +78,19 @@ export const ListTable = (props) => {
setPopoverShow(e => ({...e, anchor: undefined}))
}

const groupedWarnings = (warnings) => {
const result = {};

warnings.forEach(warning => {
if (!result[warning.type]) {
result[warning.type] = [];
}
result[warning.type].push(warning.message);
});

return result;
}

return (
<TablePagination
component="div"
@@ -245,17 +259,38 @@ export const ListTable = (props) => {
<Typography sx={{pl: 3, pt: 1}}
variant="h6"
color="orange">
Warning
Warnings <SeverityPill
color="warning">
{item.warnings?.length}
</SeverityPill>
</Typography>
<Divider/>
<List sx={{pl: 3}}>
{item.warnings.map((warning, key) =>
<ListItem
key={'warning' + key}>
{warning}
</ListItem>
<Box sx={{pl: 3}}>
{Object.entries(groupedWarnings(item.warnings)).map(([type, warnings]) =>
<Box sx={{pb: 3}}>
<Typography
variant="bod1"
sx={{fontWeight: 'bold'}}
>
{type} <SeverityPill
color="warning">
{warnings?.length}
</SeverityPill>
</Typography>

<List>
{warnings.map((warning, key) =>
<ListItem
key={'warning_' + type + '_' + key}>
{warning}
</ListItem>
)}
</List>

<Divider/>
</Box>
)}
</List>
</Box>
</>}
</Grid>
</CardContent>