Spike - OpenSearch Observability plugin and Wazuh metrics assessment #195
Comments
asteriscos
changed the title
asteriscos
changed the title
Update 2024-07-03The code of the Observability plugin is investigated and analyzed focusing on the idea of using the Notebook
Something to note that may be interesting is that they use DashboardContainerByValueRenderer, similar to the current rendering of Dashboards
|
Update 2024-07-04Notebook rendering tracking is deepened. Flowchart from the Notebook main
OutputBody VISUALIZATION code casecase 'VISUALIZATION':
let from = moment(visInput?.timeRange?.from).format(dateFormat);
let to = moment(visInput?.timeRange?.to).format(dateFormat);
from = from === 'Invalid date' ? visInput.timeRange.from : from;
to = to === 'Invalid date' ? visInput.timeRange.to : to;
return (
<>
<EuiText size="s" style={{ marginLeft: 9 }}>
{`${from} - ${to}`}
</EuiText>
<DashboardContainerByValueRenderer
key={key}
input={visInput}
onInputUpdated={setVisInput}
/>
</>
);visInput{
"viewMode": "view",
"panels": {
"1": {
"gridData": {
"x": 0,
"y": 0,
"w": 48,
"h": 20,
"i": "1"
},
"type": "visualization",
"explicitInput": {
"id": "1",
"savedObjectId": "c6182e90-3a46-11ef-9824-2bce77daa33d"
}
}
},
"isFullScreenMode": false,
"filters": [],
"useMargins": false,
"id": "ie5b34eb1-3a46-11ef-b2f0-61dbd60e329b",
"visSavedObjId": "c6182e90-3a46-11ef-9824-2bce77daa33d",
"timeRange": {
"to": "2024-07-04T20:49:10.429Z",
"from": "2024-06-04T20:49:10.429Z"
},
"title": "embed_viz_ie5b34eb1-3a46-11ef-b2f0-61dbd60e329b",
"query": {
"query": "",
"language": "lucene"
},
"refreshConfig": {
"pause": true,
"value": 15
}
}Note Although in this case inputVis uses a savedObjectId, in theory it should also support the definition of a visualization as currently used in the different dashboards. |
Update 2024/07/05Based on what was previously investigated, progress is made in testing whether a Original Threat Hunting screen
Panels with visualizations of the Threat Hunting center in a Notebook
|
Update 2024-07-08Based on the Notebook CRUD components, it is observed that they use an API to create and store data. What's more, when loading notebook examples, the API is used to generate the Notebooks, as well as the corresponding visualizations are generated at the moment to assign them to the Notebooks' paragraphs. Taking into account the following files we can see how the Notebooks are managed:
Below is the method OSD uses to add the Notebooks exampleaddSampleNotebooks = async () => {
try {
this.setState({ loading: true });
const flights = await this.props.http
.get('../api/saved_objects/_find', {
query: {
type: 'index-pattern',
search_fields: 'title',
search: 'opensearch_dashboards_sample_data_flights',
},
})
.then((resp) => resp.total === 0);
const logs = await this.props.http
.get('../api/saved_objects/_find', {
query: {
type: 'index-pattern',
search_fields: 'title',
search: 'opensearch_dashboards_sample_data_logs',
},
})
.then((resp) => resp.total === 0);
if (flights || logs) this.setToast('Adding sample data. This can take some time.');
await Promise.all([
flights ? this.props.http.post('../api/sample_data/flights') : Promise.resolve(),
logs ? this.props.http.post('../api/sample_data/logs') : Promise.resolve(),
]);
const visIds: string[] = [];
await this.props.http
.get('../api/saved_objects/_find', {
query: {
type: 'visualization',
search_fields: 'title',
search: '[Logs] Response Codes Over Time + Annotations',
},
})
.then((resp) => visIds.push(resp.saved_objects[0].id));
await this.props.http
.get('../api/saved_objects/_find', {
query: {
type: 'visualization',
search_fields: 'title',
search: '[Logs] Unique Visitors vs. Average Bytes',
},
})
.then((resp) => visIds.push(resp.saved_objects[0].id));
await this.props.http
.get('../api/saved_objects/_find', {
query: {
type: 'visualization',
search_fields: 'title',
search: '[Flights] Flight Count and Average Ticket Price',
},
})
.then((resp) => visIds.push(resp.saved_objects[0].id));
await this.props.http
.post(`${NOTEBOOKS_API_PREFIX}/note/addSampleNotebooks`, {
body: JSON.stringify({ visIds }),
})
.then((res) => {
const newData = res.body.map((notebook: any) => ({
path: notebook.name,
id: notebook.id,
dateCreated: notebook.dateCreated,
dateModified: notebook.dateModified,
}));
this.setState((prevState) => ({
data: [...prevState.data, ...newData],
}));
});
this.setToast(`Sample notebooks successfully added.`);
} catch (err: any) {
this.setToast('Error adding sample notebooks.', 'danger');
console.error(err.body.message);
} finally {
this.setState({ loading: false });
}
};How to create a Notebook// Creates a new notebook
createNotebook = (newNoteName: string) => {
if (newNoteName.length >= 50 || newNoteName.length === 0) {
this.setToast('Invalid notebook name', 'danger');
window.location.assign('#/');
return;
}
const newNoteObject = {
name: newNoteName,
};
return this.props.http
.post(`${NOTEBOOKS_API_PREFIX}/note`, {
body: JSON.stringify(newNoteObject),
})
.then(async (res) => {
this.setToast(`Notebook "${newNoteName}" successfully created!`);
window.location.assign(`#/${res}`);
})
.catch((err) => {
this.setToast(
'Please ask your administrator to enable Notebooks for you.',
'danger',
<EuiLink href={NOTEBOOKS_DOCUMENTATION_URL} target="_blank">
Documentation
</EuiLink>
);
console.error(err);
});
};How to load a NotebookloadNotebook = () => {
this.showParagraphRunning('queue');
this.props.http
.get(`${NOTEBOOKS_API_PREFIX}/note/` + this.props.openedNoteId)
.then(async (res) => {
this.setBreadcrumbs(res.path);
let index = 0;
for (index = 0; index < res.paragraphs.length; ++index) {
// if the paragraph is a query, load the query output
if (res.paragraphs[index].output[0]?.outputType === 'QUERY') {
await this.loadQueryResultsFromInput(res.paragraphs[index]);
}
}
this.setState(res, this.parseAllParagraphs);
})
.catch((err) => {
this.props.setToast(
'Error fetching notebooks, please make sure you have the correct permission.',
'danger'
);
console.error(err);
});
};How to add a paragraph in a Notebook addPara = (index: number, newParaContent: string, inpType: string) => {
const addParaObj = {
noteId: this.props.openedNoteId,
paragraphIndex: index,
paragraphInput: newParaContent,
inputType: inpType,
};
return this.props.http
.post(`${NOTEBOOKS_API_PREFIX}/paragraph/`, {
body: JSON.stringify(addParaObj),
})
.then((res) => {
const paragraphs = [...this.state.paragraphs];
paragraphs.splice(index, 0, res);
const newPara = this.parseParagraphs([res])[0];
newPara.isInputExpanded = true;
const parsedPara = [...this.state.parsedPara];
parsedPara.splice(index, 0, newPara);
this.setState({ paragraphs, parsedPara });
this.paragraphSelector(index);
if (this.state.selectedViewId === 'output_only')
this.setState({ selectedViewId: 'view_both' });
})
.catch((err) => {
this.props.setToast(
'Error adding paragraph, please make sure you have the correct permission.',
'danger'
);
console.error(err);
});
};Paragraph rendering based on typeif (typeOut !== undefined) {
switch (typeOut) {
case 'QUERY':
const inputQuery = para.inp.substring(4, para.inp.length);
const queryObject = JSON.parse(val);
if (queryObject.hasOwnProperty('error')) {
return <EuiCodeBlock key={key}>{val}</EuiCodeBlock>;
} else {
const columns = createQueryColumns(queryObject.schema);
const data = getQueryOutputData(queryObject);
return (
<div>
<EuiText key={'query-input-key'} className="wrapAll" data-test-subj="queryOutputText">
<b>{inputQuery}</b>
</EuiText>
<EuiSpacer />
<QueryDataGridMemo
key={key}
rowCount={queryObject.datarows.length}
queryColumns={columns}
dataValues={data}
/>
</div>
);
}
case 'MARKDOWN':
return (
<EuiText
key={key}
className="wrapAll markdown-output-text"
data-test-subj="markdownOutputText"
>
<MarkdownRender source={val} />
</EuiText>
);
case 'VISUALIZATION':
let from = moment(visInput?.timeRange?.from).format(dateFormat);
let to = moment(visInput?.timeRange?.to).format(dateFormat);
from = from === 'Invalid date' ? visInput.timeRange.from : from;
to = to === 'Invalid date' ? visInput.timeRange.to : to;
return (
<>
<EuiText size="s" style={{ marginLeft: 9 }}>
{`${from} - ${to}`}
</EuiText>
<DashboardContainerByValueRenderer
key={key}
input={visInput}
onInputUpdated={setVisInput}
/>
</>
);
case 'OBSERVABILITY_VISUALIZATION':
let fromObs = moment(visInput?.timeRange?.from).format(dateFormat);
let toObs = moment(visInput?.timeRange?.to).format(dateFormat);
fromObs = fromObs === 'Invalid date' ? visInput.timeRange.from : fromObs;
toObs = toObs === 'Invalid date' ? visInput.timeRange.to : toObs;
const onEditClick = (savedVisualizationId: string) => {
window.location.assign(`observability-logs#/explorer/${savedVisualizationId}`);
};
return (
<>
<EuiText size="s" style={{ marginLeft: 9 }}>
{`${fromObs} - ${toObs}`}
</EuiText>
<div style={{ height: '300px', width: '100%' }}>
<VisualizationContainer
http={getOSDHttp()}
editMode={false}
visualizationId={''}
onEditClick={onEditClick}
savedVisualizationId={para.visSavedObjId}
pplService={getPPLService()}
fromTime={para.visStartTime}
toTime={para.visEndTime}
onRefresh={false}
pplFilterValue={''}
usedInNotebooks={true}
/>
</div>
</>
);
case 'HTML':
return (
<EuiText key={key}>
{/* eslint-disable-next-line react/jsx-pascal-case */}
<Media.HTML data={val} />
</EuiText>
);
case 'TABLE':
return <pre key={key}>{val}</pre>;
case 'IMG':
return <img alt="" src={'data:image/gif;base64,' + val} key={key} />;
default:
return <pre key={key}>{val}</pre>;
}
} else {
console.log('output not supported', typeOut);
return <pre />;
}TO DO: Analyze how to combine the use of Notifications Channels to send a report based on a notebook |
Update 2024-08-08Researching Traces indexes for view operation and added scripts to generate sample data in that view. To the branch |
Update 2024-08-09investigating what indexes were allowed for the Observability > Metrics view, I could find in the code that it is filtering for indexes with the name ss4o_metrics-*-*. Then I tried to add documents and I had problems with the template so I investigated the fields and the format of each field and I found this file where it said so. so I started building the template and testing the examples that are in the same folder. (Then I found that the templates are assembled here)
|
|
|
Branch wazuh-dashboard-plugin:
Opensearch repository of a demo of opentelemetry: https://github.com/opensearch-project/opentelemetry-demo/tree/main Metrics fields and samples: https://github.com/opensearch-project/opensearch-catalog/blob/main/docs/schema/observability/metrics/metrics.md Metrics configuration blog: https://opensearch.org/blog/opentelemetry-metrics-visualization/#visualizations-with-tsvb |
Description
For the next major release of Wazuh, we want to incorporate metrics and traces about the different components of Wazuh in the dashboard.
To achieve this, we want to leverage the OpenSearch Observability plugin, as it provides a framework to work with metrics and traces. This framework is works with OpenTelemetry, which will be used in other Wazuh components.
The goal of this issue is to identify the capabilities and restrictions of the OpenSearch Observability plugin to generate Wazuh metrics and traceability reports. Within the observability plugin lies the Notebooks application which allows the enhancement of standard dashboards with code snippets, live visualizations, and narrative text.
These Notebooks can be used to generate complex reports.
We need to:
References:
Functional Requirements
Implementation Restrictions
Plan
Objective
The text was updated successfully, but these errors were encountered: