Skip to content

Commit

Permalink
Added typed parameters to all notebooks (#3925)
Browse files Browse the repository at this point in the history 
This allows all notebooks (collection, hunt, event and global) to have
typed parameters. For non global notebooks, the parameters of the
original collection are preserved so they are available inside the
notebook.

The new GUI allows to update these parameters at any time - this give
the benefit of:

1. Notebook cells can be written to depend on active parameters that the
user can manipulate.
2. Parameters are now discoverable with the GUI easily showing which
additional parameters are available to the VQL inside the cell.
  • Loading branch information
@scudette
scudette authored Nov 28, 2024
1 parent 6357f70 commit 79219ac
Show file tree
Hide file tree
Showing 34 changed files with 1,971 additions and 423 deletions.
4 changes: 4 additions & 0 deletions .github/workflows/windows.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,10 @@ jobs:
uses: actions/setup-go@v4
with:
go-version: 1.23

# Caching seems to really slow down the build due to the time
# taken to save the cache.
cache: false
id: go

- name: Configure test environment
Expand Down
11 changes: 8 additions & 3 deletions artifacts/definitions/Demo/Plugins/GUI.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -142,9 +142,14 @@ sources:
**Each of the below cells should have a H2 heading**
## Check that notebook environment variables are populated
{{ $x := Query "SELECT * FROM items(\
item=dict(NotebookId=NotebookId, ClientId=ClientId,\
FlowId=FlowId, ArtifactName=ArtifactName))" | Expand }}
Some of these are populated from the artifact parameters.
{{ $x := Query "LET X = scope() SELECT * FROM items(\
item=dict(NotebookId=X.NotebookId, ClientId=X.ClientId,\
FlowId=X.FlowId, ArtifactName=X.ArtifactName, \
ChoiceSelector=X.ChoiceSelector, StartDate=X.StartDate, \
HuntId=X.HuntId))" | Expand }}
{{ range $x }}
* {{ Get . "_key" }} - {{ Get . "_value" }}
Expand Down
4 changes: 2 additions & 2 deletions go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ require (
github.com/Velocidex/grpc-go-pool v1.2.2-0.20241016164850-ff0cb80037a8
github.com/Velocidex/json v0.0.0-20220224052537-92f3c0326e5a
github.com/Velocidex/pkcs7 v0.0.0-20230220112103-d4ed02e1862a
github.com/Velocidex/sflags v0.3.1-0.20231011011525-620ab7ca8617
github.com/Velocidex/sflags v0.3.1-0.20241126160332-cc1a5b66b8f1
github.com/Velocidex/ttlcache/v2 v2.9.1-0.20240517145123-a3f45e86e130
github.com/Velocidex/yaml/v2 v2.2.8
github.com/Velocidex/zip v0.0.0-20210101070220-e7ecefb7aad7
Expand Down Expand Up @@ -69,7 +69,7 @@ require (
github.com/sebdah/goldie/v2 v2.5.3
github.com/sergi/go-diff v1.2.0
github.com/sirupsen/logrus v1.8.1
github.com/stretchr/testify v1.9.0
github.com/stretchr/testify v1.10.0
github.com/xor-gate/ar v0.0.0-20170530204233-5c72ae81e2b7 // indirect
github.com/xor-gate/debpkg v1.0.0
go.starlark.net v0.0.0-20230925163745-10651d5192ab
Expand Down
18 changes: 4 additions & 14 deletions go.sum
View file
Original file line number Diff line number Diff line change
Expand Up @@ -96,8 +96,8 @@ github.com/Velocidex/ordereddict v0.0.0-20230909174157-2aa49cc5d11d h1:fn372EqKy
github.com/Velocidex/ordereddict v0.0.0-20230909174157-2aa49cc5d11d/go.mod h1:+MqO5UMBemyFSm+yRXslbpFTwPUDhFHUf7HPV92twg4=
github.com/Velocidex/pkcs7 v0.0.0-20230220112103-d4ed02e1862a h1:H7dVazNcaE80V8cy99TF7LPpwzvr1uJ4I2nDjb5ek7E=
github.com/Velocidex/pkcs7 v0.0.0-20230220112103-d4ed02e1862a/go.mod h1:/fy/Eg4TQz9KkJduvZfGCnbWTQ/LKaknS2wYB52cU6c=
github.com/Velocidex/sflags v0.3.1-0.20231011011525-620ab7ca8617 h1:pxAOaYTYwbWhoSwRoJOT3TJmUAjD2D011A1c8M2yyEE=
github.com/Velocidex/sflags v0.3.1-0.20231011011525-620ab7ca8617/go.mod h1:TTYBEgQFkjJvyBOC2s7G1+mNPZ3IHuqLZmVFRzyPfq4=
github.com/Velocidex/sflags v0.3.1-0.20241126160332-cc1a5b66b8f1 h1:fLJ2AjY0dtDZEBhekp3bxyxGpWh5D/+NfOVlGXB5ROA=
github.com/Velocidex/sflags v0.3.1-0.20241126160332-cc1a5b66b8f1/go.mod h1:UpFVihkMZWl2JRkVRiZYie0e2l7Ry+vjlCHCs6XVKGU=
github.com/Velocidex/sigma-go v0.0.0-20241025122940-1b771d3d57a9 h1:YavusWZ+svpybfVg8cKk9MuSyeI479a0EGJ07VGpSm8=
github.com/Velocidex/sigma-go v0.0.0-20241025122940-1b771d3d57a9/go.mod h1:ukLFs2t1+ud7MC4oN+zImhtTRP/eQHaDL3TwLs58uUA=
github.com/Velocidex/tracee_velociraptor v0.0.0-20241111012344-d72a7f2cb144 h1:eDr8kkCPy4lCYBSc+AyuNz8aML3fnDqvEz3VVW8hX6A=
Expand All @@ -116,9 +116,7 @@ github.com/alecthomas/assert v1.0.0/go.mod h1:va/d2JC+M7F6s+80kl/R3G7FUiW6JzUO+h
github.com/alecthomas/colour v0.0.0-20160524082231-60882d9e2721/go.mod h1:QO9JBoKquHd+jz9nshCh40fOfO+JzsoXy8qTHF68zU0=
github.com/alecthomas/colour v0.1.0 h1:nOE9rJm6dsZ66RGWYSFrXw461ZIt9A6+nHgL7FRrDUk=
github.com/alecthomas/colour v0.1.0/go.mod h1:QO9JBoKquHd+jz9nshCh40fOfO+JzsoXy8qTHF68zU0=
github.com/alecthomas/kingpin v2.2.6+incompatible h1:5svnBTFgJjZvGKyYBtMB0+m5wvrbUHiqye8wRJMlnYI=
github.com/alecthomas/kingpin v2.2.6+incompatible/go.mod h1:59OFYbFVLKQKq+mqrL6Rw5bR0c3ACQaawgXx0QYndlE=
github.com/alecthomas/kingpin/v2 v2.3.2/go.mod h1:0gyi0zQnjuFk8xrkNKamJoyUo382HRL7ATRpFZCw6tE=
github.com/alecthomas/kingpin/v2 v2.4.0 h1:f48lwail6p8zpO1bC4TxtqACaGqHYA22qkHjHpqDjYY=
github.com/alecthomas/kingpin/v2 v2.4.0/go.mod h1:0gyi0zQnjuFk8xrkNKamJoyUo382HRL7ATRpFZCw6tE=
github.com/alecthomas/kong v0.2.4/go.mod h1:kQOmtJgV+Lb4aj+I2LEn40cbtawdWJ9Y8QLq+lElKxE=
Expand Down Expand Up @@ -152,7 +150,6 @@ github.com/andybalholm/cascadia v1.3.2 h1:3Xi6Dw5lHF15JtdcmAHD3i1+T8plmv7BQ/nsVi
github.com/andybalholm/cascadia v1.3.2/go.mod h1:7gtRlve5FxPPgIgX36uWBX58OdBsSS6lUvCFb+h7KvU=
github.com/araddon/dateparse v0.0.0-20210429162001-6b43995a97de h1:FxWPpzIjnTlhPwqqXc4/vE0f7GvRjuAsbW+HOIe8KnA=
github.com/araddon/dateparse v0.0.0-20210429162001-6b43995a97de/go.mod h1:DCaWoUhZrYW9p1lxo/cm8EmUOOzAPSEZNGF2DK1dJgw=
github.com/asaskevich/govalidator v0.0.0-20210307081110-f21760c49a8d/go.mod h1:WaHUgvxTVq04UNunO+XhnAqY/wQc+bxr74GqbsZ/Jqw=
github.com/aws/aws-sdk-go-v2 v1.25.2 h1:/uiG1avJRgLGiQM9X3qJM8+Qa6KRGK5rRPuXE0HUM+w=
github.com/aws/aws-sdk-go-v2 v1.25.2/go.mod h1:Evoc5AsmtveRt1komDwIsjHFyrP5tDuF1D1U+6z6pNo=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.1 h1:gTK2uhtAPtFcdRRJilZPx8uJLL2J85xK11nKtWL0wfU=
Expand Down Expand Up @@ -220,7 +217,6 @@ github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDk
github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGXZJjfX53e64911xZQV5JYwmTeXPW+k8Sc=
github.com/coreos/go-oidc/v3 v3.11.0 h1:Ia3MxdwpSw702YW0xgfmP1GVCMA9aEFWu12XUZ3/OtI=
github.com/coreos/go-oidc/v3 v3.11.0/go.mod h1:gE3LgjOgFoHi9a4ce4/tJczr0Ai2/BoDhf0r5lltWI0=
github.com/cpuguy83/go-md2man/v2 v2.0.2/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
github.com/creack/pty v1.1.17 h1:QeVUsEDNrLBW4tMgZHvxy18sKtr6VI492kBhUfhDJNI=
github.com/creack/pty v1.1.17/go.mod h1:MOBLtS5ELjhRRrroQr9kyvTxUAFNvYEK993ew/Vr4O4=
Expand Down Expand Up @@ -410,7 +406,6 @@ github.com/imdario/mergo v0.3.11/go.mod h1:jmQim1M+e3UYxmgPu/WyfjB3N3VflVyUjjjwH
github.com/imdario/mergo v0.3.12 h1:b6R2BslTbIEToALKP7LxUvijTsNI9TAe80pLWN2g/HU=
github.com/imdario/mergo v0.3.12/go.mod h1:jmQim1M+e3UYxmgPu/WyfjB3N3VflVyUjjjwH0dnCYA=
github.com/inconshreveable/log15 v0.0.0-20170622235902-74a0988b5f80/go.mod h1:cOaXtrgN4ScfRrD9Bre7U1thNq5RtJ8ZoP4iXVGRj6o=
github.com/inconshreveable/mousetrap v1.0.1/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
github.com/jackwakefield/gopac v1.0.2 h1:TdHTGtC/kMc5kyYw7PEO5kqAHxcpuU4RFY4ztfg4nAM=
Expand Down Expand Up @@ -532,8 +527,6 @@ github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e/go.mod h1:zD1mROLA
github.com/nsf/termbox-go v0.0.0-20190121233118-02980233997d/go.mod h1:IuKpRQcYE1Tfu+oAQqaLisqDeXgjyyltCfsaoYN18NQ=
github.com/nsf/termbox-go v1.1.1 h1:nksUPLCb73Q++DwbYUBEglYBRPZyoXJdrj5L+TkjyZY=
github.com/nsf/termbox-go v1.1.1/go.mod h1:T0cTdVuOwf7pHQNtfhnEbzHbcNyCEcVU4YPpouCbVxo=
github.com/octago/sflags v0.2.0 h1:XceYzkRXGAHa/lSFmKLcaxSrsh4MTuOMQdIGsUD0wlk=
github.com/octago/sflags v0.2.0/go.mod h1:G0bjdxh4qPRycF74a2B8pU36iTp9QHGx0w0dFZXPt80=
github.com/olekukonko/tablewriter v0.0.5 h1:P2Ga83D34wi1o9J6Wh1mRuqd4mF/x/lgBS7N7AbDhec=
github.com/olekukonko/tablewriter v0.0.5/go.mod h1:hPp6KlRPjbx+hW8ykQs1w3UBbZlj6HuIJcUGPhkA7kY=
github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
Expand Down Expand Up @@ -608,10 +601,8 @@ github.com/spf13/afero v0.0.0-20170901052352-ee1bd8ee15a1/go.mod h1:j4pytiNVoe2o
github.com/spf13/cast v1.1.0/go.mod h1:r2rcYCSwa1IExKTDiTfzaxqT2FNHs8hODu4LnUfgKEg=
github.com/spf13/cast v1.3.1 h1:nFm6S0SMdyzrzcmThSipiEubIDy8WEXKNZ0UOgiRpng=
github.com/spf13/cast v1.3.1/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE=
github.com/spf13/cobra v1.6.0/go.mod h1:IOw/AERYS7UzyrGinqmz6HLUo219MORXGxhbaJUqzrY=
github.com/spf13/jwalterweatherman v0.0.0-20170901151539-12bd96e66386/go.mod h1:cQK4TGJAtQXfYWX+Ddv3mKDzgVb68N+wFjFa4jdeBTo=
github.com/spf13/pflag v1.0.1-0.20170901120850-7aff26db30c1/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4=
github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/spf13/viper v1.0.0/go.mod h1:A8kyI5cUJhb8N+3pkfONlcEcZbueH6nhAm0Fq7SrnBM=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.2.0/go.mod h1:qt09Ya8vawLte6SNmTgCsAVtYtaKzEcn8ATUoHMkEqE=
Expand All @@ -629,10 +620,10 @@ github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
github.com/stretchr/testify v1.8.2/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
github.com/stretchr/testify v1.9.0 h1:HtqpIVDClZ4nwg75+f6Lvsy/wHu+3BoSGCbBAcpTsTg=
github.com/stretchr/testify v1.9.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
github.com/stretchr/testify v1.10.0 h1:Xv5erBjTwe/5IxqUQTdXv5kgmIvbHo3QQyRwhJsOfJA=
github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
github.com/syndtr/goleveldb v1.0.0 h1:fBdIW9lB4Iz0n9khmH8w27SJ3QEJ7+IgjPEwGSZiFdE=
github.com/syndtr/goleveldb v1.0.0/go.mod h1:ZVVdQEZoIme9iO1Ch2Jdy24qqXrMMOU6lpPAyBWyWuQ=
github.com/tidwall/pretty v1.0.0/go.mod h1:XNkn88O1ChpSDQmQeStsy+sBenx6DDtFZJxhVysOjyk=
Expand All @@ -642,7 +633,6 @@ github.com/tklauser/numcpus v0.3.0 h1:ILuRUQBtssgnxw0XXIjKUC56fgnOrFoQQ/4+DeU2bi
github.com/tklauser/numcpus v0.3.0/go.mod h1:yFGUr7TUHQRAhyqBcEg0Ge34zDBAsIvJJcyE6boqnA8=
github.com/ulikunitz/xz v0.5.11 h1:kpFauv27b6ynzBNT/Xy+1k+fK4WswhN/6PN5WhFAGw8=
github.com/ulikunitz/xz v0.5.11/go.mod h1:nbz6k7qbPmH4IRqmfOplQw/tblSgqTqBwxkY0oWt/14=
github.com/urfave/cli v1.20.0/go.mod h1:70zkFmudgCuE/ngEzBv17Jvp/497gISqfk5gWijbERA=
github.com/valyala/bytebufferpool v1.0.0 h1:GqA5TC/0021Y/b9FG4Oi9Mr3q7XYx6KllzawFIhcdPw=
github.com/valyala/bytebufferpool v1.0.0/go.mod h1:6bBcMArwyJ5K/AmCkWv1jt77kVWyCJ6HpOuEn7z0Csc=
github.com/valyala/fastjson v1.6.4 h1:uAUNq9Z6ymTgGhcm0UynUAB6tlbakBrz6CQFax3BXVQ=
Expand Down
55 changes: 31 additions & 24 deletions gui/velociraptor/src/components/events/event-notebook.jsx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ import NotebookRenderer from '../notebooks/notebook-renderer.jsx';
import _ from 'lodash';
import UserConfig from '../core/user.jsx';
import moment from 'moment';
import Spinner from '../utils/spinner.jsx';


const POLL_TIME = 5000;

Expand Down Expand Up @@ -66,10 +68,15 @@ export default class EventNotebook extends React.Component {

let notebook_id = get_notebook_id(this.props.artifact,
this.props.client_id);
this.setState({loading: true});
let old_notebook_id = this.state.notebook &&
this.state.notebook.notebook_id;
if(old_notebook_id !== notebook_id) {
this.setState({loading: true});
}
api.get("v1/GetNotebooks", {
notebook_id: notebook_id,
}, this.source.token).then(response=>{
this.setState({loading: false});
if (response.cancel) return;
let notebooks = response.data.items || [];

Expand All @@ -81,43 +88,43 @@ export default class EventNotebook extends React.Component {
let request = {
name: "Notebook for Event Artifact " + this.props.artifact,
notebook_id: notebook_id,
context: {
type: "event",
event_artifact: this.props.artifact,
client_id: this.props.client_id,
start_time: parseInt(this.props.start_time || 0),
},
// Hunt notebooks are all public.
public: true,

// By default scope the notebook to the range selected
// by the timeline.
env: [
{key: "ArtifactName", value: this.props.artifact},
{key: "ClientId", value: this.props.client_id},
{key: "StartTime",
value: this.formatTime(this.props.start_time)},
{key: "EndTime",
value: this.formatTime(this.props.end_time)},
{key: "NotebookId", value: notebook_id},
],
};

api.post('v1/NewNotebook', request, this.source.token).then((response) => {
if (response.cancel) return;
let cell_metadata = response.data && response.data.cell_metadata;
if (_.isEmpty(cell_metadata)) {
return;
}
this.setState({loading: true});
api.post('v1/NewNotebook', request,
this.source.token).then((response) => {
this.setState({loading: false});
if (response.cancel) return;

let cell_metadata = response.data &&
response.data.cell_metadata;
if (_.isEmpty(cell_metadata)) {
return;
}

this.fetchNotebooks();
});
this.fetchNotebooks();
});
});
}

render() {
return (
<NotebookRenderer
notebook={this.state.notebook}
fetchNotebooks={this.fetchNotebooks}
/>
);
return <>
<Spinner loading={this.state.loading } />
<NotebookRenderer
notebook={this.state.notebook}
fetchNotebooks={this.fetchNotebooks}
/>
</>;
}
};
52 changes: 45 additions & 7 deletions gui/velociraptor/src/components/events/events.jsx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ import DeleteNotebookDialog from '../notebooks/notebook-delete.jsx';
import T from '../i8n/i8n.jsx';
import ToolTip from '../widgets/tooltip.jsx';
import VeloLog from "../widgets/logs.jsx";
import { EditNotebook } from '../notebooks/new-notebook.jsx';

import { withRouter } from "react-router-dom";

Expand Down Expand Up @@ -371,13 +372,40 @@ class EventMonitoring extends React.Component {

<ButtonGroup className="float-right">
{ this.state.mode === mode_notebook &&
<ToolTip tooltip={T("Delete Notebook")}>
<Button onClick={() => this.setState({showDeleteNotebook: true})}
variant="default">
<FontAwesomeIcon icon="trash"/>
<span className="sr-only">{T("Delete Notebook")}</span>
</Button>
</ToolTip>
<>
<ToolTip tooltip={T("Edit Notebook")}>
<Button onClick={()=>{
this.setState({loading: true});
api.get("v1/GetNotebooks", {
include_uploads: true,
notebook_id: get_notebook_id(
this.state.artifact.artifact, client_id),

}, this.source.token).then(resp=>{
let items = resp.data.items;
if (_.isEmpty(items)) {
return;
}

this.setState({notebook: items[0],
loading: false,
showEditNotebookDialog: true});
});
}}
variant="default">
<FontAwesomeIcon icon="wrench"/>
<span className="sr-only">{T("Edit Notebook")}</span>
</Button>
</ToolTip>

<ToolTip tooltip={T("Delete Notebook")}>
<Button onClick={() => this.setState({showDeleteNotebook: true})}
variant="default">
<FontAwesomeIcon icon="trash"/>
<span className="sr-only">{T("Delete Notebook")}</span>
</Button>
</ToolTip>
</>
}
<Dropdown title="mode" variant="default">
<Dropdown.Toggle variant="default">
Expand Down Expand Up @@ -448,6 +476,16 @@ class EventMonitoring extends React.Component {
this.setState({showDeleteNotebook: false});
}}/>
}

{ this.state.showEditNotebookDialog &&
<EditNotebook
notebook={this.state.notebook}
updateNotebooks={()=>{
this.setState({showEditNotebookDialog: false});
}}
closeDialog={() => this.setState({showEditNotebookDialog: false})}
/>
}
</>
);
}
Expand Down
Loading

0 comments on commit 79219ac

Please sign in to comment.