Skip to content

Commit

Permalink
Gh-380: Update user guide querying section (#393)
Browse files Browse the repository at this point in the history
* Added in new user guide section

* Added new development guide structure

* Added in new reference structure

* Added new administration guide structure

* Added new change notes structure

* Added example deployment to developer docs

* Added Endpoints guide

* add pages on filtering and operations

* add section on transformation

* add aggregation section

* tidy the import/export page

* add examples for java and python operations

---------

Co-authored-by: l46978 <[email protected]>
Co-authored-by: cn337131 <[email protected]>
Co-authored-by: GCHQDeveloper314 <[email protected]>
  • Loading branch information
4 people authored Sep 21, 2023
1 parent c295e33 commit a515d2f
Show file tree
Hide file tree
Showing 19 changed files with 1,032 additions and 234 deletions.
24 changes: 12 additions & 12 deletions docs/change-notes/changelist/v2-changes.md
Original file line number Diff line number Diff line change
@@ -1,37 +1,37 @@
# Gaffer 2 Changelist
# Gaffer 2 Changelist

Below is a summary of changes that have been made in Gaffer version 2.
Below is a summary of changes that have been made in Gaffer version 2.

### Accumulo 2 Support
The Accumulo store now supports Accumulo 2 and Hadoop 3 by default, with support for Accumulo 1 and Hadoop 2 retained. See the [Accumulo Migration page](../migrating-from-v1-to-v2/accumulo-migration.md) for more information about this change.

### Federated Store Improvements
The Federated Operation was added to greatly improve flexibility of using a Federated Store.
The Federated Operation was added to greatly improve flexibility of using a Federated Store.
!!! danger "Breaking change"
To migrate, please see the [Federated Store Changes page](../migrating-from-v1-to-v2/federation-changes.md).

### Cache Improvements and fixes
All "caches" within Gaffer received a lot of bug fixes which should make them significantly more stable and consistent over time. This should improve usability of FederatedStores, NamedOperations and NamedViews.
!!! danger "Breaking change"
The cache will need to be reloaded, as the new internal cache interface has changed. To do this, export all of the contents of your cache, upgrade, then re-add everything manually.
The cache will need to be reloaded, as the new internal cache interface has changed. To do this, export all of the contents of your cache, upgrade, then re-add everything manually.

### Removal of Deprecated code
All of Gaffer 1's deprecated code has been removed.
All of Gaffer 1's deprecated code has been removed.
!!! danger "Breaking change"
To migrate, please see the [deprecations](../migrating-from-v1-to-v2/deprecations.md) page.

### Dependency Upgrades
Dependencies have been updated, where possible to the latest version, removing vulnerabilities.
Dependencies have been updated, where possible to the latest version, removing vulnerabilities.
!!! danger "Breaking change"
You will need to migrate your dependencies to be compatible with Gaffer 2's new dependency versions. Please see the [dependencies](../migrating-from-v1-to-v2/dependencies.md) page for full details.

### Federated and Proxy store fixes
A lot of bugs have been fixed that should facilitate FederatedStores with ProxyStores in them.
A lot of bugs have been fixed that should facilitate FederatedStores with ProxyStores in them.
!!! danger "Breaking change"
The unique store trait `DYNAMIC_SCHEMA` has been removed from Gaffer. Simply removing it from custom FederatedStore implementations should be an adequate fix.

### Removal of CloseableIterable
The `CloseableIterable` class has been removed so Operations like `GetAllElements` now return an `Iterable` instead, but the result still implements `Closeable`.
The `CloseableIterable` class has been removed so Operations like `GetAllElements` now return an `Iterable` instead, but the result still implements `Closeable`.
!!! danger "Breaking change"
Everywhere `CloseableIterable` was used in client code should be replaced with an `Iterable`:
```java
Expand All @@ -42,9 +42,9 @@ The `CloseableIterable` class has been removed so Operations like `GetAllElement
```

### Removal of HBase and Parquet stores
The HBase and Parquet stores have been removed from Gaffer in version 2. We made posts for both the [HBase](https://github.com/gchq/Gaffer/issues/2367) and [Parquet](https://github.com/gchq/Gaffer/discussions/2557) stores to understand the levels of usage. It was then decided to remove both stores as this would make introducing various improvements easier in the long term. HBase and Parquet remain available in Gaffer version 1. In the future, they could be reimplemented for Gaffer 2, though we do not plan to currently.
The HBase and Parquet stores have been removed from Gaffer in version 2. We made posts for both the [HBase](https://github.com/gchq/Gaffer/issues/2367) and [Parquet](https://github.com/gchq/Gaffer/discussions/2557) stores to understand the levels of usage. It was then decided to remove both stores as this would make introducing various improvements easier in the long term. HBase and Parquet remain available in Gaffer version 1. In the future, they could be reimplemented for Gaffer 2, though we do not plan to currently.
!!! danger "Breaking change"
We would recommend instead using an Accumulo Store. If you would like these store implementations in Gaffer 2, or any other potential store for that matter, please [make an issue](https://github.com/gchq/Gaffer/issues/new?assignees=&labels=enhancement&projects=&template=feature_request.md&title=) on GitHub.
We would recommend instead using an Accumulo Store. If you would like these store implementations in Gaffer 2, or any other potential store for that matter, please [make an issue](https://github.com/gchq/Gaffer/issues/new?assignees=&labels=enhancement&projects=&template=feature_request.md&title=) on GitHub.

### Gaffer now builds with Java 8 and Java 11
There is now a maven profile that will swap dependency versions so you can build Gaffer with Java 11. The code has also been updated to build with both Java versions.
Expand All @@ -53,7 +53,7 @@ There is now a maven profile that will swap dependency versions so you can build
The Accumulo store now supports authenticating to Accumulo and HDFS using Kerberos, in addition to username/password. For more information, see the [Kerberos support page](../migrating-from-v1-to-v2/accumulo-kerberos.md).

### CSV Import and Export
Basic support for importing and exporting [CSVs](../../user-guide/query/api-querying/import-export-data.md) has been added.
Basic support for importing and exporting [CSVs](../../user-guide/query/gaffer-syntax/import-export/csv.md) has been added.

### All operations can now be used within NamedOperations
Previously, `GetElementsBetweenSets` could not be used within a NamedOperation as it used `inputB`. `GetElementsBetweenSets` and `inputB` have both been deprecated and instead you should use `GetElementsBetweenSetsPairs`.
Expand Down Expand Up @@ -127,4 +127,4 @@ This will mean subgraphs added to FederatedStores can have additional operation
```

1. Schema left empty for brevity
2. This example operation enables file import. Read more in the [CSV](../../user-guide/query/api-querying/import-export-data.md) docs.
2. This example operation enables file import. Read more in the [CSV](../../user-guide/query/gaffer-syntax/import-export/csv.md) docs.
2 changes: 1 addition & 1 deletion docs/change-notes/migrating-from-v1-to-v2/deprecations.md
Original file line number Diff line number Diff line change
Expand Up @@ -464,7 +464,7 @@ Now set in `serialiser`:
.library(myGraphLibrary)
.view(myView)
.addHook(customHook)
.build())
.build())
.addSchema(mySchema) // (1)!
.storeProperties(storeProperties) // (2)!
.build();
Expand Down
2 changes: 1 addition & 1 deletion docs/development-guide/example-deployment/project-setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ the graph in [Neo4j syntax](https://neo4j.com/labs/apoc/4.4/export/csv/#export-d
!!! note ""
Please note that Gaffer often requires additional information about the data such as,
`:String` on the column headers to help with typing of the values. This is demonstrated below
in the raw file. There's more detail on this in the [OpenCypher documentation](../../user-guide/query/api-querying/import-export-data.md#opencypher-formats).
in the raw file. There's more detail on this in the [OpenCypher documentation](../../user-guide/query/gaffer-syntax/import-export/csv.md#opencypher-formats).

=== "Table"
| _id | name | age | lang | _labels | _start | _end | _type | weight |
Expand Down
9 changes: 5 additions & 4 deletions docs/development-guide/example-deployment/using-the-api.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
# Using the API

As covered in the [Getting Started API page](../../user-guide/api.md) the main POST request used in the API is
`/graph/operations/execute`. This part of the guide will cover the general usage of this part of the
API and walk through some general operations you might want to use to load data or query.
As covered in the [Getting Started API page](../../user-guide/apis/rest-api.md) the main POST
request used in the API is `/graph/operations/execute`. This part of the guide will cover the
general usage of this part of the API and walk through some general operations you might want to
use to load data or query.

!!! note
The [Gaffer operations reference guide](../../reference/operations-guide/operations.md)
Expand Down Expand Up @@ -61,7 +62,7 @@ from the neo4j formatted CSV file.

The operation chain above essentially mirrors the stages in the previous diagram. In the example
chain we first ingest the data via the `ImportFromLocalFile` class (an additional operation we added
via the `operationsDeclarations.json`, which
via the `operationsDeclarations.json`), which
streams the data from the CSV file into the next `GenerateElements` operation.

For the generator we have selected the built in `Neo4jCsvElementGenerator` class, this is already
Expand Down
84 changes: 42 additions & 42 deletions docs/reference/operations-guide/misc.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ Gets data from an endpoint. [Javadoc](https://gchq.github.io/Gaffer/uk/gov/gchq/
```

=== "JSON"

``` json
{
"class" : "GetFromEndpoint",
Expand All @@ -38,10 +38,10 @@ Gets data from an endpoint. [Javadoc](https://gchq.github.io/Gaffer/uk/gov/gchq/
```

=== "Python"

``` python
g.GetFromEndpoint(
endpoint="https://mydata.io:8443/mydata"
g.GetFromEndpoint(
endpoint="https://mydata.io:8443/mydata"
)
```

Expand All @@ -68,7 +68,7 @@ This operation requires Store configuration to be set before it can be used. See
```

=== "JSON"

``` json
{
"class" : "ScoreOperationChain",
Expand All @@ -89,34 +89,34 @@ This operation requires Store configuration to be set before it can be used. See
```

=== "Python"

``` python
g.ScoreOperationChain(
operation_chain=g.OperationChain(
operations=[
g.GetElements(),
g.NamedOperation(
operation_name="namedOp"
),
g.Limit(
result_limit=3,
truncate=True
)
]
)
g.ScoreOperationChain(
operation_chain=g.OperationChain(
operations=[
g.GetElements(),
g.NamedOperation(
operation_name="namedOp"
),
g.Limit(
result_limit=3,
truncate=True
)
]
)
)
```

Results:

=== "Java"

``` java
5
```

=== "JSON"

``` json
5
```
Expand All @@ -140,7 +140,7 @@ Here we have added a NamedOperation to the NamedOperationCache, with a custom sc
```

=== "JSON"

``` json
{
"class" : "ScoreOperationChain",
Expand All @@ -164,38 +164,38 @@ Here we have added a NamedOperation to the NamedOperationCache, with a custom sc
```

=== "Python"

``` python
g.ScoreOperationChain(
operation_chain=g.OperationChain(
operations=[
g.NamedOperation(
input=[
g.EntitySeed(
vertex=1
)
],
operation_name="1-hop"
),
g.Limit(
result_limit=3,
truncate=True
)
]
)
g.ScoreOperationChain(
operation_chain=g.OperationChain(
operations=[
g.NamedOperation(
input=[
g.EntitySeed(
vertex=1
)
],
operation_name="1-hop"
),
g.Limit(
result_limit=3,
truncate=True
)
]
)
)
```

Results:

=== "Java"

``` java
5
```

=== "JSON"

``` json
5
```
94 changes: 0 additions & 94 deletions docs/user-guide/api.md

This file was deleted.

Loading

0 comments on commit a515d2f

Please sign in to comment.