Skip to content

Commit

Permalink
Merge branch '1918-update-multiple-tables-system-tests' into 1790-com…
Browse files Browse the repository at this point in the history 
…paction-contention-system-test
  • Loading branch information
@patchwork01
patchwork01 authored Mar 13, 2024
2 parents b1b330a + 232cfe2 commit 198170a
Show file tree
Hide file tree
Showing 17 changed files with 204 additions and 231 deletions.
17 changes: 16 additions & 1 deletion code-style/checkstyle-idea/checkstyle.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -190,8 +190,23 @@
<module name="IllegalImport">
<property name="illegalPkgs" value="org.junit.jupiter.api.Assertions"/>
</module>
<module name="Indentation">
<property name="id" value="indentation"/>
</module>
<module name="UnnecessarySemicolonAfterOuterTypeDeclaration">
<property name="id" value="unnecessarySemicolonAfterOuterTypeDeclaration"/>
</module>
<module name="UnnecessarySemicolonAfterTypeMemberDeclaration">
<property name="id" value="unnecessarySemicolonAfterTypeMemberDeclaration"/>
</module>
<module name="UnnecessarySemicolonInEnumeration">
<property name="id" value="unnecessarySemicolonInEnumeration"/>
</module>
<module name="UnnecessarySemicolonInTryWithResources">
<property name="id" value="unnecessarySemicolonInTryWithResources"/>
</module>
</module>
<module name="UniqueProperties">
<property name="id" value="uniqueProperties"/>
</module>
</module>
</module>
15 changes: 15 additions & 0 deletions code-style/checkstyle.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -196,6 +196,21 @@
<property name="illegalClasses"
value="^org\.junit\.jupiter\.api\.Assertions, ^org\.junit\.(?!jupiter|platform).+"/>
</module>
<module name="Indentation">
<property name="id" value="indentation"/>
</module>
<module name="UnnecessarySemicolonAfterOuterTypeDeclaration">
<property name="id" value="unnecessarySemicolonAfterOuterTypeDeclaration"/>
</module>
<module name="UnnecessarySemicolonAfterTypeMemberDeclaration">
<property name="id" value="unnecessarySemicolonAfterTypeMemberDeclaration"/>
</module>
<module name="UnnecessarySemicolonInEnumeration">
<property name="id" value="unnecessarySemicolonInEnumeration"/>
</module>
<module name="UnnecessarySemicolonInTryWithResources">
<property name="id" value="unnecessarySemicolonInTryWithResources"/>
</module>
</module>
<module name="UniqueProperties">
<property name="id" value="uniqueProperties"/>
Expand Down
4 changes: 2 additions & 2 deletions code-style/eclipse-style.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -163,7 +163,7 @@
<setting id="org.eclipse.jdt.core.formatter.alignment_for_arguments_in_annotation" value="0"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_comma_in_enum_declarations" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.alignment_for_annotations_on_package" value="0"/>
<setting id="org.eclipse.jdt.core.formatter.indent_switchstatements_compare_to_switch" value="false"/>
<setting id="org.eclipse.jdt.core.formatter.indent_switchstatements_compare_to_switch" value="true"/>
<setting id="org.eclipse.jdt.core.formatter.insert_new_line_before_else_in_if_statement" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_assignment_operator" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_new_line_after_label" value="do not insert"/>
Expand Down Expand Up @@ -339,7 +339,7 @@
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_colon_in_conditional" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_closing_paren_in_try" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_semicolon_in_try_resources" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.continuation_indentation_for_array_initializer" value="2"/>
<setting id="org.eclipse.jdt.core.formatter.continuation_indentation_for_array_initializer" value="1"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_question_in_wildcard" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_opening_paren_in_record_declaration" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.alignment_for_superinterfaces_in_enum_declaration" value="16"/>
Expand Down
20 changes: 11 additions & 9 deletions docs/14-release-process.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,25 +5,27 @@ The following steps explain how to prepare and publish a release for Sleeper, by

1. Update CHANGELOG.md with a summary of the issues fixed and improvements made in this version.

2. Create an issue for the release, and create a branch for that issue.
2. Update the [roadmap](16-roadmap.md) and remove any planned features that have been implemented in this release.

3. Set the new version number using `./scripts/dev/updateVersionNumber.sh`, e.g.
3. Create an issue for the release, and create a branch for that issue.

4. Set the new version number using `./scripts/dev/updateVersionNumber.sh`, e.g.

```bash
VERSION=0.12.0
./scripts/dev/updateVersionNumber.sh ${VERSION}
```

4. Push the branch to GitHub and open a pull request so that the tests run. If there are any failures, fix them.
5. Push the branch to GitHub and open a pull request so that the tests run. If there are any failures, fix them.

5. Get the performance figures from the nightly system tests.
6. Get the performance figures from the nightly system tests.

There should be a cron job configured to run these nightly. Running it manually and retrieving the results is documented
in the [system tests guide](13-system-tests.md#nightly-test-scripts).

Update the performance figures in the [system tests guide](13-system-tests.md#performance-benchmarks).

6. Run a deployment of the deployAll system test to test the functionality of the system. Note that it is best to
7. Run a deployment of the deployAll system test to test the functionality of the system. Note that it is best to
provide a fresh instance ID that has not been used before:

```bash
Expand All @@ -37,7 +39,7 @@ The following tests can be used as a quick check that all is working correctly.
all aspects of the system. Any changes made by pull requests should be tested by doing a system test deployment on AWS
if they are likely to either affect performance or involve changes to the way the system is deployed to AWS.

7. Test a simple query:
8. Test a simple query:

```bash
./scripts/utility/query.sh ${ID}
Expand All @@ -47,7 +49,7 @@ Choose a range query, choose 'y' for the first two questions and then choose a r
As the data that is ingested is random, it is not possible to say exactly how many results will be returned, but it
should be in the region of 900 results.

8. Test a query that will be executed by lambda:
9. Test a query that will be executed by lambda:

```bash
./scripts/utility/lambdaQuery.sh ${ID}
Expand All @@ -57,13 +59,13 @@ Choose the S3 results bucket option and then choose the same options as above. I
The first query executed by lambda is a little slower than subsequent ones due to the start-up costs. The second query
should be quicker.

9. Test a query that will be executed by lambda with the results being returned over a websocket:
10. Test a query that will be executed by lambda with the results being returned over a websocket:

```bash
./scripts/utility/webSocketQuery.sh ${ID}
```

Choose the same options as above, and results should be returned.

10. Once the above tests have been done, merge the pull request into main. Then checkout the main branch,
11. Once the above tests have been done, merge the pull request into main. Then checkout the main branch,
set the tag to `v${VERSION}` and push the tag using `git push --tags`.
8 changes: 0 additions & 8 deletions docs/16-roadmap.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,6 @@
This section documents some of the planned functionality and improvements:

- Query optimisation:
- Separate the query lambda into two. The first will be the query planner that needs to know
the information from the state store. The second executes the subqueries. This second
lambda does not need to know any information from the state store and so should be able to
start up quickly. By splitting the query functionality into two lambdas we can reduce the
number of calls to the statestore.
- Add an option for the system to automatically keep the lambdas warm by periodically
calling them with dummy queries.
- Optimise the start-up time of the lambdas, see https://docs.aws.amazon.com/lambda/latest/dg/snapstart.html
Expand Down Expand Up @@ -41,9 +36,6 @@ This section documents some of the planned functionality and improvements:
of Arrow / Parquet types. A Sleeper schema could be specified as an Arrow schema
with additional information about which are the row keys and sort keys.

- Full compactions: add the ability to trigger a full compaction, i.e. to guarantee
that every record in a table is scheduled to be read in a compaction job.

- Service that maintains an up-to-date cache of the statestore: Various parts of the
system need to query the state store. We could potentially reduce the cost and
latency of these queries if we had a long-running service that maintained an
Expand Down
111 changes: 51 additions & 60 deletions java/build/src/main/java/sleeper/build/maven/dependencydraw/DrawDependencyGraph.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -48,48 +48,44 @@ public class DrawDependencyGraph {
LayoutPaintable.BalloonRings balloonLayoutRings;
LayoutPaintable.RadialRings radialLayoutRings;

String instructions =
"<html>"
+ "<h3>Graph Transformation:</h3>"
+ "<ul>"
+ "<li>Mousewheel scales with a crossover value of 1.0.<p>"
+ " - scales the graph layout when the combined scale is greater than 1<p>"
+ " - scales the graph view when the combined scale is less than 1"
+ "<li>Mouse1+drag pans the graph"
+ "<li>Mouse1 double click on the background resets all transforms"
+ "</ul>"
+ "<h3>Vertex/Edge Selection:</h3>"
+ "<ul>"
+ "<li>Mouse1+MENU on a vertex or edge selects the vertex or edge and deselects any others"
+ "<li>Mouse1+MENU+Shift on a vertex toggles selection of the vertex"
+ "<li>Mouse1+MENU on a selected edge toggles selection of the edge"
+ "<li>Mouse1+MENU+drag elsewhere selects vertices in a region"
+ "<li>Mouse1+Shift+drag adds selection of vertices in a new region"
+ "</ul>"
+ "<h3>Vertex Transformation:</h3>"
+ "<ul>"
+ "<li>Mouse1+MENU+drag on a selected vertex moves all selected Vertices"
+ "</ul>"
+ "Note that MENU == Command on a Mac, MENU == CTRL on a PC"
+ "</html>";
String instructions = "<html>"
+ "<h3>Graph Transformation:</h3>"
+ "<ul>"
+ "<li>Mousewheel scales with a crossover value of 1.0.<p>"
+ " - scales the graph layout when the combined scale is greater than 1<p>"
+ " - scales the graph view when the combined scale is less than 1"
+ "<li>Mouse1+drag pans the graph"
+ "<li>Mouse1 double click on the background resets all transforms"
+ "</ul>"
+ "<h3>Vertex/Edge Selection:</h3>"
+ "<ul>"
+ "<li>Mouse1+MENU on a vertex or edge selects the vertex or edge and deselects any others"
+ "<li>Mouse1+MENU+Shift on a vertex toggles selection of the vertex"
+ "<li>Mouse1+MENU on a selected edge toggles selection of the edge"
+ "<li>Mouse1+MENU+drag elsewhere selects vertices in a region"
+ "<li>Mouse1+Shift+drag adds selection of vertices in a new region"
+ "</ul>"
+ "<h3>Vertex Transformation:</h3>"
+ "<ul>"
+ "<li>Mouse1+MENU+drag on a selected vertex moves all selected Vertices"
+ "</ul>"
+ "Note that MENU == Command on a Mac, MENU == CTRL on a PC"
+ "</html>";

public void drawGraph(GraphModel model) {
Graph<GraphNode, GraphEdge> g =
GraphTypeBuilder.<GraphNode, GraphEdge>directed().buildGraph();
Graph<GraphNode, GraphEdge> g = GraphTypeBuilder.<GraphNode, GraphEdge>directed().buildGraph();
model.getNodes().forEach(g::addVertex);
model.getEdges().forEach(edge -> g.addEdge(edge.getFrom(model), edge.getTo(model), edge));

VisualizationViewer<GraphNode, GraphEdge> vv =
VisualizationViewer.<GraphNode, GraphEdge>builder(g)
.build();
VisualizationViewer<GraphNode, GraphEdge> vv = VisualizationViewer.<GraphNode, GraphEdge>builder(g)
.build();

// use html to break long labels into multi-line and center-align the text
vv.getRenderContext().setVertexLabelFunction(v -> "<html><b><center>" +
v.toString().replaceAll("/", "/<br>"));

PickedNodeState picked = new PickedNodeState(model);
vv.getRenderContext().getSelectedVertexState().addItemListener(event ->
picked.updatePicked(vv.getRenderContext().getSelectedVertexState().getSelected()));
vv.getRenderContext().getSelectedVertexState().addItemListener(event -> picked.updatePicked(vv.getRenderContext().getSelectedVertexState().getSelected()));
vv.getRenderContext().setEdgeDrawPaintFunction(edge -> picked.calculateEdgeColor(edge, showTransitiveDependencies));
vv.getRenderContext().setArrowDrawPaintFunction(edge -> picked.calculateArrowColor(edge, showTransitiveDependencies));
vv.getRenderContext().setArrowFillPaintFunction(edge -> picked.calculateArrowColor(edge, showTransitiveDependencies));
Expand All @@ -103,35 +99,30 @@ public void drawGraph(GraphModel model) {

final JComboBox layoutComboBox = new JComboBox(combos);
layoutComboBox.addActionListener(
e ->
SwingUtilities.invokeLater(
() -> {
LayoutHelperDirectedGraphs.Layouts layoutBuilderType =
(LayoutHelperDirectedGraphs.Layouts) layoutComboBox.getSelectedItem();
LayoutAlgorithm.Builder layoutAlgorithmBuilder =
layoutBuilderType.getLayoutAlgorithmBuilder();
LayoutAlgorithm<GraphNode> layoutAlgorithm = layoutAlgorithmBuilder.build();
vv.removePreRenderPaintable(balloonLayoutRings);
vv.removePreRenderPaintable(radialLayoutRings);
layoutAlgorithm.setAfter(vv::scaleToLayout);
if (animateLayoutTransition.isSelected()) {
LayoutAlgorithmTransition.animate(vv, layoutAlgorithm, vv::scaleToLayout);
} else {
LayoutAlgorithmTransition.apply(vv, layoutAlgorithm, vv::scaleToLayout);
}
if (layoutAlgorithm instanceof BalloonLayoutAlgorithm) {
balloonLayoutRings =
new LayoutPaintable.BalloonRings(
vv, (BalloonLayoutAlgorithm) layoutAlgorithm);
vv.addPreRenderPaintable(balloonLayoutRings);
}
if (layoutAlgorithm instanceof RadialTreeLayoutAlgorithm) {
radialLayoutRings =
new LayoutPaintable.RadialRings(
vv, (RadialTreeLayoutAlgorithm) layoutAlgorithm);
vv.addPreRenderPaintable(radialLayoutRings);
}
}));
e -> SwingUtilities.invokeLater(
() -> {
LayoutHelperDirectedGraphs.Layouts layoutBuilderType = (LayoutHelperDirectedGraphs.Layouts) layoutComboBox.getSelectedItem();
LayoutAlgorithm.Builder layoutAlgorithmBuilder = layoutBuilderType.getLayoutAlgorithmBuilder();
LayoutAlgorithm<GraphNode> layoutAlgorithm = layoutAlgorithmBuilder.build();
vv.removePreRenderPaintable(balloonLayoutRings);
vv.removePreRenderPaintable(radialLayoutRings);
layoutAlgorithm.setAfter(vv::scaleToLayout);
if (animateLayoutTransition.isSelected()) {
LayoutAlgorithmTransition.animate(vv, layoutAlgorithm, vv::scaleToLayout);
} else {
LayoutAlgorithmTransition.apply(vv, layoutAlgorithm, vv::scaleToLayout);
}
if (layoutAlgorithm instanceof BalloonLayoutAlgorithm) {
balloonLayoutRings = new LayoutPaintable.BalloonRings(
vv, (BalloonLayoutAlgorithm) layoutAlgorithm);
vv.addPreRenderPaintable(balloonLayoutRings);
}
if (layoutAlgorithm instanceof RadialTreeLayoutAlgorithm) {
radialLayoutRings = new LayoutPaintable.RadialRings(
vv, (RadialTreeLayoutAlgorithm) layoutAlgorithm);
vv.addPreRenderPaintable(radialLayoutRings);
}
}));

layoutComboBox.setSelectedItem(LayoutHelperDirectedGraphs.Layouts.SUGIYAMA);

Expand Down
Loading

0 comments on commit 198170a

Please sign in to comment.