Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add DocumentDB support #2

Open
wants to merge 7 commits into
base: master
Choose a base branch
from
Open

Add DocumentDB support #2

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
d0ff49a
Typo
slayer1551 Jun 1, 2022
b936d3b
Fix new job window modal from not closing
slayer1551 Jun 1, 2022
9e14bd6
AWS Document DB Support
slayer1551 Jun 1, 2022
d90ea5f
Fix totalPage count
slayer1551 Jun 1, 2022
32b3687
Update agendash.js
slayer1551 Jun 1, 2022
a76b9e5
Temp: Remove the attempt to do a $facet query
jaywink Sep 15, 2022
53e2739
Temp: Apply a 1000 document limit to the jobs query
jaywink Sep 15, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
165 changes: 107 additions & 58 deletions lib/controllers/agendash.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ module.exports = function (agenda, options) {
if (job) {
preMatch.name = job;
}

if (options.query && options.property) {
if (options.isObjectId) {
preMatch[options.property] = ObjectId(options.query);
Expand All @@ -48,6 +48,7 @@ module.exports = function (agenda, options) {
} else {
preMatch[options.property] = { $regex: options.query, $options: "i" };
}

}

const postMatch = {};
Expand All @@ -56,69 +57,117 @@ module.exports = function (agenda, options) {
}

const collection = agenda._collection.collection || agenda._collection;
return collection
.aggregate([
{ $match: preMatch },
{
$sort: {
nextRunAt: -1,
lastRunAt: -1,
lastFinishedAt: -1,
},
const aggregateQuery = [
{ $match: preMatch },
{
$sort: {
nextRunAt: -1,
lastRunAt: -1,
lastFinishedAt: -1,
},
{
$project: {
job: "$$ROOT",
_id: "$$ROOT._id",
running: {
$and: ["$lastRunAt", { $gt: ["$lastRunAt", "$lastFinishedAt"] }],
},
scheduled: {
$and: ["$nextRunAt", { $gte: ["$nextRunAt", new Date()] }],
},
queued: {
$and: [
"$nextRunAt",
{ $gte: [new Date(), "$nextRunAt"] },
{ $gte: ["$nextRunAt", "$lastFinishedAt"] },
],
},
completed: {
$and: [
"$lastFinishedAt",
{ $gt: ["$lastFinishedAt", "$failedAt"] },
],
},
failed: {
$and: [
"$lastFinishedAt",
"$failedAt",
{ $eq: ["$lastFinishedAt", "$failedAt"] },
],
},
repeating: {
$and: ["$repeatInterval", { $ne: ["$repeatInterval", null] }],
},
},
{
$project: {
job: "$$ROOT",
_id: "$$ROOT._id",
running: {
$and: ["$lastRunAt", { $gt: ["$lastRunAt", "$lastFinishedAt"] }],
},
scheduled: {
$and: ["$nextRunAt", { $gte: ["$nextRunAt", new Date()] }],
},
queued: {
$and: [
"$nextRunAt",
{ $gte: [new Date(), "$nextRunAt"] },
{ $gte: ["$nextRunAt", "$lastFinishedAt"] },
],
},
completed: {
$and: [
"$lastFinishedAt",
{ $gt: ["$lastFinishedAt", "$failedAt"] },
],
},
failed: {
$and: [
"$lastFinishedAt",
"$failedAt",
{ $eq: ["$lastFinishedAt", "$failedAt"] },
],
},
repeating: {
$and: ["$repeatInterval", { $ne: ["$repeatInterval", null] }],
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The whitespace changes make this difficult to review

},
},
{ $match: postMatch },
{
$facet: {
pages: [
{ $count: "totalMatchs" },
{
$project: {
totalPages: {
$ceil: { $divide: ["$totalMatchs", options.limit] },
},
},
{ $match: postMatch }


];
const standardMongoDBQuery = [

{
$facet: {
pages: [
{ $count: "totalMatchs" },
{
$project: {
totalPages: {
$ceil: { $divide: ["$totalMatchs", options.limit] },
},
},
],
filtered: [{ $skip: options.skip }, { $limit: options.limit }],
},
],
filtered: [{ $skip: options.skip }, { $limit: options.limit }],
}
}
]
const awsDocumentDBQuery = [
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we explain the limitation with this query, for some reason we're trying to run a query, failing and then trying to run another query that's docDB compatible.

{
$group: {
_id: null,
totalMatchs: {
$sum: 1
},
},
])
.toArray();
results: {
$push: '$$ROOT'
}
}
},
{
$project: {
pages: [{totalPages: '$totalMatchs'}],
filtered: {
$slice: [
'$results',
options.skip,
options.limit
]
}
}
}
]
return collection
.aggregate([...aggregateQuery,...standardMongoDBQuery])
.toArray()
.then((result) => {
return result;
})
.catch((error) => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is gross, it's an extra round trip to AWS every time we run this query, and I'm not sure amazon will thank us for that. Can we just make this a configuration option (or somehow smartly detect the version of Mongo, rather than this awful thing?)

We're not even looking at the error response to detect why it failed 😮‍💨

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm hoping to avoid us maintaining our own code until upstream implements this properly. This works and it's only doing this query every so and so seconds. I hardly think AWS will notice.

We can reimplement this patch but I'm going to postpone it after staging migration, if ok.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To clarify the reason for not spending engineering time now: when upstream implements documentdb support, we're going to want to use that implementation anyway, not whatever we came up with. As such I'd avoid making anything fancy at this stage. Given we only need documentdb support, I could simplify the pr - but that carries the issue if the upstream patch gets new commits, we'll be in a merge conflict against that.

return collection.aggregate([...aggregateQuery,...awsDocumentDBQuery])
.toArray()
.then((result) => {
result[0].pages[0].totalPages = Math.ceil(result[0].pages[0].totalPages / options.limit);
return result;
})
.catch((error) => {
return error;
})


});

};

const getOverview = async () => {
Expand Down
6 changes: 3 additions & 3 deletions public/app/js/confirms.js
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
const popupmessage = Vue.component("popup-message", {
props: ["job", "deletec", "requeuec", "createc"],
template: `
<div v-if="deletec" class="alert alert-success popupmessage">Job Deleted successfull</div>
<div v-else-if="requeuec" class="alert alert-success popupmessage">Job Requeue successfull</div>
<div v-else-if="createc" class="alert alert-success popupmessage">Job Created successfull</div>
<div v-if="deletec" class="alert alert-success popupmessage">Job Deleted successful</div>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Job Deleted successfully, surely?

<div v-else-if="requeuec" class="alert alert-success popupmessage">Job Requeue successful</div>
<div v-else-if="createc" class="alert alert-success popupmessage">Job Created successful</div>
`,
});
3 changes: 1 addition & 2 deletions public/app/js/newJob.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,6 @@ const newJob = Vue.component("new-job", {
.then((data) => {
this.$emit("popup-message");
this.$emit("refresh-data");
this.$refs.Close.click();
this.clear();
})
.catch(console.log);
Expand Down Expand Up @@ -79,7 +78,7 @@ const newJob = Vue.component("new-job", {
</form>
</div>
<div class="modal-footer">
<button type="button" class="btn btn-info" @click="create()">Create Job</button>
<button type="button" class="btn btn-info" data-dismiss="modal" @click="create()">Create Job</button>
<button type="button" class="btn btn-secondary" data-dismiss="modal">Cancel</button>
</div>
</div>
Expand Down