Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Massive performance improvement... #363

Open
wants to merge 8 commits into
base: master
Choose a base branch
from
Open

Massive performance improvement... #363

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
917b9c6
Gigantic performance improvemet
typhoonsimon Mar 14, 2020
8f9945b
Rewritten peers sync way
typhoonsimon Mar 14, 2020
0a10086
Hybrid display fix
typhoonsimon Mar 14, 2020
45c5f53
Hybrid display fix
typhoonsimon Mar 14, 2020
ca20343
Hybrid display fix
typhoonsimon Mar 14, 2020
5a28894
Update README.md
typhoonsimon Mar 14, 2020
263140e
Update README.md
typhoonsimon Mar 14, 2020
4f7f61b
Add files via upload
typhoonsimon Mar 14, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ An open source block explorer written in node.js.
* [Sugarchain Explorer](https://1explorer.sugarchain.org/)
* [Florincoin](https://florincoin.info/info)
* [Maxcoin Explorer 1](https://explorer.maxcoinproject.net/)
* [Litecoin Plus Explorer 1](http://explorer.litecoinplus.co/)


*Note: If you would like your instance mentioned here contact me*
Expand Down
29 changes: 17 additions & 12 deletions lib/database.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -435,28 +435,33 @@ module.exports = {
},

get_last_txs_ajax: function(start, length, min, cb) {
Tx.countDocuments({'total': {$gte: min}}, function(err, count){
Tx.find({'total': {$gte: min}}).sort({blockindex: 'desc'}).skip(Number(start)).limit(Number(length)).exec(function(err, txs){
if (err) {
return cb(err);
} else {
return cb(txs, count);
}
});
lib.get_blockcount(function(blockcount) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@typhoonsimon As per the function's name "get_last_txs_ajax" - this function does NOT show latest blocks - it shows Latest Transactions, we cannot use the blockcount call here.

This does look very efficient though and would work almost as is for an additional function for latest blocks, which we could then set in the config for the user to display latest TX or latest BLOCKS on the front page.

In terms of making this code work for latest TXs what's the performance difference of Tx.countDocuments vs Tx.count here?

tx_count might be a stat we should keep updated via block indexing?

var blockFrom = blockcount - Number(start);
if (settings.index.last_txs != 0) {
blockcount = settings.index.last_txs;
}
var q = {$and: [{total: {$gt: Number(min)}}, {blockindex: {$lte: blockFrom}}]};
Tx.find(q).sort({blockindex: -1}).limit(Number(length)).exec(function(err, txs) {
if (err) {
return cb(err);
} else {
return cb(txs, blockcount);
}
});
});
},

get_address_txs_ajax: function(hash, start, length, cb) {
var totalCount = 0;
AddressTx.find({a_id: hash}).countDocuments({}, function(err, count){
AddressTx.find({a_id: hash}).count(function(err, count){
if(err) {
return cb(err);
} else {
totalCount = count;
AddressTx.aggregate([
{ $match: { a_id: hash } },
{ $match: { a_id: hash } },
{ $sort: {blockindex: -1} },
{ $skip: Number(start) },
{ $skip: Number(start) },
{
$group: {
_id: '',
Expand All @@ -474,7 +479,7 @@ module.exports = {
if (err) {
return cb(err);
} else {
AddressTx.find({a_id: hash}).sort({blockindex: 'desc'}).skip(Number(start)).limit(Number(length)).exec(function (err, address_tx) {
AddressTx.find({a_id: hash}).sort({blockindex: -1}).skip(Number(start)).limit(Number(length)).exec(function (err, address_tx) {
if (err) {
return cb(err);
} else {
Expand Down
49 changes: 26 additions & 23 deletions scripts/peers.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,37 +23,40 @@ mongoose.connect(dbString, function(err) {
console.log('Aborting');
exit();
} else {
var peers = Array();
var cnt = 0;
request({uri: 'http://127.0.0.1:' + settings.port + '/api/getpeerinfo', json: true}, function (error, response, body) {
lib.syncLoop(body.length, function (loop) {
var i = loop.iteration();
var address = body[i].addr.split(':')[0];
var port = body[i].addr.split(':')[1];
db.find_peer(address, function(peer) {
if (peer) {
if (isNaN(peer['port']) || peer['port'].length < 2 || peer['country'].length < 1) {
db.drop_peers(function() {
console.log('Saved peers missing ports or country, dropping peers. Re-reun this script afterwards.');
exit();
});
}
// peer already exists
loop.next();
} else {
request({uri: 'https://freegeoip.app/json/' + address, json: true}, function (error, response, geo) {
db.create_peer({
address: address,
port: port,
protocol: body[i].version,
version: body[i].subver.replace('/', '').replace('/', ''),
country: geo.country_name
}, function(){
loop.next();
});
});
request({uri: 'https://freegeoip.app/json/' + address, json: true}, function (error, response, geo) {
if (address.startsWith('10.') || address.startsWith('192.168') || address.startsWith('172.16')) {
geo.country_name = '[private address]';
}
peers[cnt++] = {
address: address,
port: port,
protocol: body[i].version,
version: body[i].subver.replace('/', '').replace('/', ''),
country: geo.country_name
};
loop.next();
});
}, function() {
exit();

// insert all at once after creation
Copy link
Collaborator

@TheHolyRoger TheHolyRoger Jul 16, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks good, much better way of doing it

db.drop_peers(function() {
console.log('Dropped, rebuilding...');
lib.syncLoop(cnt, function (loop) {
var i = loop.iteration();
db.create_peer(peers[i], function() {
loop.next();
});
}, function() {
exit();
});
});
});
});
}
Expand Down
2 changes: 1 addition & 1 deletion views/layout.pug
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ html
function update_stats(){
$.ajax({url: '/ext/summary', success: function(json){
$("#supply").text(parseInt(parseFloat(json.data[0].supply).toFixed(0)).toLocaleString('en'));
$("#difficulty").text(parseFloat(json.data[0].difficulty).toFixed(2));
$("#difficulty").text(json.data[0].difficulty);
Copy link
Collaborator

@TheHolyRoger TheHolyRoger Jul 16, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we changing this? If you want this formatted differently then we should add config options for it in a separate PR
Edit: sorry my fault I broke this for hybrid diff

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in b7180dc

$("#difficultyHybrid").text(json.data[0].difficultyHybrid);
$("#hashrate").text(parseFloat(json.data[0].hashrate).toLocaleString('en'));
$("#lastPrice").text(parseFloat(json.data[0].lastPrice).toFixed(8) + ' #{settings.markets.exchange}'.toUpperCase());
Expand Down