Massive performance improvement... #363

typhoonsimon · 2020-03-14T08:22:21Z

.. for index page and address view page. Don't forget that these additional mongo indexes must be created to achieve the level of performance of http://lcp.altcoinwarz.com/

db.txes.createIndex({total: 1})
db.txes.createIndex({total: -1})
db.txes.createIndex({blockindex: 1})
db.txes.createIndex({blockindex: -1})

addresstxes
db.addresstxes.createIndex({a_id: 1, blockindex: -1})```

... over large chains. Added indexes and brought back last_txs setting (set to 0 will display the whole chain in the index, but without sacrificing server or mongo performance).

typhoonsimon · 2020-03-14T10:04:57Z

Ah sorry, that was a debug, I will fix it.

typhoonsimon · 2020-03-14T10:08:23Z

OK I have put back the correct port settings, sorry about that.

typhoonsimon · 2020-03-14T11:01:08Z

it should be noted that there is a diff between count and countDocuments, and by virtue of mongos documentation this should be avoided.

Do you want me to set it back to countDocuments ?

uaktags · 2020-03-14T15:17:27Z

it should be noted that there is a diff between count and countDocuments, and by virtue of mongos documentation this should be avoided.

Do you want me to set it back to countDocuments ?

I'm not entirely sure yet. I need to go through and double check its usage. My fear is improper shutdowns will cause the count to be off, but this is simply the total count of txes for an address. It's not something I deem mission critical compared to say the actual values of transactions/amounts/etc.

Performance wise, count is indeed much faster, but it's also susceptible to inaccuracies.

typhoonsimon · 2020-03-15T03:25:40Z

it should be noted that there is a diff between count and countDocuments, and by virtue of mongos documentation this should be avoided.

Do you want me to set it back to countDocuments ?

I'm not entirely sure yet. I need to go through and double check its usage. My fear is improper shutdowns will cause the count to be off, but this is simply the total count of txes for an address. It's not something I deem mission critical compared to say the actual values of transactions/amounts/etc.

Performance wise, count is indeed much faster, but it's also susceptible to inaccuracies.

That's exactly what I thought. It's the same as count without lock on MySQL InnoDB, the count is not accurate but is very fast. In this case, worst that can happen is the the page number display is not accurate. But transactions per address are slow, I guess we could live with that given the cost of the countDocuments().

uaktags · 2020-03-15T19:35:00Z

it should be noted that there is a diff between count and countDocuments, and by virtue of mongos documentation this should be avoided.

Do you want me to set it back to countDocuments ?

I'm not entirely sure yet. I need to go through and double check its usage. My fear is improper shutdowns will cause the count to be off, but this is simply the total count of txes for an address. It's not something I deem mission critical compared to say the actual values of transactions/amounts/etc.
Performance wise, count is indeed much faster, but it's also susceptible to inaccuracies.

That's exactly what I thought. It's the same as count without lock on MySQL InnoDB, the count is not accurate but is very fast. In this case, worst that can happen is the the page number display is not accurate. But transactions per address are slow, I guess we could live with that given the cost of the countDocuments().

I agree. I would however counter to provide a settings config so that explorer admins can choose to have accuracy. The performance impact from if/then statements would be negligible.

allavett · 2020-06-01T10:35:03Z

Any update on the ETA of this pull request being finalized and merged?

Also, @typhoonsimon where does the need come to index both descending and ascending directions, does it really give the desired result?

.. Don't forget that these additional mongo indexes must be created to achieve the level of performance of http://lcp.altcoinwarz.com/
db.txes.createIndex({total: 1})
db.txes.createIndex({total: -1})
db.txes.createIndex({blockindex: 1})
db.txes.createIndex({blockindex: -1})

uaktags · 2020-07-09T04:07:59Z

I'm going to add this to my testing and merges with uaktags:explorer#58. Once testing is complete, I'll push accept and merge this. Hopefully end of week for an eta

TheHolyRoger

As per my comments this needs adjusting and/or duplicating into latest TX and latest BLOCK function calls

TheHolyRoger · 2020-07-16T13:12:52Z

views/layout.pug

@@ -27,7 +27,7 @@ html
        function update_stats(){
          $.ajax({url: '/ext/summary', success: function(json){
            $("#supply").text(parseInt(parseFloat(json.data[0].supply).toFixed(0)).toLocaleString('en'));
-            $("#difficulty").text(parseFloat(json.data[0].difficulty).toFixed(2));
+            $("#difficulty").text(json.data[0].difficulty);


Why are we changing this? If you want this formatted differently then we should add config options for it in a separate PR
Edit: sorry my fault I broke this for hybrid diff

Fixed in b7180dc

TheHolyRoger · 2020-07-16T13:13:25Z

scripts/peers.js

        });
      }, function() {
-        exit();
+
+      // insert all at once after creation


this looks good, much better way of doing it

TheHolyRoger · 2020-07-16T13:18:56Z

lib/database.js

-            return cb(txs, count);
-          }
-        });
+    lib.get_blockcount(function(blockcount) {


@typhoonsimon As per the function's name "get_last_txs_ajax" - this function does NOT show latest blocks - it shows Latest Transactions, we cannot use the blockcount call here.

This does look very efficient though and would work almost as is for an additional function for latest blocks, which we could then set in the config for the user to display latest TX or latest BLOCKS on the front page.

In terms of making this code work for latest TXs what's the performance difference of Tx.countDocuments vs Tx.count here?

tx_count might be a stat we should keep updated via block indexing?

TheHolyRoger · 2020-07-16T14:04:16Z

Merged the index declarations to 'main' here: 3f81b75 - these alone seem to have improved paging performance for me so thanks for that @typhoonsimon

TheHolyRoger · 2020-07-16T16:36:39Z

d611f8f This patch seems to add a mild speed improvement until we resolve this PR but thanks for your help so far @typhoonsimon

TheHolyRoger · 2020-07-16T19:35:02Z

Difficulty display fixed: b7180dc

typhoonsimon added 7 commits March 14, 2020 16:13

Gigantic performance improvemet

917b9c6

... over large chains. Added indexes and brought back last_txs setting (set to 0 will display the whole chain in the index, but without sacrificing server or mongo performance).

Rewritten peers sync way

8f9945b

Hybrid display fix

0a10086

Hybrid display fix

45c5f53

Hybrid display fix

ca20343

Update README.md

5a28894

Update README.md

263140e

uaktags linked an issue Mar 14, 2020 that may be closed by this pull request

Please help, paging doesn't work, I am getting MAD ! #360

Open

uaktags requested a review from TheHolyRoger March 14, 2020 09:15

Add files via upload

4f7f61b

buzzkillb mentioned this pull request Apr 29, 2020

Not loading transactions #367

Closed

uaktags added a commit to uaktags/explorer that referenced this pull request Jul 10, 2020

Testing iquidus#363

0915b39

TheHolyRoger requested changes Jul 16, 2020

View reviewed changes

This was linked to issues Jul 19, 2020

Blocks Page not Loading #391

Open

Add option for Latest Txs vs Latest Blocks on front page #238

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Massive performance improvement... #363

Massive performance improvement... #363

typhoonsimon commented Mar 14, 2020 •

edited by uaktags

Loading

typhoonsimon commented Mar 14, 2020

typhoonsimon commented Mar 14, 2020

typhoonsimon commented Mar 14, 2020

uaktags commented Mar 14, 2020

typhoonsimon commented Mar 15, 2020

uaktags commented Mar 15, 2020

allavett commented Jun 1, 2020 •

edited

Loading

uaktags commented Jul 9, 2020

TheHolyRoger left a comment

TheHolyRoger Jul 16, 2020 •

edited

Loading

TheHolyRoger Jul 16, 2020

TheHolyRoger Jul 16, 2020 •

edited

Loading

TheHolyRoger Jul 16, 2020

TheHolyRoger commented Jul 16, 2020 •

edited

Loading

TheHolyRoger commented Jul 16, 2020

TheHolyRoger commented Jul 16, 2020

Massive performance improvement... #363

Are you sure you want to change the base?

Massive performance improvement... #363

Conversation

typhoonsimon commented Mar 14, 2020 • edited by uaktags Loading

typhoonsimon commented Mar 14, 2020

typhoonsimon commented Mar 14, 2020

typhoonsimon commented Mar 14, 2020

uaktags commented Mar 14, 2020

typhoonsimon commented Mar 15, 2020

uaktags commented Mar 15, 2020

allavett commented Jun 1, 2020 • edited Loading

uaktags commented Jul 9, 2020

TheHolyRoger left a comment

Choose a reason for hiding this comment

TheHolyRoger Jul 16, 2020 • edited Loading

Choose a reason for hiding this comment

TheHolyRoger Jul 16, 2020

Choose a reason for hiding this comment

TheHolyRoger Jul 16, 2020 • edited Loading

Choose a reason for hiding this comment

TheHolyRoger Jul 16, 2020

Choose a reason for hiding this comment

TheHolyRoger commented Jul 16, 2020 • edited Loading

TheHolyRoger commented Jul 16, 2020

TheHolyRoger commented Jul 16, 2020

typhoonsimon commented Mar 14, 2020 •

edited by uaktags

Loading

allavett commented Jun 1, 2020 •

edited

Loading

TheHolyRoger Jul 16, 2020 •

edited

Loading

TheHolyRoger Jul 16, 2020 •

edited

Loading

TheHolyRoger commented Jul 16, 2020 •

edited

Loading