Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(export): use createWriteStream to export json #277

Merged
merged 7 commits into from
Feb 12, 2025

Conversation

D-Sketon
Copy link
Member

@D-Sketon D-Sketon commented Jan 30, 2025

check list

  • Add test cases for the changes.
  • Passed the CI test.

Description

I'm wondering why you don't just splice the strings when exporting? It's faster and takes less memory.
before
{06AA44C6-E458-43BE-929D-E9006A14191C}
{200754C4-1DE4-4149-AE1D-CC8069ECA165}
You can see the memory rise at the end of hexo g.
after
{D5249C5F-668C-482D-845B-45133F9ED3C1}
{F14BF233-E8C8-4D04-A85E-C684E15436EA}

Additional information

Sorry, something went wrong.

@coveralls
Copy link

coveralls commented Jan 30, 2025

Pull Request Test Coverage Report for Build 13085724941

Details

  • 59 of 61 (96.72%) changed or added relevant lines in 3 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage increased (+0.06%) to 98.353%

Changes Missing Coverage Covered Lines Changed/Added Lines %
src/util.ts 22 24 91.67%
Totals Coverage Status
Change from base Build 12828819518: 0.06%
Covered Lines: 4480
Relevant Lines: 4555

💛 - Coveralls

src/database.ts Outdated
Comment on lines 37 to 41
let prefix = '';
if (i) {
prefix = ',';
}
await handle.write(`${prefix}"${key}":${models[key]._export()}`);
Copy link
Member

@SukkaW SukkaW Jan 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we are targeting the modern Node.js versions now, maybe we could try fs.createWriteStream?

let p: Promise<unknown> | undefined;
const writeStream = fs.createWriteStream(filePath);

p = asyncWriteToStream(writeStream, prefix + '\n');
if (p) await p;

for (let i = 0; i < itemsLength; i++) {
  p = asyncWriteToStream(writeStream, items[i] + '\n');
  // eslint-disable-next-line no-await-in-loop -- stream high water mark
  if (p) await p;
}

// You can find `asyncWriteToStream` implementation at https://github.com/SukkaW/foxts/blob/a7f4e40434b1a46d58d5062b9df7713486f677b2/src/async-write-to-stream/index.ts#L17

@D-Sketon D-Sketon marked this pull request as draft January 30, 2025 11:13
@D-Sketon D-Sketon marked this pull request as ready for review January 30, 2025 16:47
@D-Sketon D-Sketon requested a review from SukkaW January 30, 2025 16:50
const raw = data[keys[i]];
if (raw) {
const prefix = i === 0 ? '' : ',';
p = asyncWriteToStream(writeStream, `${prefix}${JSON.stringify(schema._exportDatabase(cloneDeep(raw) as object))}`);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This new function writes each cloned object directly to stream, making it more friendly for escape analysis

src/database.ts Outdated Show resolved Hide resolved
Co-authored-by: Sukka <[email protected]>
Signed-off-by: D-Sketon <[email protected]>
SukkaW
SukkaW previously approved these changes Jan 31, 2025
@D-Sketon
Copy link
Member Author

I wonder why all the tests on windows are failing :(

@D-Sketon
Copy link
Member Author

D-Sketon commented Feb 1, 2025

I wonder why all the tests on windows are failing :(

fixed

SukkaW
SukkaW previously approved these changes Feb 1, 2025
Copy link
Member

@SukkaW SukkaW left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I wonder about the new implementation's performance though.

@D-Sketon D-Sketon changed the title perf(export): Speed up and reduce memory usage refactor(export): use createWriteStream to export json Feb 1, 2025
@D-Sketon
Copy link
Member Author

D-Sketon commented Feb 1, 2025

@SukkaW
1x hexo-many-posts

Cold Processing

Index Before Cost time (s) After Cost time (s)
Save Database 0.19s 0.18s

Hot Processing

Index Before Cost time (s) After Cost time (s)
Save Database 0.18s 0.17s

Another Cold Processing

Index Before Cost time (s) After Cost time (s)
Save Database 0.21s 0.19s

{E41C823B-F573-4D39-B577-AA1E015A9B8E}

4x hexo-many-posts

Cold Processing

Index Before Cost time (s) After Cost time (s)
Save Database 0.85s 0.68s

Hot Processing

Index Before Cost time (s) After Cost time (s)
Save Database 0.66s 0.69s

Another Cold Processing

Index Before Cost time (s) After Cost time (s)
Save Database 0.93s 0.70s

{0D03F0F9-4B2A-4491-9FDC-E624B19ECED3}

8x hexo-many-posts

Cold Processing

Index Before Cost time (s) After Cost time (s)
Save Database 1.94s 1.42s

Hot Processing

Index Before Cost time (s) After Cost time (s)
Save Database 1.28s 1.34s

Another Cold Processing

Index Before Cost time (s) After Cost time (s)
Save Database 1.45s 1.44s

{D6F4E5C8-CDC5-4231-81F0-8A83118B4A7B}

12x hexo-many-posts

Cold Processing

Index Before Cost time (s) After Cost time (s)
Save Database 2.12s 2.05s

Hot Processing

Index Before Cost time (s) After Cost time (s)
Save Database 1.96s 2.04s

Another Cold Processing

Index Before Cost time (s) After Cost time (s)
Save Database 2.21s 2.11s

{525263B0-ADA1-493D-9E47-4A74FBF52089}

@D-Sketon
Copy link
Member Author

D-Sketon commented Feb 1, 2025

BTW, I realised that coverage was not working, so I locked mocha to version 11.0.2

@D-Sketon
Copy link
Member Author

D-Sketon commented Feb 2, 2025

This PR solves part of the situation in hexojs/hexo#4922 (it's still possible to throw an exception if the word count of a post is too long)

@D-Sketon D-Sketon merged commit 4ad0460 into hexojs:master Feb 12, 2025
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants