Performance Improvment

[RandallFlagg]

@Hirogen I am going to put here a few thing that I think can improve the code performance, resources, and general improvements to the code base.
I will number them in order to have easier track. If code change happens, I will open a new issue and link to it from here.

1. 1. AppDomain.CurrentDomain.AssemblyResolve += ColumnizerResolveEventHandler; - I would like to remove this event registration and instead call the code directly without the event. Do you see a problem doing that?
2. 2. OnDataGridView_CellPainting - Why is this event registered? Why not to do the logic on the SelectionChanged event?
3. 3. The DelayedTrigger mechanism is causing a big delay when navigating over the lines in the DataGridView. I suggest removing it and replace it with a simple event mechanism. Solved in PR Status line update - Remove the thread and use event instead #333
4. 4. In ReadToBufferList function in LogFileReader there is a while condition that makes the loading super slow due to a lot of notifications being sent. Is there a reason for limiting the buffer with lines and not with size? Why is the default so small - 500? If I put 100000 to the "linesPerBuffer" setting the time goes down from 30 sec. to about 1sec. Solved in PR Improve loading time of a file by changing the default from 500 to 50000 lines per buffer #345
5. 5. When closing a tab there seems to be a small delay. Need to check and see why and if it can be improved. Solved in PR Removed the thread that watches file changes #366
6. 6. I am trying to improve the ipc(remove the grpc.core). I am trying to test the 1 instance(enable/disable) but can't seem to make it work. @Hirogen can you help? I am on the Devlopment branch. I get an exception on line 159 in Program.cs
client.NewWindowOrLockedWindow(new Grpc.FileNames { FileNames_ = { args } });
      Exception: The SSL connection could not be established, see inner exception.
      InnerException: Cannot determine the frame size or a corrupted frame was received. - Continues in Change from gRPC to Named Pipes #374

[Hirogen]

2 => Well if you look closely this draws the cell with an addition to other elements that are needed (bookmarks, highlight entries) and this is also called multiply times when other events fire, so this can not just be called when a selection changes. For Instance if a pattern hits (regex etc.) the cell is painted with the highlights. Calling CellPainting yourself and doing the work, can be faster then letting the grid do it, also its the only way to customize it as you wish without the restriction of the datagridview. Logexpert uses also its own implementation of the DataGridView (BufferedDataGridView) because the datagridview is slow. I wouldn't touch that, its quite integral to the highlight system and this needs to be treated carefully, otherwise nothing works again ^^ (I have been there, its not so nice)

1 => As far as I understand the code in the pluginregistry, this is a backup solution, or it seems to be a backup solution to reload a plugin dll, if it could not be resolve the first time. you could try to change that and see if the plugins are still loaded, but be sure to check the error state as well, if a plugin can not be loaded. I haven't touched the plugin system at all, probably needs an overhaul.

But just fork and create pr if you have performances improvements 👍 don't be shy, just make sure everything works as expected :D

[Hirogen]

Yeah, 3 seems to be old Even-Driven-Implementation, there are newer and better ways. If this is causing a delay and a simple mechanism helps, I'm all for it

[RandallFlagg]

Yeah, 3 seems to be old Even-Driven-Implementation, there are newer and better ways. If this is causing a delay and a simple mechanism helps, I'm all for it

Solved in PR #333

[Hirogen]

4. should be changed differently, TPL would be the best https://learn.microsoft.com/en-us/dotnet/standard/parallel-programming/task-parallel-library-tpl, but that is a major rework, not sure why the 100 Line limit is there, but I guess this is from an older implementation as well, don't forget logexpert was started before .net 3.5 so there is a lot of legacy code that is probably replaceable by newer concepts.

TPL would be able to use all cores to load/filter a file

If changed to 100k and it loads faster, is the question, what is the downside? Processing time? What are your preconditions, how many CPU Cores, how much ram, what harddisk?

First thing I try would be changing the _lruCacheDict to a concurrent dictionary (https://learn.microsoft.com/en-us/dotnet/api/system.collections.concurrent.concurrentdictionary-2?view=net-9.0) this will probably safe a little bit time, because it's designed to be used in multithread situations

[RandallFlagg]

4. should be changed differently, TPL would be the best https://learn.microsoft.com/en-us/dotnet/standard/parallel-programming/task-parallel-library-tpl, but that is a major rework, not sure why the 100 Line limit is there, but I guess this is from an older implementation as well, don't forget logexpert was started before .net 3.5 so there is a lot of legacy code that is probably replaceable by newer concepts.

TPL would be able to use all cores to load/filter a file

If changed to 100k and it loads faster, is the question, what is the downside? Processing time? What are your preconditions, how many CPU Cores, how much ram, what harddisk?

First thing I try would be changing the _lruCacheDict to a concurrent dictionary (https://learn.microsoft.com/en-us/dotnet/api/system.collections.concurrent.concurrentdictionary-2?view=net-9.0) this will probably safe a little bit time, because it's designed to be used in multithread situations

Regarding the TPL, I can write a new file loading mechanisim. I will take a look and update if I see that I can do it in a reasonable time.

I looked at the _lruCacheDict and it looked like something that I don't want to touch at this point in time. Luckily, @TheNicker is on it in PR #344.

Regarding the change I don't think it is suppose to affect anyone with a reasonable modern hardware. I am using a laptop from 2018 with 16gb and nvme.

The improved speed made me think about 2 things regarding this and that they are easy to do:

1. change the default
2. Take it out to settings

What do you say? I don't really see a downside for this 2 changes.

You can see the changes in PR #345

[Hirogen]

I wouldn't take it out of the settings, just so people can tweak it, but updating the default settings, is definitely a good idea

yeah a new file loading mechanism would be good, but that is a major rework and not sure if this would be good time spent, since the current one works.

[RandallFlagg]

I wouldn't take it out of the settings, just so people can tweak it, but updating the default settings, is definitely a good idea

yeah a new file loading mechanism would be good, but that is a major rework and not sure if this would be good time spent, since the current one works.

This is done in PR #345. I meant to take out the change to the setting window not to take it out of the setting window.
Good. We are on the same page. Moving on.

[Hirogen]

6. 6. [ ]  I am trying to improve the rpc(remove the grpc.core). I am trying to test the 1 instance(enable/disable) but can't seem to make it work. [@Hirogen](https://github.com/Hirogen) can you help? I am on the Devlopment branch. I get an exception on line _159_ in `Program.cs`
      `client.NewWindowOrLockedWindow(new Grpc.FileNames { FileNames_ = { args } });`
      **Exception:** The SSL connection could not be established, see inner exception.
      **InnerException:** Cannot determine the frame size or a corrupted frame was received.

in about 2 weeks, I have a few weeks free time of my family, then I can dig into grpc and why that thing is not working, my plan is to implement that feature then, maybe earlier, not sure yet, but definitely in 2 weeks

[RandallFlagg]

6. 6. [ ]  I am trying to improve the rpc(remove the grpc.core). I am trying to test the 1 instance(enable/disable) but can't seem to make it work. [@Hirogen](https://github.com/Hirogen) can you help? I am on the Devlopment branch. I get an exception on line _159_ in `Program.cs`
      `client.NewWindowOrLockedWindow(new Grpc.FileNames { FileNames_ = { args } });`
      **Exception:** The SSL connection could not be established, see inner exception.
      **InnerException:** Cannot determine the frame size or a corrupted frame was received.

in about 2 weeks, I have a few weeks free time of my family, then I can dig into grpc and why that thing is not working, my plan is to implement that feature then, maybe earlier, not sure yet, but definitely in 2 weeks

I managed to upgrade it but the reason I didn't continue is because it didn't do anything. I will push and let you know the branch, so you don't need to start from scratch.

[RandallFlagg]

6. 6. [ ]  I am trying to improve the rpc(remove the grpc.core). I am trying to test the 1 instance(enable/disable) but can't seem to make it work. [@Hirogen](https://github.com/Hirogen) can you help? I am on the Devlopment branch. I get an exception on line _159_ in `Program.cs`
      `client.NewWindowOrLockedWindow(new Grpc.FileNames { FileNames_ = { args } });`
      **Exception:** The SSL connection could not be established, see inner exception.
      **InnerException:** Cannot determine the frame size or a corrupted frame was received.

in about 2 weeks, I have a few weeks free time of my family, then I can dig into grpc and why that thing is not working, my plan is to implement that feature then, maybe earlier, not sure yet, but definitely in 2 weeks

I managed to upgrade it but the reason I didn't continue is because it didn't do anything. I will push and let you know the branch, so you don't need to start from scratch.

This is a vaunarbility issue. Now it is also important: https://osv.dev/vulnerability/GHSA-9hxf-ppjv-w6rq

[Hirogen]

6. 6. [ ]  I am trying to improve the rpc(remove the grpc.core). I am trying to test the 1 instance(enable/disable) but can't seem to make it work. [@Hirogen](https://github.com/Hirogen) can you help? I am on the Devlopment branch. I get an exception on line _159_ in `Program.cs`
      `client.NewWindowOrLockedWindow(new Grpc.FileNames { FileNames_ = { args } });`
      **Exception:** The SSL connection could not be established, see inner exception.
      **InnerException:** Cannot determine the frame size or a corrupted frame was received.

in about 2 weeks, I have a few weeks free time of my family, then I can dig into grpc and why that thing is not working, my plan is to implement that feature then, maybe earlier, not sure yet, but definitely in 2 weeks

I managed to upgrade it but the reason I didn't continue is because it didn't do anything. I will push and let you know the branch, so you don't need to start from scratch.

This is a vaunarbility issue. Now it is also important: https://osv.dev/vulnerability/GHSA-9hxf-ppjv-w6rq

should be fine if we upgrade the nuget package

[RandallFlagg]

6. 6. [ ]  I am trying to improve the rpc(remove the grpc.core). I am trying to test the 1 instance(enable/disable) but can't seem to make it work. [@Hirogen](https://github.com/Hirogen) can you help? I am on the Devlopment branch. I get an exception on line _159_ in `Program.cs`
      `client.NewWindowOrLockedWindow(new Grpc.FileNames { FileNames_ = { args } });`
      **Exception:** The SSL connection could not be established, see inner exception.
      **InnerException:** Cannot determine the frame size or a corrupted frame was received.

in about 2 weeks, I have a few weeks free time of my family, then I can dig into grpc and why that thing is not working, my plan is to implement that feature then, maybe earlier, not sure yet, but definitely in 2 weeks

I managed to upgrade it but the reason I didn't continue is because it didn't do anything. I will push and let you know the branch, so you don't need to start from scratch.

This is a vaunarbility issue. Now it is also important: https://osv.dev/vulnerability/GHSA-9hxf-ppjv-w6rq

should be fine if we upgrade the nuget package

You can't. That package is in maintenance mode. This is not a real solution.
I am talking about:
<PackageVersion Include="Grpc.Core" Version="2.46.6" />

Regardless of that, why is gRPC used for communication between the different instances? Why not other solutions such as "Named Pipes"?

[Hirogen]

it was the first thing that popped up, while I was searching for a suitable replacement for ipcserverchannel replacement, that is not supported in .net 8, also named pipes are windows specific, and I wanted to use something, that is not depending on a windows technology that could be retired (like wcf or the ipcserverchannels)

[Hirogen]

@RandallFlagg I tested a little bit, the logging experience and I need to confirm it, with a older version.
I created a small program that can create log statments (single lines), and writes them to a file, without exclusive access. And I let it write every 10 ms a line. After that I open it in Logexpert and Logexpert crawls to a halt, so something is up with reading the rolling logfile. I need to test this also with the new changes of the branche, but currently something is up.

I used older versions, with rolling logfiles before, and there we got 10-20 mb/s for a file, and this worked fine, so something is wrong or performance has degenerated.

I'll test a little more

[RandallFlagg]

it was the first thing that popped up, while I was searching for a suitable replacement for ipcserverchannel replacement, that is not supported in .net 8, also named pipes are windows specific, and I wanted to use something, that is not depending on a windows technology that could be retired (like wcf or the ipcserverchannels)

@Hirogen I just managed to run on Linux "Named Pipes". If you want we can switch to that. I think it will make the code much more simple, faster and less dependant on external maintainers. What do you say?

[Hirogen]

it was the first thing that popped up, while I was searching for a suitable replacement for ipcserverchannel replacement, that is not supported in .net 8, also named pipes are windows specific, and I wanted to use something, that is not depending on a windows technology that could be retired (like wcf or the ipcserverchannels)

@Hirogen I just managed to run on Linux "Named Pipes". If you want we can switch to that. I think it will make the code much more simple, faster and less dependant on external maintainers. What do you say?

sure :) open to everything that makes it simpler and working on different Win and Linux :)