#265 Update MemoryMappedPageManager memory mapped initiation

[ssonthal]

#265

Two changes:

1. Creating the MemoryMappedFile without setting the total file size upfront

_mapped = MemoryMappedFile.CreateFromFile(_file, null, 0, MemoryMappedFileAccess.ReadWrite, HandleInheritability.None, true);

2. Instead of mapping the whole file, mapping only a portion of it dynamically as needed

_whole = _mapped.CreateViewAccessor(0, historyDepth * Page.PageSize);

[ssonthal]

Hi @Scooletz, pls let me know if these changes are in the right direction.

[Scooletz]

Provided comments. I don't understand how this could work without significant changes or using the NT method mentioned in the referred issue. Maybe we should add some tests for the MemoryMappedPageManager to ensure that it works properly? Also, add some matrix for OS dependent tests?

[Scooletz]

The _whole is used as both, the view to ForceFlush when a synchronization happens and as an accessor that provides the pointer _ptr for the whole file. This is why it was set to cover whole file.

[ssonthal]

Can we introduce a new variable called _viewAccessor for accessing the memory mapped portion as a pointer. And use _whole as it is for ForceFlush?

[Scooletz]

I'm not following. ForceFlush semantics should be that all the pages that were allocated are flushed down and then FSYNC is called. I believe we could create a one time use accessor and flush it. This might work.

[ssonthal]

You mean, inside the ForceFlush method, we can get another accessor for _file that covers the whole file and flush it?

[Scooletz]

AFAIK, the mapping will get size from the _file, but if the file is just created, it won't work as expected as one will need to set the length. I don't understand how this should work.

[ssonthal]

what if we remove this line -

_file.setLength(size);

while creating a new file. We can set it to some safe initial size.

[Scooletz]

I don't know how to answer. We may do all the things as long as the file is properly mapped and written and flushed 😅

[ssonthal]

Would you recommend -

1. initialize the _file with an initial size of say, 4 MB.
2. modifying the write and flush operations in a way it first checks the size of the _file and only if it's large enough to store incoming data, proceed as it is. Otherwise, can double the file size until it can accommodate incoming data.

[ssonthal]

Would you recommend -

1. initialize the _file with an initial size of say, 4 MB.
2. modifying the write and flush operations in a way it first checks the size of the _file and only if it's large enough to store incoming data, proceed as it is. Otherwise, can double the file size until it can accommodate incoming data.

@Scooletz What do you think about this approach?

[ssonthal]

Provided comments. I don't understand how this could work without significant changes or using the NT method mentioned in the referred issue. Maybe we should add some tests for the MemoryMappedPageManager to ensure that it works properly? Also, add some matrix for OS dependent tests?

I read somewhere that the .NET memory-mapped file API abstracts the complexity of memory management, so we don’t need to use lower-level functions like NtCreateSection in C#.

Might be completely wrong here.

Do you suggest using ntdll.dll and create a custom class like this to get the same behavior as lmdb? -

public static class NativeMethods
{
    [DllImport("ntdll.dll")]
    public static extern NTSTATUS NtCreateSection(
        out IntPtr SectionHandle,
        ACCESS_MASK DesiredAccess,
        ref OBJECT_ATTRIBUTES ObjectAttributes,
        ref LARGE_INTEGER MaximumSize,
        uint SectionPageProtection,
        uint AllocationAttributes,
        IntPtr FileHandle);
}

Although since ntdll.dll is an unmanaged dll, on Linux machines, it will not work.

[Scooletz]

Although since ntdll.dll is an unmanaged dll, on Linux machines, it will not work.

The issue is pure Windows issue (see #265 that you linked). On Linux, mmap behaves nicely not preallocating whole file up front. The only method that I partially investigated was the mentioned NtCreateSection, but it might be the case that there are other ways. One way or the other, the crux is that on Windows it bloats to the file size defined at the beginning.

[Scooletz]

Any comments @shubham-sonthalia ?