Replace official MongoDB Driver

[Turnerj]

Due to the numerous issues with the driver which make working with it far harder than it needs to be and more limiting, this issue is for the eventual removal of the official MongoDB Driver with a custom implementation.

This does mean a large loss in feature compatibility between MongoFramework and the official driver however this may be a good thing. Removing the limitations of the driver allows for more advanced querying.

Some of the reasons for this change include:

• Lack of full DateTime support (try running MyDate.Date in a LINQ query)
  Per James Kovacs on Twitter, this is due to no 1-to-1 mapping of MyDate.Date in particular. However using new DateTime(year, month, day) gives the desired result so it is certainly possible.
• Lack of full URI support (try running a LINQ query against the Host or Path on a URI type).
  This issue is likely unfair to the driver as there is no direct translations for the properties on URI to a MongoDB query counterpart. For me to implement it, I'd need to use $regexMatch and other such queries which have their own limitations.
• Issues with type descriptors being inconsistent
• The freezing of class maps (seems entirely unnecessary, especially when methods like BsonClassMap.LookupClassMap freeze the map unexpectedly). Freezing also seems to affect concurrent class map registration, requiring its own work arounds.
• How complicated it is to reset the driver to the default state (primarily a problem for my own testing)
  This would be due to lots of static types that have their own internal caches and registries etc. Even knowing what classes etc are like this, I can only access the properties to reset via reflection as they are all private types.
• No method of handling renaming of types (MongoFramework might be able to work around this via a new custom PreviouslyKnownAs attribute without a full replacement of the driver though)
• Not handling generics properly through discriminators (there might be semi-workarounds for it but the core driver doesn't handle it by itself)
• Limitations in extending LINQ support for more advanced aggregation queries
• ResultTransformer (used for .Any() queries) isn't generating efficient queries, instead sending a whole entity back (it does limit it to one entity but it screws up entity tracking - it should send the smallest payload back - maybe a projection to a single value) (see 8c1fcc6 as collateral)
• SetExtraElementsMember is a little finicky about the instance of the BsonMemberMap instance (see this commit for workaround)
• Performance and memory issues - not enough that people may notice but enough to be measurable.
  I don't know specific areas in the driver where it happens (query building vs serialization vs connection logic). I do know that MongoFramework wrapping the driver doesn't help but comparatively looking at how it compares in a caching solution, it seems that there is quite a bit of room for improvement. Once more progress is made on Custom LINQ engine #112 then we will see how LINQ processing plays a part in this. Partial benchmarks for the official driver can be found here, pending future comparison to custom implementation.
• Missing support for text score and sorting via LINQ
  There are ways to do this via the driver but not specifically through LINQ as far as I can tell. This will be achievable with a replacement to the core LINQ system as the queries needed can be generated directly in BSON for the aggregation pipeline.
• GetType() is not support in queries, for example in a query like GroupBy(e => e.GetType())

The list of reasons here isn't exhaustive, just the ones that come to mind as I write this

Sure, a few of these things are just bugs in the MongoDB Driver and could be fixed but some are fundamental flaws with how the driver works.

This change is definitely not scheduled for Version 1 as it is a huge change - this issue is really just for putting it on the cards of things that need looking at in the future.

[Turnerj]

Current status:

• Remove BsonClassMap.AutoMap from Mapping Registration Process #106 (completed) tracks dropping BsonClassMap.AutoMap from MongoFramework
• Custom LINQ Provider #87 tracks replacing the LINQ core of the MongoDB driver
• Custom Entity Serializer #109 tracks replacing the serializer core of the MongoDB driver

In addition to these, there would still need to be tasks for:

• Low-level command writing (translating insert/update/replace etc to write queries - replacing the internals of what BulkWrite etc do - this would include writing indexes etc)
• Replace connection core (all the lowest level communication to MongoDB)

[ahmednfwela]

This issue was opened in 2019, is this still relevant when we are almost 2023 ? or can this package depend on the official driver again ?

[Turnerj]

Hey @ahmednfwela - currently MongoFramework still does depend on the MongoDB Driver in a number of significant ways. That said, a number of the issues I've raised above still remain in the driver.

There is some movement for things to be better and more extendable including some major changes to their internal LINQ implementation (which I plan to utilise more heavily in future releases) however I'd still pursue moving more pieces out of the driver's hands and into MongoFramework where I can introduce more control and flexibility of behaviour.

[ahmednfwela]

Hey @Turnerj , thanks for the response.

Replacing the official driver is fine and all, but I have been running into conflicts trying to use the official driver LINQ and MongoFramework LINQ in the same file, I can't even use ToListAsync() on IMongoQueryable because the driver is using extensions on IQueryable as well

[Turnerj]

Yeah, extension conflicts from importing both MongoFramework.Linq and the driver's equivalent in the same file is definitely going to have problems. There isn't a lot I can really do about that with the biggest thing to help, having my LINQ extensions tucked behind their own namespace, is already in place versus having them in the global namespace.

One fairly average suggestion would be to split the class you're working in into separate files to avoid it - like a partial class. Then in one you can reference anything MongoFramework related and in the other, anything driver related. Obviously this breaks down quite a bit if it is a single method where you need both plus partial classes can get messy.

Out of curiousity, what were you needing to fall back to the driver for? Perhaps there is a way MongoFramework can cover it so you don't need to?

[Turnerj]

Oh, I should probably state, even if I wasn't going to replace the driver any more, those LINQ extensions are probably one of the areas that would remain regardless. To provide some of the functionality in MongoFramework, I have to wrap the driver in certain ways as many types and interfaces are internal to the driver.

[ahmednfwela]

Out of curiousity, what were you needing to fall back to the driver for? Perhaps there is a way MongoFramework can cover it so you don't need to?

I am migrating an old project from (MSSQL [EF core] + MongoDB) to MongoDB Only, and I was replacing EF core context with MongoFramework contexts, where I started facing this IQueryable conflict issue.

So I decided to build my own context class and use raw IMongoCollection instead.

I think that a simple solution to the conflicts would be to create a new Queryable type, e.g.: IMongoFrameworkQueryable<T> that extends IMongoQueryable<T> and build all the extensions on it