Current status of the project - september 23

[jaytaph]

At the beginning of september 2023, there was no gosub. It started as a (al-be-it) futile protest against big tech corps (see the blogpost and related hackernews post).

In the few weeks where I had some time, i've set up the following:

• a basic html5 tokenizer that can convert a string of html code into tokens.
• a basic html5 parser that ALMOST can convert these tokens into a node tree.
• have some basic test programs that checks the tokenizer and parser against a test suite called html5lib-tests.

There is still a lot to do, but it's a bit difficult to organize everything at the moment. There are too many things we don't know yet. But for now, the main plan will be:

Make sure the parser can tokenize the tokens into a node-tree.

This work is in progress and some tests are passing. However, the parser is a very complex system where a lot of things can go wrong, which result in endless debugging. The parser is not the fastest parser, and probably has lots of issues.. but the main goal is to be able to parse html5. At a later point, we can see if we can also make it fast, nice and hopefully keep the memory footprint low.

Have a different system (the user-agent (UA)) that receives the node-tree in some form, and prints this only the screen (textmode, maybe graphical).

I want the engine to be separated from the actual browser (the user-agent). This allows us to use the gosub-engine in other applications that needs a html5-browser, or even create a completely different browser based on just the engine. For now my goal would be to have a simple user-agent that receives the node tree that the parser creates, and display this tree on the (graphical/text) screen.

This could be a completely standalone project that does not depend on the parser to be ready. It just needs to have a document with a (correct) html5 tree, which can be mocked.

Analyze the current code and see we can fix all the beginner Rust mistakes i've made.

I have only 3 weeks (21 days) of experience with rust, so there is a lot of mistakes i'm making codewise. We need people to look at the code, try not to laugh at it, and try and fix the mistakes i've made.

generate a DOM tree from the given node tree (or maybe better, convert the node-tree so it can be used as the DOM tree.

Once we have a node tree generated by the parser, we should create a DOM tree. This is fairly similar to the actual node tree, but has all kind of mandatory fields so it can be used by the render engine to display things on screen. There are two ways to deal with this:

• generate a DOM tree directly from the html5 parser.
• generate a DOM tree from the node tree.

If it doesn't make things too complex, i'm for option 1, otherwise, we might keep things a bit simpler by using option 2. Note however, that when we incorporate a javascript engine (v8) into gosub, the parser can execute javascript as well, which needs a dom tree. So at that point, it might even make sense to have a dom tree right in the parser.

Hopefully, we have these points done by the end of next month. But in the meantime, there is enough discussion needed in order to create a real vision for this browser effort.

[AR-DEV-1]

Cool

[mihna123]

What about css?

[jaytaph]

css parsing will be a part of the project, but for the time being not the most important component. We need more research in how to incorporate css parsing (from what I understand, css parsing itself is quite easy).

[mihna123]

I would like to try and create the css parser one of these days. Sounds fun!

[ynewmark]

Hey, can I try my hand at a simple text based user-agent? Would a first version that just prints the text inside the nodes be useful?

[jaytaph]

Absolutely! That would be a great first start. You could use gosub-engine as the engine.. and have your user-agent read a html page from an url. Give this to the inputstream (which in turn is needed for the parser), and display the document that is returned by the parser.

Should be fairly straight forward, except for the fact that the parser is not quite done yet. There will be things missing and crashing even, but simple html should be possible.

[shellheim]

Woah, a damm browser in rust? That sounds pretty exciting!

[Kiyoshika]

What a coincidence, I've been thinking of starting a browser project myself. I wrote a custom markup language parser and renderer for the terminal in C (kind of like HTML but for the terminal using ncurses) but I don't really know Rust. I'd like to somehow contribute to this at some point

[AR-DEV-1]

I also have started a Web Browser Project;
https://github.com/AR-DEV-1/CyberWave

[emwalker]

I'm a kind of a beginner with Rust myself. I probably have 1-2 years of side-project experience in Rust, depending on how you add things up. But I've worked with nom a little and am quite impressed with it. I see that we're doing some pretty low-level stuff in the initial pass of the HTML5 parser. I was wondering whether it made sense to evaluate nom for doing the low-level parsing instead. Here is an example of what it looks like in practice. It has a streaming mode, so the requirement of a streaming parser shouldn't present an issue.

The appeals of nom are (1) the project's aim of being zero copy, (2) the combinator style and (3) the ease in unit testing at a very granular level (more granular than in my example). I suspect it is also fast, but I do not have data to back this up.

It is easy to consider changes like this at this stage of the project. It will become harder as the project advances along.

[emwalker]

Judging from the reply to this question on Reddit, I'm thinking nom and the other parser libraries might be too restrictive for the use case of a browser, although I'm not 100 percent confident of that. If nom can handle things like ambiguous grammars, backtracking and error recovery, it might still be suitable (probably safe to say it doesn't attempt all of these things).

[smorelk]

Pretty dope!

[kastil300]

very nice