Deserializing Serde JSON AST to Program

[shahryarjb]

Hi everyone, apologies in advance as I'm new to Rust and have been working on this project for a little while.
I've written this function to convert JavaScript code into an AST:

pub fn source_to_ast<'a>(file_path: &str, allocator: &'a Allocator) -> Result<Program<'a>, String> {
    if !Path::new(file_path).exists() {
        return Err(format!("File does not exist"));
    }

    let source_text = fs::read_to_string(file_path)
        .map_err(|e| format!("Failed to read the JavaScript file: {}", e))?
        .into_boxed_str();
    let source_text: &'a str = Box::leak(source_text);

    let source_type = SourceType::from_path(file_path)
        .map_err(|e| format!("Failed to determine source type: {:?}", e))?;

    let parser = Parser::new(allocator, source_text, source_type).with_options(ParseOptions {
        parse_regular_expression: true,
        ..ParseOptions::default()
    });

    let parse_result = parser.parse();
    Ok(parse_result.program)
}

And I convert the AST to JSON using the following code:

let allocator = Allocator::default(); // Create an OXC allocator
match source_to_ast(&file_path, &allocator) {
    Ok(ast) => {
        // Attempt to serialize the AST into JSON
        let ast_json_result: Result<Value, String> =
            to_value(&ast).map_err(|e| format!("Serialization error: {}", e));

        let (status, message) = match ast_json_result {
            // If serialization is successful, modify the JSON object and return it
            Ok(mut ast_json) => {
                if let Value::Object(ref mut map) = ast_json {
                    map.insert(
                        "source_text".to_string(),
                        Value::String(ast.source_text.to_string()),
                    );
                }
                (atoms::ok(), ast_json.to_string())
            }
            // If serialization fails, send the error message to Elixir
            Err(serialization_error) => (atoms::error(), serialization_error),
        };

        encode_response(env, status, atoms::source_to_ast_nif(), message)
    }
    // If the source_to_ast function fails, send the error message to Elixir
    Err(msg) => encode_response(env, atoms::error(), atoms::source_to_ast_nif(), msg),
}

Now, I want to do the reverse: convert JSON back into a Program. However, I suspect that deserialization (Deserialize) might not be supported.

I got some inspiration from this file in the project on how to add it, but I wasn't completely successful:
https://github.com/oxc-project/oxc/blob/38ba2f640c059cae5c27c126f343a5dd44aecfd7/crates/oxc_linter/src/config/overrides.r

Any suggestions or guidance would be greatly appreciated! 😊

[overlookmotel]

I'm sorry but we've not implemented Deserialize on the AST yet. It's a tricky problem as we have to "massage" our Rust AST to align with ESTree in the JSON AST. So those custom modifications need to be made in reverse when going from JSON back to Rust, which is not so easy.

Our current work in this area is around creating a framework for serialization/deserialization on top of serde (#6347), which will make it easier for us to achieve ESTree compatibility in JSON AST. We will tackle going in the opposite direction (JSON back to Rust) after that work is complete.

So TDLR: This is not possible at present, and I'm sorry to say it will not be coming in immediate future. But better interop with JS/JSON is something we consider important, and we will be aiming to make progress on it in the new year.

Out of interest, why are you needing to covert AST to JSON and back?

[shahryarjb]

Hello, Dear @overlookmotel , I hope you're doing well. Thank you for your quick and thorough response 🙏🏻♥️.

I'm planning to gradually port the entire OXC library to Elixir using the Rustler library. The goal is to give users the ability to access the AST of a JS file in Elixir, apply their own modifications, and then send it to Rust. To ensure minimal data transformation, I convert it to JSON and provide the output.

I've placed the relevant code at the following link:
https://github.com/shahryarjb/mishka_chelekom/blob/rustler-nifs/native/mishka_parser_nif/src/lib.rs#L34-L48

This project heavily depends on this functionality. Without it, it's not feasible to make the required AST modifications directly in Elixir and then send them to Rust for operations such as Codegen.

I sincerely apologize for taking up your time. Since I’m quite new to Rust, there might be a better approach. I haven’t tested it yet, but it seems that the intermediary library (Rustler/#structs) has something called NifStruct. I’m not sure if it can be used in this case or not.

Thank you in advance

[overlookmotel]

No worries. Happy to help (except that I'm not being very helpful!)

I don't know anything about Elixir, but I think I understand what you're trying to do, and great that you're looking to port Oxc.

However, I'm sorry to say I don't think what you need is possible at present.

What you've suggested is likely the best approach - using serde to serialize/deserialize to a format (e.g. JSON) which is supported in both Rust and Erlang.

But problem is that it requires Deserialize to be implemented on AST types on Rust side. We will do that, but we are not in a position to do it yet. It's hard to do properly, and I'm afraid we would not want to accept a PR for a "partial" implementation, as then it'd be a maintenance burden, and make it harder for us to make progress with our current focus, which is sorting out transfer in the other direction (Rust to JSON).

So I'm afraid you're blocked at present.

If you really want to move forwards with this in short term, you could implement Serialize and Deserialize as a wrapper around Oxc in your own Rust code (treating Oxc's AST types as "foreign types"). But I must warn you, that would not be a small undertaking, as there are a lot of AST types.

Sorry to be the bearer of bad news. I for one feel strongly that cross-language interop is really important, and I've been working on a new serialization scheme which is way faster than JSON. But our focus up to now has been mainly on making everything work well, correctly, and performantly on the Rust side, so interop is unfortunately an undeveloped area at present. There's only so much time in the day!

[overlookmotel]

PS I have no idea if Rustler offers an alternative solution. I completely lack the knowledge to understand what it does (I don't even know what a NIF is). Maybe it can help you if it has some alternative serialization/deserialization mechanism, but I can't advise on that.

[shahryarjb]

So, there's no choice but to wait 😂. Thank you for your responses. I hope this feature will also be added to OXC. It truly has great potential to improve many libraries that need to modify JS in Elixir.
Thank you!