Ruby: add support for extracting overlay databases #19684

nickrolfe · 2025-06-06T11:30:33Z

There will need to be followup changes on the QL side before querying will work, but this is enough to be able to build an overlay database for Ruby.

Copilot

Pull Request Overview

This PR adds support for overlay database extraction in Ruby by introducing a new databaseMetadata relation and path‐prefix transformation logic.

Introduces a --add-metadata-relation flag to the shared extractor generator and implements create_database_metadata().
Adds PathTransformer and normalize_and_transform_path in the shared extractor for prefix rewrites.
Updates the Ruby extractor to skip unchanged files, load overlay changes, apply path transformations, and emit an empty base‐metadata file.

Reviewed Changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
shared/tree-sitter-extractor/src/generator/mod.rs	Added `add_metadata_relation` flag and `create_database_metadata`
shared/tree-sitter-extractor/src/file_paths.rs	Implemented `PathTransformer`, `load_path_transformer`, and path normalization with transformation
shared/tree-sitter-extractor/src/extractor/*.rs	Propagated transformer through file population and trap writing
ruby/ql/lib/ruby.dbscheme	Added new `databaseMetadata` relation definition
config/dbscheme-fragments.json	Registered the `Database metadata` fragment
ruby/extractor/src/extractor.rs	Added overlay‐change filtering, transformer usage, and metadata output
ruby/extractor/src/generator.rs	Enabled metadata relation in the Ruby generator

Comments suppressed due to low confidence (1)

ruby/extractor/src/extractor.rs:347

[nitpick] New get_overlay_changed_files logic and load_path_transformer are not covered by existing tests. Consider adding unit tests for parsing the JSON changes file and path transformation behavior.

fn get_overlay_changed_files() -> Option<HashSet<PathBuf>> {

shared/tree-sitter-extractor/src/file_paths.rs

Copilot · 2025-06-10T16:49:33Z

ruby/extractor/src/extractor.rs

+        .map(|change| {
+            change
+                .as_str()
+                .map(|s| PathBuf::from(s).canonicalize().ok())
+                .flatten()
+        })
+        .collect()


Collecting an iterator of Option<PathBuf> will not produce HashSet<PathBuf>. You should use filter_map to drop None entries and wrap the resulting HashSet in Some(...) before returning.

Suggested change

.map(|change| {

change

.as_str()

.map(|s| PathBuf::from(s).canonicalize().ok())

.flatten()

})

.collect()

.filter_map(|change| {

change

.as_str()

.and_then(|s| PathBuf::from(s).canonicalize().ok())

})

.collect::<HashSet<PathBuf>>()

.into()

nickrolfe · 2025-06-12T15:31:56Z

@aibaars do you have capacity to review this?

aibaars

Looks fine to me. A few comments though. I think it would be fine to add the new features to all extractors .

aibaars · 2025-06-19T08:26:59Z

shared/tree-sitter-extractor/src/generator/mod.rs

@@ -32,6 +33,16 @@ pub fn generate(

    writeln!(dbscheme_writer, include_str!("prefix.dbscheme"))?;

+    // Eventually all languages will have the metadata relation (for overlay support), at which


I think we can just shove the metadata tables directly in prefix.dbscheme. An unused table shouldn't hurt.

shared/tree-sitter-extractor/src/extractor/simple.rs

aibaars · 2025-06-19T08:32:11Z

rust/extractor/src/trap.rs

@@ -212,7 +212,7 @@ impl TrapFile {
        );
    }
    pub fn emit_file(&mut self, absolute_path: &Path) -> Label<generated::File> {
-        let untyped = extractor::populate_file(&mut self.writer, absolute_path);
+        let untyped = extractor::populate_file(&mut self.writer, absolute_path, None);


I would just add support for path transformers to the Rust extractor.

aibaars · 2025-06-19T08:32:40Z

rust/extractor/src/archive.rs

@@ -15,7 +15,7 @@ impl Archiver {
    }

    fn try_archive(&self, source: &Path) -> std::io::Result<()> {
-        let dest = file_paths::path_for(&self.root, source, "");
+        let dest = file_paths::path_for(&self.root, source, "", None);


I would just add support for path transformers to the Rust extractor.

shared/tree-sitter-extractor/src/file_paths.rs

aibaars · 2025-06-19T08:40:37Z

shared/tree-sitter-extractor/src/file_paths.rs

+        .filter(|line| !line.is_empty())
+        .collect::<Vec<String>>();
+
+    if lines.len() != 2 {


How hard would it be to just implement path transformers completely?

nickrolfe · 2025-06-19T15:02:55Z

I've force-pushed a couple of the suggested fixes. Regarding the others:

prefix.dbscheme: since this will change the QL-for-QL dbscheme, I'll do it as a quick follow-up PR. That way, if any issues arise, it can be reverted without affecting Ruby overlay support.
similarly, I'll add the Rust path-transformer support as a separate PR.
for implementing full path-transformer support, I don't think it would be an enormous amount of work to port either the existing Java or C++ implementations, along with their unit tests, but it's probably enough work that we discussed within our team and decided not to do it (since it's not on the critical path for overlay support).

This is required for overlay support.

Supports only a minimal subset of the project layout specification; enough to work with the transformers produced by the CLI when building an overlay database.

nickrolfe added the Ruby label Jun 6, 2025

github-actions bot added QL-for-QL Rust Pull requests that update Rust code labels Jun 6, 2025

nickrolfe force-pushed the nickrolfe/ruby-overlay-extraction branch 2 times, most recently from 33c2550 to fb89f22 Compare June 6, 2025 14:03

nickrolfe marked this pull request as ready for review June 10, 2025 16:45

Copilot AI review requested due to automatic review settings June 10, 2025 16:45

nickrolfe requested review from a team as code owners June 10, 2025 16:45

nickrolfe mentioned this pull request Jun 10, 2025

Ruby: generate overlay discard predicates #19719

Open

Copilot AI reviewed Jun 10, 2025

View reviewed changes

nickrolfe force-pushed the nickrolfe/ruby-overlay-extraction branch 2 times, most recently from 576f0d2 to 4029b89 Compare June 11, 2025 11:45

nickrolfe assigned aibaars Jun 12, 2025

aibaars reviewed Jun 19, 2025

View reviewed changes

nickrolfe force-pushed the nickrolfe/ruby-overlay-extraction branch from 4029b89 to fa24ee6 Compare June 19, 2025 14:57

nickrolfe added 3 commits June 19, 2025 16:34

Ruby: add support for extracting overlays

c4ccc55

Ruby: add databaseMetadata relation to dbscheme

1bd7c4f

This is required for overlay support.

Ruby: add minimal path transformer support

665df4b

Supports only a minimal subset of the project layout specification; enough to work with the transformers produced by the CLI when building an overlay database.

nickrolfe force-pushed the nickrolfe/ruby-overlay-extraction branch from fa24ee6 to 665df4b Compare June 19, 2025 15:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Ruby: add support for extracting overlay databases #19684

Ruby: add support for extracting overlay databases #19684

nickrolfe commented Jun 6, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI Jun 10, 2025

Uh oh!

nickrolfe commented Jun 12, 2025

Uh oh!

aibaars left a comment

Uh oh!

aibaars Jun 19, 2025

Uh oh!

Uh oh!

aibaars Jun 19, 2025

Uh oh!

aibaars Jun 19, 2025

Uh oh!

Uh oh!

aibaars Jun 19, 2025

Uh oh!

nickrolfe commented Jun 19, 2025

Uh oh!

Uh oh!

		@@ -32,6 +33,16 @@ pub fn generate(

		writeln!(dbscheme_writer, include_str!("prefix.dbscheme"))?;

		// Eventually all languages will have the metadata relation (for overlay support), at which

Ruby: add support for extracting overlay databases #19684

Are you sure you want to change the base?

Ruby: add support for extracting overlay databases #19684

Conversation

nickrolfe commented Jun 6, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Copilot AI Jun 10, 2025

Choose a reason for hiding this comment

Uh oh!

nickrolfe commented Jun 12, 2025

Uh oh!

aibaars left a comment

Choose a reason for hiding this comment

Uh oh!

aibaars Jun 19, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

aibaars Jun 19, 2025

Choose a reason for hiding this comment

Uh oh!

aibaars Jun 19, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

aibaars Jun 19, 2025

Choose a reason for hiding this comment

Uh oh!

nickrolfe commented Jun 19, 2025

Uh oh!

Uh oh!