Add opt-in language features to the Qsharp.json manifest (#1089)

Closes #543 

## Overview

This PR adds two things:
1. The ability to opt-in to language features via the project manifest,
`qsharp.init()`, or command-line flags
2. A simple feature, `v2-preview-syntax`, ostensibly created to allow
users to opt-in to breaking syntax that's coming in the future, but
really created so I could wire up and show how a real language feature
works. Only one syntax change was made in this feature: removal of
scoped qubit blocks. This is because a scoped qubit block is identical
to a scoped block with a qubit in it. `use qubit = Qubit () {};` is
equivalent to `{ use qubit = Qubit(); }`. We therefore prefer the
latter.


## Implementation Notes

This PR includes the addition of `ParserContext`. It became clear to me
that language features need to be globally available to _at least_ the
parser, and probably other areas of the compiler later. Instead of
plumbing this all around manually as an argument, I promoted `Scanner`
into a `ParserContext`.

This PR also refactors manifest loading in `projectSystem.ts` -- we had
some duplicate code in `getManifestThrowsOnParseFailure` and
`getManifest`. I DRY'd that up.


## FAQ

### Why opt-in features?

In our discussions of evolving the language, it became clear that we
prioritize both improving the language, including in breaking ways, and
maintaining compatibility. As a result, introducing the notion of being
able to opt-in to a specific language feature became a blocker for most
further language design work.

### How will packages with different feature sets work together?

In general, we should aim to write features that don't break
compatibility with other packages. The language server and all compiler
scenarios assess language features on a per-package basis, so as long as
the individual compilation units are sane and well-formed, their feature
sets should be interoperable.

## Tests

These tests are considered _passing_ if turning on the language feature
`v2-preview-syntax` causes the compiler to reject scoped qubit block
syntax.

✅ Histogram:


https://github.com/microsoft/qsharp/assets/12157751/631dded2-d016-4933-83fe-bdff67852f50


✅ Language Service


https://github.com/microsoft/qsharp/assets/12157751/964574f6-c6bd-4225-acd4-6eb2afaec41c


✅  Jupyter Notebook


https://github.com/microsoft/qsharp/assets/12157751/fc5cc8b8-1c47-4dd9-b478-a233f16fc9e3



✅ `qsc`


https://github.com/microsoft/qsharp/assets/12157751/48e746fc-f7fe-4e9a-a658-d428b640d062



✅ `qsi`



https://github.com/microsoft/qsharp/assets/12157751/8154de70-80c5-444c-ac4e-f33d775a2fa9

✅  Debugger



https://github.com/microsoft/qsharp/assets/12157751/5aa7fb5a-3ec4-43fe-b256-6b75c808fe7e


✅ Language service inside of notebook cells


![image](https://github.com/microsoft/qsharp/assets/12157751/b710cc71-ce7d-4732-8acb-f315778239a1)


✅  QIR Generation




https://github.com/microsoft/qsharp/assets/12157751/eed50ca5-85c4-4434-9443-a6c67ef10337

Author: sezna

Cargo.lock

@@ -54,7 +54,7 @@ probability = "0.20"
 indenter = "0.3"
 regex-lite = "0.1"
 rustc-hash = "1.1.0"
-serde = "1.0"
+serde = { version = "1.0", features = [ "derive" ] }
 serde-wasm-bindgen = "0.6"
 wasm-bindgen = "0.2"
 wasm-bindgen-futures = "0.4"

Cargo.toml

@@ -4,6 +4,7 @@
 use criterion::{criterion_group, criterion_main, Criterion};
 use indoc::indoc;
 use qsc::{interpret::Interpreter, PackageType};
+use qsc_data_structures::language_features::LanguageFeatures;
 use qsc_eval::output::GenericReceiver;
 use qsc_frontend::compile::{RuntimeCapabilityFlags, SourceMap};
 
@@ -20,6 +21,7 @@ pub fn teleport(c: &mut Criterion) {
             sources,
             PackageType::Exe,
             RuntimeCapabilityFlags::all(),
+            LanguageFeatures::default(),
         )
         .expect("code should compile");
         b.iter(move || {
@@ -38,6 +40,7 @@ pub fn deutsch_jozsa(c: &mut Criterion) {
             sources,
             PackageType::Exe,
             RuntimeCapabilityFlags::all(),
+            LanguageFeatures::default(),
         )
         .expect("code should compile");
         b.iter(move || {
@@ -56,6 +59,7 @@ pub fn large_file(c: &mut Criterion) {
             sources,
             PackageType::Exe,
             RuntimeCapabilityFlags::all(),
+            LanguageFeatures::default(),
         )
         .expect("code should compile");
         b.iter(move || {
@@ -86,6 +90,7 @@ pub fn array_append(c: &mut Criterion) {
             sources,
             PackageType::Exe,
             RuntimeCapabilityFlags::all(),
+            LanguageFeatures::default(),
         )
         .expect("code should compile");
         b.iter(move || {
@@ -116,6 +121,7 @@ pub fn array_update(c: &mut Criterion) {
             sources,
             PackageType::Exe,
             RuntimeCapabilityFlags::all(),
+            LanguageFeatures::default(),
         )
         .expect("code should compile");
         b.iter(move || {
@@ -134,6 +140,7 @@ pub fn array_literal(c: &mut Criterion) {
             sources,
             PackageType::Exe,
             RuntimeCapabilityFlags::all(),
+            LanguageFeatures::default(),
         )
         .expect("code should compile");
         b.iter(move || {
@@ -169,6 +176,7 @@ pub fn large_nested_iteration(c: &mut Criterion) {
             sources,
             PackageType::Exe,
             RuntimeCapabilityFlags::all(),
+            LanguageFeatures::default(),
         )
         .expect("code should compile");
         b.iter(move || {

compiler/qsc/benches/eval.rs

@@ -3,6 +3,7 @@
 
 use criterion::{criterion_group, criterion_main, Criterion};
 use qsc::compile::{self, compile};
+use qsc_data_structures::language_features::LanguageFeatures;
 use qsc_frontend::compile::{PackageStore, RuntimeCapabilityFlags, SourceMap};
 use qsc_passes::PackageType;
 
@@ -20,6 +21,7 @@ pub fn large_file(c: &mut Criterion) {
                 sources,
                 PackageType::Exe,
                 RuntimeCapabilityFlags::all(),
+                LanguageFeatures::default(),
             );
             assert!(reports.is_empty());
         })

compiler/qsc/benches/large.rs

@@ -9,6 +9,7 @@ use log::info;
 use miette::{Context, IntoDiagnostic, Report};
 use qsc::compile::compile;
 use qsc_codegen::qir_base;
+use qsc_data_structures::language_features::LanguageFeatures;
 use qsc_frontend::{
     compile::{PackageStore, RuntimeCapabilityFlags, SourceContents, SourceMap, SourceName},
     error::WithSource,
@@ -55,6 +56,10 @@ struct Cli {
     /// Path to a Q# manifest for a project
     #[arg(short, long)]
     qsharp_json: Option<PathBuf>,
+
+    /// Language features to compile with
+    #[arg(short, long)]
+    features: Vec<String>,
 }
 
 #[derive(Copy, Clone, Debug, PartialEq, Eq, PartialOrd, Ord, ValueEnum)]
@@ -79,6 +84,8 @@ fn main() -> miette::Result<ExitCode> {
         dependencies.push(store.insert(qsc::compile::std(&store, capabilities)));
     }
 
+    let mut features = LanguageFeatures::from_iter(cli.features);
+
     let mut sources = cli
         .sources
         .iter()
@@ -93,12 +100,23 @@ fn main() -> miette::Result<ExitCode> {
             let mut project_sources = project.sources;
 
             sources.append(&mut project_sources);
+
+            features.merge(LanguageFeatures::from_iter(
+                manifest.manifest.language_features,
+            ));
         }
     }
 
     let entry = cli.entry.unwrap_or_default();
     let sources = SourceMap::new(sources, Some(entry.into()));
-    let (unit, errors) = compile(&store, &dependencies, sources, package_type, capabilities);
+    let (unit, errors) = compile(
+        &store,
+        &dependencies,
+        sources,
+        package_type,
+        capabilities,
+        features,
+    );
     let package_id = store.insert(unit);
     let unit = store.get(package_id).expect("package should be in store");
 

compiler/qsc/src/bin/qsc.rs

@@ -9,6 +9,7 @@ use miette::{Context, IntoDiagnostic, Report, Result};
 use num_bigint::BigUint;
 use num_complex::Complex64;
 use qsc::interpret::{self, InterpretResult, Interpreter};
+use qsc_data_structures::language_features::LanguageFeatures;
 use qsc_eval::{
     output::{self, Receiver},
     val::Value,
@@ -47,6 +48,10 @@ struct Cli {
     /// Path to a Q# manifest for a project
     #[arg(short, long)]
     qsharp_json: Option<PathBuf>,
+
+    /// Language features to compile with
+    #[arg(short, long)]
+    features: Vec<String>,
 }
 
 struct TerminalReceiver;
@@ -80,6 +85,8 @@ fn main() -> miette::Result<ExitCode> {
         .map(read_source)
         .collect::<miette::Result<Vec<_>>>()?;
 
+    let mut features = LanguageFeatures::from_iter(cli.features);
+
     if sources.is_empty() {
         let fs = StdFs;
         let manifest = Manifest::load(cli.qsharp_json)?;
@@ -88,6 +95,10 @@ fn main() -> miette::Result<ExitCode> {
             let mut project_sources = project.sources;
 
             sources.append(&mut project_sources);
+
+            features.merge(LanguageFeatures::from_iter(
+                manifest.manifest.language_features,
+            ));
         }
     }
     if cli.exec {
@@ -96,6 +107,7 @@ fn main() -> miette::Result<ExitCode> {
             SourceMap::new(sources, cli.entry.map(std::convert::Into::into)),
             PackageType::Exe,
             RuntimeCapabilityFlags::all(),
+            features,
         ) {
             Ok(interpreter) => interpreter,
             Err(errors) => {
@@ -115,6 +127,7 @@ fn main() -> miette::Result<ExitCode> {
         SourceMap::new(sources, None),
         PackageType::Lib,
         RuntimeCapabilityFlags::all(),
+        features,
     ) {
         Ok(interpreter) => interpreter,
         Err(errors) => {

compiler/qsc/src/bin/qsi.rs

@@ -2,6 +2,7 @@
 // Licensed under the MIT License.
 
 use miette::{Diagnostic, Report};
+use qsc_data_structures::language_features::LanguageFeatures;
 use qsc_frontend::{
     compile::{CompileUnit, PackageStore, RuntimeCapabilityFlags, SourceMap},
     error::WithSource,
@@ -27,8 +28,15 @@ pub fn compile(
     sources: SourceMap,
     package_type: PackageType,
     capabilities: RuntimeCapabilityFlags,
+    language_features: LanguageFeatures,
 ) -> (CompileUnit, Vec<Error>) {
-    let mut unit = qsc_frontend::compile::compile(store, dependencies, sources, capabilities);
+    let mut unit = qsc_frontend::compile::compile(
+        store,
+        dependencies,
+        sources,
+        capabilities,
+        language_features,
+    );
     let mut errors = Vec::new();
     for error in unit.errors.drain(..) {
         errors.push(WithSource::from_map(&unit.sources, error.into()));

compiler/qsc/src/compile.rs

@@ -3,6 +3,7 @@
 
 use crate::compile::{self, compile, core, std};
 use miette::Diagnostic;
+use qsc_data_structures::language_features::LanguageFeatures;
 use qsc_frontend::{
     compile::{OpenPackageStore, PackageStore, RuntimeCapabilityFlags, SourceMap},
     error::WithSource,
@@ -37,6 +38,7 @@ impl Compiler {
         sources: SourceMap,
         package_type: PackageType,
         capabilities: RuntimeCapabilityFlags,
+        language_features: LanguageFeatures,
     ) -> Result<Self, Errors> {
         let core = core();
         let mut store = PackageStore::new(core);
@@ -47,15 +49,27 @@ impl Compiler {
             dependencies.push(id);
         }
 
-        let (unit, errors) = compile(&store, &dependencies, sources, package_type, capabilities);
+        let (unit, errors) = compile(
+            &store,
+            &dependencies,
+            sources,
+            package_type,
+            capabilities,
+            language_features,
+        );
         if !errors.is_empty() {
             return Err(errors);
         }
 
         let source_package_id = store.insert(unit);
         dependencies.push(source_package_id);
 
-        let frontend = qsc_frontend::incremental::Compiler::new(&store, dependencies, capabilities);
+        let frontend = qsc_frontend::incremental::Compiler::new(
+            &store,
+            dependencies,
+            capabilities,
+            language_features,
+        );
         let store = store.open();
 
         Ok(Self {

compiler/qsc/src/incremental.rs

@@ -26,6 +26,7 @@ use num_bigint::BigUint;
 use num_complex::Complex;
 use qsc_codegen::qir_base::BaseProfSim;
 use qsc_data_structures::{
+    language_features::LanguageFeatures,
     line_column::{Encoding, Range},
     span::Span,
 };
@@ -123,12 +124,13 @@ impl Interpreter {
         sources: SourceMap,
         package_type: PackageType,
         capabilities: RuntimeCapabilityFlags,
+        language_features: LanguageFeatures,
     ) -> Result<Self, Vec<Error>> {
         let mut lowerer = qsc_eval::lower::Lowerer::new();
         let mut fir_store = fir::PackageStore::new();
 
-        let compiler =
-            Compiler::new(std, sources, package_type, capabilities).map_err(into_errors)?;
+        let compiler = Compiler::new(std, sources, package_type, capabilities, language_features)
+            .map_err(into_errors)?;
 
         for (id, unit) in compiler.package_store() {
             fir_store.insert(
@@ -359,8 +361,15 @@ impl Debugger {
         sources: SourceMap,
         capabilities: RuntimeCapabilityFlags,
         position_encoding: Encoding,
+        language_features: LanguageFeatures,
     ) -> Result<Self, Vec<Error>> {
-        let interpreter = Interpreter::new(true, sources, PackageType::Exe, capabilities)?;
+        let interpreter = Interpreter::new(
+            true,
+            sources,
+            PackageType::Exe,
+            capabilities,
+            language_features,
+        )?;
         let source_package_id = interpreter.source_package;
         Ok(Self {
             interpreter,

compiler/qsc/src/interpret.rs

@@ -5,6 +5,7 @@
 
 use indoc::indoc;
 use miette::Result;
+use qsc_data_structures::language_features::LanguageFeatures;
 use qsc_eval::{output::CursorReceiver, val::Value};
 use qsc_frontend::compile::{RuntimeCapabilityFlags, SourceMap};
 use qsc_passes::PackageType;
@@ -70,6 +71,7 @@ fn stack_traces_can_cross_eval_session_and_file_boundaries() {
         source_map,
         PackageType::Lib,
         RuntimeCapabilityFlags::all(),
+        LanguageFeatures::default(),
     )
     .expect("Failed to compile base environment.");
 
@@ -144,6 +146,7 @@ fn stack_traces_can_cross_file_and_entry_boundaries() {
         source_map,
         PackageType::Exe,
         RuntimeCapabilityFlags::all(),
+        LanguageFeatures::default(),
     )
     .expect("Failed to compile base environment.");