Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: String templates #2630

Merged
merged 36 commits into from
Dec 1, 2023
Merged
Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
ca09099
Initial scanner and parser
timotheeguerin Oct 27, 2023
3b9ba72
Progress
timotheeguerin Oct 27, 2023
e13b6f5
Unindent after
timotheeguerin Nov 2, 2023
ca98178
Merge branch 'main' of https://github.com/Microsoft/typespec into fea…
timotheeguerin Nov 2, 2023
ab49c95
Scanner and parser working
timotheeguerin Nov 3, 2023
2eb4eb7
Checker
timotheeguerin Nov 3, 2023
d1c5c04
String template checker
timotheeguerin Nov 3, 2023
0300483
Json Schema handle template string
timotheeguerin Nov 3, 2023
a572001
add tm language
timotheeguerin Nov 3, 2023
71741b1
Colorization
timotheeguerin Nov 3, 2023
e027aa5
Fix issue with multiple segments
timotheeguerin Nov 3, 2023
dad6fd7
Add parsing test for model expression in string template
timotheeguerin Nov 6, 2023
ad23a85
Format
timotheeguerin Nov 6, 2023
a84c0a6
.
timotheeguerin Nov 6, 2023
58f4802
Merge branch 'main' into feature/string-template
timotheeguerin Nov 8, 2023
c3b4911
Merge branch 'main' into feature/string-template
timotheeguerin Nov 13, 2023
f98ab52
update grammar
timotheeguerin Nov 13, 2023
a3f281f
Merge branch 'main' into feature/string-template
timotheeguerin Nov 13, 2023
16e7f8b
Auto marshaling of values
timotheeguerin Nov 13, 2023
a4b6d9c
Add string template support
timotheeguerin Nov 14, 2023
45c8a05
Docs and prism-js
timotheeguerin Nov 14, 2023
0bbe5aa
Merge branch 'main' into feature/string-template
timotheeguerin Nov 14, 2023
07338eb
Merge branch 'main' into feature/string-template
timotheeguerin Nov 14, 2023
b0101fe
Update grammar
timotheeguerin Nov 16, 2023
f66688b
Update common/changes/@typespec/compiler/feature-string-template_2023…
timotheeguerin Nov 16, 2023
a8ace5c
Add test
timotheeguerin Nov 16, 2023
f103e5a
Add test and docs on decorators
timotheeguerin Nov 16, 2023
d023fa8
Merge branch 'feature/string-template' of https://github.com/timothee…
timotheeguerin Nov 16, 2023
36248c4
Merge branch 'main' into feature/string-template
timotheeguerin Nov 16, 2023
91c4fdc
Merge branch 'main' into feature/string-template
timotheeguerin Nov 16, 2023
9867f86
Fix grammar, add support for openapi3
timotheeguerin Nov 16, 2023
d0ae965
openapi3 changelog
timotheeguerin Nov 16, 2023
a9f761e
Merge branch 'main' into feature/string-template
timotheeguerin Nov 17, 2023
6c18405
ADd sample and fix issue with template param
timotheeguerin Nov 17, 2023
08e7b3f
Merge branch 'main' into feature/string-template
markcowl Nov 21, 2023
ae003dd
Merge with main
timotheeguerin Dec 1, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions packages/compiler/lib/reflection.tsp
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,4 @@ model Operation {}
model Scalar {}
model Union {}
model UnionVariant {}
model StringTemplate {}
73 changes: 69 additions & 4 deletions packages/compiler/src/core/checker.ts
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,14 @@ import {
StdTypes,
StringLiteral,
StringLiteralNode,
StringTemplate,
Copy link
Member Author

@timotheeguerin timotheeguerin Nov 3, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do the names look good? Alternatives could have been

  • TemplateString
  • TemplateStringLiteral
  • TemplateLiteral
  • StringTemplateLiteral

Copy link
Contributor

@daviwil daviwil Nov 16, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

StringTemplateLiteral is probably more accurate

EDIT: Ahh, I didn't see the rest of the names. Probably don't want all of them to be StringTemplateLiteralX so StringTemplate is fine IMO

StringTemplateExpressionNode,
StringTemplateHeadNode,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you split this into head/middle/end so that you only capture the part that needs to be computed as the middle?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Those are technically slightly different nodes and that's how typescript also structure their template so thought be good to keep inline as they must have had a reason to do that. But right now the data inside is basically the same

StringTemplateMiddleNode,
StringTemplateSpan,
StringTemplateSpanLiteral,
StringTemplateSpanValue,
StringTemplateTailNode,
Sym,
SymbolFlags,
SymbolLinks,
Expand Down Expand Up @@ -641,6 +649,8 @@ export function createChecker(program: Program): Checker {
return checkTupleExpression(node, mapper);
case SyntaxKind.StringLiteral:
return checkStringLiteral(node);
case SyntaxKind.StringTemplateExpression:
return checkStringTemplateExpresion(node, mapper);
case SyntaxKind.ArrayExpression:
return checkArrayExpression(node, mapper);
case SyntaxKind.UnionExpression:
Expand Down Expand Up @@ -2382,6 +2392,48 @@ export function createChecker(program: Program): Checker {
return getMergedSymbol(aliasType.node!.symbol) ?? aliasSymbol;
}
}

function checkStringTemplateExpresion(
node: StringTemplateExpressionNode,
mapper: TypeMapper | undefined
): StringTemplate {
const spans: StringTemplateSpan[] = [createTemplateSpanLiteral(node.head)];
for (const span of node.spans) {
spans.push(createTemplateSpanValue(span.expression, mapper));
spans.push(createTemplateSpanLiteral(span.literal));
}
const type = createType({
kind: "StringTemplate",
node,
spans,
});

return type;
}

function createTemplateSpanLiteral(
node: StringTemplateHeadNode | StringTemplateMiddleNode | StringTemplateTailNode
): StringTemplateSpanLiteral {
return createType({
kind: "StringTemplateSpan",
node: node,
isInterpolated: false,
type: getLiteralType(node),
});
}

function createTemplateSpanValue(
node: Expression,
mapper: TypeMapper | undefined
): StringTemplateSpanValue {
return createType({
kind: "StringTemplateSpan",
node: node,
isInterpolated: true,
type: getTypeForNode(node, mapper),
});
}

function checkStringLiteral(str: StringLiteralNode): StringLiteral {
return getLiteralType(str);
}
Expand Down Expand Up @@ -4058,7 +4110,13 @@ export function createChecker(program: Program): Checker {
return finishTypeForProgramAndChecker(program, typePrototype, typeDef);
}

function getLiteralType(node: StringLiteralNode): StringLiteral;
function getLiteralType(
node:
| StringLiteralNode
| StringTemplateHeadNode
| StringTemplateMiddleNode
| StringTemplateTailNode
): StringLiteral;
function getLiteralType(node: NumericLiteralNode): NumericLiteral;
function getLiteralType(node: BooleanLiteralNode): BooleanLiteral;
function getLiteralType(node: LiteralNode): LiteralType;
Expand Down Expand Up @@ -4870,16 +4928,23 @@ export function createChecker(program: Program): Checker {
} as const);
}

function createLiteralType(value: string, node?: StringLiteralNode): StringLiteral;
function createLiteralType(
value: string,
node?:
| StringLiteralNode
| StringTemplateHeadNode
| StringTemplateMiddleNode
| StringTemplateTailNode
): StringLiteral;
function createLiteralType(value: number, node?: NumericLiteralNode): NumericLiteral;
function createLiteralType(value: boolean, node?: BooleanLiteralNode): BooleanLiteral;
function createLiteralType(
value: string | number | boolean,
node?: StringLiteralNode | NumericLiteralNode | BooleanLiteralNode
node?: LiteralNode
): StringLiteral | NumericLiteral | BooleanLiteral;
function createLiteralType(
value: string | number | boolean,
node?: StringLiteralNode | NumericLiteralNode | BooleanLiteralNode
node?: LiteralNode
): StringLiteral | NumericLiteral | BooleanLiteral {
if (program.literalTypes.has(value)) {
return program.literalTypes.get(value)!;
Expand Down
1 change: 1 addition & 0 deletions packages/compiler/src/core/helpers/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,5 +3,6 @@ export { getLocationContext } from "./location-context.js";
export * from "./operation-utils.js";
export * from "./path-interpolation.js";
export * from "./projected-names-utils.js";
export { stringTemplateToString } from "./string-template-utils.js";
export * from "./type-name-utils.js";
export * from "./usage-resolver.js";
39 changes: 39 additions & 0 deletions packages/compiler/src/core/helpers/string-template-utils.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
import { createDiagnostic } from "../messages.js";
import { Diagnostic, StringTemplate } from "../types.js";
import { getTypeName } from "./type-name-utils.js";

/**
* Convert a string template to a string value.
* Only literal interpolated can be converted to string.
* Otherwise diagnostics will be reported.
*
* @param stringTemplate String template to convert.
*/
export function stringTemplateToString(
stringTemplate: StringTemplate
): [string, readonly Diagnostic[]] {
const diagnostics: Diagnostic[] = [];
const result = stringTemplate.spans
.map((x) => {
if (x.isInterpolated) {
switch (x.type.kind) {
case "String":
case "Number":
case "Boolean":
return String(x.type.value);
default:
diagnostics.push(
createDiagnostic({
code: "non-literal-string-template",
target: x.node,
})
);
return getTypeName(x.type);
}
} else {
return x.type.value;
}
})
.join("");
return [result, diagnostics];
}
7 changes: 7 additions & 0 deletions packages/compiler/src/core/messages.ts
Original file line number Diff line number Diff line change
Expand Up @@ -586,6 +586,13 @@ const diagnostics = {
"Projections are experimental - your code will need to change as this feature evolves.",
},
},
"non-literal-string-template": {
severity: "error",
messages: {
default:
"Value interpolated in this string template cannot be converted to a string. Only literal types can be automatically interpolated.",
},
},

/**
* Binder
Expand Down
132 changes: 132 additions & 0 deletions packages/compiler/src/core/parser.ts
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,11 @@ import {
SourceFile,
Statement,
StringLiteralNode,
StringTemplateExpressionNode,
StringTemplateHeadNode,
StringTemplateMiddleNode,
StringTemplateSpanNode,
StringTemplateTailNode,
Sym,
SyntaxKind,
TemplateParameterDeclarationNode,
Expand Down Expand Up @@ -1340,6 +1345,8 @@ function createParser(code: string | SourceFile, options: ParseOptions = {}): Pa
return parseReferenceExpression();
case Token.StringLiteral:
return parseStringLiteral();
case Token.StringTemplateHead:
return parseStringTemplateExpression();
case Token.TrueKeyword:
case Token.FalseKeyword:
return parseBooleanLiteral();
Expand Down Expand Up @@ -1446,6 +1453,119 @@ function createParser(code: string | SourceFile, options: ParseOptions = {}): Pa
};
}

function parseStringTemplateExpression(): StringTemplateExpressionNode {
const pos = tokenPos();
const head = parseStringTemplateHead();
const spans = parseStringTemplateSpans(head.tokenFlags);
const last = spans[spans.length - 1];

if (head.tokenFlags & TokenFlags.TripleQuoted) {
const [indentationsStart, indentationEnd] = scanner.findTripleQuotedStringIndent(
last.literal.pos,
last.literal.end
);
mutate(head).value = scanner.unindentAndUnescapeTripleQuotedString(
head.pos,
head.end,
indentationsStart,
indentationEnd,
Token.StringTemplateHead,
head.tokenFlags
);
for (const span of spans) {
mutate(span.literal).value = scanner.unindentAndUnescapeTripleQuotedString(
span.literal.pos,
span.literal.end,
indentationsStart,
indentationEnd,
span === last ? Token.StringTemplateTail : Token.StringTemplateMiddle,
head.tokenFlags
);
}
}
return {
kind: SyntaxKind.StringTemplateExpression,
head,
spans,
...finishNode(pos),
};
}

function parseStringTemplateHead(): StringTemplateHeadNode {
const pos = tokenPos();
const flags = tokenFlags();
const text = flags & TokenFlags.TripleQuoted ? "" : tokenValue();

parseExpected(Token.StringTemplateHead);

return {
kind: SyntaxKind.StringTemplateHead,
value: text,
tokenFlags: flags,
...finishNode(pos),
};
}

function parseStringTemplateSpans(tokenFlags: TokenFlags): readonly StringTemplateSpanNode[] {
const list: StringTemplateSpanNode[] = [];
let node: StringTemplateSpanNode;
do {
node = parseTemplateTypeSpan(tokenFlags);
list.push(node);
} while (node.literal.kind === SyntaxKind.StringTemplateMiddle);
return list;
}

function parseTemplateTypeSpan(tokenFlags: TokenFlags): StringTemplateSpanNode {
const pos = tokenPos();
const expression = parseExpression();
const literal = parseLiteralOfTemplateSpan(tokenFlags);
return {
kind: SyntaxKind.StringTemplateSpan,
literal,
expression,
...finishNode(pos),
};
}
function parseLiteralOfTemplateSpan(
headTokenFlags: TokenFlags
): StringTemplateMiddleNode | StringTemplateTailNode {
const pos = tokenPos();
const flags = tokenFlags();
const text = flags & TokenFlags.TripleQuoted ? "" : tokenValue();

if (token() === Token.CloseBrace) {
nextStringTemplateToken(headTokenFlags);
return parseTemplateMiddleOrTemplateTail();
} else {
parseExpected(Token.StringTemplateTail);
return {
kind: SyntaxKind.StringTemplateTail,
value: text,
tokenFlags: flags,
...finishNode(pos),
};
}
}

function parseTemplateMiddleOrTemplateTail(): StringTemplateMiddleNode | StringTemplateTailNode {
const pos = tokenPos();
const flags = tokenFlags();
const text = flags & TokenFlags.TripleQuoted ? "" : tokenValue();
const kind =
token() === Token.StringTemplateMiddle
? SyntaxKind.StringTemplateMiddle
: SyntaxKind.StringTemplateTail;

nextToken();
return {
kind,
value: text,
tokenFlags: flags,
...finishNode(pos),
};
}

function parseNumericLiteral(): NumericLiteralNode {
const pos = tokenPos();
const valueAsString = tokenValue();
Expand Down Expand Up @@ -2575,6 +2695,10 @@ function createParser(code: string | SourceFile, options: ParseOptions = {}): Pa
scanner.scanDoc();
}

function nextStringTemplateToken(tokenFlags: TokenFlags) {
scanner.reScanStringTemplate(tokenFlags);
}

function createMissingIdentifier(): IdentifierNode {
const pos = tokenPos();
previousTokenEnd = pos;
Expand Down Expand Up @@ -3162,7 +3286,15 @@ export function visitChildren<T>(node: Node, cb: NodeCallback<T>): T | undefined
case SyntaxKind.DocUnknownTag:
return visitNode(cb, node.tagName) || visitEach(cb, node.content);

case SyntaxKind.StringTemplateExpression:
return visitNode(cb, node.head) || visitEach(cb, node.spans);
case SyntaxKind.StringTemplateSpan:
return visitNode(cb, node.expression) || visitNode(cb, node.literal);

// no children for the rest of these.
case SyntaxKind.StringTemplateHead:
case SyntaxKind.StringTemplateMiddle:
case SyntaxKind.StringTemplateTail:
case SyntaxKind.StringLiteral:
case SyntaxKind.NumericLiteral:
case SyntaxKind.BooleanLiteral:
Expand Down
Loading
Loading