ScalaDays

Implementing a Macro

in Scala 3

Nicolas Stucki

LAMP / EPFL

Overview

Practical walkthrough of a macro implementation

Running example: string interpolator for JSON DSL
Implement a string interpolator as a macro
Implement a string interpolator extractor as a macro
Use transparent macro and type refinements
Precise error reporting on string interpolators

GitHub project (link + QR in last slide)

JSON specification (ECMA-404)

JSON value

object: name/value pairs
array: sequence of values
string
number
literals: true, false , null

JSON representation in Scala

For our purposes we want to have types for each JSON value
Directly use String
Custom JsonObject and JsonArray


                type Json = JsonObject | JsonArray | String

JsonArray

Trivial wrapper over a sequence


              class JsonArray(values: Json*):
                def apply(idx: Int): Json = values(idx)
                def length: Int = values.length

JsonObject

Wrapper over a name/value map
Uses scala.Selectable


              import scala.language.dynamics

              class JsonObject(nameValues: Map[String, Json]) extends scala.Selectable:
                def selectDynamic(name: String): Json | Undefined =
                  nameValues.getOrElse(name, Undefined)

              type Undefined = Undefined.type
              object Undefined

`scala.Selectable`

If a member does not exist on a class with scala.Selectable,

selectDynamic


              val jsonObject: JsonObject = ...

              val name: Json | Undefined = jsonObject.name // jsonObject.selectDynamic("name")
              val date: Json | Undefined = jsonObject.children // jsonObject.selectDynamic("children")

Similar for applications with applyDynamic
Can be enhanced with refinement types

String interpolator macro


              val firstTalk: Json =
                json""" { "name": "Resource anagement Made Easy", "speaker": "Julien Truffaut" } """
              val secondTalk: Json =
                json""" { "name": Implementing a Macro", "speaker": "Nicolas Stucki" } """
              val talks: Json =
                json" [ $firstTalk, $secondTalk ] "

scala.StringContext


                json" [ $firstTalk, $secondTalk ] "

String interpolation is desugared into StringContext

json


                StringContext(" [ ", ", ", "] ").json(firstTalk, secondTalk)

Usually implemented as an extension method

Multi-stage programming


              def helloExpr(nameExpr: Expr[String])(using Quotes): Expr[String] =
                val hello: Expr[String] = '{ "Hello" }
                val helloName: Expr[String] = '{ ${hello} + " " + ${nameExpr}  }
                helloName // '{ "Hello" + " " + name  }

Expr[T]: Value representing code of type T

Multi-stage programming


              def helloExpr(nameExpr: Expr[String])(using Quotes): Expr[String] =
                val hello: Expr[String] = '{ "Hello" }
                val helloName: Expr[String] = '{ ${hello} + " " + ${nameExpr}  }
                helloName // '{ "Hello" + " " + name  }

Expr[T]: Value representing code of type T
'{ expr }: Delays the computation of expr

Multi-stage programming


              def helloExpr(nameExpr: Expr[String])(using Quotes): Expr[String] =
                val hello: Expr[String] = '{ "Hello" }
                val helloName: Expr[String] = '{ ${hello} + " " + ${nameExpr}  }
                helloName // '{ "Hello" + " " + name  }

Expr[T]: Value representing code of type T
'{ expr }: Delays the computation of expr
${ expr }: Evaluates expr now and insert the code

Multi-stage programming


              def helloExpr(nameExpr: Expr[String])(using Quotes): Expr[String] =
                val hello: Expr[String] = Expr("Hello") // '{ "Hello" }
                val helloName: Expr[String] = '{ ${hello} + " " + ${nameExpr}  }
                helloName // '{ "Hello" + " " + name  }

Expr[T]: Value representing code of type T
'{ expr }: Delays the computation of expr
${ expr }: Evaluates expr now and insert the code
Expr(value): Lifts the value into and Expr[T]

Multi-stage programming


              def helloExpr(nameExpr: Expr[String])(using Quotes): Expr[String] =
                ...

              inline def hello(name: String): String =
                ${ helloExpr('{name}) }

${ expr }: Evaluates expr now and insert the code
Macro is a ${ expr } in an inline method (not in '{...})

Macro definition


              extension (inline sc: StringContext)
                inline def json(inline args: Json*): Json =
                  ...

Macro defined as extension inline method
Use inline StringContext to avoid its instantiation
Use inline arguments to elide the creation of a sequence

Macro definition


              extension (inline sc: StringContext)
                inline def json(inline args: Json*): Json =
                  ${ jsonExpr('sc, 'args) }

              def jsonExpr(sc: Expr[StringContext], argsExpr: Expr[Seq[Json]])(using Quotes): Expr[Json] =
                ...

'sc is a shorthand for '{sc}

Macro implementation


              def jsonExpr(sc: Expr[StringContext], argsExpr: Expr[Seq[Json]])(using Quotes): Expr[Json] =
                ...

Macro implementation


              def jsonExpr(sc: Expr[StringContext], argsExpr: Expr[Seq[Json]])(using Quotes): Expr[Json] =
                val jsonPattern: Pattern = parsed(sc)
                ...

Parse and validate the strings of the StringContext


              StringContext(" [ ", ", ", "] ")
                .json(firstTalk, secondTalk)

Macro implementation


              def jsonExpr(sc: Expr[StringContext], argsExpr: Expr[Seq[Json]])(using Quotes): Expr[Json] =
                val jsonPattern: Pattern = parsed(sc)
                val Varargs(argExprs) = argsExpr
                ...

Parse and validate the strings of the StringContext
Extract individual arguments from varargs


              StringContext(" [ ", ", ", "] ")
                .json(firstTalk, secondTalk)

Macro implementation


              def jsonExpr(sc: Expr[StringContext], argsExpr: Expr[Seq[Json]])(using Quotes): Expr[Json] =
                val jsonPattern: Pattern = parsed(sc)
                val Varargs(argExprs) = argsExpr
                toJsonExpr(jsonPattern, argExprs)

Parse and validate the strings of the StringContext
Extract individual arguments from varargs
Compile the pattern and arguments into an Expr[Json]


                StringContext(" [ ", ", ", "] ")
                  .json(firstTalk, secondTalk)


                  '{ JsonArray(firstTalk, secondTalk) }

Macro implementation


              def jsonExpr(sc: Expr[StringContext], argsExpr: Expr[Seq[Json]])(using Quotes): Expr[Json] =
                val jsonPattern: Pattern = parsed(sc)
                val Varargs(argExprs) = argsExpr
                toJsonExpr(jsonPattern, argExprs)

Parsing the `StringContext`


              def jsonExpr(sc: Expr[StringContext], argsExpr: Expr[Seq[Json]])(using Quotes): Expr[Json] =
                val jsonPattern: Pattern = parsed(sc)
                val Varargs(argExprs) = argsExpr
                toJsonExpr(jsonPattern, argExprs)


              def parsed(scExpr: Expr[StringContext])(using Quotes): Pattern =
                ...

Parsing the `StringContext`


              def parsed(scExpr: Expr[StringContext])(using Quotes): Pattern =
                val sc: StringContext = scExpr.valueOrAbort

Use valueOrAbort to get the value using FromExpr[StringContext]

Use value to get an Option[StringContext]
Use Expr(v) in pattern position


              scExpr = '{ StringContext(" [ ", ", ", "] ") }

↓


              sc = StringContext(" [ ", ", ", "] ")

Parsing the `StringContext`


              def parsed(scExpr: Expr[StringContext])(using Quotes): Pattern =
                val sc: StringContext = scExpr.valueOrAbort
                val parts: Seq[String] = sc.parts.map(scala.StringContext.processEscapes)
                ...

Get the string parts of the interpolator


              sc = StringContext(" [ ", ", ", "] ")

↓


              parts = Seq(" [ ", ", ", "] ")

Parsing the `StringContext`


              def parsed(scExpr: Expr[StringContext])(using Quotes): Pattern =
                val sc: StringContext = scExpr.valueOrAbort
                val parts: Seq[String] = sc.parts.map(scala.StringContext.processEscapes)
                Parser(parts).parse() match
                  case Parsed(pattern) => pattern
                  case ParseError(msg, location) => // report error

Parse the string parts into a Pattern AST


                enum Pattern:
                  case ...

JSON interpolator AST


              enum Pattern:
                case Str(value: String)
                case Arr(patterns: Pattern*)
                case Obj(namePatterns: (String, Pattern)*)
                case InterpolatedValue

JSON interpolator AST


              enum Pattern:
                case Str(value: String)
                case Arr(patterns: Pattern*)
                case Obj(namePatterns: (String, Pattern)*)
                case InterpolatedValue

The following interpolation


                json""" { "name": $talkName, "track": "2" } """


                Obj("name" -> InterpolatedValue, "track" -> Str("2"))

Compiling the pattern


              def jsonExpr(sc: Expr[StringContext], argsExpr: Expr[Seq[Json]])(using Quotes): Expr[Json] =
                val jsonPattern: Pattern = parsed(sc)
                val Varargs(argExprs) = argsExpr
                toJsonExpr(jsonPattern, argExprs)


              def toJsonExpr(ast: Pattern, args: Seq[Expr[Json]])(using Quotes): Expr[Json] =
                ...

Compiling the pattern


              def toJsonExpr(ast: Pattern, args: Seq[Expr[Json]])(using Quotes): Expr[Json] =
                def rec(ast: Pattern): Expr[Json] =
                  ast match
                    ...
                rec(ast)

Recursively transform the Pattern AST into an Expr[Json]

Compiling the pattern


              def toJsonExpr(ast: Pattern, args: Seq[Expr[Json]])(using Quotes): Expr[Json] =
                def rec(ast: Pattern): Expr[Json] =
                  ast match
                    case Pattern.Str(value) => Expr(value)
                    ...
                rec(ast)

Use ToExpr[String] to lift the value

Compiling the pattern


              def toJsonExpr(ast: Pattern, args: Seq[Expr[Json]])(using Quotes): Expr[Json] =
                def rec(ast: Pattern): Expr[Json] =
                  ast match
                    ...
                    case Pattern.Arr(values*) =>
                      val valueExprs: Seq[Expr[Json]] = values.map(rec)
                      val valuesExpr: Expr[Seq[Json]] = Varargs(valueExprs)
                      '{ JsonArray($valuesExpr*) }
                    ...
                rec(ast)

Transform each value recursively
Create an expression the varargs sequence
Instantiate JsonArray with varargs

Compiling the pattern


              def toJsonExpr(ast: Pattern, args: Seq[Expr[Json]])(using Quotes): Expr[Json] =
                def rec(ast: Pattern): Expr[Json] =
                  ast match
                    ...
                    case Pattern.Obj(nameValues*) =>
                      val nameExprValueExprs: Seq[(Expr[String], Expr[Json])]  =
                        for (name, value) <- nameValues yield (Expr(name), rec(value))
                      val nameValueExprs: Seq[Expr[(String, Json)]] = nameExprValueExprs.map(Expr.ofTuple)
                      val nameValuesExpr: Expr[Seq[(String, Json)]] = Varargs(nameValueExprs)
                      '{ JsonObject($nameValuesExpr*) }
                    ...
                rec(ast)

Transform each key and value recursively into expressions
Use Expr.ofTuple to create a tuple from expressions

Compiling the pattern


              def toJsonExpr(ast: Pattern, args: Seq[Expr[Json]])(using Quotes): Expr[Json] =
                val argsIterator = args.iterator
                def rec(ast: Pattern): Expr[Json] =
                  ast match
                    ...
                    case Pattern.InterpolatedValue =>
                      argsIterator.next()
                rec(ast)

Get the i-th interpolated value

Compiling the pattern


              def toJsonExpr(ast: Pattern, args: Seq[Expr[Json]])(using Quotes): Expr[Json] =
                val argsIterator = args.iterator
                def rec(ast: Pattern): Expr[Json] =
                  ast match
                    case Pattern.Str(value) => Expr(value)
                    case Pattern.Arr(values*) =>
                      val valueExprs: Seq[Expr[Json]] = values.map(rec)
                      val valuesExpr: Expr[Seq[Json]] = Varargs(valueExprs)
                      '{ JsonArray($valuesExpr*) }
                    case Pattern.Obj(nameValues*) =>
                      val nameExprValueExprs: Seq[(Expr[String], Expr[Json])]  =
                        for (name, value) <- nameValues yield (Expr(name), rec(value))
                      val nameValueExprs: Seq[Expr[(String, Json)]] = nameExprValueExprs.map(Expr.ofTuple)
                      val nameValuesExpr: Expr[Seq[(String, Json)]] = Varargs(nameValueExprs)
                      '{ JsonObject($nameValuesExpr*) }
                    case Pattern.InterpolatedValue =>
                      argsIterator.next()
                rec(ast)

Now we know how to

define string interpolator macros


              val firstTalk: Json =
                json""" { "name": "Resource Management Made Easy", "speaker": "Julien Truffaut" } """
              val secondTalk: Json =
                json""" { "name": Implementing a Macro", "speaker": "Nicolas Stucki" } """
              val talks: Json =
                json" [ $firstTalk, $secondTalk ] "

String interpolator extractor macro


              talk match
                case json"" " { "name": $name, "speaker": $speaker } """ =>
                  println(s"$name by $speaker")

scala.StringContext


                    json" [ $firstTalk, $secondTalk ] "


                  case json" [ $firstTalk, $secondTalk ] " =>

scala.StringContext


                    json" [ $firstTalk, $secondTalk ] "

↓


                  case json" [ $firstTalk, $secondTalk ] " =>

↓


              StringContext(" [ ", ", ", "] ").json(firstTalk, secondTalk)

scala.StringContext


              StringContext(" [ ", ", ", "] ").json(firstTalk, secondTalk)

↓


                  StringContext(" [ ", ", ", "] ")

                    .json.apply(firstTalk, secondTalk)

↓


                  StringContext(" [ ", ", ", "] ")

                    .json.unapply(firstTalk, secondTalk)

String interpolator macro V2


              type JsonStringContext

              extension (sc: scala.StringContext)
                @compileTimeOnly("...")
                def json: JsonStringContext = ???

Wrapper over StringContext

String interpolator macro V2


              type JsonStringContext

              extension (sc: scala.StringContext)
                @compileTimeOnly("...")
                def json: JsonStringContext = ???

              extension (inline jsonSC: JsonStringContext)
                inline def apply(inline args: Json*): Json =
                  ${ jsonExpr('jsonSC, 'args) }

                inline def unapplySeq(scrutinee: Json): Option[Seq[Json]] =
                  ${ jsonUnapplySeqExpr('jsonSC, 'scrutinee) }

Multi-stage programming

pattern matching


              def nameOf(nameExpr: Expr[String])(using Quotes): Expr[String] =
                nameExpr match
                  case '{ "Hello" } => Expr("No name")
                  ...

case '{ expr } matches the expression

Multi-stage programming

pattern matching


              def nameOf(nameExpr: Expr[String])(using Quotes): Expr[String] =
                nameExpr match
                  case '{ "Hello" } => Expr("No name")
                  case '{ "Hello " + ($nameExpr: String) } => nameExpr
                  ...

case '{ expr } matches the expression
$name extracts a sub-expression
$name:T extracts a sub-expression of type T

Macro implementation


              def jsonExpr(scExpr: Expr[StringContext],
                           argsExpr: Expr[Seq[Json]])(using Quotes): Expr[Json] =
                val jsonPattern: Pattern = parsed(scExpr)
                val Varargs(argExprs) = argsExpr
                toJsonExpr(jsonPattern, argExprs)

Macro implementation V2


              def jsonExpr(jsonSCExpr: Expr[JsonStringContext],
                           argsExpr: Expr[Seq[Json]])(using Quotes): Expr[Json] =
                val '{ ($scExpr: StringContext).json } = jsonSCExpr
                val jsonPattern: Pattern = parsed(scExpr)
                val Varargs(argExprs) = argsExpr
                toJsonExpr(jsonPattern, argExprs)

Quote pattern to extract the StringContex

Macro implementation V2


              def jsonExpr(jsonSCExpr: Expr[JsonStringContext],
                           argsExpr: Expr[Seq[Json]])(using Quotes): Expr[Json] =
                val '{ ($scExpr: StringContext).json } = jsonSCExpr
                val jsonPattern: Pattern = parsed(scExpr)
                val Varargs(argExprs) = argsExpr
                toJsonExpr(jsonPattern, argExprs)

              def jsonUnapplySeqExpr(jsonSCExpr: Expr[JsonStringContext],
                                     scrutinee: Expr[Json])(using Quotes): Expr[Option[Seq[Json]]] =
                ...

Macro implementation V2


              def jsonExpr(jsonSCExpr: Expr[JsonStringContext],
                           argsExpr: Expr[Seq[Json]])(using Quotes): Expr[Json] =
                val '{ ($scExpr: StringContext).json } = jsonSCExpr
                val jsonPattern: Pattern = parsed(scExpr)
                val Varargs(argExprs) = argsExpr
                toJsonExpr(jsonPattern, argExprs)

              def jsonUnapplySeqExpr(jsonSCExpr: Expr[JsonStringContext],
                                     scrutinee: Expr[Json])(using Quotes): Expr[Option[Seq[Json]]] =
                val '{ ($scExpr: StringContext).json } = jsonSCExpr
                val jsonPattern: Pattern = parsed(scExpr)
                ...

Macro implementation V2


              def jsonExpr(jsonSCExpr: Expr[JsonStringContext],
                           argsExpr: Expr[Seq[Json]])(using Quotes): Expr[Json] =
                val '{ ($scExpr: StringContext).json } = jsonSCExpr
                val jsonPattern: Pattern = parsed(scExpr)
                val Varargs(argExprs) = argsExpr
                toJsonExpr(jsonPattern, argExprs)

              def jsonUnapplySeqExpr(jsonSCExpr: Expr[JsonStringContext],
                                     scrutinee: Expr[Json])(using Quotes): Expr[Option[Seq[Json]]] =
                val '{ ($scExpr: StringContext).json } = jsonSCExpr
                val jsonPattern: Pattern = parsed(scExpr)
                val jsonPatternExpr: Expr[Pattern] = Expr(jsonPattern)
                ...

Macro implementation V2


              def jsonExpr(jsonSCExpr: Expr[JsonStringContext],
                           argsExpr: Expr[Seq[Json]])(using Quotes): Expr[Json] =
                val '{ ($scExpr: StringContext).json } = jsonSCExpr
                val jsonPattern: Pattern = parsed(scExpr)
                val Varargs(argExprs) = argsExpr
                toJsonExpr(jsonPattern, argExprs)

              def jsonUnapplySeqExpr(jsonSCExpr: Expr[JsonStringContext],
                                     scrutinee: Expr[Json])(using Quotes): Expr[Option[Seq[Json]]] =
                val '{ ($scExpr: StringContext).json } = jsonSCExpr
                val jsonPattern: Pattern = parsed(scExpr)
                val jsonPatternExpr: Expr[Pattern] = Expr(jsonPattern)
                '{ $jsonPatternExpr.unapplySeq($scrutinee) }


              enum Pattern:
                ...
                def unapplySeq(json: Json): Option[Seq[Json]] = ...

Macro implementation V2


              def jsonExpr(jsonSCExpr: Expr[JsonStringContext],
                           argsExpr: Expr[Seq[Json]])(using Quotes): Expr[Json] =
                val '{ ($scExpr: StringContext).json } = jsonSCExpr
                val jsonPattern: Pattern = parsed(scExpr)
                val Varargs(argExprs) = argsExpr
                toJsonExpr(jsonPattern, argExprs)

              def jsonUnapplySeqExpr(jsonSCExpr: Expr[JsonStringContext],
                                     scrutinee: Expr[Json])(using Quotes): Expr[Option[Seq[Json]]] =
                val '{ ($scExpr: StringContext).json } = jsonSCExpr
                val jsonPattern: Pattern = parsed(scExpr)
                val jsonPatternExpr: Expr[Pattern] = Expr(jsonPattern)
                '{ $jsonPatternExpr.unapplySeq($scrutinee) }

Lifting a `Pattern`


              given ToExpr[Pattern] with
                def apply(pattern: Pattern)(using Quotes): Expr[Pattern] =
                  ...

Lifting a `Pattern`


              given ToExpr[Pattern] with
                def apply(pattern: Pattern)(using Quotes): Expr[Pattern] =
                  pattern match
                    case Pattern.InterpolatedValue => '{ Pattern.InterpolatedValue }
                    ...

Trivial cases with a quoted expressions

Lifting a `Pattern`


              given ToExpr[Pattern] with
                def apply(pattern: Pattern)(using Quotes): Expr[Pattern] =
                  pattern match
                    case Pattern.InterpolatedValue => '{ Pattern.InterpolatedValue }
                    case Pattern.Str(value) => '{ Pattern.Str(${Expr(value)}) }
                    ...

Use ToExpr[String] for Expr(value)

Lifting a `Pattern`


              given ToExpr[Pattern] with
                def apply(pattern: Pattern)(using Quotes): Expr[Pattern] =
                  pattern match
                    case Pattern.InterpolatedValue => '{ Pattern.InterpolatedValue }
                    case Pattern.Str(value) => '{ Pattern.Str(${Expr(value)}) }
                    case Pattern.Arr(patterns*) => '{ Pattern.Arr(${Expr(patterns)}*) }
                    case Pattern.Obj(namePatterns*) => '{ Pattern.Obj(${Expr(namePatterns)}*) }

Use ToExpr[Seq[T]], and ToExpr[(T, U)] from Standard library
Combined with this ToExpr[Pattern], and ToExpr[String]

Now we know how to

define string interpolator extractor macros


              talk match
                case json"" " { "name": $name, "speaker": $speaker } """ =>
                  println(s"$name by $speaker")

Refining the types


              val secondTalk =
                json""" { "name": "Implementing a Macro", "speaker": "Nicolas Stucki" } """

              val name: String = secondTalk.name

`scala.Selectable` and refinement types


              class JsonObject(...) extends scala.Selectable:
                ...


              val jsonObject: JsonObject { val name: String } = ...

              val name: String = jsonObject.name

Refined interpolated types


              val secondTalk: JsonObject { val name: String; val speaker: String } =
                json""" { "name": Implementing a Macro", "speaker": "Nicolas Stucki" } """

Refined interpolated types


              val firstTalk:  JsonObject { val name: String; val speaker: String } = ...
              val secondTalk: JsonObject { val name: String; val speaker: String } =
                json""" { "name": Implementing a Macro", "speaker": "Nicolas Stucki" } """

              val talks: JsonArray {
                def apply(idx: Int): JsonObject { val name: String; val speaker: String }
              } =
                json" [ $firstTalk, $secondTalk ] "

Transparent macro definition


              extension (inline jsonSC: JsonStringContext)
                transparent inline def apply(inline args: Json*): Json =
                  ${ jsonExpr('jsonSC, 'args) }

Make the macro transparent and refine the type in the implementation

Transparent macro definition


              extension (inline jsonSC: JsonStringContext)
                transparent inline def apply(inline args: Json*): Json =
                  ${ jsonExpr('jsonSC, 'args) }

                transparent inline def unapply(inline scrutinee: Json): Option[Tuple] =
                  ${ jsonUnapplyExpr('jsonSC, 'scrutinee) }

Use transparent unapply instead of unapplySeq
Refine Tuple into tuple of known size and refined element types

Macro implementation V2


              def jsonExpr(jsonSCExpr: Expr[JsonStringContext],
                           argsExpr: Expr[Seq[Json]])(using Quotes): Expr[Json] =
                val '{ ($scExpr: StringContext).json } = jsonSCExpr
                val jsonPattern: Pattern = parsed(scExpr)
                val Varargs(argExprs) = argsExpr
                toJsonExpr(jsonPattern, argExprs)

Can refines result into String, JsonArray, or JsonObject

Macro implementation V3


              def jsonExpr(jsonSCExpr: Expr[JsonStringContext],
                           argsExpr: Expr[Seq[Json]])(using Quotes): Expr[Json] =
                val '{ ($scExpr: StringContext).json } = jsonSCExpr
                val jsonPattern: Pattern = parsed(scExpr)
                val Varargs(argExprs) = argsExpr
                val refinedJsonType: Type[?] = refinedType(jsonPattern, argExprs)
                toJsonExpr(jsonPattern, argExprs)

Refinements of JsonObject { val name: ... }
Refinements of JsonArray { def apply(idx: Int): ... }

Multi-stage programming

runtime types


              def showType[T](using Type[T])(using Quotes): String =
                Type.of[T]
                Type.of[String]
                Type.of[List[T]]
                ...

Type[T]: Runtime representation of T
Type.of[T]: Construct Type[T]

Statically known type
Given implicitly

Multi-stage programming

type pattern matching


              def showType[T](using Type[T])(using Quotes): String =
                Type.of[T] match
                  case '[String] => "String"
                  case '[List[t]] => "List[" + showType[t] + "]"
                  case '[t] => "?"

case '[T] => : Matches the type type T
t is type variable (lower-case type name)
Type[t] is implicitly given be the pattern

Macro implementation V3


              def jsonExpr(jsonSCExpr: Expr[JsonStringContext],
                           argsExpr: Expr[Seq[Json]])(using Quotes): Expr[Json] =
                val '{ ($scExpr: StringContext).json } = jsonSCExpr
                val jsonPattern: Pattern = parsed(scExpr)
                val Varargs(argExprs) = argsExpr
                val refinedJsonType: Type[?] = refinedType(jsonPattern, argExprs)
                val expr: Expr[Json] = toJsonExpr(jsonPattern, argExprs)
                refinedJsonType match
                  case '[t] => '{ $expr.asInstanceOf[t] }.asExprOf[Json]

Give the name t to the refined type
Cast the expression with this refinement

Computing the refined type


              def refinedType(pattern: Pattern, args: Seq[Expr[Json]])(using Quotes): Type[?] =
                val jsonSchema: Schema = schema(pattern, args)
                schemaToType(jsonSchema)

Compute a Schema
Convert the schema into a type

Computing the refined type


              def refinedType(pattern: Pattern, args: Seq[Expr[Json]])(using Quotes): Type[?] =
                val jsonSchema: Schema = schema(pattern, args)
                schemaToType(jsonSchema)


              enum Schema:
                case Value // Json
                case Str   // String
                case Arr(elemSchema: Schema)             // JsonArray { def apply(idx: Int): ... }
                case Obj(nameSchemas: (String, Schema)*) // JsonObject { ... }

Computing the `Schema`


              def schema(pattern: Pattern, args: Seq[Expr[Json]])(using Quotes): Schema =
                def rec(pattern: Pattern): Schema =
                  ...
                rec(pattern)

Computing the `Schema`


              def schema(pattern: Pattern, args: Seq[Expr[Json]])(using Quotes): Schema =
                def rec(pattern: Pattern): Schema =
                  pattern match
                    case Pattern.Str(_) => Schema.Str
                    ...
                rec(pattern)

Computing the `Schema`


              def schema(pattern: Pattern, args: Seq[Expr[Json]])(using Quotes): Schema =
                def rec(pattern: Pattern): Schema =
                  pattern match
                    ...
                    case Pattern.Arr() => Schema.Arr(Schema.Value)
                    case Pattern.Arr(patterns*) =>
                      val elementSchema: Schema = patterns.map(rec).reduce(union)
                      Schema.Arr(elementSchema)
                    ...
                rec(pattern)

Computing the `Schema`


              def schema(pattern: Pattern, args: Seq[Expr[Json]])(using Quotes): Schema =
                def rec(pattern: Pattern): Schema =
                  pattern match
                    ...
                    case Pattern.Obj(nameValues*) =>
                      val nameSchemas: Seq[(String, Schema)] =
                        for (name, value) <- nameValues yield (name, rec(value))
                      Schema.Obj(nameSchemas*)
                    ...
                rec(pattern)

Computing the `Schema`


              def schema(pattern: Pattern, args: Seq[Expr[Json]])(using Quotes): Schema =
                def rec(pattern: Pattern): Schema =
                  pattern match
                    ...
                    case Pattern.InterpolatedValue => Schema.Value
                rec(pattern)

Computing the `Schema`


              def schema(pattern: Pattern, args: Seq[Expr[Json]])(using Quotes): Schema =
                val argsIterator = args.iterator
                def rec(pattern: Pattern): Schema =
                  pattern match
                    ...
                    case Pattern.InterpolatedValue =>
                      argsIterator.next() match
                        case '{ $argExpr : t } => schemaOf[t]
                rec(pattern)

Use the type variable t to extract the precise type of the expression


              private def schemaOf[T](using Type[T])(using Quotes): Schema =
                ...

Extracting information from `Type[T]`


              private def schemaOf[T](using Type[T])(using Quotes): Schema =
                ...

Extracting information from `Type[T]`


              private def schemaOf[T](using Type[T])(using Quotes): Schema =
                Type.of[T] match
                  case '[String] => Schema.Str
                  ...

Extracting information from `Type[T]`


                private def schemaOf[T](using Type[T])(using Quotes): Schema =
                  Type.of[T] match
                    ...
                    case '[JsonArray] =>
                      import quotes.reflect.*
                      val refinedElementSchema: Schema =
                        TypeRepr.of[T].widen match
                          case Refinement(parent, "apply", MethodType(_, _, resType)) =>
                            resType.asType match
                              case '[t] => schemaOf[t]
                          case _ => Schema.Value
                      Schema.Arr(refinedElementSchema)
                    ...

Use reflection API
TypeRepr.of[T]: Structured representation of the type T
asType: Convert TypeRepr into Type[?]

Extracting information from `Type[T]`


                private def schemaOf[T](using Type[T])(using Quotes): Schema =
                  Type.of[T] match
                    ...
                    case '[JsonObject] =>
                      import quotes.reflect.*
                      def refinements(tpe: TypeRepr): Vector[(String, Schema)] =
                        tpe match
                          case Refinement(parent, name, info) =>
                            val refinedSchema = info.asType match
                              case '[t] => schemaOf[t]
                            refinements(parent) :+ (name, refinedSchema)
                          case _ => Vector()
                      Schema.Obj(refinements(TypeRepr.of[T].widen)*)
                    ...

Extracting information from `Type[T]`


                private def schemaOf[T](using Type[T])(using Quotes): Schema =
                  Type.of[T] match
                    ...
                    case _ => Schema.Value

From schema to `Type[T]`


                def refinedType(pattern: Pattern, args: Seq[Expr[Json]])(using Quotes): Type[?] =
                  val jsonSchema: Schema = schema(pattern, args)
                  schemaToType(jsonSchema)


                def schemaToType(schema: Schema)(using Quotes): Type[?] =
                  ...

From schema to `Type[T]`


                def schemaToType(schema: Schema)(using Quotes): Type[?] =
                  schema match
                    case Schema.Value => Type.of[Json]
                    case Schema.Str   => Type.of[String]
                    ...

From schema to `Type[T]`


                def schemaToType(schema: Schema)(using Quotes): Type[?] =
                  schema match
                    ...
                    case Schema.Arr(elemSchema) =>
                      refinedType(elemSchema) match
                        case '[t] => Type.of[ JsonArray { def apply(idx: Int): t } ]
                    ...

From schema to `Type[T]`


                def schemaToType(schema: Schema)(using Quotes): Type[?] =
                  schema match
                    ...
                    case Schema.Obj(nameSchemas*) =>
                      import quotes.reflect.*
                      val refined = nameSchemas.foldLeft(TypeRepr.of[JsonObject]) {
                        case (acc, (name, schema)) =>
                          refinedType(schema) match
                            case '[t] => Refinement(acc, name, TypeRepr.of[t])
                      }
                      refined.asType

From schema to `Type[T]`


                def schemaToType(schema: Schema)(using Quotes): Type[?] =
                  schema match
                    case Schema.Value => Type.of[Json]
                    case Schema.Str   => Type.of[String]
                    case Schema.Arr(elemSchema) =>
                      refinedType(elemSchema) match
                        case '[t] => Type.of[ JsonArray { def apply(idx: Int): t } ]
                    case Schema.Obj(nameSchemas*) =>
                      import quotes.reflect.*
                      val refined = nameSchemas.foldLeft(TypeRepr.of[JsonObject]) {
                        case (acc, (name, schema)) =>
                          refinedType(schema) match
                            case '[t] => Refinement(acc, name, TypeRepr.of[t])
                      }
                      refined.asType

Now we know how to

use type refinements in macros


                val secondTalk =
                  json""" { "name": "Implementing a Macro", "speaker": "Nicolas Stucki" } """

                val name: String = secondTalk.name

Precise error reporting


                json""" { "name": $name, "speaker" = $speaker } """
                                                   ^
                                                   expected `:` but found `=`

Location in string interpolator


                json""" { "name": $name, "speaker" = $speaker } """
                       ^^^^^^^^^^^     ^^^^^^^^^^^^^^        ^^^
                       part 0          part 1                part 2

Location in string interpolator


                json""" { "name": $name, "speaker" = $speaker } """
                       ^^^^^^^^^^^     ^^^^^^^^^^^^^^        ^^^
                       part 0          part 1      |         part 2
                                                   └ Location(partIndex = 1, offset = 12)


                class Location(val partIndex: Int, val offset: Int)

Location in string interpolator


              json""" { "name": $name, "speaker" = $speaker } """
                                                 └ Location(partIndex = 1, offset = 12)


              def parsed(scExpr: Expr[StringContext])(using Quotes): Pattern =
                val sc: StringContext = scExpr.valueOrAbort
                val parts = sc.parts.map(scala.StringContext.processEscapes)
                Parser(parts).parse() match
                  case Parsed(pattern) => pattern
                  case ParseError(msg, location) =>
                    ...

Location in string interpolator


              json""" { "name": $name, "speaker" = $speaker } """
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                                 └ Location(partIndex = 1, offset = 12)


              def parsed(scExpr: Expr[StringContext])(using Quotes): Pattern =
                val sc: StringContext = scExpr.valueOrAbort
                val parts = sc.parts.map(scala.StringContext.processEscapes)
                Parser(parts).parse() match
                  case Parsed(pattern) => pattern
                  case ParseError(msg, location) =>
                    quotes.reflect.report.errorAndAbort(msg)

Location in string interpolator


              json""" { "name": $name, "speaker" = $speaker } """
                                     ^^^^^^^^^^^^^^
                                                 └ Location(partIndex = 1, offset = 12)


              def parsed(scExpr: Expr[StringContext])(using Quotes): Pattern =
                val sc: StringContext = scExpr.valueOrAbort
                val parts = sc.parts.map(scala.StringContext.processEscapes)
                Parser(parts).parse() match
                  case Parsed(pattern) => pattern
                  case ParseError(msg, location) =>
                    val '{ scala.StringContext(${Varargs(partExprs)}: _*) } = scExpr
                    val partWithError: Expr[String] = partExprs(location.partIndex)
                    quotes.reflect.report.errorAndAbort(msg, partWithError)

Location in string interpolator


              json""" { "name": $name, "speaker" = $speaker } """
                                                 ^
                                                 └ Location(partIndex = 1, offset = 12)


              def parsed(scExpr: Expr[StringContext])(using Quotes): Pattern =
                val sc: StringContext = scExpr.valueOrAbort
                val parts = sc.parts.map(scala.StringContext.processEscapes)
                Parser(parts).parse() match
                  case Parsed(pattern) => pattern
                  case ParseError(msg, location) =>
                    val '{ scala.StringContext(${Varargs(partExprs)}: _*) } = scExpr
                    val partWithError: Expr[String] = partExprs(location.partIndex)
                    import quotes.reflect.*
                    val sourceFile = scExpr.asTerm.pos.sourceFile
                    val baseOffset = partWithError.asTerm.pos.start
                    val offset = baseOffset + location.offset
                    report.errorAndAbort(msg, Position(sourceFile, offset, offset + 1))

Now we know how to

report errors in string interpolators


              json""" { "name": $name, "speaker" = $speaker } """
                                                 ^
                                                 expected `:` but found `=`

Scala

Days

2023

Thank you

https://github.com/nicolasstucki/scala-days-2023

Implementing a Macro

in Scala 3

Overview

JSON specification (ECMA-404)

JSON representation in Scala

JsonArray

JsonObject

scala.Selectable

String interpolator macro

scala.StringContext

Multi-stage programming

Multi-stage programming

Multi-stage programming

Multi-stage programming

Multi-stage programming

Macro definition

Macro definition

Macro implementation

Macro implementation

Macro implementation

Macro implementation

Macro implementation

Parsing the StringContext

Parsing the StringContext

Parsing the StringContext

Parsing the StringContext

JSON interpolator AST

JSON interpolator AST

Compiling the pattern

Compiling the pattern

Compiling the pattern

Compiling the pattern

Compiling the pattern

Compiling the pattern

Compiling the pattern

Now we know how to

define string interpolator macros

String interpolator extractor macro

scala.StringContext

scala.StringContext

scala.StringContext

String interpolator macro V2

String interpolator macro V2

Multi-stage programming

pattern matching

Multi-stage programming

pattern matching

Macro implementation

Macro implementation V2

Macro implementation V2

Macro implementation V2

Macro implementation V2

Macro implementation V2

Macro implementation V2

Lifting a Pattern

Lifting a Pattern

Lifting a Pattern

Lifting a Pattern

Now we know how to

define string interpolator extractor macros

Refining the types

scala.Selectable and refinement types

Refined interpolated types

Refined interpolated types

Transparent macro definition

Transparent macro definition

Macro implementation V2

Macro implementation V3

Multi-stage programming

runtime types

Multi-stage programming

type pattern matching

Macro implementation V3

Computing the refined type

Computing the refined type

Computing the Schema

Computing the Schema

Computing the Schema

Computing the Schema

Computing the Schema

`scala.Selectable`

Parsing the `StringContext`

Parsing the `StringContext`

Parsing the `StringContext`

Parsing the `StringContext`

Lifting a `Pattern`

Lifting a `Pattern`

Lifting a `Pattern`

Lifting a `Pattern`

`scala.Selectable` and refinement types

Computing the `Schema`

Computing the `Schema`

Computing the `Schema`

Computing the `Schema`

Computing the `Schema`

Computing the `Schema`

Extracting information from `Type[T]`

Extracting information from `Type[T]`

Extracting information from `Type[T]`

Extracting information from `Type[T]`

Extracting information from `Type[T]`

From schema to `Type[T]`

From schema to `Type[T]`

From schema to `Type[T]`

From schema to `Type[T]`

From schema to `Type[T]`