Skip to content

Test Expectations

Test expectations allow you to specify which component of the language under test should be tested, and what the expected output of the test will be.

We have already seen the parse succeeds expectation in action, and briefly mentioned that a test case without any test expectations is the same as a test case with a parse succeeds expectation. We will now list the syntax and semantics of all the currently supported test expectations:

Parse Expectations

Parse expectations can be used to test the syntax of your language. They indicate that the input fragment of the test case should be parsed and allow you to specify what you expect the result to be. As parsing is the preliminary step to all other language components (e.g., analysis and transformations) they are treated differently from other expectations. If no parse expectation is present on a test case, even if another expectation (e.g. an analysis expectation) is present, a parse succeeds expectation will be added to the test case.

Expectation.ParseSucceeds = <parse succeeds>
Parse the fragment and expect the parsing to succeed, with no parse errors and no ambiguities.
Expectation.ParseFails = <parse fails>
Parse the fragment and expect the parsing to fail, with at least one parse error.
Expectation.ParseAmbiguous = <parse ambiguous>
Parse the fragment and expect the parsing to succeed with one or more ambiguities.
Expectation.ParseToAterm = <parse to <ATerm>>

Parse the fragment, expect parsing to succeed, and compare it to the given ATerm AST. When using test fixtures, the ATerm should only be the AST of the fragment of the test, not of the entire test fixture. Please note that if you want to specify a List in the ATerm, the square brackets of the list may interfere with the markers of a fragment. Therefore, to specify a list as the expected output, prepend it with the keyword !ATerm. For example:

parse to !ATerm ["5"]
Expectation.ParseToFragment = <parse to <Language?> <OpenMarker> <Fragment> <CloseMarker>]]>

Parse the fragment, expect parsing to succeed, and compare it to the result of parsing the given Fragment with the given Language. When the Language is omitted the language under test will be used to parse the given fragment. When using test fixtures, only the test's input fragment will be combined with the test fixture. The fragment in this expectation (i.e., the output fragment) will not be combined with it, even if the language under test is used to parse it. To counteract this, the entire AST (including the nodes from the fixture) will be compared to the expectation's fragment's AST.

This expectation can be useful to test disambiguations, or to test your language against a reference implementation. An example test case for disambiguation in MiniJava would be: 

module disambiguation
language MiniJava
start symbol Exp

test plus is left associative [[
  1 + 2 + 3
]] parse to [[
  (1 + 2) + 3
]]

Analysis Expectations

Analysis expectations specify that the analyzer should be tested. We will first discuss the most generic analysis expectations: the message expectations. These expectations can be used to test the entire static semantic analysis process. Then we will look at test expectations that are more specific: the name analysis expectations.

Analysis Message Expectations

Analysis message expectations will cause the input fragment to be parsed and analyzed (e.g., name and type analysis and static error checking). Finally, the resulting messages (i.e. errors, warnings, or notes) will be compared to the expectation. Note that messages of the expected type are not allowed to appear in the test fixture, if one is present. This is to prevent test from succeeding, when a message that would make it fail appears in an unexpected location. Only the messages within the test's fragment will be compared to the expectation.

Expectation.MessageExpectation = <<Operator?> <INT> <Severity>>
These expectations will check if the number of messages of the given severity generated by the analysis matches the given number. Any other messages (of different severity or located in the test fixture) will be ignored by this expectation.

The optional operator can be used to losen the strictness of the check. For example the expectation > 2 errors allows any number of errors bigger than 2. If no operator is specified, it will default to =. The allowed operators are =, >, >=, <, <=.

The allowed message severities are: error or errors for error messages, warning or warnings for warning messages, and note or notes for note messages. Please note that error messages in this expectation refer only to analysis errors (not parse errors). Also when you are testing for warnings, for example, any analysis messages of different severity (including errors) will not cause the test to fail. For example, the following test will succeed, regardless of the blatant type error, because we only test for warnings:

module test-for-warnings
language MiniJava

fixture [[
  class Main {
    public static void main(String[] args) {
      System.out.println([[...]]);
    }
  }
]]

test no warnings [[1 + new A()]] 0 warnings

These analysis message expectations may be followed by the at keyword and a list of one or more comma separated references to selections of the test's fragment: #<INT>, #<INT>, #<INT>, .... As this is the first time we encounter references to selections, let's look at an example:

module error-locations
language MiniJava

test duplicate classes [[
  class Main {
    public static void main(String[] args) {
      System.out.println(42);
    }
  }
  class [[A]]{}
  class [[A]]{}
]] 2 errors at #1, #2

This test will cause SPT to check if the specified messages appeared at the location of the given selection references. The selections are the classnames A that are selected by wrapping them in an open and close marker. Selections are referenced by the order in which they appear, starting at 1, from left to right and top to bottom.

It is allowed to give less selection references than the number of expected messages. In this case SPT assumes you don't care about the location of the other messages. If the same selection is referenced more than once, multiple messages will be expected at that location. mFor example 3 errors at #1,#1 expects 3 errors, 2 of which should be at the location of selection number 1. The other error may be anywhere within the test fragment.

Expectation.MessageContent = <<Severity> like <STRING>>

This expectation specifies that there should be at least 1 message of the given severity that contains the given String. For example error like "duplicate class name" expects there to be at least 1 error in the fragment whose message contains duplicate class name.

This expectation can also be followed by the at keyword, and a single selection reference, to indicate where you expect the message with the given content.

Expectation.AnalysisSucceeds = <analysis succeeds>
This expectation is syntactic sugar for 0 errors.
Expectation.AnalysisFails = <analysis fails>
This expectation is syntactic sugar for > 0 errors.

Name Analysis Expectations

Name analysis expectations will check if use sites can be resolved and, if required, if they resolve to the correct definition. The fragment will be parsed and analyzed, but any number and severity of analysis messages are allowed.

Expectation.Resolve = <resolve #<INT>>
Try to resolve the AST node at the given selection. Expect it to successfully resolve to any definition site.
Expectation.ResolveTo = <resolve #<INT> to #<INT>>
Try to resolve the AST node at the first given selection. Expect it to successfully resolve to the location marked by the second given selection.

Note that selections can only occur in the test's fragment, not in the test fixture. So name analysis can only be tested within a test's fragment.

Transformation Expectations

A transformation transforms an AST to another AST. The idea within Spoofax is that a transformation has a name, and can be nested within a structure of menu's. Furthermore, it can have additional information about whether it transforms the raw AST (i.e. the parse result) or the analyzed AST (i.e. the result of desugaring and analysis). In languages created with Spoofax, transformations are Stratego strategies that are registered in the Menus.esv file.

Transformation expectations will first look up a given transformation using the name under which it was registered. Note that, for Spoofax languages, this is not necessarily the name of the Stratego strategy, but the name under which it is registered in the Menus.esv file. If this name is not unique, the menu structure can be used to look up the proper transformation.

Once the transformation is found, SPT will determine if it requires the raw AST, or the analyzed AST. If the raw AST is required, it will only parse the fragment. If the analyzed AST is required, it will also analyze the parse result. However, analysis is allowed to produce any number and severity of messages. Then, SPT will run the transformation on the entire AST, including the nodes from the test fixture, if there was one.

Expectation.Transform = <transform <STRING>>
The STRING should be delimited by double quotes and contain the name of the transformation. If the name is not unique, the menu structure can be included as well, seperated by ->. For example: transform "Menu name -> transformation name" to Some(Result()). As long as the transformation returns a result, this expectation passes.
Expectation.TransformToAterm = <transform <STRING> to <ATerm>>
Same as Transform, but the result of the transformation is compared to the given AST.
Expectation.TransformToFragment = <transform <STRING> to <Language?> <OpenMarker> <Fragment> <CloseMarker>>
Does the same as TransformToAterm, but compares the result of the transformation to the AST of the given fragment. If the applied transformation required the raw AST, the given fragment will only be parsed with the given language. If no language is given, the language under test will be used. If the applied transformation required an analyzed AST, the given fragment will be parsed and analyzed.

Run Stratego Expectations

These test expectations are really only applicable to languages that use Stratego strategies in their implementation. They will parse and analyze the fragment and run a given Stratego strategy (with no arguments) and compare its output to the expectation.

Expectation.Run = <run <STRATEGY>>
This expectation will lookup the given strategy name and run it on the AST node in the test's fragment. If the fragment contains multiple nodes (e.g., it's a list of Statements but some Statements were in the test fixture) the strategy will be run on each of these nodes. Either until it completes successfully, or until it failed on all these nodes. Note that it wil not be executed on the nodes in the test fixture, if there was one.
Expectation.RunWithArgs = <run <STRATEGY>(|<TermArgs>)>

This expectation will run a strategy that expects term arguments. String literals, integer literals and selection references are permitted as term arguments.:

test rename variable without type [[
  let
    var msg := "Hello World"
  in
    print([[msg]])
  end
]] run rename(|#1, "message", 0) to [[
  let
    var message := "Hello World"
  in
    print(message)
  end
]]
Expectation.RunFails = <run <STRATEGY> fails>
This expectation checks if the given strategy fails.
Expectation.RunOn = <run <STRATEGY> on #<INT>>
This expectation does the same as Run, except it runs the strategy on the nodes at the given selection instead of the nodes of the test's fragment.
Expectation.RunToAterm = <run <STRATEGY> to <ATerm>>
shorty
Expectation.RunToAtermOn = <run <STRATEGY> on #<INT> to <ATerm>>
These expectations are similar to the first two, but they require the result of running the strategy to match the given AST.
Expectation.RunToFragment = <run <STRATEGY> to <Language?> <OpenMarker> <Fragment> <CloseMarker>>
Expectation.RunToFragmentOn = <run <STRATEGY> on #<INT> to <Language?> <OpenMarker> <Fragment> <CloseMarker>>
These expectations are similar to the first two, but they require the result of running the strategy to match the result of analyzing the given fragment with the given language. If no language is given, the language under test is used.

Origin Location Expectations

Expectation.HasOrigins = <has origin locations>
This expectation parses and analyzes the fragment. It then checks if all AST nodes in the test's fragment (except for Lists in Spoofax) have a source region (an origin) associated with them. It does not check the AST nodes in the test fixture.

When using Spoofax, there are some strategies that will break the origin information when used. This can lead to desugarings that create AST nodes without origin information, which can cause problems when trying to create messages at their location and with other services. This expectation can be used to check that your analysis is origin preserving.

Last update: October 17, 2024
Created: October 17, 2024