Formula interpreter (DSL)

published: ⏰ 16

Quite often projects require the ability to customize some formulas or scripts available to the user. The main idea is to change the behavior of parts of the application without having to go through the entire development cycle with delivery of a new release. There are different options for how to do this, this will be discussed below.

Introduction

Let me make a small digression to understand what we are talking about and why ANTLR4. A long time ago, in 2005-2006, I was interested in developing various scripting languages ​​and DSLs. I couldn’t find the " red/green dragon book" (Principles of Compiler Design) then, but there was a small selection from other sources and examples. Unfortunately, tools for generating like ANTLR passed me by and all the parsers, lexical and semantic analyzers were written manually, re-transferring them from project to project. On the one hand, it was quite a tedious job, but it seemed that the final speed of work was worth it (universal solutions could be slower).

Often the main need was for a lightweight scripting engine or DSL for the project requirements (data description formats, simple logic, etc.). With the first one, I somehow couldn’t find anything suitable, but I had a pretty interesting experience developing a compiler with an interpreter. I made simple stack VMs (Delphi / C++) with bytecode and calling native functions (like JNA) for embedding in native applications.

And in the case of DSL, I always wrote my own solutions, since there couldn’t be any ready-made ones. Here I always wanted to try ready-made tools, a couple of decades have passed and their time has come :).

Task

I will say right away that the example is very simple and this is a side activity within the development of tekBlueprint.plans. Formulas that can be specified by the user will be used to calculate the costs of completing tasks. Basic requirements for formulas:

  1. High speed of operation is required. Considering that the calculation is actually performed for matrix cells, it is important to minimize the calculation time. The number of cells can reach tens of thousands and they must be recalculated without delays.
  2. Formula calculation and the ability to expand them are necessary. Formulas can contain various operators, functions and user variables.
  3. Ensure code isolation. In the future, the code must be executed on an application server (SaaS), which means certain restrictions for formulas on access to the execution environment.

There are several possible implementation options here:

  1. Using scripting languages – JavaScript (Nashorn,Rhino Engine), Groovy and others. Here, requirements 1 and 3 are not met.
  2. Using template engines – Freemarker, MVEL, JStachio and others. Here, requirements 1 and 2 are not met (for example, not all template engines allow performing arithmetic calculations).
  3. Do it yourself, since it is already clear that there are “fatal flaws” in the previous options :) This will allow you to fully control the formula syntax and ensure that all requirements are met.

Next, we will implement the third option, that is, create our own parser of formulas in the Java language. For this, we will use the grammar analyzer ANTLR4.

Connecting ANTLR4 to the project

ANTLR4 generates the necessary classes based on pre-defined grammars. The project uses Maven, so this can be done using a plugin (there is a similar one for Gradle).

Add Maven plugin
XML

<build>
    <plugins>
        <plugin>
            <groupId>org.antlr</groupId>
            <artifactId>antlr4-maven-plugin</artifactId>
            <version>4.13.2</version>
            <executions>
                <execution>
                    <id>antlr</id>
                    <goals>
                        <goal>antlr4</goal>
                    </goals>
                </execution>
            </executions>
            <configuration>
                <visitor>true</visitor>
            </configuration>
        </plugin>
    </plugins>
</build>

The file with grammars should be in the directory src/main/antlr4. In our case, the file will be in the directory src/main/antlr4/com/company/antlr4/EffortFormulaExpr.g4, the final package for classes will be com.company.antlr4.

Grammar and parser generation

As I said earlier, the formula is quite simple – arithmetic operations, user variables and functions. Therefore, the grammar (EffortFormulaExpr.g4) will be as follows:

Antlrv4
grammar EffortFormulaExpr;

expression  : expression (MUL | DIV) expression
            | expression (PLUS | MINUS) expression
            | LPAREN expression RPAREN
            | func
            | (MINUS | PLUS)* atom
            ;

func        : VARIABLE LPAREN (expression (',' expression)*)? RPAREN
            ;

atom        : NUMBER
            | VARIABLE
            ;

NUMBER      : '0'..'9'+ ('.' '0'..'9'+)?
            ;
MUL         : '*'
            ;
DIV         : '/'
            ;
PLUS        : '+'
            ;
MINUS       : '-'
            ;
VARIABLE    : [a-zA-Z_][a-zA-Z0-9_]*
            ;
LPAREN      : '('
            ;
RPAREN      : ')'
            ;
WS          : [ \r\n\t]+ -> skip
            ;

The grammar is based on arithmetic.g4.

Here, the highest priority is given to multiplication and division operations, then addition and subtraction. Functions (func) can take any number of arguments separated by commas. Variables (VARIABLE) begin with a letter or underscore and can contain letters, numbers, and underscores.

After describing the grammar, you can run the Maven command to generate classes: mvn antlr4:antlr4. As a result, we will get the following classes, which will be needed further:

  • EffortFormulaExprLexer.java – lexical for parsing the formula into tokens.
  • EffortFormulaExprParser.java – a parser for building a parse tree of a formula (AST).
  • EffortFormulaExprBaseVisitor.java – a base visitor class that can be used to traverse the parse tree. To create this class, you need to enable <visitor>true</visitor> in the plugin.

Abstract Syntax Tree

Now you can create a visitor class that will build an AST based on a formula. To simplify it, I combined it with the expression compiler class and added the compile method, which takes a string with a formula and returns the root node of the AST.

Java
/**
 * Compiler class responsible for parsing and compiling effort formula expressions.
 *
 * @author Sergey Grachev
 */
@AllArgsConstructor
public class Compiler extends EffortFormulaExprBaseVisitor<Node> {

    /**
     * Compiles the given expression string into an abstract syntax tree (AST) node.
     *
     * @param expression The effort formula expression to compile.
     * @return The root node of the compiled AST.
     */
    public static Node compile(final String expression) {
        final var tokens = new CommonTokenStream(new EffortFormulaExprLexer(CharStreams.fromString(expression)));
        final var parser = new EffortFormulaExprParser(tokens);
        return new Compiler().visit(parser.expression());
    }

    /**
     * Visits an expression context and constructs the corresponding AST node.
     *
     * @param ctx The expression context to visit.
     * @return The constructed AST node for the given expression.
     */
    @Override
    public Node visitExpression(final EffortFormulaExprParser.ExpressionContext ctx) {
        if (ctx.atom() != null) {
            if (!ctx.PLUS().isEmpty()) {
                return new UnaryPlusNode(visit(ctx.atom()));
            }
            if (!ctx.MINUS().isEmpty()) {
                return new UnaryMinusNode(visit(ctx.atom()));
            }
        }
        if (!ctx.PLUS().isEmpty()) {
            return new BinaryNode(
                    visit(ctx.expression(0)), BinaryNode.PLUS, visit(ctx.expression(1)));
        }
        if (!ctx.MINUS().isEmpty()) {
            return new BinaryNode(
                    visit(ctx.expression(0)), BinaryNode.MINUS, visit(ctx.expression(1)));
        }
        if (ctx.MUL() != null) {
            return new BinaryNode(
                    visit(ctx.expression(0)), BinaryNode.MULTIPLY, visit(ctx.expression(1)));
        }
        if (ctx.DIV() != null) {
            return new BinaryNode(
                    visit(ctx.expression(0)), BinaryNode.DIVIDE, visit(ctx.expression(1)));
        }
        if (ctx.LPAREN() != null) {
            return visit(ctx.expression(0));
        }
        return super.visitExpression(ctx);
    }

    /**
     * Visits an atom context and constructs the corresponding AST node.
     *
     * @param ctx The atom context to visit.
     * @return The constructed AST node for the given atom.
     */
    @Override
    public Node visitAtom(final EffortFormulaExprParser.AtomContext ctx) {
        if (ctx.NUMBER() != null) {
            return new NumberNode(Double.parseDouble(ctx.NUMBER().getText()));
        }
        if (ctx.VARIABLE() != null) {
            return new VariableNode(ctx.VARIABLE().getText());
        }
        return super.visitAtom(ctx);
    }

    /**
     * Visits a function context and constructs the corresponding AST node.
     *
     * @param ctx The function context to visit.
     * @return The constructed AST node for the given function.
     */
    @Override
    public Node visitFunc(final EffortFormulaExprParser.FuncContext ctx) {
        final var fnName = ctx.VARIABLE().getText();
        final var contexts = ctx.expression();
        final var args = new Node[contexts.size()];
        int index = 0;
        for (final var arg : contexts) {
            args[index++] = visit(arg);
        }
        return new FunctionNode(fnName, args);
    }
}

Each of the methods creates a new node in the binary tree, where the node is the operation, and the leaves are the arguments of the operation. That is, in order to get the result of the calculations, you just need to recursively perform all the operations starting from the root. For simplicity, all calculations use the double type.

Below are the AST nodes

Java
/**
 * Represents a node in an abstract syntax tree (AST) for formulas.
 *
 * <p>This interface extends the {@link Formula} interface, indicating that each node
 * in the AST can be evaluated as part of a formula.</p>
 *
 * @author Sergey Grachev
 */
public interface Node extends Formula {

}
Java
/**
 * Represents a node in the abstract syntax tree (AST) that holds a numeric value.
 *
 * @author Sergey Grachev
 */
@Value
public class NumberNode implements Node {

    /**
     * The numeric value stored in this node.
     */
    private final double value;

    /**
     * Evaluates the node, which simply returns the stored numeric value.
     *
     * @param context the formula evaluation context (not used for NumberNode)
     * @return the numeric value of this node
     */
    @Override
    public double evaluate(final FormulaContext context) {
        return value;
    }
}
Java
/**
 * Represents a variable node in an abstract syntax tree (AST) for formula evaluation.
 *
 * <p>This class is used to represent variables within a formula, which can be resolved
 * to their corresponding values using the provided {@link FormulaContext}.
 *
 * @author Sergey Grachev
 */
@Value
public final class VariableNode implements Node {

    /**
     * The name of the variable represented by this node.
     */
    private final String name;

    /**
     * Evaluates the variable node using the given formula context.
     *
     * <p>This method retrieves the value of the variable from the provided {@link FormulaContext}.
     *
     * @param context the formula context containing variable values
     * @return the evaluated value of the variable
     */
    @Override
    public double evaluate(final FormulaContext context) {
        return context.var(name);
    }
}
Java
/**
 * Represents a unary minus operation in an abstract syntax tree (AST).
 *
 * <p>This node contains a single child node, which represents the operand to be negated.
 * When evaluated, this node returns the negative of its child's value.</p>
 *
 * @author Sergey Grachev
 */
@Value
public class UnaryMinusNode implements Node {
    /**
     * The child node representing the operand to be negated.
     */
    private final Node child;

    /**
     * Evaluates the unary minus operation using the provided formula context.
     *
     * <p>This method evaluates the child node and returns its value negated.</p>
     *
     * @param context the formula context used for evaluation
     * @return the negative of the evaluated child node's value
     */
    @Override
    public double evaluate(final FormulaContext context) {
        return -child.evaluate(context);
    }
}
Java
/**
 * Represents a unary plus node in an abstract syntax tree (AST) for formula evaluation.
 * The unary plus operation does not change the value of its operand, but it can be used
 * to ensure that the operand is treated as a positive number.
 *
 * @author Sergey Grachev
 */
@Value
public class UnaryPlusNode implements Node {
    /**
     * The child node representing the operand to which the unary plus operation is applied.
     */
    private final Node child;

    /**
     * Evaluates the unary plus node by evaluating its child node and returning the result.
     * Since the unary plus does not change the value, this method simply returns the evaluated
     * value of the child node.
     *
     * @param context The formula context containing variables and their values.
     * @return The evaluated value of the child node.
     */
    @Override
    public double evaluate(final FormulaContext context) {
        return +child.evaluate(context);
    }
}
Java
/**
 * Represents a function node in the abstract syntax tree (AST) of a formula.
 *
 * <p>This class encapsulates a function call, including its name and arguments,
 * and provides functionality to evaluate the function within a given context.</p>
 *
 * @author Sergey Grachev
 */
@RequiredArgsConstructor
public class FunctionNode implements Node {
    /**
     * An empty array of doubles used when no values are needed.
     */
    private static final double[] EMPTY = new double[0];

    /**
     * The name of the function to be called.
     */
    private final String name;

    /**
     * The arguments passed to the function, represented as an array of {@link Node} objects.
     */
    private final Node[] arguments;

    /**
     * An array to store evaluated values of the arguments. This is initialized if there are any arguments.
     */
    private final @Nullable double[] values;

    /**
     * The {@link FormulaFunction} object representing the function to be called.
     * It is lazily initialized when first needed.
     */
    private @Nullable FormulaFunction fn = null;

    /**
     * Constructs a new FunctionNode with the specified name and arguments.
     *
     * <p>The values array is initialized if there are any arguments.</p>
     *
     * @param name      The name of the function to be called.
     * @param args      The arguments passed to the function, represented as an array of {@link Node} objects.
     */
    public FunctionNode(final String name, final Node... args) {
        this.name = name;
        this.arguments = args;
        this.values = arguments != null && arguments.length > 0 ? new double[args.length] : null;
    }

    /**
     * Evaluates the function within the given context.
     *
     * <p>If the function has not been resolved yet, it is looked up in the provided {@link FormulaContext}.
     * If the function is not found, an {@link IllegalStateException} is thrown.</p>
     *
     * <p>The arguments are evaluated and their values stored in the values array.
     * The function is then called with these values.</p>
     *
     * @param context   The {@link FormulaContext} within which to evaluate the function.
     * @return The result of the function call as a double.
     */
    @Override
    public double evaluate(final FormulaContext context) {
        if (fn == null) {
            fn = context.findFunction(name);
            if (fn == null) {
                throw new IllegalStateException("Unknown function: " + name);
            }
        }

        if (values == null) {
            return fn.call(EMPTY);
        }

        int index = 0;
        for (final var argument : arguments) {
            values[index++] = argument.evaluate(context);
        }

        return fn.call(values);
    }
}
Java
/**
 * Represents a binary operation node in an abstract syntax tree (AST) for formula evaluation.
 *
 * <p>This class encapsulates two child nodes and an operator that defines the binary operation
 * to be performed between the values of these nodes.</p>
 *
 * @author Sergey Grachev
 */
@Value
public class BinaryNode implements Node {

    /**
     * Constant representing the addition (+) operator.
     */
    public static final byte PLUS = 0;

    /**
     * Constant representing the subtraction (-) operator.
     */
    public static final byte MINUS = 1;

    /**
     * Constant representing the multiplication (*) operator.
     */
    public static final byte MULTIPLY = 2;

    /**
     * Constant representing the division (/) operator.
     */
    public static final byte DIVIDE = 3;

    /**
     * The left operand node in the binary operation.
     */
    private final Node left;

    /**
     * The right operand node in the binary operation.
     */
    private final Node right;

    /**
     * The operator defining the binary operation to be performed between the left and right nodes.
     */
    private final byte operator;

    /**
     * Constructs a new BinaryNode with the specified left and right child nodes and operator.
     *
     * @param left     the left operand node
     * @param operator the operator defining the binary operation
     * @param right    the right operand node
     */
    public BinaryNode(final Node left, final byte operator, final Node right) {
        this.left = left;
        this.right = right;
        this.operator = operator;
    }

    /**
     * Evaluates the binary operation represented by this node using the provided formula context.
     *
     * @param context the formula context containing variable values and other evaluation data
     * @return the result of the binary operation as a double value
     * @throws RuntimeException if an unknown operator is encountered
     */
    @Override
    public double evaluate(final FormulaContext context) {
        final var lVal = left.evaluate(context);
        final var rVal = right.evaluate(context);

        return switch (operator) {
            case PLUS -> lVal + rVal;
            case MINUS -> lVal - rVal;
            case MULTIPLY -> lVal * rVal;
            case DIVIDE -> lVal / rVal;
            default -> throw new RuntimeException("Unknown operator: " + operator);
        };
    }
}

Here:

  • Node is an abstract class representing a node in the expression tree.
  • NumberNode is a numeric constant.
  • VariableNode is a variable whose value will be obtained from the context when calculating the formula.
  • UnaryMinusNode, UnaryPlusNode are changing the sign of the expression. UnaryPlusNode is actually useless, you can get rid of it for optimization.
  • BinaryNode is a binary operation (for example, addition or multiplication) performed on other nodes of the tree.
  • FunctionNode is a function call with arguments. The function is described by the Function interface, which accepts an array of values ​​and returns the result.

Additionally, you need to create extension classes.

Java
/**
 * Represents a context for evaluating formulas, containing variables and functions.
 *
 * @author Sergey Grachev
 */
@Getter
public final class FormulaContext {

    /**
     * The default value to return when a variable is not found in the context.
     */
    private static final Double DEFAULT_VALUE = (double) 0;

    /**
     * The current value of the formula context.
     */
    private double value;

    /**
     * A map containing variables and their corresponding values.
     */
    private final Map<String, Double> variables = new HashMap<>();

    /**
     * A map containing functions and their corresponding implementations.
     */
    private final Map<String, FormulaFunction> functions = new HashMap<>();

    /**
     * Constructs a new FormulaContext with the default value.
     */
    public FormulaContext() {
        this.value = DEFAULT_VALUE;
    }

    /**
     * Constructs a new FormulaContext with the specified initial value.
     *
     * @param value The initial value for the formula context.
     */
    public FormulaContext(final double value) {
        this.value = value;
    }

    /**
     * Adds a variable to the context with the given name and value.
     *
     * @param name  The name of the variable.
     * @param value The value of the variable.
     * @return This instance of FormulaContext for method chaining.
     */
    public FormulaContext addVariable(final String name, final double value) {
        variables.put(name, value);
        return this;
    }

    /**
     * Adds a function to the context with the given name and implementation.
     *
     * @param name The name of the function.
     * @param func The implementation of the function.
     * @return This instance of FormulaContext for method chaining.
     */
    public FormulaContext addFunction(final String name, final FormulaFunction func) {
        functions.put(name, func);
        return this;
    }

    /**
     * Calls a function in the context with the given name and arguments.
     *
     * @param name The name of the function to call.
     * @param args The arguments to pass to the function.
     * @return The result of the function call.
     * @throws IllegalStateException If the function is not found in the context.
     */
    public double call(final String name, final double... args) {
        final var fn = functions.get(name);
        if (fn == null) {
            throw new IllegalStateException("Unknown function: " + name);
        }
        return fn.call(args);
    }

    /**
     * Retrieves the value of a variable from the context.
     *
     * @param name The name of the variable to retrieve.
     * @return The value of the variable, or the default value if the variable is not found.
     */
    public double var(final String name) {
        final var v = variables.get(name);
        return v == null ? DEFAULT_VALUE : v;
    }

    /**
     * Finds a function in the context by its name.
     *
     * @param name The name of the function to find.
     * @return The function if found, or null if not found.
     */
    @Nullable
    public FormulaFunction findFunction(final String name) {
        return functions.get(name);
    }
}
Java
/**
 * Represents a functional interface for mathematical formulas that can be applied to an array of double values.
 *
 * <p>This interface is designed to be used with lambda expressions or method references, allowing
 * for the definition of custom formula functions that take an array of double arguments and return a double result.</p>
 *
 * @author Sergey Grachev
 */
@FunctionalInterface
public interface FormulaFunction {

    /**
     * Applies this function to the given array of double values.
     *
     * @param args an array of double values representing the input parameters for the formula
     * @return the result of applying this function to the input arguments as a double value
     */
    double call(double[] args);
}

Here:

  • FormulaFunction – interface for functions in formulas.
  • FormulaContext – context that stores variables and functions.

Formula calculation check

Everything is ready and can be tested, the test result should be 52.0

Groovy
class CompilerTest extends Specification {

    def 'compile expression'() {
        given:
        var context = new FormulaContext(5)
        context.addFunction('FN1', args -> 5 * args[0]);
        context.addFunction('FN2', args -> 3 * args[0]);
        context.addVariable('VA1', 4);
        context.addVariable('VA2', 3);

        when:
        var expr = Compiler.compile('-1 + 2 * +3 + 5 + (VA1 * VA2) + FN1(FN2(2))')

        then:
        expr != null;
        expr.evaluate(context) == -1 + 2 * +3 + 5 + (4 * 3) + (5 * (3 * 2))
    }
}

Comparison of execution speed

And finally, a small comparison of the calculation speed using different approaches. Below you can see that Groovy turned out to be the slowest, which is generally expected. MVEL/Freemarker templates are a little better, as they are more lightweight compared to a full-fledged script engine. Our AST interpreter is 10 times faster, but significantly behind native Java code.

Text
Benchmark                            Mode  Cnt       Score   Error  Units
FormulaBenchmark.benchGroovy        thrpt         2983.378          ops/s
FormulaBenchmark.benchMvelTemplate  thrpt        11770.380          ops/s
FormulaBenchmark.benchFreemarker    thrpt        12311.651          ops/s
FormulaBenchmark.benchPlan          thrpt       147752.201          ops/s
FormulaBenchmark.benchJava          thrpt       725310.418          ops/s

In the future, the execution speed can be brought closer to native Java using certain optimizations:

  1. Remove unnecessary nodes – UnaryPlusNode.
  2. Pre-calculate constant expressions (analogous to constexpr in C++).
  3. Transform AST into bytecode using ASM/Javassist, but this is not always possible for some JVMs (for example, GraalVM).

Conclusion

In this article, I showed a simple example of creating an expression interpreter in Java using ANTLR4: defining a grammar, generating a parser, assembling an AST and calculating a value based on it. The result is a simple but functional interpreter that will be expanded in the future to meet the needs of tekBlueprint.plans.

ℹ️ Note

Sample project available at GitHub.

Java Antlr Dsl Parser Formula Devmix