Reusing results from previous runs

Autory can calculate some projection nodes in a run while reusing the results of other projection nodes from a previous run.

This is useful when one part of a model changes, but its precedent projection nodes have already been calculated.

Changes from Autory 2 to Autory 3

In Autory 3, reused projection node results are read from the .feather output files in the results folder under OutputFolder.

This is different from Autory 2, where reused projection node results were read from the .xlsx projection node output workbooks.

The .xlsx output workbooks are still useful for reviewing results manually, but they are not the source used by Autory 3 when it reuses projection node results from disk.

When reuse happens

The main control is RunProjectionNodesFilter. This formula decides which projection nodes must be recalculated.

If a projection node is excluded by RunProjectionNodesFilter, Autory asks one more question:

Is this projection node still needed by a projection node that will be recalculated?

• If yes, Autory reuses the projection node's previous results from disk.
• If no, Autory ignores the projection node for this run.

In this context, "needed" means that another projection node depends on it through a formula such as:

• mapProjectionValues,
• mapExternalProjectionValues,
• cascadeProjectionValues,
• createProjectionNodes, or
• projectionResultsToTable.

Child nodes included only because RunChildNodes is TRUE are different. If such a child projection node is filtered out by RunProjectionNodesFilter and is not also needed by a formula, it is ignored rather than reused.

The following flow diagram summarizes the decision making process for each projection node:

Where Autory looks

Autory looks in the current run's OutputFolder. If OutputFolder is relative, it is relative to the model workbook's folder.

For each reused projection node, Autory expects the previous result files to be in a folder like this:

OutputFolder/
  results/
    hierarchy-path-and-level/
      valuation-type-and-date/
        properties.feather
        scalars.feather
        time_vectors.feather
        scalar_iterations.feather
        time_vector_iterations.feather

The exact folder names are generated from the projection node's hierarchy path, hierarchy level, valuation type, and valuation date.

Autory checks for properties.feather before it starts recalculating projection nodes. If this file is missing for a reused node, the run stops with an error. The scalars.feather and time_vectors.feather files are read when present.

Typical workflow

1. Run the precedent projection nodes first.
2. Use an OutputProjectionNodeFilter that writes the projection nodes you will want to reuse later.
3. Run the dependent projection nodes later, using the same OutputFolder.
4. Use RunProjectionNodesFilter in the later run to recalculate only the dependent projection nodes.

For example, a first run might calculate lower-level projection nodes:

RunProjectionNodesFilter = hLevel=1
OutputProjectionNodeFilter = TRUE

A later run might calculate only a summary node that maps values from those lower-level nodes:

RunProjectionNodesFilter = hLevel=2
OutputProjectionNodeFilter = TRUE

The second run recalculates the level-2 node. If that node maps values from the level-1 projection nodes, Autory reads the level-1 results from the .feather files created by the first run.

For RunMultiple batches, put producer runs in an earlier group than consumer runs. Runs in the same group may execute at the same time, so a consumer run should not rely on files that another run in the same group is still writing.

Common mistakes

Filtering out output that a later run needs

OutputProjectionNodeFilter controls which projection nodes write result files.

If a projection node is calculated but excluded by OutputProjectionNodeFilter, its results are available in memory to other projection nodes in the same run, but they will not be available on the disk for re-use by a later run.

Use RunProjectionNodesFilter to avoid recalculating nodes, and use OutputProjectionNodeFilter only when you are sure the omitted result files will not be needed later.

Reusing partial output

If a run was interrupted, treat its result files as partial unless the run summary says the run completed. See how to stop a run.

Reusing filtered time steps

OutputTimeStepFilter controls which time-vector rows are written to time_vectors.feather.

If a previous run wrote only selected time steps, a later run can still reuse those files, but missing time steps will not have the original variable values. For best results, write all time steps for projection nodes that will be reused by later runs.

Related commands

The autory convert results command also reads Autory 3 .feather result files. Use it when a downstream process needs CSV, SQLite, JSON, etc.