Dynamic systems

The dynamic system module

The dynamic_systems module is essential for GrainLearning to execute computational models and encapsulate simulation and observation (reference) data in a single DynamicSystem class. Currently, the dynamic_systems module contains

a DynamicSystem class that executes simulations handles the simulation and observation data in a Python environment,
an IODynamicSystem class that sends instructions to third-party software (from the command line) and retrieves simulation data from the output files of the software.

Note that the dynamic system classes defined in GrainLearning are also known as state-space models that consist of the predicted states \(\vec{x}_t\) (DynamicSystem.sim_data) and experimental observables \(\vec{y}_t\) (DynamicSystem.obs_data).

\[\begin{split}\begin{align} \vec{x}_t & =\mathbb{F}(\vec{x}_{t-1})+\vec{\nu}_t \label{eq:dynaModel},\\ \vec{y}_t & =\mathbb{H}(\vec{x}_t)+\vec{\omega}_t \label{eq:obsModel} \end{align}\end{split}\]

where \(\mathbb{F}\) represents the third-party software model that takes the previous model state \(\vec{x}_{t-1}\) to make predictions for time \(t\). If all observables \(\vec{y}_t\) are independent and have a one-to-one relationship with \(\vec{y}_t\), the observation model \(\mathbb{H}\) reduces to the identity matrix \(\mathbf{I}_d\), with \(d\) being the number of independent observables.

The simulation and observation errors \(\vec{\nu}_t\) and \(\vec{\omega}_t\) are random variables and assumed to be normally distributed with zero means. We consider both errors together in the covariance matrix with SMC.cov_matrices. In fact, \(\vec{x}_t\) and \(\vec{y}_t\) are also random variables whose distributions are updated by the inference module.

Interact with third-party software

Interaction with external “software” models can be done via the callback of the DynamicSystem class. You can define your own DynamicSystem.callback and pass samples (combinations of parameters) to the model implemented in Python or to the software from the command line. The following gives an example of the callback where the “software” model \(\mathbb{F}\) is a Python function.

A callback function implemented in Python

def run_sim(system: Type["DynamicSystem"], **kwargs):
    data = []
    for params in system.param_data:
        # a linear function y = a*x + b
        y_sim = params[0] * system.ctrl_data + params[1]
        data.append(np.array(y_sim, ndmin=2))
    # assign model output to the dynamic system class
    model.set_sim_data(data)

The IODynamicSystem class

The IODynamicSystem inherits from DynamicSystem and is intended to work with third-party software packages. The IODynamicSystem.run function overrides the DynamicSystem.run function of the DynamicSystem. to write samples in a text file and run the IODynamicSystem.callback to execute the third-party software model at all sample points. Below is an example of the callback where parameter samples are passed as command-line arguments to an external executable.

A callback function that interacts with external software

executable = './software'

def run_sim(system, **kwargs):
    from math import floor, log
    import os
    # keep the naming convention consistent between iterations
    mag = floor(log(system.num_samples, 10)) + 1
    curr_iter = kwargs['curr_iter']
    # loop over and pass parameter samples to the executable
    for i, params in enumerate(system.param_data):
        description = 'Iter'+str(curr_iter)+'-Sample'+str(i).zfill(mag)
        print(" ".join([executable, '%.8e %.8e'%tuple(params), description]))
        os.system(' '.join([executable, '%.8e %.8e'%tuple(params), description]))

Data format and directory structure

GrainLearning can read .npy (for backward compatibility) and plain text formats. When using IODynamicSystem, the directory IODynamicSystem.sim_data_dir must exist and contains the observation data file IODynamicSystem.obs_data_file. Subdirectories with name iter<curr_iter> will be created in IODynamicSystem.sim_data_dir. In these subdirectories, you find

simulation data file: <sim_name>_Iter<curr_iter>-Sample<sample_ID>_sim.txt
parameter data file: <sim_name>_Iter<curr_iter>-Sample<sample_ID>_param.txt,

where <sim_name> is IODynamicSystem.sim_name, <curr_iter> is BayesianCalibration.curr_iter, and <sample_ID> is the index of the IODynamicSystem.param_data sequence.

For example, the observation data stored in a text file IODynamicSystem.obs_data_file should look like this.

Similarly, in a simulation data file linear_Iter0-Sample00_sim.txt, you find

# f
741666667
023635417
3056041669999998
587572917
869541667
151510417

Note the simulation data doesn’t contain the DynamicSystem.ctrl_data sequence.

Therefore, when using IODynamicSystem the user needs to provide the keys to the data sequence of the control and observation group. These keys are also used to extract the corresponding data from the simulation data files.

# name of the control variable
"ctrl_name": 'u',
# name of the output variables of the model
"obs_names": ['f'],