The validation of simulation models

Citation

Hicks, C. and Earl, C.F. (2001), "The validation of simulation models", Assembly Automation, Vol. 21 No. 3. https://doi.org/10.1108/aa.2001.03321caa.002

The validation of simulation models

The validation of simulation models

Christian Hicks and Christopher F. Earl

Keywords: Assembly, Simulation, Modelling

Simulation models may be used to predict how a system will behave. Conventional engineering science has often tried to use formulae to do this. However, for complex systems it is often not possible to adequately describe the system using formulae alone. Computer simulations use rules to model system changes and apply them repeatedly to predict behaviour. In many cases, there is an element of uncertainty about the changes produced by a rule and repeated simulations are used to model patterns of behaviour. Many simulation models provide animated graphics that can help provide an insight into the system and its dynamic behaviour.

Assembly and manufacturing systems have been simulated in different ways and are often regarded as useful. But can we trust the results of a simulation? Casti, in his analysis of simulation and complexity (1997), highlights this "Can you trust it?" problem. However, this problem is not new to simulation practitioners grappling with complex manufacturing and assembly systems.

The first step in simulation is to identify the target system. This is then represented in terms of an abstract model that typically includes objects, relationships and behaviour patterns. An appropriate balance must be drawn between model complexity and accuracy. The model is then developed into a computer program using some language or software package. The simulation then needs to be validated to ensure that an inference from the model is a correct prediction for the actual process.

Simulation is an inductive, experimental technique in which solutions are sought through a finite number of experiments. The experimental frame defines a set of circumstances, observed variables, their levels, initial state, input schedules and termination conditions. Stochastic simulations are sampling experiments, which observe the principles of statistical design. Statistical methods may therefore be used for the design and analysis of experiments.

There are several ways to establish the validity of a simulation. First, does it use a generally accepted representation of reality, such as a widely accepted theory? Second, rigorously test the assumptions that underpin the model. Third, triangulation may be used to compare results with complementary studies, or theory. Sensitivity analysis may be used to establish the significance of various factors within the simulation. Finally, the simulation can be viewed as a "black box" and it can be validated in terms of its input-output transformations using statistical methods.

Naylor and Finger (1967) point out that all statistical tests make assumptions about the nature of a process. The validity of the statistical tests may therefore be questionable. Classical statistical tests establish the probabilities of rejecting a good model (type I error) or accepting a bad model (type II error). However, statistical tests may be inconclusive, particularly when there are only limited real world data. Fishman (1978) points out that statistical validation, while desirable, is not always possible.

Figure 1 shows a framework for validating simulation models proposed by Schlesinger (1980). Analysis generates a conceptual model. Model qualification determines the adequacy of the conceptual model. The next step produces a computer program. Model verification confirms that the computerised model represents the conceptual model within specified limits of accuracy using carefully chosen test cases. The final stage, validation, tests the input-output transformation of the simulation by statistical analysis. The strength of this framework is that it combines three different mechanisms for establishing confidence in a simulation.

Figure 1 The Validation of simulation models

When using simulation, it is always necessary to consider the issue of model validity. With simulation packages, the verification stage is the responsibility of the software supplier. However, it is the user of the simulation who is responsible for ensuring that the conceptual model and the input-output transformations are appropriate for the application.

References

Casti, J. (1997), Would-be Worlds: How Simulation Is Changing the Frontiers of Science, John Wiley & Sons, New York, NY.Fishman, G.S. (1978), Principles of Discrete Event Simulation, John Wiley & Sons, London.Naylor, T.H. and Finger, J.M. (1967), "Verification of computer models", Management Science, Vol. 14, p. 92Schlesinger, S. (1980), "Terminology for model credibility", Simulation, Vol. 34, pp. 101-5.