High-performance modelling

Until a few years ago, software applications automatically benefitted automatically from technical improvements of microprocessors. Higher CPU clock speeds meant faster execution, or in case of PCRaster, shorter runtimes of environmental models. But the increase in CPU speed slowed down, and increasing the number of cores per CPU instead has become a common practice by the manufacturers. Unfortunately, software is not able to automatically use multi-core CPUs. Given that data sizes and model complexity tend to increase, there is a growing demand for new modelling software that is capable of using all available hardware resources.

This project is about making a PCRaster-like modelling engine that executes models faster. This can be done by making the software scalable. This involves rewriting the individual model algorithms into concurrent algorithms that are capable of using multiple CPU cores in a computer. To take this one step further, the software should be able to view a model as a collection of concurrent tasks that can be distributed over all compute resources, ranging from simple laptops to high-end clusters and supercomputers. This allows users to develop models on a local machine and, once ready, run the same model unchanged on a cluster or a supercomputer.

Besides the computational tasks purely performed by CPUs, modelling projects often involve large amounts of I/O, e.g. because of the use of large datasets, or checkpointing. When the performance of the model algorithms increase, the overall performance of the model execution becomes dominated by the I/O. This is especially the case when the model is executed on a distributed cluster or supercomputer. In such a case the model engine should use parallel I/O strategies to leverage the availability of the parallel file system often present on such systems. That way, I/O performance can also scale up with the number of compute nodes allocated to the model.

In short, this high-performance modelling project focusses on three aspects of high-performance computing:

  • Design and implementation of concurrent modelling algorithms. This will make models scale better within a single compute node.

  • Design and implementation of an architecture for the distribution of whole models. This will make models scale over multiple compute nodes.

  • Design and implementation of an approach for doing parallel I/O. This will scale I/O over multiple compute nodes.

In the end, the goal of this project is to design and implement a modelling engine that is scalable over all relevant hardware architectures. This modelling engine will be usable through a high level scripting language like Python. This will make it possible for domain experts to perform modelling studies involving larger data volumes and/or more and more detailed represented environmental processes, and/or perform modeling studies faster. And they will be able to do this without knowing anything about concurrent or distributed programming.

Currently, we are working on the first sub-project mentioned above, in a project called High-performance geocomputation, funded by the Dutch research programme called Maps4Society.

Addition information: