Vacancy PhD student: parallel algorithm design for distributed geoscience applications

We have a vacancy for a PhD student working at the intersection of computer sciences and geosciences, with important contributions to the next generation modelling software. Information about the position can be found at: https://www.uu.nl/en/organisation/working-at-utrecht-university/jobs Search for (select ‘Academic’ and ‘Faculty of Geosciences’): PhD Parallel algorithm design for distributed geoscience applications The vacancy is no longer available

FOSS4G in Bonn

foss4g 2016 in Bonn

Last week I attended the FOSS4G conference in Bonn. There were plenty of interesting presentations, and the venue (World Conference Center Bonn) was really impressive. If you are interested in our current work on estimation of human exposure to air pollution you can download our presentation slides and watch my presentation (ca. 25 min; mp4, webm). You can also contact us directly for more information.

Vacancy: PhD position

We have a vacancy for a PhD student working at the intersection of geocomputation and environmental health science. The successful candidate will work in a multidisciplinary team under supervision of Dr Derek Karssenberg (PCRaster research team, Faculty of Geosciences, Utrecht University) and Dr Ilonca Vaartjes (University Medical Center Utrecht). Detailed information is at http://www.werkenbijumcutrecht.nl/international/vacancies/Pages/phd-gghdc%20(2016-0557).aspx

Bring Your Own Linux

pcraster_open_source_code In a recent post we outlined that we will no longer distribute compiled versions of PCRaster for Linux systems but modellers need to build their own versions. We now completed most of the work on our build infrastructure and included the components from previous release packages that were yet missing (such as documentation) in the source tree. A few things still need to be solved on our side, e.g. further reducing the number of  external libraries we build ourselves instead of using system libraries only; and fixing some unit test failures as well. Still, the current setup should result in a complete PCRaster package, and therefore we think that we can now provide instructions to the users to compile and install their own PCRaster version. However, we recommend to not use the current development branch to generate a package and use it in production mode, please consider it still as beta for the upcoming PCRaster release. Please note that building your own PCRaster package does not require software development experience on your side, but requires some system administration steps (and endurance). Also, trying to build your own PCRaster might be adventurous at the moment. There is a myriad of Linux distributions out there, each with different versions (and combinations thereof) of external libraries. Your build might therefore fail in the first attempt. However, we very much appreciate your feedback with respect to your builds. Please share your experiences on our mailing list. And we will use your feedback to streamline the build and installation process.

Building prerequisites

We assume that you use a recent 64-bit Linux distribution (such as Ubuntu 16.04 x86_64), and we will outline the required steps by means of the apt package manager. In case you use another distribution and package manager you need to adapt the steps accordingly. First, you will need to install a compiler and a set of libraries and applications required to build PCRaster. Perform the following three steps: sudo apt update sudo apt upgrade sudo apt install cmake gcc g++ git libboost-all-dev libgdal-dev libpython2.7-dev libncurses5-dev libgeos++-dev imagemagick libxslt1-dev xsltproc qt4-dev-tools qt4-qmake libqwt-dev xsdcxx libxml2 libxml2-utils python-numpy

Building PCRaster

We provide a script that performs the necessary steps to build PCRaster, you need to download that one first. We assume that you are going to build the package in the folder 'pcraster_4_2_beta' in your home directory (you can choose another location and directory name if you prefer). Execute the following three steps: cd $HOME wget http://pcraster.geo.uu.nl/download/build_pcraster.py python build_pcraster.py pcraster_4_2_beta --build-gdal That's all, but it will take a while to complete. You will see something like this: ~$ python build_pcraster.py pcraster_4_2_beta Initialising directories Obtaining source code Building utility libraries Building PCRaster (this can take about one hour...) Installing PCRaster Testing pcrcalc May 26 2016 (linux/x86_64) USAGE: pcrcalc [options] "expression" or pcrcalc [options] -f scriptFile ( or #!: pcrcalc -F [options]+) other flags: s # : set seed (integer > 0) for random generator default is based on current time b f : overrule script bindings 1 : update timeseries files at end of each timestep r f : set run directory d f : debug mode, check MV creation on assignment comparing against clone or areamap boolean mask c : strict Case significant filename check (Unix portability) p : print profile information m : optimize with areamap MV compression l : use less memory but more temporary disk storage t : test argument substitution Great, you just built your own PCRaster version! To use is type: source $HOME/bash_pcraster_4_2_beta.sh You can now use your own version it for first experiments, e.g. import the PCRaster Python module, run a few models and compare the output with the previous releases; or fix some bugs and send patches to us... Anyhow, let us know whether you were successful. And 'it just works' is feedback we appreciate as well ;-) In case of a build failure check the pcraster_install.log if you can fix some obvious problems, otherwise the logfile could provide us with valuable information. Meanwhile we will continue working towards the PCRaster 4.2 release. Most likely we will make the build script more configurable as well such that e.g. you can specify if you want to build the documentation or not. More in our next post!

apt-get update

screenshot_osx We released the PCRaster version 4.1 already some time ago. We’ve been busy the last year with several projects, and we are finally getting closer to the next release. A lot still needs to be done in the next few weeks, but it is time for a short update. While we will continue to distribute binary packages for Windows systems, Linux users will face some changes:

Dropping LSB support

Until now, we provided Linux versions of the PCRaster software conforming to the Linux Standard Base (LSB). This approach is convenient for users as our executables will run on virtually any Linux distribution. For us developers, however, building LSB software is quite tedious. We were forced to use specific (outdated) compiler versions and we needed to compile and distribute a lot (about 25) of third party libraries ourselves. In addition, the LSB is rather conservative with respect to supporting new compiler versions, therefore not enabling us to use new and very useful features of the C++ programming language. However, we need those new C++ features for our current projects. As PCRaster is open-source for quite some time now there is also no reason to distribute binaries any longer. We will instead focus on improving and extending our software and on distributing the code. We changed our whole build infrastructure to use more libraries installed on the system, and further try to reduce the dependencies to third party libraries making it easier for us to develop and add new features. However, no longer providing binaries will require the Linux users to compile the PCRaster software themselves. Don’t fear; we will provide build scripts that smooth the compilation process.

Distributing experimental features

We develop new software due to several research and development projects. Last summer we worked on a multicore module for the PCRaster point operations within the maps4society project. We continue our work on high performance computing elaborating distributed computing to further increase runtime performance of environmental models and parallel I/O. We are also busy with a new file format that is capable of handling heterogeneous data. These exciting projects result in new, volatile software rather than stable versions. But we are convinced that our developments are of interest and useful for adventurous modellers. However, previously it was very difficult to distribute those new software developments to our users. The new release will include an optional flag to build these experimental features such that modellers can benefit from our latest projects more easily.

Supporting new platforms

Reducing dependencies to external libraries should result in a straightforward and easier build process for Linux distributions. The PCRaster software will therefore continue to run on all kinds of Linux machines (and yours). We would also like to add the PCRaster software to the OSGeo live CD to streamline the first time experience of new users. In the very near future we also will be able to support other UNIX based systems (the attentive reader might have noticed the operating system used for the screenshot). In a later stage we consider distributing our software via package managers such as apt or macports.

Improving runtime performance

PCRaster Modflow groundwater model of Indonesia During the summer we have been busy with tailoring PCRaster towards high-performance computing, and we already have rewritten parts of the PCRaster algorithms due to our Maps4Society project. This refactoring required significant efforts and code changes. Sometimes, however, appending just a single word can yield performance improvements as well... as our colleague Edwin Sutanudjaja mentioned, a simple NOPRINT statement in several packages prevents Modflow to write a large amount of redundant information. For a model running the extent shown above (see also here; 21600000 cells, with 3920887 drains and 21600000 river cells) the Modflow runtime reduces from about real 5m39.229s user 5m10.392s sys 0m25.511s to about real 3m33.181s user 3m1.913s sys 0m25.666s Nice savings, especially if you need to run several timesteps! The improvements will be included in the upcoming PCRaster release.