- 1 Contributing with python code to the UMEP plugin in QGIS
- 2 SUEWS wrapper
- 3 f2py
- 4 Upcoming Developments
- 5 Benchmarking system
- 6 Coding Guidelines
- 7 How to setup your development environment on Windows
- 8 SUEWS Prepare Developer
Contributing with python code to the UMEP plugin in QGIS
Here, instructions and recommendations on how to make your own plugin to go with the UMEP plugin will be described as well as tips and tricks on how to make use of the GIS capabilities included in QGIS to go with your plugin.
Gary Sherman (creator of QGIS) has produced a number of useful tools for developers that can be used to make your own contribution to the software. Below, a number of tool are listed that come very handy when you want to create your own plugin. There is also a number of resources online that is very useful. One is the PyQGIS Developer Cookbook available from the QGIS webpage and another is a book written by Gary Sherman called The PyQGIS Programmer's Guide.
- Start off by creating a plugin using the PluginBuilder in QGIS. This is a plugin that sets up all necessary files and folders for your plugin.
- Another useful plugin is the PluginReloader which makes it possible to reload a plugin in QGIS without restarting the software.
- pb_tool is another useful program to use for installing your plugin in the QGIS plugin folder as well as cleaning etc.
Please use the python libraries that comes with the QGIS installation as much as possible without including external libraries when developing your plugin. All libraries are not included when a common installation of QGIS is installed. Go to our Getting started section for further instructions.
External Python libraries currently included in the UMEP plugin (Utilities folder):
- f90nml - for reading and writing of fortran namelists
- Pysolar - for calculation of Sun positions
To use the f90nml library located in the Utilities folder:
from ..Utilities import f90nml
The pandas library is not install by default so a simple install pandas cannot be used use instead a try statement:
try: import pandas except Exception, e: QMessageBox.critical(None, 'Error', 'The WATCH data download/extract feature requires the pandas package ' 'to be installed. Please consult the FAQ in the manual for further information') return
The same goes for matplotlib and other libraries that you are uncertain of.
The main file of the python wrapper for the SUEWS model is called SUEWSwrapper.py. To change version of SUEWS when running the wrapper, simple go in to SUEWSwrapper.py and activate the line which is calling the appropriate wrapper (e.g. SUEWSwrapper_v2016a) and comment out other versions.
In SUEWS v2017b - this is no longer used/needed
A possibility to make use of fortran subroutines in python. See here for documetation.
- When developing the fortran code for compiling with f2py kind=1d0 will not work for function parameters to be passed in from python. Should use KIND=8.
Setting up Windows machine for running f2py
Please note that the steps below contain some links that over time could change, however, the basics should remain the same. This has been tested and established for an Intel 64-bit machine running NT operating system. This is still to be tested on other versions of Windows operating system. If you carry out on both OS, please add to the list below or report to team.
The process can also be applied to a 32-bit machine, but the choice of python2.7 and set up for MinGW-w64 will be different. This has not been tested, and perhaps shouldn’t be encouraged if SUEWS is being developed and tested on/for 64-bit architecture.
Before starting any of the required steps it is recommended that any version of Python, mingw, and Cygwin be removed from the machine if possible. If you already have Python 2.7 and MinGW installed, or have followed the procedure below then go to the instructions on developing fortran-python interface.
- Download Python 2.7.XX : download from: here and install to directory C:\Python27
- Add to PATH environment variable:
- Add to C_INCLUDE_PATH environment variable:
Note: The C:\Python27\Scripts directory is required for use of pip in subsequent steps.
- Install latest NumPY package to get latest f2py
(i) Download numpy-1.11.0rc1+mkl-cp27-cp27m-win_amd64.whl from here
(ii) Open command prompt and change directory to where numpy wheel is installed (e.g. Downloads folder). Run the following command:
Pip install numpy-1.11.0rc1+mkl-cp27-cp27m-win_amd64.whl
(iii) Check the directories for numpy and numpy-1.11.0rc1.dist-info are under C:\Python27\Lib\site-packages. Dates and times of directory should indicate these are new from wheel.
- Download Mingw-w64 from here
Note: This is suitable for 64 and 32 bit architecture.
- Run mingw-w64-install.exe (found in directory to which you have downloaded it to).
(i) In the install procedure set:
Version: 5.3.0 (or what ever is latest)
Architecture: x86_64 (for 64 bit machine)
Build revision: 0
(ii) Set the destination folder to: C:\mingw-w64_x86 when prompted.
- Add to environment variable:
(i) C_INCLUDE_PATH: C:\mingw-w64_x86\mingw64\include
(ii) PATH: C:\mingw-w64_x86\mingw64\bin
- This step is required to create/replace the import library found under directory C:\Python27\libs. The import library is libpython27.a.
(i) Download the pexports binary pexports-0.47-mingw32-bin.tar.xz from here.
Note: pexports-0.47 could change for subsequent versions.
(ii) Unpack the tar file and put pexports.exe in C:\Python27\libs.
- Open a command prompt and run the following command:
pexports C:\Windows\System32\python27.dll > USERDIR\python27.def
Note: USERDIR is the user directory you put the file in. As it is an intermediary step and a temporary file, the user directory you use shouldn’t matter, however, don’t try to put it in Windows\System32 directory, or any other directory in the system Path.
- Create the import library libpython27.a for helping the linker of MinGW link to the correct python DLL.
(i) Open Command Prompt and change directory to where python27.def was created in step 8 (i.e. USERDIR).
(ii) Run command:
dlltool –D python27.dll –d python.def –l libpython27.a
(iii) Move resulting libpython27.a into C:\Python27\libs , replacing any existing version of this file in the directory
Making Fortran-Python Interface ‘dll’ (.pyd file) with F2PY
- This is shown using a makefile (named Makefile) that is called from the command line as follows:
mingw32-make –f Makefile
Note: This should be called from within the directory that the Makefile and source code is in.
- Basic Makefile:
CC = gnu95 CCO= x86_64-w64-mingw32-gfortran FFLAGS = -fPIC TARGET = INTENDED_NAME_OF_PYD MODULES = nameOfModules.o main: NAMEOFMAINPROGRAMFILE.f95 $(MODULES) f2py.py –c –-fcompiler=$(CC) –-compiler=mingw32 –m $(TARGET) NAMEOFMAINPROGRAMFILE.f95 $(MODULES) $(MODULES): nameOfModules.f95 $(CCO) –c $(FFLAGS) nameOfModules.f95 cleanall: -del $(MODULES)
Note: A .pyd file should have been created on the completion of compilation from command called in step 1.
- Create a directory to store all created .pyd files in (e.g. C:\PythonPYD) and add to PATH environment variable.
This ensures the .pyd files are picked up and used by python scripts.
Distributing f2py Modules for Windows
Note: all the .dll files (including those used to make python library from MinGW)
need to be packaged up so that a machine without MinGW can use the developed python libraries.
The .dll files to include are: NEED TO LIST THEM
Importing and using in Python
- Import the module into python script in the same way you would import any other module:
If your module is called SolweigShadow, for example, then import SolweigShadow as SS will enable you to access the functions of the module by SS.functionName().
Note: The parentheses are needed regardless of whether the function has parameter inputs/outputs.
- To see what functions are available for the imported module, use the command
|SUEWS||Convective boundary layer development||Completed||Göteborg Univ|
|SUEWS/SOLWEIG||Mean radiant temperature model||Active||Göteborg Univ|
|SUEWS||Storage Heat flux - ESTM||Completed||Göteborg Univ /Reading|
|SUEWS||Storage Heat flux - AnOHM||Active||Reading/Tsinghua|
|SUEWS||Anthropogenic Heat fluxes||Actve||Reading|
|Wind||Pedestrian wind speed||Active||Göteborg Univ/Reading|
|Multi||Downscaling data *download WATCH||Active||Lingbo Xue (Reading)/TS|
|Multi||Downscaling data *precip mass check||Active||TK/ LX (LJ/TS)|
|Multi||Downscaling data *precip intensity||Active||AG/ LX|
|SUEWS/SOLWEIG||Radiation coupling||Active||Göteborg Univ/Reading|
If you are interested in contributing to the code please contact Sue Grimmond.
- Code written in Fortran – currently Fortran 95
- Names should be defined at least in one place in the code – ideally when defined
- Implicit None should be used in all subroutines
- Variable name should include units. e.g. Temp_C, Temp_K
- Output variable attributes should be provided in the
TYPEstructure defined in the
ctrl_outputmodule as follows:
- TYPE varAttr
- CHARACTER(len = 15) :: header ! short name in headers
- CHARACTER(len = 12) :: unit ! unit
- CHARACTER(len = 14) :: fmt ! output format
- CHARACTER(len = 50) :: longNm ! long name for detailed description
- CHARACTER(len = 1) :: aggreg ! aggregation method
- CHARACTER(len = 10) :: group ! group: datetime, default, ESTM, Snow, etc.
- INTEGER :: level ! output priority level: 0 for highest (defualt output)
- END TYPE varAttr
- Code should be written generally
- Data set for testing should be provided
- Demonstration that the model performance has improved when new code has been added or that any deterioration is warranted.
- Additional requirements for modelling need to be indicated in the manual
- All code should be commented in the program (with initials of who made the changes – name specified somewhere and institution)
- The references used in the code and in the equations will be collected to a webpage
- Current developments that are being actively worked on
How to setup your development environment on Windows
gfortran with NetBeans
- Go to Cygwin and install 64-bit. You need to make sure that you install gfortran, g++, gdb, make and gcc. I am not really sure what is needed so I tend to install too many packages rather that too few. Install in c:\cygwin64
- Go to your Environment Variables in advanced system settings in windows and include C:\cygwin64\bin;C:\cygwin64\usr\bin;C:\cygwin64\usr\local\bin;C:\cygwin64\lib;C:\cygwin64\usr\lib in your Path.
- Install NetBeans from www.netbeans.org. You only need to download the C/C++ version.
- If you don’t have the correct Java, follow the link presented to you and install correct version.
- Copy your code to a folder of your choice.
- Create a new project (C/C++ from Existing Source) and use you folder as the project folder. Keep all other settings.
- You are ready to work.
NOTE: Another nice thing to do is to use gfortran from your cluster on your windows PC. Do the following:
- In Netbeans, go to Tools>Options>C/C++ and click Edit next to localhost. Click Add… and write metcl2. Just keep on clicking until you need to give your username and password for the cluster.
- Now you should be able to run GNU on the cluster from your windows PC.
Python and PyCharm (Not so good alternative)
- Install python 2.7.X, 64 bit from python.org (Windows x86-64 MSI installer). Install with default settings.
- Visit JetBrain, Pycharm website and obtain a student account (go to Discounted and Complimentary Licenses, https://www.jetbrains.com/pycharm/buy/). Click on For Students and Teachers, go to bottom of the page and click Apply Now. Choose either a student or a teacher status. You will get an email where you activate your license.
- Create a folder which you can use as a project folder. Copy the python code (*.py) from the suews repository and put it the folder. If you don’t have access to the repository talk to Fredrik Lindberg.
- Download PyCharm professional (https://www.jetbrains.com/pycharm/download/) and install.
- Start PyCharm and activate license using your JetBrains account.
- Create a new project (Pure python) and choose the created folder (3) as your project folder and use your python installation as interpreter. Click ok in the next message box.
- Go to File>Settings >Project Interpreter. Add a new package by clicking the green plus sign. Search for numpy and install package. If you get errors, you probably need correct version of Visual studio. There is an address of a website where you can download it in the error message when you tried to install numpy.
- Also install matplotlib (used for plotting)
- Run mainfileLondon.py to do stuff.
Python and PyCharm (good alternative)
- Go to qgis.org and click on download. Choose the installation for advanced users (64-bit). Choose the advanced desktop installation and make sure that qgis-ltr is included. Keep other default settings. This give you a python installation with everything you need (pretty much). IF you are missing python libraries after the installation, you can restart the installation file and add more components.
- If you haven’t installed PyCharm, follow set 2 through 5 above.
- Create a .bat-file (e.g. PyCharmWithQgis.bat) with the following content (put it in your folder created earlier and edit it so that the paths on line 1 and 5 is correct):
SET OSGEO4W_ROOT=C:\OSGeo4W64 SET QGISNAME=qgis SET QGIS=%OSGEO4W_ROOT%\apps\%QGISNAME% SET QGIS_PREFIX_PATH=%QGIS% SET PYCHARM="C:\Program Files (x86)\JetBrains\PyCharm 2017.3.5\bin\pycharm.exe" CALL %OSGEO4W_ROOT%\bin\o4w_env.bat SET PATH=%PATH%;%QGIS%\bin SET PYTHONPATH=%QGIS%\python;%PYTHONPATH% start "PyCharm aware of QGIS" /B %PYCHARM% %*
- Run the bat-file.
How to make standalone application using py2exe (this is not used, see below)
- In PyCharm, add the pip package (if not already there). See bullet point 6. Above.7.
- Go to http://www.lfd.uci.edu/~gohlke/pythonlibs/ and download the appropriate py2exe package (.whl).
- Open a command prompt and go to the folder where you download the py2exe package and write:
- Create a file called setup.py in your working directory with the following code:
from distutils.core import setup import py2exe
- From a command prompt (can use terminal in PyCharm) write:
python setup.py install
- Then write:
python setup.py py2exe
- All files and folders needed are now created in a subfolder call dist. You also have to add the SUEWS executable and all files needed to run the model.
How to make standalone application using Pyinstaller (use this)
- Add the pip package (see above)
- You need to add the path to where pip.exe is located (usually C:\Python27\Scripts\). If you don’t know how to add a path in your environment settings you can temporarily add one in a command prompt by writing:
- In the same command prompt, write:
pip install pyinstaller
- Locate yourself where you have your script and write:
SUEWS Prepare Developer
This is for advanced users regarding SUEWS Prepare plugin in UMEP. The information in should help with translating the plugin, adding new tabs or adding new variables.
|most important files for making changes to the plugin||excel documents SUEWS_init.xlsx, SUEWS_SiteLibrary.xls and SUEWS_SiteSelect.xlsx.|
|files are located||as a part of the plugin in the folder named “Input” (by default in C:\Users\your_username\.qgis2\python\plugins\SUEWSPrepare\Input).|
|SUEWS Prepare uses these files||for example to generate the amount of site library tabs and the contents of those tabs.|
|Take care||any changes made to these documents will be lost if they are replaced (e.g. reinstalling or updating the plugin). This can be prevented by making backups of the excel documents before reinstalling or updating.|
|SUEWS_init.xlsx||This file handles the amount of site library tabs in the plugin, the name of these tabs and their connection to other excel sheets and text documents. Each sheet represents one tab.|
|SUEWS_SiteLibrary.xls||This file contains all the different information connected to different site. Each excel sheet is connected to a different kind of information like vegetation and water data and each line in a sheet represents a different area or site. This information is used to determine what kind of information and variable will be present in a widget of a site library tab.|
|SUEWS_SiteSelect.xlsx||This file contains an example of one line of output from the plugin. It is used by the plugin to check the order of the outputs. It can be considered the least important and useful for developers.|
|Modifying the plugin||How to work with the excel documents to make changes to existing information inside the plugin such as titles. This could be required for translation or to fix spelling errors.|
|Changes available through SUEWS_init.xlsx|
|A detailed look at the SUEWS_init document||The SUEWS_init determines the number of site library tabs as well as the number of widgets in these tabs and where the widgets will fetch their content.
The document contains a number of sheets and every sheet represents one site library tab. The names of the sheets will determine the title of the site library tab. The first one is an example of how the layout of a working sheet should look.
Each row of a sheet represents a new widget. Every column of the row is used to determine the specific characteristics of the widget.
|Change the variables in the variable box of a widget||The content of a widget is decided by what sheet in the document SUEWS_SiteLibrary.xls it is connected to. This connection is created by the information in the first column of a sheet in SUEWS_init. To make changes edit the text in the first column to match the name of the sheet you want to fetch information from.
Example: Let’s say for the purposes of this example that we want the content of the tab named “Paved” to have the same content of the tab named “Evergreen”. To do this we must change the connection in the paved sheet of SUEWS_init to match that of the evergreen sheet. In the evergreen sheet we can see it’s connected to a sheet in SUEWS_SiteLibrary called SUEWS_Veg. If we change the text of the first column in the paved sheet to match this, the content of the tab will change to the same as the evergreen tab. PICTURE? this needs attention</ span style="color: red">
|Change the default parameters for a widget||fourth and fifth columns are optional information and decide if there are any default parameters for a widget.
The number in the fourth column decides if there is an identification code for the tab. This identification code is used to exclude entries from the site library. Many tabs might link to the same site library sheet and if there is an identification code only the entries that match the code will be shown in the widget. If there is a number in the fifth column the tab will try to match this number against the site codes (not to be confused with the identification code). The side codes are the codes that fill out the drop down box in the widget marked “code” and each code represent one site library entry. If there exist a default site code for a tab this code will be selected in the drop down menu on the plugin start up. Example: Let’s keep making changes to the paved sheet. Right now the identification code for the sheet is “1” and the default site code is “661”. If we change the identification code (fourth column) to “4” a different set of site entries will be available for selection in the widget. One of the site codes that are now available is “662”. By changing the content of the fifth tab to “662” this will now be the default site code for the widget. PICTURE? this needs attention</ span style="color: red">
|Change the order of the widget site code in the final output of the plugin||A widget’s contribution to the final output of the plugin will be the selected site code in the widget. This code will be placed somewhere on a predetermined place in a long list of variables. The sixth column in a SUEWS_init sheet represents this position in the final output. To change a widget’s output order edit the number in the sixth column. Take care to make sure changing the position doesn’t overwrite any other information. The order of the final output is also closely tied to the document SUEWS_SiteSelect, see more [[#XLSX].|
|Editing a tab name||The name of the tabs in the SUEWS Prepare main window correspond to the names of the sheets in the excel document SUEWS_init. To edit a tab name simply change the name of the sheet.
Example: After all the changes made to the paved sheet in SUEWS_init the name “paved” as a description of the tab no longer fit. By renaming the sheet to “vegetation” the tab will have a more fitting name. this needs attention</ span style="color: red"> PICTURE?
Changes available through SUEWS_SiteLibrary.xls
| What can be made through the SUEWS_SiteLibrary.xls.
The SUEWS_SiteLibrary document is what defines the variables inside a tab. This document defines the titles and tooltips for the variables as well as the values for the variables on different sites.
|Variable index||first row of a site library sheet is an index of the variables in the sheet.|
|Variable and metadata titles||second row contains the titles of the variables. The first cell is always the title “Code”. After all the variable titles follows a blank cell. The cells that follows will be titles for metadata, it is also possible that there is no metadata for the sheet. The row always end with the titles “Photo”, “LC_previous” and “LC_code” in that order.|
|Variable tooltips||third row contains tooltips or longer descriptions of the variable titles.|
|Site entries||A site entry represents one complete set of values for all the variables in the sheet. One row represents one site entry.
The first cell of a site entry always contains the site code. This code is used to differentiate between different site entries and needs to be a unique integer number for the sheet. The following cells contain values for different variables until an exclamation mark marks the end of variables. If there are any metadata descriptions these will be in the cells following the exclamation mark. The last three cells are in order: a photo url if there is one otherwise the cell is left blank, a blank cell and lastly the identification code if there is one (otherwise the cell is left blank). The two last rows: The two last rows of the sheet contains a single “-9” in the first cell. These rows are used by the plugin to signify the end of the data in the sheet and nothing below these rows will be read.
|Change the title of a variable||To change the title of a variable, first navigate to the correct sheet in SUEWS_SiteLibrary. The titles of all variables are decided by the text in the second row. Replace the text in a column to change the name of a single variable or for example translation purposes replace every word in the second row with its translation.|
|Change the tooltip of a variable||
The tooltip of a variable is a longer description than the title that shows up when the user hovers over the variable text box.
The third row of a SUEWS_SiteLibrary sheet defines the tooltip of a variable. To changes it, replace the text for the relevant column in the third row.
Changes available through SUEWS_SiteSelect.xlsx
|The document SUEWS_SiteSelect.xlsx is mainly connected to the final output of the plugin. Most developers won’t need to make any changes to it. Developers mainly concerned with the layout of the SUEWS Prepare plugin will not need to be concerned about SUEWS_SiteSelect.|
|Change the order of the final output||
The second row of the sheet SUEWS_SiteSelect contains text strings that are used by the plugin to identify a variables place in the final output of the plugin. Changing the order of the strings in the second row will similarly affect the final output.
|Adding to the plugin||How to make additions to the plugin (e.g. adding new tabs). Earlier information will be useful when adding to the plugin. i.e. read earlier sections before reading this one.|
|Adding a new tab to the plugin||As discussed (#XLSX) the excel document SUEWS_init.xlsx is closely tied to how the plugin generates tabs. The plugin will generate tabs according to the number of sheets in this excel document and according to the information in the sheets.
A single sheet represents one new tab. Every row in a sheet represents a widget that will be added to the tab. Every column in a sheet contains certain information that decides the specifics for a widget such as what variables will be added. The first sheet of the excel document is an example sheet that can be used as a quick reference for the content of the columns. For a more detailed description see #XLSX.
|To add a new tab to the plugin:||#Create a new sheet in the SUEWS_init document. The order of the sheets will match the order of the tabs in the plugin. Do not place the sheet first in the excel document as this is used as a placeholder for the example sheet. The name of the sheet will become the title of the tab.
|<div id="ADD" Adding a new set of site variables to the plugin</div>||As discussed in #XLS the variables of a site (and consequently the variables that appear in a widget connected to this site) are generated from the excel document SUEWS_SiteLibrary. One sheet represents the variables of a type of site and can be connected to multiple widgets and tabs.
A new site sheet must fulfil certain conditions. The first row of the sheet should be an index of the variables in the sheet that ranges from one to the amount of variables. The second row should contain the titles for the variables and the first column should always be “Code”. Furthermore the second row should always end with the titles “Photo”, “LC_previous” and “LC_code” in that order. The third row should contain longer tooltips or descriptions of the variables. The rows following the third row should each represent one site entry. Lastly the sheet should end with two rows that just contains “-9” in the first column. For a more detailed description see #XLS.
There are two options when adding site entries; it can be done manually directly in the sheet or through the plugin when the sheet has been connected to a widget. (See Section 6.1 and 3.3.2)
When adding a site entry manually certain conditions must be followed:
Each new sheet needs a matching text document located in the “Output” directory of the plugin. This text document needs to mimic most of the excel sheet. Instead of columns separating the variables the text document should use tab indents and each line in the text document represents a row in the sheet. The first line of the text document should be an index of the variables. The second line should be the variable titles. The text document should not contain the variable tooltips therefore the site entries should start on the third line of the text document as opposed to the fourth row of the excel sheet. Any site entries added manually to the excel sheet needs to be manually entered to the text document as well. The two last lines of the text document should just contain a single “-9”. To add a new site library sheet use the methodology above and follow these steps: