Development Notes

From OpenM++
Jump to: navigation, search


OpenM++ development notes

This page contains various notes only for OpenM++ developers. There is no useful information on that page for anyone else. It is a notes, they are not in any specific order and may not true. OK, you have been warned.

Git layout

  • bin - empty folder, used for OpenM++ compiled binaries and third party tools
  • include - includes for public interfaces of compiler and libraries
    • libopenm - model runtime library public interface
    • omc - model compiler public interface
  • licenses - third party lincences
  • models - test models, for example:
    • NewCaseBased - simple test model
    • NewTimeBased - simple test model
    • modelOne - test model for runtime library, does not use OpenM++ compiler
  • openm - OpenM++ core source code
    • libopenm - model runtime library (libopenm) and compiler library (libopenm_omc_db)
      • common - common helper routines, for example: log
      • db - data access classes
      • include - includes for libopenm and libopenm_omc_db
      • model - model base classes
      • msg - message passing library
    • main.cpp - models main() entry point
    • libsqlite - SQLite with extension functions such as SQRT()
    • omc - OpenM++ compiler
  • Perl - perl scripts
  • props - VC++ project includes to build the models
  • R - openMpp R library: integration between OpenM++ and R
  • sql - sql scripts to create openM++ database
    • db2 - DB2 version of openM++ database scripts
    • mssql - Microsoft SQL Server version of openM++ database scripts
    • mysql - MySql version of openM++ database scripts
    • postgresql - PostgreSql version of openM++ database scripts
    • sqlite - SQLite version of openM++ database scripts

OpenM++ logs and trace

As it is now model executable output log messages into three streams:

  • standard output (console)
  • "last" log file: /current/working/dir/modelExeName.log
  • "stamped" log file: /current/working/dir/

Model trace output is similar to log output but works much faster. Trace output is buffered and may be lost if something goes wrong and model crushed.

You can adjust output log and trace output inside of main() by changing: theLog->init(...); parameters. It is also be controlled by .ini options file.

Defines for OpenM++

You may need to change defines to build OpenM++ from source code:

  • OM_DB_SQLITE: use SQLite as database provider (only one supported at the moment)
  • OM_MSG_MPI: use MPI as for message passing library (see below)
  • OM_MSG_EMPTY: use empty version message passing library (default value)
  • OM_UCVT_MSSTL: use c++11 STL to convert strings to UTF-8 (default on Windows)
  • OM_UCVT_ICONV: use glibc iconv to convert strings and file content to UTF-8 (default on Linux)

Please note:

  • OM_MSG_MPI and OM_MSG_EMPTY mutually exclusive
  • to set defines properly change (on Windows) or use make OM_MSG_USE=MPI (on Linux)
  • OM_UCVT_MSSTL and OM_UCVT_ICONV mutually exclusive
  • OM_UCVT_MSSTL tested on Windows with VC++2012 and account for Microsoft-specific implementation of STL codecvt classes.

Defines and other changes for VC++

Defines to compile libsqlite library with extension functions: SQLITE_ENABLE_COLUMN_METADATA; SQLITE_OMIT_LOAD_EXTENSION; HAVE_ACOSH; HAVE_ASINH; HAVE_ATANH;

To avoid innumerous compatibility errors and warnings following must be defined: _CRT_SECURE_NO_WARNINGS and _CRT_NONSTDC_NO_WARNINGS.

OpenM++ data library notes

IDbExec interface is db-connection wrapper and only the place where real SQL operations executed. All other classes are to wrap OpenM++ database tables and implement "business logic".

Data library is NOT thread-safe by design, do not pass it objects between model threads without proper guards.

Difference between OpenM++ database schema and Modgen schema:

  • support multiple models and multiple versions of the same model
  • support multiple run results of each model
  • tends to be more "relational", i.e.:
    • language-specific rows moved to separate tables
    • sub-samples are in rows not in columns

Database schema "read-only" compatible with Modgen database. For each Modgen table corresponding view created which allow to read from OpenM++ database as from Modgen database. If OpenM++ database contains multiple models (or versions) then it not be exposed to Modgen compatibility views.

OpenM++ database notes

If database connection string is not specified then model try to open SQLite database with name ModelName.sqlite (i.e.: modelOne.sqlite) in current working directory. Other word, default database connection strig is:

Database=ModelName.sqlite; Timeout=86400; OpenMode=ReadWrite;

Database can be created by following commands:

sqlite3 ModelName.sqlite < ../sql/sqlite/create_db_sqlite.sql
sqlite3 ModelName.sqlite < ModelName_create_model.sql
sqlite3 ModelName.sqlite < ModelName_insert_parameters.sql

On Linux slite3 executable most likely in your PATH. On Windows you must download sqlite3.exe from SQLite web-site.

OpenM++ data library notes: SQLite

Following parameters allowed for SQLite database connection:

  • Database - (required) database file name or URI, file name can be empty
  • Timeout - (optional) table lock "busy" timeout in seconds, default=0
  • OpenMode - (optional) database file open mode: ReadOnly, ReadWrite, Create, default=ReadOnly
  • DeleteExisting - (optional) if true then delete existing database file, default: false

If OpenMode=Create specified then database file created if not exist, which is default SQLite behavior.

Note: minimal connection string syntax for SQLite provider is: "Database=" and in that case SQLite will open temporary database. That kind of connection string does not really make sense for OpenM++ models because temporary database will be deleted after model exit.

OpenM++ message passing library notes

Message passing library (a.k.a. execute library) used for:

  • broadcast metadata and input parameters from root process to slave modeling processes
  • gather output modeling results from all modeling processes into root process

That library has two versions:

  • define OM_MSG_MPI: MPI-based version which does the job as described above (MPI component must be installed)
  • define OM_MSG_EMPTY: empty version of library, which does nothing and don't required anything installed

When empty version of library can useful?

To develop and debug your model without having MPI installed and without complexity of multi-process debugging. Obviously, some technique must be used to debug modeling logic inside of single process.

IMsgExec interface is main class for message passing library. All processes involved in the modeling must can be identified by integer process rank. Root process rank is zero.

Messaging library is NOT thread-safe, at least for now, do not pass it objects between model threads without proper guards. It may change in the future versions.

OpenM++ and UTF-8 strings

All strings inside of openM++ source code expected to be are UTF-8 encoded. If you need to pass string to openM++ API, please convert it to UTF-8 first. There is helper function which return file content converted as UTF-8 string:

string fileContent = fileToUtf8("someFile.txt");

Following rules applied to detect file encoding:

  • if byte order mark (BOM) present in the file then it converted according to BOM
  • if code page (encoding name) specified, i.e.: "English_US.1252" then it used for conversion
  • default user code page (encoding name) used to convert file content to UTF-8

You can use optional parameter to explicitly specify code page (encoding name):

string fileContent = fileToUtf8("someFile.txt", "English_Canada.1252"); // Windows: CP-1252
string fileContent = fileToUtf8("someFile.txt", "WINDOWS-1252");        // Linux:   CP-1252

That name is OS-specific and conversion would fail if name is invalid.

Please note: conversion from UTF-32 to UTF-8 not supported on Windows.

<metadesc>OpenM++: open source microsimulation platform</metadesc>