Development Setup

Introduction

The nomad infrastructure consists of a series of nomad and 3rd party services:

  • nomad worker (python): task worker that will do the processing

  • nomad app (python): the nomad app and it’s REST APIs

  • nomad gui: a small server serving the web-based react gui

  • proxy: an nginx server that reverse proxyies all services under one port

  • elastic search: nomad’s search and analytics engine

  • mongodb: used to store processing state

  • rabbitmq: a task queue used to distribute work in a cluster

All 3rd party services should be run via docker-compose (see blow). The nomad python services can be run with python to develop them. The gui can be run with a development server via yarn.

Below you will find information on how to install all python dependencies and code manually. How to use docker/docker-compose. How run 3rd-party services with docker-compose.

Keep in mind the docker-compose configures all services in a way that mirror the configuration of the python code in nomad/config.py and the gui config in gui/.env.development.

To learn about how to run everything in docker, e.g. to operate a NOMAD OASIS in production, go (here)(/app/docs/ops.html).

Install python code and dependencies

Cloning and development tools

If not already done, you should clone nomad and create a python virtual environment.

To clone the repository:

git clone git@gitlab.mpcdf.mpg.de:nomad-lab/nomad-FAIR.git
cd nomad-FAIR

C libs

Even though the NOMAD infrastructure is written in python, there is a C library required by one of our python dependencies.

libmagic

Libmagic allows to determine the MIME type of files. It should be installed on most unix/linux systems. It can be installed on MacOS with homebrew:

brew install libmagic

Virtual environment

pyenv

The nomad code currently targets python 3.7. If you host machine has an older version installed, you can use pyenv to use python 3.7 in parallel to your system’s python.

virtualenv

We strongly recommend to use virtualenv to create a virtual environment. It will allow you to keep nomad and its dependencies separate from your system’s python installation. Make sure to base the virtual environment on Python 3. To install virtualenv, create an environment and activate the environment use:

pip install virtualenv
virtualenv -p `which python3` .pyenv
source .pyenv/bin/activate

Conda

If you are a conda user, there is an equivalent, but you have to install pip and the right python version while creating the environment.

conda create --name nomad_env pip python=3.7
conda activate nomad_env

To install libmagick for conda, you can use (other channels might also work):

conda install -c conda-forge --name nomad_env libmagic

pip

Make sure you have the most recent version of pip:

pip install --upgrade pip

The next steps can be done using the setup.sh script. If you prefer to understand all the steps and run them manually, read on:

Install NOMAD-coe dependencies.

Nomad is based on python modules from the NOMAD-coe project. This includes parsers, python-common and the meta-info. These modules are maintained as their own GITLab/git repositories. To clone and initialize them run:

git submodule update --init

All requirements for these submodules need to be installed and they need to be installed themselves as python modules. Run the dependencies.sh script that will install everything into your virtual environment:

./dependencies.sh -e

The -e option will install the NOMAD-coe dependencies with symbolic links allowing you to change the downloaded dependency code without having to reinstall after.

Install nomad

Finally, you can add nomad to the environment itself (including all extras)

pip install -e .[all]

Generate GUI artifacts

The NOMAD GUI requires static artifacts that are generated from the NOMAD Python codes.

nomad dev metainfo > gui/src/metainfo.json
nomad dev searchQuantities > gui/src/searchQuantities.json
nomad dev units > gui/src/units.js
./gitinfo.sh

In additional, you have to do some more steps to prepare your working copy to run all the tests. See below.

Build and run the infrastructure with docker

Docker and nomad

Nomad depends on a set of databases, search engines, and other services. Those must run to make use of nomad. We use docker and docker-compose to create a unified environment that is easy to build and to run.

You can use docker to run all necessary 3rd-party components and run all nomad services manually from your python environment. You can also run nomad in docker, but using Python is often preferred during development, since it allows you change things, debug, and re-run things quickly. The later one brings you closer to the environment that will be used to run nomad in production. For development we recommend to skip the next step.

Docker images for nomad

Nomad comprises currently two services, the worker (does the actual processing), and the app. Those services can be run from one image that have the nomad python code and all dependencies installed. This is covered by the Dockerfile in the root directory of the nomad sources. The gui is served also served from the app which entails the react-js frontend code.

Before building the image, make sure to execute

./gitinfo.sh

This allows the app to present some information about the current git revision without having to copy the git itself to the docker build context.

Run necessary 3-rd party services with docker-compose

You can run all containers with:

cd ops/docker-compose/infrastructure
docker-compose -f docker-compose.yml -f docker-compose.override.yml up -d mongo elastic rabbitmq

To shut down everything, just ctrl-c the running output. If you started everything in deamon mode (-d) use:

docker-compose down

Usually these services only used by the nomad containers, but sometimes you also need to check something or do some manual steps.

The docker-compose can be overriden with additional seetings. See documentation section on operating NOMAD for more details. The override docker-compose.override.yml will expose all database ports to the hostmachine and should be used in development. To use it run docker-compose with -f docker-compose.yml -f docker-compose.override.yml.

ELK (elastic stack)

If you run the ELK stack (and enable logstash in nomad/config.py), you can reach the Kibana with localhost:5601. The index prefix for logs is logstash-. The ELK is only available with the docker-compose.dev-elk.yml override.

Run nomad services

API and worker

To simply run a worker with the installed nomad cli, do (from the root)

nomad admin run worker

To run it directly with celery, do (from the root)

celery -A nomad.processing worker -l info

You can also run worker and app together:

nomad admin run appworker

GUI

When you run the gui on its own (e.g. with react dev server below), you have to have the API running manually also. This inside docker API is configured for ngingx paths and proxies, which are run by the gui container. But you can run the production gui in docker and the dev server gui in parallel with an API in docker. Either with docker, or:

cd gui
yarn
yarn start

Run the tests

additional settings and artifacts

To run the tests some additional settings and files are necessary that are not part of the code base.

First you need to create a nomad.yaml with the admin password for the user management system:

keycloak:
  password: <the-password>

Secondly, you need to provide the springer.msg Springer materials database. It can be copied from /nomad/fairdi/db/data/springer.msg on our servers and should be placed at nomad/normalizing/data/springer.msg.

Thirdly, you have to provide static files to serve the docs and NOMAD distribution:

cd docs
make html
cd ..
python setup.py compile
python setup.py sdist
cp dist/nomad-lab-*.tar.gz dist/nomad-lab.tar.gz

run the necessary infrastructure

You need to have the infrastructure partially running: elastic, rabbitmq. The rest should be mocked or provided by the tests. Make sure that you do no run any worker, as they will fight for tasks in the queue.

cd ops/docker-compose
docker-compose up -d elastic rabbitmq
cd ../..
pytest -svx tests

We use pylint, pycodestyle, and mypy to ensure code quality. To run those:

nomad dev qa --skip-test

To run all tests and code qa:

nomad dev qa

This mimiques the tests and checks that the GitLab CI/CD will perform.

Setup your (I)DE

The documentation section on development guidelines details how the code is organized, tested, formatted, and documented. To help you meet these guidelines, we recomment to use a proper IDE for development and ditch any VIM/Emacs (mal-)practices.

Visual Studio Code

Here are some VSCode settings that will enable features for linting, some auto formating, line size ruler, etc.

{
    "python.venvPath": "${workspaceFolder}/.pyenv",
    "python.pythonPath": "${workspaceFolder}/.pyenv/bin/python",
    "git.ignoreLimitWarning": true,
    "editor.rulers": [90],
    "editor.renderWhitespace": "all",
    "editor.tabSize": 4,
    "[javascript]": {
        "editor.tabSize": 2
    },
    "files.trimTrailingWhitespace": true,
    "git.enableSmartCommit": true,
    "eslint.autoFixOnSave": true,
    "python.linting.pylintArgs": [
        "--load-plugins=pylint_mongoengine,nomad/metainfo/pylint_plugin",
    ],
    "python.linting.pep8Path": "pycodestyle",
    "python.linting.pep8Enabled": true,
    "python.linting.pep8Args": ["--ignore=E501,E701"],
    "python.linting.mypyEnabled": true,
    "python.linting.mypyArgs": [
        "--ignore-missing-imports",
        "--follow-imports=silent",
        "--no-strict-optional"
    ],
    "workbench.colorCustomizations": {
        "editorError.foreground": "#FF2222",
        "editorOverviewRuler.errorForeground": "#FF2222",
        "editorWarning.foreground": "#FF5500",
        "editorOverviewRuler.warningForeground": "#FF5500",
        "activityBar.background": "#4D2111",
        "titleBar.activeBackground": "#6B2E18",
        "titleBar.activeForeground": "#FDF9F7"
    },
    "files.watcherExclude": {
        "**/.git/objects/**": true,
        "**/.git/subtree-cache/**": true,
        "**/node_modules/*/**": true,
        "**/.pyenv/*/**": true,
        "**/__pycache__/*/**": true,
        "**/.mypy_cache/*/**": true,
        "**/.volumes/*/**": true,
        "**/docs/.build/*/**": true
    }
}

Here are some example launch configs for VSCode:

{
  "version": "0.2.0",
  "configurations": [
    {
      "type": "chrome",
      "request": "launch",
      "name": "Launch Chrome against localhost",
      "url": "http://localhost:3000",
      "webRoot": "${workspaceFolder}/gui"
    },
    {
      "name": "Python: API Flask (0.11.x or later)",
      "type": "python",
      "request": "launch",
      "module": "flask",
      "env": {
        "FLASK_APP": "nomad/app/__init__.py"
      },
      "args": [
        "run",
        "--port",
        "8000",
        "--no-debugger",
        "--no-reload"
      ]
    },
    {
      "name": "Python: some test",
      "type": "python",
      "request": "launch",
      "cwd": "${workspaceFolder}",
      "program": "${workspaceFolder}/.pyenv/bin/pytest",
      "args": [
        "-sv",
        "tests/test_cli.py::TestClient::test_mirror"
      ]
    },
    {
      "name": "Python: Current File",
      "type": "python",
      "request": "launch",
      "program": "${file}"
    },
    {
      "name": "Python: Attach",
      "type": "python",
      "request": "attach",
      "localRoot": "${workspaceFolder}",
      "remoteRoot": "${workspaceFolder}",
      "port": 3000,
      "secret": "my_secret",
      "host": "localhost"
    }
  ]
}