CLI Reference¶

Parse CLI¶

nomad parse¶

Run parsing and normalizing locally.

nomad parse [OPTIONS] MAINFILE

Options

--show-archive¶: Print the archive data.

--show-metadata¶: Print the extracted repo metadata.

--skip-normalizers¶: Do not run the normalizer.

--not-strict¶: Do also match artificial parsers.

--parser <parser>¶: Skip matching and use the provided parser

Arguments

MAINFILE¶: Required argument

Client CLI commands¶

nomad client¶

Commands that use the nomad API to do useful things

nomad client [OPTIONS] COMMAND [ARGS]...

Options

-n, --url <url>¶: The URL where nomad is running, default is “http://nomad-lab.eu/prod/rae/api”.

-u, --user <user>¶: the user name to login, default is “leonard.hofstadter@nomad-fairdi.tests.de” login.

-w, --password <password>¶: the password used to login.

--no-ssl-verify¶: disables SSL verificaton when talking to nomad.

--no-token¶: replaces token with basic auth, e.g. to work with v0.6.x or older API versions

integrationtests¶

Runs a few example operations as a test.

nomad client integrationtests [OPTIONS]

Options

--skip-parsers¶: Skip extensive upload and parser tests.

--skip-publish¶: Skip publish the upload. Should not be done on an production environment.

--skip-doi¶: Skip assigning a doi to a dataset.

--skip-mirror¶: Skip get mirror tests.

local¶

Run processing locally.

nomad client local [OPTIONS] CALC_ID

Options

--override¶: Override existing local calculation data.

--show-archive¶: Print the archive data.

--show-metadata¶: Print the extracted repo metadata.

--mainfile <mainfile>¶: Use this mainfile (in case mainfile cannot be retrived via API.

--skip-normalizers¶: Do not normalize.

--not-strict¶: Also match artificial parsers.

Arguments

CALC_ID¶: Required argument

mirror¶

Mirror data from another nomad deployment.

It uses its ‘client’ nature to export data from the other nomad (source) via REST API. I.e., use the -n (–url) client parameter to specify the source deployment.

The data will be added directly to the databases, filesystem, etc. of ‘this’ nomad deployment (target), i.e. it be haves like an ‘admin’ command. This means you either run it in the environement of the target deployment or use the –config nomad parameter.

nomad client mirror [OPTIONS] [QUERY]

Options

--move¶: Instead of copying the underlying upload files, we move it and replace it with a symlink.

--link¶: Instead of copying the underlying upload files, we create symlinks in the target.

--source-mapping <source_mapping>¶: A mapping in the form “dir:mapped” that replaces path prefix “dir” with “mapped” in all paths provided by the API of the source. Allows to handle local mounts paths in source deployment. E.g. use “.volumes/fs:/nomad/fairdi/<source>/fs”.

--target-mapping <target_mapping>¶: A mapping in the form “dir:mapped” that replaces path prefix “mirror” with “mapped” in all paths used for the target. Allows to handle local mounts paths in target deployment. E.g. use “.volumes/fs:/nomad/fairdi/<target>/fs”.

--dry¶: Do not actually mirror data, just fetch data and report.

--files-only¶: Will only copy/move files and not even look at the calculations. Useful, when moving metadata via mongo dump/restore.

--skip-files¶: Will not copy/move/link any files.

--migration <migration>¶: The name of a migration script used to transform the metadata.

--skip-es¶: Do not add mirrored data to elastic search

--staging¶: Mirror non published uploads. Only works with –move or –link.

--replace¶: Replace existing uploads.

Arguments

QUERY¶: Optional argument

statistics-plot¶

Generate various matplotlib charts

nomad client statistics-plot [OPTIONS]

Options

--errors¶: Two charts with relative and absolute parser/normalizer errors per code.

--x-axis <x_axis>¶: Aggregation used for x-axis, values are “code” and “time”.

--y-axis <y_axis>¶: Metrics used for y-axis, values are “entries”, “energies”, “calculations”, “uploaders”.

--cumulate¶: Cumulate over x-axis.

--title <title>¶: Override chart title with given value.

--total¶: Provide total sums of key metrics.

--save <save>¶: Save to given file instead of showing the plot.

--power <power>¶: User power scale instead of log with the given inverse power.

--open-access¶: Only consider Open-Access data.

--minimum <minimum>¶: Only consider codes with at least the given ammount of entries.

statistics-table¶

Generate table with basic statistics summary.

nomad client statistics-table [OPTIONS]

Options

--html¶: Output HTML instead of plain text table.

--geometries¶: Use geometries not unique geometries.

--public-path <public_path>¶: The path to the public data. Default is .volumes/fs/public.

synchdb¶

Synchronizes the NOMAD database with the given external database.

nomad client synchdb [OPTIONS] DB_NAME ROOT_URL

Options

--outfile <outfile>¶: File to read/write files missing in NOMAD database

--nomadfile <nomadfile>¶: File to read/write files in NOMAD database

--dbfile <dbfile>¶: File to read/write files in given database

--parallel <parallel>¶: Number of processes to spawn to download/upload files

--do-download¶: Flag to automatically download downloaded files

--do-upload¶: Flag to automatically upload downloaded files

--do-publish¶: Flag to automatically publish upload

--cleanup¶: Flag to clean up downloaded files

Arguments

DB_NAME¶: Required argument

ROOT_URL¶: Required argument

upload¶

Upload files to nomad. The given path can be a single file or a directory. All .zip files in a directory will be uploaded.

nomad client upload [OPTIONS] PATH...

Options

--name <name>¶: Optional name for the upload of a single file. Will be ignored on directories.

--offline¶: Upload files “offline”: files will not be uploaded, but processed were they are. Only works when run on the nomad host.

--publish¶: Automatically move upload out of the staging area after successful processing

Arguments

PATH¶: Required argument(s)

Admin CLI commands¶

nomad admin¶

The nomad admin commands to do nasty stuff directly on the databases. Remember: With great power comes great responsibility!

nomad admin [OPTIONS] COMMAND [ARGS]...

clean¶

Checks consistency of files and es vs mongo and deletes orphan entries.

nomad admin clean [OPTIONS]

Options

--dry¶: Do not delete anything, just check.

--skip-calcs¶: Skip cleaning calcs with missing uploads.

--skip-fs¶: Skip cleaning the filesystem.

--skip-es¶: Skip cleaning the es index.

--staging-too¶: Also clean published entries in staging, make sure these files are not due to reprocessing

--force¶: Do not ask for confirmation.

entries¶

Entry related commands

nomad admin entries [OPTIONS] COMMAND [ARGS]...

rm¶

Delete selected entries from mongo and elastic

nomad admin entries rm [OPTIONS] [ENTRIES]...

Options

--skip-es¶: Keep the elastic index version of the data.

--skip-mongo¶: Keep uploads and calcs in mongo.

Arguments

ENTRIES¶: Optional argument(s)

index-materials¶

(Re-)index all materials.

This command will is used to completely rebuild the materials index or parts of it. You can choose which source is used for building the index: In general ElasticSearch is preferred, but if it is not available also MongoDB can be used.

By default this command will start to rebuild the index from scratch. This can be done on a “live” system because a new temporary index is used. If you use the –in-place option, the indexing will be run on the same index that is currently in use.

nomad admin index-materials [OPTIONS]

Options

--threads <threads>¶: Number of threads to use.

--code <code>¶: Index only calculcations of given codes.

--dry¶: Do not index, just compute entries.

-n <n>¶: Number of calculations to process. Leave undefined to process all calculations.

--source <source>¶

Options: mongo | es

lift-embargo¶

Check and lift embargo of data with expired embargo period.

nomad admin lift-embargo [OPTIONS]

Options

--dry¶: Do not lift the embargo, just show what needs to be done.

--parallel <parallel>¶: Use the given amount of parallel processes. Default is 1.

migrate¶

Migrate data from older NOMAD versions

nomad admin migrate [OPTIONS]

Options

--mongo-db <mongo_db>¶: The database name of the existing data

ops¶

Generate scripts and commands for nomad operation.

nomad admin ops [OPTIONS] COMMAND [ARGS]...

dump¶

Dump the mongo (calculation metadata) db.

nomad admin ops dump [OPTIONS]

Options

--restore¶: Do not dump, but restore.

gui-config¶

Configures the GUIs based on NOMAD config.

nomad admin ops gui-config [OPTIONS]

nginx-conf¶

Generate an nginx.conf to serve the GUI and proxy pass to API container.

nomad admin ops nginx-conf [OPTIONS]

Options

--prefix <prefix>¶: Alter the url path prefix.

--host <host>¶: Alter the NOMAD app host.

--port <port>¶: Alter the NOMAD port host.

--server, --no-server¶: Control writing of the outer server {} block. Useful when conf file is included within another nginx.conf.

prototypes-update¶

Updates the AFLOW prototype information using the latest online version and writes the results to a python module in the given FILEPATH.

nomad admin ops prototypes-update [OPTIONS] FILEPATH

Options

--matches-only¶: Only update the match information that depends on the symmetry analysis settings. Will not perform and online update.

Arguments

FILEPATH¶: Required argument

restore¶

Restore the mongo (calculation metadata) db.

nomad admin ops restore [OPTIONS] PATH_TO_DUMP

Arguments

PATH_TO_DUMP¶: Required argument

similarity¶

Tools for managing the DOS similarity data.

nomad admin ops similarity [OPTIONS] COMMAND [ARGS]...

ingest¶

Ingests the given similarity information from an msgpack file into MongoDB.

nomad admin ops similarity ingest [OPTIONS]

Options

-i, --in <input_path>¶: Path of the ingested msgpack file.

--batch_size <batch_size>¶: Batch size for MongoDB bulk ingestion.

--verbose¶: Enable verbose output.

update¶

Updates the msgpack file containing the similarity information.

nomad admin ops similarity update [OPTIONS]

Options

-d, --dir <input_dir>¶: Path of the folder containing the raw similarity information files

-o, --out <out>¶: Path of the output msgpack file.

--verbose¶: Enable verbose output.

springer-update¶

Updates the springer database in nomad.config.normalize.springer_db_path.

nomad admin ops springer-update [OPTIONS]

Options

--max-n-query <max_n_query>¶: Number of unsuccessful springer request before returning an error. Default is 10.

--retry-time <retry_time>¶: Time in seconds to retry after unsuccessful request. Default is 120.

reset¶

Reset/remove all databases.

nomad admin reset [OPTIONS]

Options

--remove¶: Do not just reset all dbs, but also remove them.

--i-am-really-sure¶: Must be set for the command to to anything.

reset-processing¶

Reset all “stuck” in processing uploads and calc in low level mongodb operations.

nomad admin reset-processing [OPTIONS]

Options

--zero-complete-time¶: Sets the complete time to epoch zero.

run¶

Run a nomad service locally (outside docker).

nomad admin run [OPTIONS] COMMAND [ARGS]...

app¶

Run the nomad development app with all apis.

nomad admin run app [OPTIONS]

Options

--debug¶: Does run flask in debug.

--with-chaos <with_chaos>¶: Enable a certain percentage of chaos.

appworker¶

Run both app and worker.

nomad admin run appworker [OPTIONS]

worker¶

Run the nomad development worker.

nomad admin run worker [OPTIONS]

uploads¶

Upload related commands

nomad admin uploads [OPTIONS] COMMAND [ARGS]...

Options

--user <user>¶: Select uploads of user with given id

--unpublished¶: Select only uploads in staging

--published¶: Select only uploads that are publised

--outdated¶: Select published uploads with older nomad version

--code <code>¶: Select only uploads with calcs of given codes

--query-mongo¶: Select query mongo instead of elastic search.

--processing¶: Select only processing uploads

--processing-failure-uploads¶: Select uploads with failed processing

--processing-failure-calcs¶: Select uploads with calcs with failed processing

--processing-failure¶: Select uploads where the upload or any calc has failed processing

--processing-incomplete-uploads¶: Select uploads that have not yet been processed

--processing-incomplete-calcs¶: Select uploads where any calc has net yot been processed

--processing-incomplete¶: Select uploads where the upload or any calc has not yet been processed

--processing-necessary¶: Select uploads where the upload or any calc has either not been processed or processing has failed in the past

--unindexed¶: Select uploads that have no calcs in the elastic search index.

chown¶

Change the owner of the upload and all its calcs.

nomad admin uploads chown [OPTIONS] USERNAME [UPLOADS]...

Arguments

USERNAME¶: Required argument

UPLOADS¶: Optional argument(s)

edit¶

Allows to edit metadata attribute of all entries in uploads. Be aware that this only edits the attributes. E.g. if you set publish true, it won’t publish the upload, pack its files, change the upload metadata, etc.

nomad admin uploads edit [OPTIONS] [UPLOADS]...

Options

--publish <publish>¶

Set the publish attribute true and change with_embargo attribute.

Options: with-embargo | no-embargo

--unpublish¶: Set the publish attribute to false.

Arguments

UPLOADS¶: Optional argument(s)

index¶

(Re-)index all calcs of the given uploads.

nomad admin uploads index [OPTIONS] [UPLOADS]...

Options

--parallel <parallel>¶: Use the given amount of parallel processes. Default is 1.

--transformer <transformer>¶: Qualified name to a Python function that should be applied to each EntryMetadata.

Arguments

UPLOADS¶: Optional argument(s)

integrity¶

Check certain integrity criteria

nomad admin uploads integrity [OPTIONS] COMMAND [ARGS]...

dois¶

Uploads that have datasets with DOIs that do not exist.

nomad admin uploads integrity dois [OPTIONS] [UPLOADS]...

Arguments

UPLOADS¶: Optional argument(s)

ls¶

List selected uploads

nomad admin uploads ls [OPTIONS] [UPLOADS]...

Options

-c, --calculations¶: Show details about calculations.

--ids¶: Only show a list of ids.

--json¶: Output a JSON array of ids.

Arguments

UPLOADS¶: Optional argument(s)

prepare-migration¶

Prepares files for being used in the upcoming NOMAD v1.0.

nomad admin uploads prepare-migration [OPTIONS] [UPLOADS]...

Options

--dry¶: Just check, do nothing.

-f, --force¶: Ignore warnings and perform the operation regardless.

-q, --quiet¶: No output (only logs).

--upload-ids¶: Print uploads with errors.

--label <label>¶: A label to label log entries with.

Arguments

UPLOADS¶: Optional argument(s)

quarantine-raw-files¶

Moves certain files from public or restricted to quarantine in published uploads.

nomad admin uploads quarantine-raw-files [OPTIONS] [UPLOADS]...

Options

--file-pattern <file_pattern>¶: The files as .zip patterns, e.g. “*/POTCAR”. Default is all possible POTCAR patterns.

--dry¶: Just check, do nothing.

Arguments

UPLOADS¶: Optional argument(s)

re-pack¶

Repack selected uploads.

nomad admin uploads re-pack [OPTIONS] [UPLOADS]...

Options

--parallel <parallel>¶: Use the given amount of parallel processes. Default is 1.

Arguments

UPLOADS¶: Optional argument(s)

re-process¶

Reprocess selected uploads.

nomad admin uploads re-process [OPTIONS] [UPLOADS]...

Options

--parallel <parallel>¶: Use the given amount of parallel processes. Default is 1.

--reprocess-running¶: Also reprocess already running processes.

Arguments

UPLOADS¶: Optional argument(s)

reset¶

Reset the processing state.

nomad admin uploads reset [OPTIONS] [UPLOADS]...

Options

--with-calcs¶: Also reset all calculations.

--success¶: Set the tasks status to success instead of pending

--failure¶: Set the tasks status to failure instead of pending.

Arguments

UPLOADS¶: Optional argument(s)

rm¶

Delete selected upload

nomad admin uploads rm [OPTIONS] [UPLOADS]...

Options

--skip-es¶: Keep the elastic index version of the data.

--skip-mongo¶: Keep uploads and calcs in mongo.

--skip-files¶: Keep all related files.

Arguments

UPLOADS¶: Optional argument(s)

stop¶

Attempt to abort the processing of uploads.

nomad admin uploads stop [OPTIONS] [UPLOADS]...

Options

--calcs¶: Only stop calculation processing.

--kill¶: Use the kill signal and force task failure.

--no-celery¶: Do not attempt to stop the actual celery tasks

Arguments

UPLOADS¶: Optional argument(s)

users¶

Add, import, export users.

nomad admin users [OPTIONS] COMMAND [ARGS]...

import¶

Import users to keycloak from a JSON file.

nomad admin users import [OPTIONS] PATH_TO_USERS_FILE

Arguments

PATH_TO_USERS_FILE¶: Required argument