Command Line Interface (CLI)¶
nomad¶
This is the entry point to nomad's command line interface CLI. It uses a sub-command structure similar to the git command.
Usage:
Options:
-v, --verbose sets log level to info
--debug sets log level to debug
--log-label TEXT Label applied to logg entries.
--help Show this message and exit.
Subcommands
- admin: The nomad admin commands to do nasty stuff directly on the databases.
- clean: Cleanse the given path by removing empty folders.
- client: Commands that use the nomad API to do useful things
- dev: Commands related to the nomad source code.
- parse: Run parsing and normalizing locally.
nomad admin¶
The nomad admin commands to do nasty stuff directly on the databases. Remember: With great power comes great responsibility!
Usage:
Options:
Subcommands
- clean: Checks consistency of files and es vs mongo and deletes orphan entries.
- entries: Entry related commands
- lift-embargo: Check and lift embargo of data with expired embargo period.
- ops: Generate scripts and commands for nomad operation.
- reset: Reset/remove all databases.
- reset-processing: Reset all uploads and entries "stuck" in processing using level mongodb operations.
- rewrite-doi-urls:
- run: Run a nomad service locally (outside docker).
- upgrade: Commands for upgrading to a newer NOMAD version
- uploads: Upload related commands
- users: Add, import, export users.
nomad admin clean¶
Checks consistency of files and es vs mongo and deletes orphan entries.
Usage:
Options:
--dry Do not delete anything, just check.
--skip-entries Skip cleaning entries with missing uploads.
--skip-fs Skip cleaning the filesystem.
--skip-es Skip cleaning the es index.
--staging-too Also clean published entries in staging, make sure these
files are not due to reprocessing
--force Do not ask for confirmation.
--help Show this message and exit.
nomad admin entries¶
Entry related commands
Usage:
Options:
Subcommands
- rm: Delete selected entries from mongo and elastic
nomad admin entries rm¶
Delete selected entries from mongo and elastic
Usage:
Options:
--skip-es Keep the elastic index version of the data.
--skip-mongo Keep uploads and entries in mongo.
--help Show this message and exit.
nomad admin lift-embargo¶
Check and lift embargo of data with expired embargo period.
Usage:
Options:
--dry Do not lift the embargo, just show what needs to be
done.
--parallel INTEGER Use the given amount of parallel processes. Default is
1.
--help Show this message and exit.
nomad admin ops¶
Generate scripts and commands for nomad operation.
Usage:
Options:
Subcommands
- dump: Dump the mongo db.
- nginx-conf: Generate an nginx.conf to serve the GUI and proxy pass to API container.
- prototypes-update: Updates the AFLOW prototype information using the latest online version and writes the results to a python module in the given FILEPATH.
- restore: Restore the mongo db.
- springer-update: Updates the springer database in nomad.config.normalize.springer_db_path.
nomad admin ops dump¶
Dump the mongo db.
Usage:
Options:
nomad admin ops nginx-conf¶
Generate an nginx.conf to serve the GUI and proxy pass to API container.
Usage:
Options:
--prefix TEXT Alter the url path prefix.
--host TEXT Alter the NOMAD app host.
--port TEXT Alter the NOMAD port host.
--server / --no-server Control writing of the outer server {} block. Useful
when conf file is included within another
nginx.conf.
--help Show this message and exit.
nomad admin ops prototypes-update¶
Updates the AFLOW prototype information using the latest online version and writes the results to a python module in the given FILEPATH.
Usage:
Options:
--matches-only Only update the match information that depends on the
symmetry analysis settings. Will not perform an online
update.
--help Show this message and exit.
nomad admin ops restore¶
Restore the mongo db.
Usage:
Options:
nomad admin ops springer-update¶
Updates the springer database in nomad.config.normalize.springer_db_path.
Usage:
Options:
--max-n-query INTEGER Number of unsuccessful springer request before
returning an error. Default is 10.
--retry-time INTEGER Time in seconds to retry after unsuccessful request.
Default is 120.
--help Show this message and exit.
nomad admin reset¶
Reset/remove all databases.
Usage:
Options:
--remove Do not just reset all dbs, but also remove them.
--i-am-really-sure Must be set for the command to to anything.
--help Show this message and exit.
nomad admin reset-processing¶
Reset all uploads and entries "stuck" in processing using level mongodb operations.
Usage:
Options:
nomad admin rewrite-doi-urls¶
Rewrites the existing dataset URLs in existing DOI records with freshly generated dataset URLs. This is useful, if the URL layout has changed.
Usage:
Options:
--dry Just test if DOI exists and print is current
URL.
--save-existing-records TEXT A filename to store the existing DOI records
in.
--help Show this message and exit.
nomad admin run¶
Run a nomad service locally (outside docker).
Usage:
Options:
Subcommands
- app: Run the nomad development app with all apis.
- appworker: Run both app and worker.
- hub: Run the jupyter hub.
- worker: Run the nomad development worker.
nomad admin run app¶
Run the nomad development app with all apis.
Usage:
Options:
--with-gui The app will configure the gui for production and service
it.
--host TEXT Passed as host parameter.
--port INTEGER Passed as port parameter.
--log-config TEXT Passed as log-config parameter.
--gunicorn Run app with gunicorn instead of uvicorn.
--workers INTEGER Passed to uvicorn workers parameter.
--help Show this message and exit.
nomad admin run appworker¶
Run both app and worker.
Usage:
Options:
--app-host TEXT Passed as app host parameter.
--app-port INTEGER Passed as app port parameter.
--fastapi-workers INTEGER Number of FastAPI workers.
--celery-workers INTEGER Number of Celery workers.
--dev Use one worker (for dev. env.).
--help Show this message and exit.
nomad admin run hub¶
Run the jupyter hub.
Usage:
Options:
nomad admin run worker¶
Run the nomad development worker.
Usage:
Options:
nomad admin upgrade¶
Commands for upgrading to a newer NOMAD version
Usage:
Options:
Subcommands
- migrate-mongo: Converts (upgrades) records from one mongodb and migrates to another.
nomad admin upgrade migrate-mongo¶
Converts (upgrades) records from one mongodb and migrates to another. Note, it is strongly recommended to run this command with loglevel verbose, i.e.
nomad -v upgrade migrate-mongo ...
Usage:
Options:
--host TEXT The mongodb host. By default same as the
configured db.
--port INTEGER The mongodb port. By default same as the
configured db.
--src-db-name TEXT The name of the source database. [required]
--dst-db-name TEXT The name of the destination database. By
default same as the configured db.
--upload-query TEXT An mongo upload query. All uploads matching
the query will be included in the migration.
--entry-query TEXT An mongo entry query. All uploads with an
entry matching the query will be included in
the migration.
--ids-from-file TEXT Reads upload IDs from the specified file.
Cannot be used together with the --upload-
query or --entry-query options. This can for
example be used to retry just the uploads
that has previously failed (as these ids can
be exported to file using --failed-ids-to-
file). You can specify both --ids-from-file
and --failed-ids-to-file at the same time
with the same file name.
--failed-ids-to-file TEXT Write the IDs of failed and skipped uploads
to the specified file. This can for example
be used to subsequently retry just the
uploads that failed (as these ids can be
loaded from file using --ids-from-file). You
can specify both --ids-from-file and
--failed-ids-to-file at the same time with
the same file name.
--upload-update TEXT json with updates to apply to all converted
uploads
--entry-update TEXT json with updates to apply to all converted
entries
--overwrite [always|if-newer|never]
If an upload already exists in the
destination db, this option determines
whether it and its child records should be
overwritten with the data from the source
db. Possible values are "always", "if-
newer", "never". Selecting "always" always
overwrites, "never" never overwrites, and
"if-newer" overwrites if the upload either
doesn't exist in the destination, or it
exists but its complete_time (i.e. last time
it was processed) is older than in the
source db.
--fix-problems If a minor, fixable problem is encountered,
fixes it automaticall; otherwise fail.
--dry Dry run (not writing anything to the
destination database).
--help Show this message and exit.
nomad admin uploads¶
Upload related commands
Usage:
Options:
--uploads-mongo-query TEXT A query
--entries-mongo-query TEXT A query
--entries-es-query TEXT A query
--unpublished Select only uploads in staging
--published Select only uploads that are publised
--outdated Select published uploads with older nomad
version
--processing Select only processing uploads
--processing-failure-uploads Select uploads with failed processing
--processing-failure-entries Select uploads with entries with failed
processing
--processing-failure Select uploads where the upload or any entry
has failed processing
--processing-incomplete-uploads
Select uploads that have not yet been
processed
--processing-incomplete-entries
Select uploads where any entry has net yot
been processed
--processing-incomplete Select uploads where the upload or any entry
has not yet been processed
--processing-necessary Select uploads where the upload or any entry
has either not been processed or processing
has failed in the past
--unindexed Select uploads that have no entries in the
elastic search index.
--help Show this message and exit.
Subcommands
- chown: Change the owner of the upload and all its entries.
- convert-archive: Convert selected uploads to the new format.
- export: List selected uploads
- export-bundle: Export one or more uploads as bundles.
- import-bundle: Import one or more uploads from bundles. Unless specified by the user,
- index: (Re-)index all entries of the given uploads.
- integrity: Check certain integrity criteria and return a list of upload IDs.
- ls: List selected uploads
- process: Reprocess selected uploads.
- publish: Publish selected uploads.
- re-pack: Repack selected uploads.
- reset: Reset the processing state.
- rm: Delete selected upload
- stop: Attempt to abort the processing of uploads.
nomad admin uploads chown¶
Change the owner of the upload and all its entries.
Usage:
Options:
nomad admin uploads convert-archive¶
Convert selected uploads to the new format.
Usage:
Options:
-o, --overwrite Overwrite existing target files.
-d, --delete-old Delete old archives once successfully converted.
-m, --migrate Only convert v1 archive files to v1.2 archive
files.
-f, --force-repack Force repacking existing archives that are already
in the new format
-p, --parallel INTEGER Number of processes to use for conversion. Default
is os.cpu_count().
-s, --size-limit INTEGER Only handle archives under limited size in GB.
Default is -1 (no limit).
--help Show this message and exit.
nomad admin uploads export¶
List selected uploads
Usage:
Options:
--required TEXT The required in JSON format
-o, --output TEXT The file to write data to
--help Show this message and exit.
nomad admin uploads export-bundle¶
Export one or more uploads as bundles.
Usage:
Options:
--out-dir TEXT Output folder. Default value is "./bundles" (defined in
config)
--uncompressed Specify to export each bundle as an uncompressed
folder, instead of a zip-file.
--overwrite Specify to, for each bundle, overwrite the destination
file/folder if it already exists.
-s, --settings TEXT The export settings, specified as json. Settings not
specified in the dictionary will be set to the default
values.
-i, --ignore-errors Specify to ignore errors on individual uploads, and
continue exporting (the default behaviour is to abort
on first failing upload).
--help Show this message and exit.
nomad admin uploads import-bundle¶
Import one or more uploads from bundles. Unless specified by the user, the configured default import settings are used.
Usage:
Options:
--in TEXT The input path, specifying a bundle or a
folder containing multiple bundles.
-m, --multi Specify this flag if the input_path is a
folder containing multiple bundles, and all
these should be imported. If this option is
specified without specifying --in, we will
default the input path to ./bundles
-s, --settings TEXT The import settings, specified as json.
Settings not specified in the dictionary will
be set to the default values.
-e, --embargo_length INTEGER The embargo length (0-36 months). 0 means no
embargo. If unspecified, the embargo period
defined in the bundle will be used.
-c, --use-celery If specified, uses celery and the worker pool
to do the main part of the import. NOTE: this
requires that the workers can access the
bundle via the exact same path.
-i, --ignore-errors Specify this flag to ignore errors on
individual bundles, and continue importing
(the default behaviour is to abort on first
failing bundle).
--help Show this message and exit.
nomad admin uploads index¶
(Re-)index all entries of the given uploads.
Usage:
Options:
--parallel INTEGER Use the given amount of parallel processes.
Default is 1.
--transformer TEXT Qualified name to a Python function that should be
applied to each EntryMetadata.
--skip-materials Only update the entries index.
--print-progress INTEGER Prints a dot every given seconds. Can be used to
keep terminal open that have an i/o-based timeout.
--help Show this message and exit.
nomad admin uploads integrity¶
Check certain integrity criteria and return a list of upload IDs.
Usage:
Options:
--both-storages Select uploads that have both staging and public
versions.
--missing-storage Select uploads of which the corresponding raw
folder (for staging) or the raw zip archive (for
public) is missing in the file system. This only
checks the existence of folder/archive uploaded by
user. To check the contents, use the --missing-
raw-files flag.
--missing-raw-files Select uploads that any of the files listed in
metadata/files is missing. Use --check-all-entries
to check files for all entries in the upload. It
uses the indexed ES data and does not open the
msgpack archive files.
--missing-archive-files Select uploads that miss archive (msgpack) files.
--missing-index Select uploads of which the ES index information
is missing.
--entry-mismatch Select uploads that have different numbers of
entries in mongo and ES.
--nomad-version-mismatch Select uploads that have different nomad versions
in archive and ES.
--old-archive-format Select uploads that are using the old archive
format (v1).
--not-preferred-suffix Select uploads that are using the preferred
(first) suffix in the configuration.
--check-all-entries Check all entries in the upload, otherwise only
check one entry per upload.
--help Show this message and exit.
nomad admin uploads ls¶
List selected uploads
Usage:
Options:
-e, --entries Include details about upload entries in the output.
--ids Only include the upload ids in the output.
--json Output a JSON array instead of a tabulated list.
--size INTEGER Controls the maximum size of returned uploads, use -1 to
return all.
--help Show this message and exit.
nomad admin uploads process¶
Reprocess selected uploads.
Usage:
Options:
--parallel INTEGER Use the given amount of parallel processes.
Default is 1.
--process-running Also reprocess already running processes.
--setting TEXT key=value to overwrite a default reprocess config
setting.
--print-progress INTEGER Prints a dot every given seconds. Can be used to
keep terminal open that have an i/o-based timeout.
--help Show this message and exit.
nomad admin uploads publish¶
Publish selected uploads.
Usage:
Options:
--parallel INTEGER Use the given amount of parallel processes.
Default is 1.
--embargo-length INTEGER Use an embargo length (months) for the
publication.
--help Show this message and exit.
nomad admin uploads re-pack¶
Repack selected uploads.
Usage:
Options:
nomad admin uploads reset¶
Reset the processing state.
Usage:
Options:
--with-entries Also reset all entries.
--success Set the process status to success instead of pending
--failure Set the process status to failure instead of pending.
--help Show this message and exit.
nomad admin uploads rm¶
Delete selected upload
Usage:
Options:
--skip-es Keep the elastic index version of the data.
--skip-mongo Keep uploads and entries in mongo.
--skip-files Keep all related files.
--help Show this message and exit.
nomad admin uploads stop¶
Attempt to abort the processing of uploads.
Usage:
Options:
--entries Only stop entries processing.
--kill Use the kill signal and force task failure.
--no-celery Do not attempt to stop the actual celery tasks
--help Show this message and exit.
nomad admin users¶
Add, import, export users.
Usage:
Options:
Subcommands
- import: Import users to keycloak from a JSON file.
nomad admin users import¶
Import users to keycloak from a JSON file.
Usage:
Options:
nomad clean¶
Cleanse the given path by removing empty folders.
Usage:
Options:
nomad client¶
Commands that use the nomad API to do useful things
Usage:
Options:
-n, --url TEXT The URL where nomad is running, default is
"http://nomad-lab.eu/prod/v1/api".
-u, --user TEXT the user name to login, default is no login.
-w, --password TEXT the password used to login.
--token-via-api retrieve the access token from the api not keycloak.
--no-ssl-verify disables SSL verificaton when talking to nomad.
--help Show this message and exit.
Subcommands
- datatests: Metainfo compatibility tests against data in a NOMAD installation.
- integrationtests: Runs a few example operations as a test.
- local: Run processing locally.
- synchdb: Synchronizes the NOMAD database with the given external database.
- upload: Upload files to nomad. The given path can be a single file or a directory. For file(s) that are not .zip or .tar files, a temporary .zip file will be created and uploaded.
nomad client datatests¶
Metainfo compatibility tests against data in a NOMAD installation.
Usage:
Options:
nomad client integrationtests¶
Runs a few example operations as a test.
Usage:
Options:
--skip-parsers Skip extensive upload and parser tests.
--skip-publish Skip publish the upload. Should not be done on an production
environment.
--skip-doi Skip assigning a doi to a dataset.
--help Show this message and exit.
nomad client local¶
Run processing locally.
Usage:
Options:
--override Override existing local entry data.
--show-archive Print the archive data.
--show-metadata Print the extracted repo metadata.
--skip-normalizers Do not normalize.
--not-strict Also match artificial parsers.
--help Show this message and exit.
nomad client synchdb¶
Synchronizes the NOMAD database with the given external database.
Usage:
Options:
--outfile TEXT File to read/write files missing in NOMAD database
--nomadfile TEXT File to read/write files in NOMAD database
--dbfile TEXT File to read/write files in given database
--local_path TEXT Directory to which the files will be downloaded
--parallel INTEGER Number of processes to spawn to download/upload files
--do-download Flag to automatically download downloaded files
--do-upload Flag to automatically upload downloaded files
--do-publish Flag to automatically publish upload
--cleanup Flag to clean up downloaded files
--help Show this message and exit.
nomad client upload¶
Upload files to nomad. The given path can be a single file or a directory. For file(s) that are not .zip or .tar files, a temporary .zip file will be created and uploaded.
Usage:
Options:
--upload-name TEXT Optional name for the upload of a single file. Will be
ignored on directories.
--local-path Upload files "offline": files will not be uploaded,
but processed were they are. Only works when run on
the nomad host.
--publish Automatically move upload out of the staging area
after successful processing
--ignore-path-prefix Ignores common path prefixes when creating an upload.
--help Show this message and exit.
nomad dev¶
Commands related to the nomad source code.
Usage:
Options:
Subcommands
- api-model: Export an API model in JSON schema.
- example-data: Adds a few pieces of data to NOMAD.
- gui-artifacts: Generates all python-based GUI artifacts into javascript code.
- gui-config: Generates the GUI development config JS file based on NOMAD config.
- gui-env: Generates the GUI development .env file based on NOMAD config.
- gui-qa: Runs tests and linting of the nomad gui source code. Useful before committing code.
- metainfo: Generates a JSON with all metainfo.
- parser-metadata: Generates a JSON file that compiles all the parser metadata from each parser project.
- qa: Runs tests and linting of the nomad python source code. Useful before committing code.
- search-quantities: Generates a JSON with all search quantities.
- update-parser-readmes: Updates parser
s README files by combining a general template with a parsers metadata YAML file.
nomad dev api-model¶
Export an API model in JSON schema.
Usage:
Options:
nomad dev example-data¶
Adds a few pieces of data to NOMAD.
Usage:
Options:
nomad dev gui-artifacts¶
Generates all python-based GUI artifacts into javascript code.
Usage:
Options:
nomad dev gui-config¶
Generates the GUI development config JS file based on NOMAD config.
Usage:
Options:
nomad dev gui-env¶
Generates the GUI development .env file based on NOMAD config.
Usage:
Options:
nomad dev gui-qa¶
Runs tests and linting of the nomad gui source code. Useful before committing code.
Usage:
Options:
nomad dev metainfo¶
Generates a JSON with all metainfo.
Usage:
Options:
nomad dev parser-metadata¶
Generates a JSON file that compiles all the parser metadata from each parser project.
Usage:
Options:
nomad dev qa¶
Runs tests and linting of the nomad python source code. Useful before committing code.
Usage:
Options:
--skip-tests Do no tests, just do code checks.
-x, --exitfirst Stop testing after first failed test case.
--help Show this message and exit.
nomad dev search-quantities¶
Generates a JSON with all search quantities.
Usage:
Options:
nomad dev update-parser-readmes¶
Updates parsers README files by combining a general template with a parsers metadata YAML file.
Usage:
Options:
--parser TEXT Only updated the README of the given parsers subdirctory.
--help Show this message and exit.
nomad parse¶
Run parsing and normalizing locally.
Usage:
Options:
--show-archive Print the archive data.
--archive-with-meta If the archvie is printed, it would include
metadata like m_def and m_annotations.
--show-metadata Print the extracted repo metadata.
--preview-plots Preview generated plots
--skip-normalizers Do not run the normalizer.
--not-strict Do also match artificial parsers.
--parser TEXT Skip matching and use the provided parser
(format: `parsers/<name>`). Additional selection
rules still apply for parsers with multiple main
files.
--server-context Whether to use server context.
--username TEXT Username for authentication.
--password TEXT Password for authentication.
--save-plot-dir DIRECTORY Directory to save the plot
--help Show this message and exit.