Command Line Interface (CLI)¶
nomad¶
This is the entry point to nomad's command line interface CLI. It uses a sub-command structure similar to the git command.
Usage:
Options:
-v, --verbose sets log level to info
--debug sets log level to debug
--log-label TEXT Label applied to logg entries.
--help Show this message and exit.
Subcommands
- admin: The nomad admin commands to do nasty stuff directly on the databases.
- client: Commands that use the nomad API to do useful things
- dev: Commands related to the nomad source code.
- parse: Run parsing and normalizing locally.
nomad admin¶
The nomad admin commands to do nasty stuff directly on the databases. Remember: With great power comes great responsibility!
Usage:
Options:
Subcommands
- clean: Checks consistency of files and es vs mongo and deletes orphan entries.
- entries: Entry related commands
- lift-embargo: Check and lift embargo of data with expired embargo period.
- ops: Generate scripts and commands for nomad operation.
- reset: Reset/remove all databases.
- reset-processing: Reset all uploads and entries "stuck" in processing using level mongodb operations.
- rewrite-doi-urls:
- run: Run a nomad service locally (outside docker).
- upgrade: Commands for upgrading to a newer NOMAD version
- uploads: Upload related commands
- users: Add, import, export users.
nomad admin clean¶
Checks consistency of files and es vs mongo and deletes orphan entries.
Usage:
Options:
--dry Do not delete anything, just check.
--skip-entries Skip cleaning entries with missing uploads.
--skip-fs Skip cleaning the filesystem.
--skip-es Skip cleaning the es index.
--staging-too Also clean published entries in staging, make sure these
files are not due to reprocessing
--force Do not ask for confirmation.
--help Show this message and exit.
nomad admin entries¶
Entry related commands
Usage:
Options:
Subcommands
- rm: Delete selected entries from mongo and elastic
nomad admin entries rm¶
Delete selected entries from mongo and elastic
Usage:
Options:
--skip-es Keep the elastic index version of the data.
--skip-mongo Keep uploads and entries in mongo.
--help Show this message and exit.
nomad admin lift-embargo¶
Check and lift embargo of data with expired embargo period.
Usage:
Options:
--dry Do not lift the embargo, just show what needs to be
done.
--parallel INTEGER Use the given amount of parallel processes. Default is
1.
--help Show this message and exit.
nomad admin ops¶
Generate scripts and commands for nomad operation.
Usage:
Options:
Subcommands
- dump: Dump the mongo db.
- nginx-conf: Generate an nginx.conf to serve the GUI and proxy pass to API container.
- prototypes-update: Updates the AFLOW prototype information using the latest online version and writes the results to a python module in the given FILEPATH.
- restore: Restore the mongo db.
- springer-update: Updates the springer database in nomad.config.normalize.springer_db_path.
nomad admin ops dump¶
Dump the mongo db.
Usage:
Options:
nomad admin ops nginx-conf¶
Generate an nginx.conf to serve the GUI and proxy pass to API container.
Usage:
Options:
--prefix TEXT Alter the url path prefix.
--host TEXT Alter the NOMAD app host.
--port TEXT Alter the NOMAD port host.
--server / --no-server Control writing of the outer server {} block. Useful
when conf file is included within another
nginx.conf.
--help Show this message and exit.
nomad admin ops prototypes-update¶
Updates the AFLOW prototype information using the latest online version and writes the results to a python module in the given FILEPATH.
Usage:
Options:
--matches-only Only update the match information that depends on the
symmetry analysis settings. Will not perform an online
update.
--help Show this message and exit.
nomad admin ops restore¶
Restore the mongo db.
Usage:
Options:
nomad admin ops springer-update¶
Updates the springer database in nomad.config.normalize.springer_db_path.
Usage:
Options:
--max-n-query INTEGER Number of unsuccessful springer request before
returning an error. Default is 10.
--retry-time INTEGER Time in seconds to retry after unsuccessful request.
Default is 120.
--help Show this message and exit.
nomad admin reset¶
Reset/remove all databases.
Usage:
Options:
--remove Do not just reset all dbs, but also remove them.
--i-am-really-sure Must be set for the command to to anything.
--help Show this message and exit.
nomad admin reset-processing¶
Reset all uploads and entries "stuck" in processing using level mongodb operations.
Usage:
Options:
nomad admin rewrite-doi-urls¶
Rewrites the existing dataset URLs in existing DOI records with freshly generated dataset URLs. This is useful, if the URL layout has changed.
Usage:
Options:
--dry Just test if DOI exists and print is current
URL.
--save-existing-records TEXT A filename to store the existing DOI records
in.
--help Show this message and exit.
nomad admin run¶
Run a nomad service locally (outside docker).
Usage:
Options:
Subcommands
- app: Run the nomad development app with all apis.
- appworker: Run both app and worker.
- hub: Run the jupyter hub.
- logtransfer: Run service to collect and submits logs to the central Nomad instance.
- worker: Run the nomad development worker.
nomad admin run app¶
Run the nomad development app with all apis.
Usage:
Options:
--with-gui The app will configure the gui for production and service
it.
--host TEXT Passed to uvicorn host parameter.
--port INTEGER Passed to uvicorn host parameter.
--log-config TEXT Passed to uvicorn log-config parameter.
--gunicorn Run app with gunicorn.
--workers INTEGER Passed to uvicorn workers parameter.
--help Show this message and exit.
nomad admin run appworker¶
Run both app and worker.
Usage:
Options:
nomad admin run hub¶
Run the jupyter hub.
Usage:
Options:
nomad admin run logtransfer¶
Run service to collect and submits logs to the central Nomad instance.
Usage:
Options:
nomad admin run worker¶
Run the nomad development worker.
Usage:
Options:
nomad admin upgrade¶
Commands for upgrading to a newer NOMAD version
Usage:
Options:
Subcommands
- migrate-mongo: Converts (upgrades) records from one mongodb and migrates to another.
nomad admin upgrade migrate-mongo¶
Converts (upgrades) records from one mongodb and migrates to another. Note, it is strongly recommended to run this command with loglevel verbose, i.e.
nomad -v upgrade migrate-mongo ...
Usage:
Options:
--host TEXT The mongodb host. By default same as the
configured db.
--port INTEGER The mongodb port. By default same as the
configured db.
--src-db-name TEXT The name of the source database. [required]
--dst-db-name TEXT The name of the destination database. By
default same as the configured db.
--upload-query TEXT An mongo upload query. All uploads matching
the query will be included in the migration.
--entry-query TEXT An mongo entry query. All uploads with an
entry matching the query will be included in
the migration.
--ids-from-file TEXT Reads upload IDs from the specified file.
Cannot be used together with the --upload-
query or --entry-query options. This can for
example be used to retry just the uploads
that has previously failed (as these ids can
be exported to file using --failed-ids-to-
file). You can specify both --ids-from-file
and --failed-ids-to-file at the same time
with the same file name.
--failed-ids-to-file TEXT Write the IDs of failed and skipped uploads
to the specified file. This can for example
be used to subsequently retry just the
uploads that failed (as these ids can be
loaded from file using --ids-from-file). You
can specify both --ids-from-file and
--failed-ids-to-file at the same time with
the same file name.
--upload-update TEXT json with updates to apply to all converted
uploads
--entry-update TEXT json with updates to apply to all converted
entries
--overwrite [always|if-newer|never]
If an upload already exists in the
destination db, this option determines
whether it and its child records should be
overwritten with the data from the source
db. Possible values are "always", "if-
newer", "never". Selecting "always" always
overwrites, "never" never overwrites, and
"if-newer" overwrites if the upload either
doesn't exist in the destination, or it
exists but its complete_time (i.e. last time
it was processed) is older than in the
source db.
--fix-problems If a minor, fixable problem is encountered,
fixes it automaticall; otherwise fail.
--dry Dry run (not writing anything to the
destination database).
--help Show this message and exit.
nomad admin uploads¶
Upload related commands
Usage:
Options:
--uploads-mongo-query TEXT A query
--entries-mongo-query TEXT A query
--entries-es-query TEXT A query
--unpublished Select only uploads in staging
--published Select only uploads that are publised
--outdated Select published uploads with older nomad
version
--processing Select only processing uploads
--processing-failure-uploads Select uploads with failed processing
--processing-failure-entries Select uploads with entries with failed
processing
--processing-failure Select uploads where the upload or any entry
has failed processing
--processing-incomplete-uploads
Select uploads that have not yet been
processed
--processing-incomplete-entries
Select uploads where any entry has net yot
been processed
--processing-incomplete Select uploads where the upload or any entry
has not yet been processed
--processing-necessary Select uploads where the upload or any entry
has either not been processed or processing
has failed in the past
--unindexed Select uploads that have no entries in the
elastic search index.
--help Show this message and exit.
Subcommands
- chown: Change the owner of the upload and all its entries.
- export: List selected uploads
- export-bundle: Export one or more uploads as bundles.
- import-bundle: Import one or more uploads from bundles. Unless specified by the user,
- index: (Re-)index all entries of the given uploads.
- integrity: Check certain integrity criteria
- ls: List selected uploads
- process: Reprocess selected uploads.
- re-pack: Repack selected uploads.
- reset: Reset the processing state.
- rm: Delete selected upload
- stop: Attempt to abort the processing of uploads.
nomad admin uploads chown¶
Change the owner of the upload and all its entries.
Usage:
Options:
nomad admin uploads export¶
List selected uploads
Usage:
Options:
--required TEXT The required in JSON format
-o, --output TEXT The file to write data to
--help Show this message and exit.
nomad admin uploads export-bundle¶
Export one or more uploads as bundles.
Usage:
Options:
--out-dir TEXT Output folder. Default value is "./bundles" (defined in
config)
--uncompressed Specify to export each bundle as an uncompressed
folder, instead of a zip-file.
--overwrite Specify to, for each bundle, overwrite the destination
file/folder if it already exists.
-s, --settings TEXT The export settings, specified as json. Settings not
specified in the dictionary will be set to the default
values.
-i, --ignore-errors Specify to ignore errors on individual uploads, and
continue exporting (the default behaviour is to abort
on first failing upload).
--help Show this message and exit.
nomad admin uploads import-bundle¶
Import one or more uploads from bundles. Unless specified by the user, the configured default import settings are used.
Usage:
Options:
--in TEXT The input path, specifying a bundle or a
folder containing multiple bundles.
-m, --multi Specify this flag if the input_path is a
folder containing multiple bundles, and all
these should be imported. If this option is
specified without specifying --in, we will
default the input path to ./bundles
-s, --settings TEXT The import settings, specified as json.
Settings not specified in the dictionary will
be set to the default values.
-e, --embargo_length INTEGER The embargo length (0-36 months). 0 means no
embargo. If unspecified, the embargo period
defined in the bundle will be used.
-c, --use-celery If specified, uses celery and the worker pool
to do the main part of the import. NOTE: this
requires that the workers can access the
bundle via the exact same path.
-i, --ignore-errors Specify this flag to ignore errors on
individual bundles, and continue importing
(the default behaviour is to abort on first
failing bundle).
--help Show this message and exit.
nomad admin uploads index¶
(Re-)index all entries of the given uploads.
Usage:
Options:
--parallel INTEGER Use the given amount of parallel processes.
Default is 1.
--transformer TEXT Qualified name to a Python function that should be
applied to each EntryMetadata.
--skip-materials Only update the entries index.
--print-progress INTEGER Prints a dot every given seconds. Can be used to
keep terminal open that have an i/o-based timeout.
--help Show this message and exit.
nomad admin uploads integrity¶
Check certain integrity criteria
Usage:
Options:
Subcommands
- entry-index: Uploads that have more entries in mongo than in ES.
nomad admin uploads integrity entry-index¶
Uploads that have more entries in mongo than in ES.
Usage:
Options:
nomad admin uploads ls¶
List selected uploads
Usage:
Options:
-e, --entries Show details about entries.
--ids Only show a list of ids.
--json Output a JSON array of ids.
--help Show this message and exit.
nomad admin uploads process¶
Reprocess selected uploads.
Usage:
Options:
--parallel INTEGER Use the given amount of parallel processes.
Default is 1.
--process-running Also reprocess already running processes.
--setting TEXT key=value to overwrite a default reprocess config
setting.
--print-progress INTEGER Prints a dot every given seconds. Can be used to
keep terminal open that have an i/o-based timeout.
--help Show this message and exit.
nomad admin uploads re-pack¶
Repack selected uploads.
Usage:
Options:
nomad admin uploads reset¶
Reset the processing state.
Usage:
Options:
--with-entries Also reset all entries.
--success Set the process status to success instead of pending
--failure Set the process status to failure instead of pending.
--help Show this message and exit.
nomad admin uploads rm¶
Delete selected upload
Usage:
Options:
--skip-es Keep the elastic index version of the data.
--skip-mongo Keep uploads and entries in mongo.
--skip-files Keep all related files.
--help Show this message and exit.
nomad admin uploads stop¶
Attempt to abort the processing of uploads.
Usage:
Options:
--entries Only stop entries processing.
--kill Use the kill signal and force task failure.
--no-celery Do not attempt to stop the actual celery tasks
--help Show this message and exit.
nomad admin users¶
Add, import, export users.
Usage:
Options:
Subcommands
- import: Import users to keycloak from a JSON file.
nomad admin users import¶
Import users to keycloak from a JSON file.
Usage:
Options:
nomad client¶
Commands that use the nomad API to do useful things
Usage:
Options:
-n, --url TEXT The URL where nomad is running, default is
"http://nomad-lab.eu/prod/v1/api".
-u, --user TEXT the user name to login, default is no login.
-w, --password TEXT the password used to login.
--token-via-api retrieve the access token from the api not keycloak.
--no-ssl-verify disables SSL verificaton when talking to nomad.
--help Show this message and exit.
Subcommands
- datatests: Metainfo compatibility tests against data in a NOMAD installation.
- integrationtests: Runs a few example operations as a test.
- local: Run processing locally.
- synchdb: Synchronizes the NOMAD database with the given external database.
- upload: Upload files to nomad. The given path can be a single file or a directory. All .zip files in a directory will be uploaded.
nomad client datatests¶
Metainfo compatibility tests against data in a NOMAD installation.
Usage:
Options:
nomad client integrationtests¶
Runs a few example operations as a test.
Usage:
Options:
--skip-parsers Skip extensive upload and parser tests.
--skip-publish Skip publish the upload. Should not be done on an production
environment.
--skip-doi Skip assigning a doi to a dataset.
--help Show this message and exit.
nomad client local¶
Run processing locally.
Usage:
Options:
--override Override existing local entry data.
--show-archive Print the archive data.
--show-metadata Print the extracted repo metadata.
--skip-normalizers Do not normalize.
--not-strict Also match artificial parsers.
--help Show this message and exit.
nomad client synchdb¶
Synchronizes the NOMAD database with the given external database.
Usage:
Options:
--outfile TEXT File to read/write files missing in NOMAD database
--nomadfile TEXT File to read/write files in NOMAD database
--dbfile TEXT File to read/write files in given database
--local_path TEXT Directory to which the files will be downloaded
--parallel INTEGER Number of processes to spawn to download/upload files
--do-download Flag to automatically download downloaded files
--do-upload Flag to automatically upload downloaded files
--do-publish Flag to automatically publish upload
--cleanup Flag to clean up downloaded files
--help Show this message and exit.
nomad client upload¶
Upload files to nomad. The given path can be a single file or a directory. All .zip files in a directory will be uploaded.
Usage:
Options:
--upload-name TEXT Optional name for the upload of a single file. Will be
ignored on directories.
--local-path Upload files "offline": files will not be uploaded, but
processed were they are. Only works when run on the
nomad host.
--publish Automatically move upload out of the staging area after
successful processing
--help Show this message and exit.
nomad dev¶
Commands related to the nomad source code.
Usage:
Options:
Subcommands
- example-data: Adds a few pieces of data to NOMAD.
- example-upload-metadata: Generates a JSON file from example-uploads metadata in the YAML file.
- gui-artifacts: Generates all python-based GUI artifacts into javascript code.
- gui-config: Generates the GUI development config JS file based on NOMAD config.
- gui-env: Generates the GUI development .env file based on NOMAD config.
- gui-qa: Runs tests and linting of the nomad gui source code. Useful before committing code.
- metainfo: Generates a JSON with all metainfo.
- parser-metadata: Generates a JSON file that compiles all the parser metadata from each parser project.
- qa: Runs tests and linting of the nomad python source code. Useful before committing code.
- search-quantities: Generates a JSON with all search quantities.
- toolkit-metadata: Generate toolkit tutorial metadata from analytics submodules.
- update-parser-readmes: Updates parser
s README files by combining a general template with a parsers metadata YAML file.
nomad dev example-data¶
Adds a few pieces of data to NOMAD.
Usage:
Options:
nomad dev example-upload-metadata¶
Generates a JSON file from example-uploads metadata in the YAML file.
Usage:
Options:
nomad dev gui-artifacts¶
Generates all python-based GUI artifacts into javascript code.
Usage:
Options:
nomad dev gui-config¶
Generates the GUI development config JS file based on NOMAD config.
Usage:
Options:
nomad dev gui-env¶
Generates the GUI development .env file based on NOMAD config.
Usage:
Options:
nomad dev gui-qa¶
Runs tests and linting of the nomad gui source code. Useful before committing code.
Usage:
Options:
nomad dev metainfo¶
Generates a JSON with all metainfo.
Usage:
Options:
nomad dev parser-metadata¶
Generates a JSON file that compiles all the parser metadata from each parser project.
Usage:
Options:
nomad dev qa¶
Runs tests and linting of the nomad python source code. Useful before committing code.
Usage:
Options:
--skip-tests Do no tests, just do code checks.
-x, --exitfirst Stop testing after first failed test case.
--help Show this message and exit.
nomad dev search-quantities¶
Generates a JSON with all search quantities.
Usage:
Options:
nomad dev toolkit-metadata¶
Generate toolkit tutorial metadata from analytics submodules.
Usage:
Options:
nomad dev update-parser-readmes¶
Updates parsers README files by combining a general template with a parsers metadata YAML file.
Usage:
Options:
--parser TEXT Only updated the README of the given parsers subdirctory.
--help Show this message and exit.
nomad parse¶
Run parsing and normalizing locally.
Usage:
Options:
--show-archive Print the archive data.
--archive-with-meta If the archvie is printed, it would include metadata
like m_def and m_annotations.
--show-metadata Print the extracted repo metadata.
--skip-normalizers Do not run the normalizer.
--not-strict Do also match artificial parsers.
--parser TEXT Skip matching and use the provided parser (format:
`parsers/<name>`). Additional selection rules still
apply for parsers with multiple main files.
--server-context Whether to use server context.
--username TEXT Username for authentication.
--password TEXT Password for authentication.
--help Show this message and exit.