Write NOMAD Schemas in YAML¶
This guide explains how to write and upload NOMAD schemas in our
.archive.yaml format. For more information on how an archive file is composed, visit the learn section on schemas.
Let's assume we want to describe chemical compositions via elements contained in a
The following structured data (in this example as a
.yaml document) could describe the composition of water.
In structured data formats (such as
.json), data is put into combinations
of primitive values (e.g.
1.141), objects (a set of keys and value pairs, where values can be objects, lists, or primitive values), or lists of values.
In a schema, we want to describe the structure of data, i.e. what are the allowed combinations of objects, lists, and primitive values. The key element here is to define what keys certain types of objects can have and what the possible values for each key might be.
In NOMAD, we call objects sections and we define types of objects with section definitions. Since objects can be nested, sections become like the sections and sub-sections of a book or paper. Sections are a representation of data and they are the building blocks for archives. Section definitions form a schema and they are the building blocks for the metainfo.
In the above example, we have to types of objects, one for elements (with keys for
isotopes) and the overall
object for structures (with keys for
elements). Let's start with
the definition for elements. This is what the section definition looks like in NOMAD's yaml-based schema format:
A section definition provides all the available keys for a section that instantiates
this definition. For each key, e.g.
isotopes, it provides
more information on the possible values.
Let's have a look at the overall definition for our chemical composition:
Again, all possible keys (
elements) are defined. But now we see
that there are two different types of keys, quantities and sub-sections. We
say that section definitions can have properties (e.g. the keys they define) and
there are two distinct types of properties.
Quantities define possible primitive values. The basic properties that go into a quantity definition are:
- type: what kind of primitive value can be used, e.g.
- shape: what is the shape of the value, e.g. scalar or list (
- unit: what is the physical meaning of the value
The names of quantity definitions serve as the key, used in respective section objects.
This is a list of supported quantity types.
||Numpy based integer with 32 bits.|
||Numpy based integer with 64 bits.|
||Numpy based float with 32 bits.|
||Numpy based float with 64 bits.|
||A type for NOMAD users as values.|
||A complex type for author information.|
|To define a quantity that is a reference to a specific section.|
The shape of a quantitiy is a list of dimensions, where each dimension defines the possible size of that dimension. The empty list (or no shape) describes a scalar value, a list with one dimension a list or vector, a list with two dimensions a matrix, etc.
Dimensions can be given as:
- an integer number to define a fixed size, e.g. a 3x3 matrix would have shape
- the string
'*'to denote am arbitrary sized dimension, e.g. a list quantity would have shape
- A string that describes the name of a sibling quantity with an integer type, e.g.
NOMAD manages units and data with units via the Pint Python package. A unit is given as a string that is parsed by pint. These strings can
be simple units (or their aliases) or complex expressions. Here are a few examples:
While you can use all kinds of units in your uploaded schemas, the built-in NOMAD schema (Metainfo) uses only SI units.
Sub-sections define a part-of-relationship between two sections. Sub-section definitions are properties of the parent section definition and name a child
section definition. In the data, we can now contain instances of the target (e.g.
Element) in instances of the source (e.g.
Composition). A sub-section can be
defined as repeating to allow many child sections of the same type. In our example,
Composition can contain many
The names of sub-section definitions serve as the key, used in respective section objects.
NOMAD archive files allow you to upload data in NOMAD's native file format. An archive
file can be a .yaml or .json file. It ends with
Archive files are mainly used to convey data. Since schemas are also "just" data, archive
files can also be used to convey a schema.
You can upload schemas and data in separate files.
Or, you can upload schemas and data in the same file:
definitions: sections: Element: quantities: label: type: str density: type: np.float64 unit: g/cm**3 isotopes: type: int shape: ['*'] Composition: quantities: composition: type: str sub_sections: elements: section: Element repeats: true data: m_def: Composition composition: H2O elements: - label: H density: 8.375e-05 isotopes: [1, 2, 3] - label: O density: 1.141 isotopes: [16, 17, 18]
We already saw that we can define a part-of relationship between sections. When we want to represent highly inter-linked data, this is often insufficient. References allow us to create a more lose relationship between sections.
A reference is a uni-directional link between a source section and a target section. References can be defined in a schema as a quantity in the source section definition that uses the target section definition as a type.
Instead of connecting the elements in a composition with sub-sections, we can also connect a composition section to elements with a quantity:
type: Element refers to the section definition
Element, very similar to
section: Element in a sub-section definition.
We saw above that sub-sections are represented as nested objects in data (forcing a
part-of relationship). References are represented as string-typed primitive values
in serialized data. Here is an example
Composition with references to elements:
These string-references determine the target section's place in the same archive.
/-separated segment represents a key. A reference starts from the
root object and following the sequence of keys to a specific object (i.e. section).
Here is the full archive data:
If you follow the keys
0, you reach the
section that represent hydrogen. Keep in mind that lists use index-numbers as keys.
References can look different depending on the context. Above we saw simple references
that point from one data section to another. But, you also already a saw a different
type of reference. Schema's themselves contain references: when we
type: Element or
section: Element to refer to a section definition, we were
writing down references that point to a section definition. Here we can use a convenience representation:
Element simply replaces the otherwise cryptic
So far, we never discussed the use of
m_def. In the examples you might have seen this
as a special key in some objects. Whenever we cannot determine the section definition
for a section by its context (e.g. the key/sub-section used to contain it in a parent section), we use
m_def to provide a reference to the section definition.
Different forms of references¶
Depending on where references are used, they might take a different serialized form. Here are a few examples for different reference syntax:
||Reference to a section within the sub-section hierarchy of the same archive.|
||Reference to a section definition in the same archive. Can only be used to target section definitions.|
||Reference to a section definition that was written in Python and is part of the NOMAD code. Can only be used to target section definitions.|
||Reference to a section in a different
||Reference to a section in a processed archive given by entry mainfile.|
||Reference to a section in a processed archive given by entry-id.|
||Reference to a section in an entry of a different upload.|
||Reference to a section in an entry in a different NOMAD installation.|
References across entries¶
A references in the archive of one entry can point to a section in a different entry's archive. The following two example files, exemplify this use of reference between two NOMAD entries.
definitions: sections: Element: quantities: label: type: str density: type: np.float64 unit: g/cm**3 isotopes: type: int shape: ['*'] PeriodicTable: sub_sections: elements: repeats: true section: Element data: m_def: PeriodicTable elements: - label: H density: 0.00008375 isotopes: [1, 2, 3] - label: O density: 1.141 isotopes: [16, 17, 18]
definitions: sections: Composition: quantities: composition: type: str elements: type: ../upload/raw/periodic_table.archive.yaml#Element shape: ['*'] data: m_def: Composition composition: 'H2O' elements: - ../upload/raw/periodic_table.archive.yaml#data/elements/0 - ../upload/raw/periodic_table.archive.yaml#data/elements/1
These inter-entry references have two parts:
<entry>#<section>, where entry
is a path or URL denoting the target entry and section a path within the target entry's sub-section containment hierarchy.
Please note that also schemas can be spread over multiple files. In the above example, one file contained the schema and data for a periodic table and another file contained schema and data for the composition of water (using the periodic table).
Base sections and inheritance¶
We add a relationship between section definitions that allows us to create more specialized definitions from more abstract definitions. Here the properties of the abstract definition are inherited by the more specialized definitions
Here is a simple schema with two specialization of the same abstract section definition:
The two specialized definitions
Evaporation define the abstract
Process via the
base_section property. With this
inherit the quantity
time. We do not need to repeat quantities from the base section, and we can add more properties. Here is an example
Evaporation using both the inherited
and added quantity:
What happens if we reference abstract definitions in sub-sections or reference quantities?
Here is an sub-section example. In one schema, we define the relationship between
Process. In another schema, we want to add more specializations to what a process is.
The section definition use in the sub-section
processes defines what a contained
section has to be "at least". Meaning that any section based on a specialization of
Process would be a valid
The fact that a sub-section or reference target can have different "forms" (i.e. based on different specializations) is called polymorphism in object-oriented data modelling.
NOMAD provides a series of built-in section definitions. For example, there is
EntryArchive, a definition for the top-level object in all NOMAD archives (e.g.
.archive.yaml files). Here is a simplified except of the main NOMAD schema
Compare this to the previous examples: we used the top-level keys
data without really explaining why. Here you can see why. The
us to put a metainfo package (i.e. a NOMAD schema) into our archives. And
data allows us to put data into archives that is a
EntryData definition is empty. It is merely an abstract placeholder that allows you to add specialized data sections to your archive.
Therefore, all section definitions that define a top-level data section, should
nomad.datamodel.EntryData as a base section. This would be the first "correct"
Here are a few other built-in section definitions and packages of definitions:
|Section definition or package||Purpose|
|nomad.datamodel.EntryArchive||Used for the root object of all NOMAD entries|
|nomad.datamodel.EntryMetadata||Used to add standard NOMAD metadata such as ids, upload, processing, or author information to entries.|
|nomad.datamodel.EntryData||An abstract section definition for the
|nomad.datamodel.ArchiveSection||Allows to put
|nomad.datamodel.metainfo.eln.*||A package of section definitions to inherit commonly used quantities for ELNs. These quantities are indexed and allow specialization to utilize the NOMAD search.|
|nomad.parsing.tabular.TableData||Allows to inherit parsing of references .csv and .xls files.|
|nomad.datamodel.metainfo.simulation.*||A package of section definitions use by NOMAD's electronic structure code parsers to produce simulator "run" based data|
|nomad.metainfo.*||A package that contains all definitions of definitions, e.g. NOMAD's "schema language". Here you find definitions for what a sections, quantity, sub-sections, etc. is.|
Separating data and schema¶
As we saw above, a NOMAD entry can contain schema
data at the
same time. To organize your schemas and data efficiently, it is often necessary to re-use
schemas and certain data in other entries. You can use references to spread your
schemas and data over multiple entries and connect the pieces via references.
Here is a simple schema, stored in a NOMAD entry with mainfile name
Now, we can re-use this schema in many entries via references. Here, we extend
the schema and instantiate definitions is a separate mainfile
definitions: sections: SpecialElement: # Extending the definition from another entry base_section: '../upload/raw/schema.archive.yaml#Element' quantities: atomic_weight: type: float unit: 'g/mol' data: # Instantiating the definition from another entry m_def: '../upload/raw/schema.archive.yaml#Composition' composition: 'H2O' elements: # Implicitly instantiate Element as defined for Composition.elements - label: H # Explicitly instantiate SpecialElement as a polymorph substitute - m_def: SpecialElement label: O atomic_weight: 15.9994
Here is a last example that re-uses the schema and references data from the two entries above:
You cannot create definitions that lead to circular loading of
definitions section in an NOMAD entry represents a schema package. Each package
needs to be fully loaded and analyzed before it can be used by other packages in other entries.
Therefore, two packages in two entries cannot reference each other.