Schema annotations¶

Definitions in a schema can have annotations. These annotations provide additional information that NOMAD can use to alter its behavior around these definitions. Annotations are named blocks of key-value pairs:

definitions:
  sections:
    MyAnnotatedSection:
      m_annotations:
        annotation_name:
          key1: value
          key2: value

Many annotations control the representation of data in the GUI. This can be for plots or data entry/editing capabilities.

ELN annotations¶

These annotations control how data can be entered and edited. Use the key eln to add this annotations. For example:

class Sample(EntryData):
    sample_id = Quantity(type=str, a_eln=dict(component='StringEditQuantity'))`)

or in YAML schemas:

Sample:
  quantities:
    sample_id:
      type: str
      m_annotations:
        eln:
          component: StringEditQuantity

An eln annotation can be added to section and quantity definitions to different effects. In both cases, it controls how sections and quantities are represented in the GUI with different parameters; see below.

The UI gives an overview about all ELN edit annotations and components here.

name	type
component	`str`	The form field component that is used to make the annotated quantity editable. If no component is given, the quantity won't be editable. This can be used on quantities only. The supported values are: `StringEditQuantity`: For editing simple short string values. `URLEditQuantity`: For editing strings that are validated to be URLs. `EnumEditQuantity`: For Editing enum values. Uses a dropdown list with enum values. This component may be used for short enumerates. `RadioEnumEditQuantity`: For Editing enum values. Uses radio buttons. `AutocompleteEditQuantity`: For editing enum values. Uses an autocomplete form with dropdown list. This component may be used for longer enumerates. `FileEditQuantity`: For editing a reference to a file. Will allow to choose a file or upload a file. `BoolEditQuantity`: For editing boolean choices. `NumberEditQuantity`: For editing numbers with our without unit. `SliderEditQuantity`: For editing numbers with a horizontal slider widget. `DateTimeEditQuantity`: For editing datetimes. `RichTextEditQuantity`: For editing long styled text with a rich text editor. `ReferenceEditQuantity`: For editing references to other sections. `UserEditQuantity`: For entering user information. Lets you choose a nomad user or enter information manually. `AuthorEditQuantity`: For entering author information manually. options: - `StringEditQuantity` - `URLEditQuantity` - `EnumEditQuantity` - `RadioEnumEditQuantity` - `AutocompleteEditQuantity` - `FileEditQuantity` - `BoolEditQuantity` - `NumberEditQuantity` - `SliderEditQuantity` - `DateTimeEditQuantity` - `RichTextEditQuantity` - `ReferenceEditQuantity` - `UserEditQuantity` - `AuthorEditQuantity`
label	`str`	Custom label for the quantity shown on the form field.
props	`Dict[str, typing.Any]`	A dictionary with additional props that are passed to the editcomponent.
default	`typing.Any`	Prefills any set form field component with the given value. This is different from the quantities `default` property. The quantities default is not stored in the data; the default value is assumed if no other value is given. The ELN form field default value will be stored, even if not changed.
defaultDisplayUnit	`str`	Allows to define a default unit to initialize a `NumberEditQuantity` with. The unit has to be compatible with the unit of the annotation quantity and the annotated quantity must have a unit. Only applies to quantities and with `component=NumberEditQuantity`.
minValue	`Union[int, float]`	Allows to specify a minimum value for quantity annotations with number type. Will show an error, if outside numbers are entered. Only works on quantities and in conjunction with `component=NumberEditQuantity`.
maxValue	`Union[int, float]`	Allows to specify a maximum value for quantity annotations with number type. Will show an error, if outside numbers are entered. Only works on quantities and in conjunction with `component=NumberEditQuantity`.
showSectionLabel	`int`	To customize the ReferenceEditQuantity behaviour. If true the section label will be shown instead of referenced file name and the path to the section.
hide	`List[str]`	The annotation "hide" is deprecated. Use "visible" key of "properties" annotation instead. Allows you to hide certain quantities from a section editor. Give a list of quantity names. Quantities must exist in the section that this annotation is added to. Can only be used in section annotations. deprecated
overview	`int`	Shows the annotation section on the entry's overview page. Can only be used on section annotations.
lane_width	`Union[str, int]`	Value to overwrite the css width of the lane used to render the annotation section and its editor.
properties	`SectionProperties`	The value to customize the quantities and sub sections of the annotation section. The supported keys: `visible`: To determine the visible quantities and sub sections by their names `editable`: To render things visible but not editable, e.g. in inheritance situations `order`: # To order things, properties listed in that order first, then the rest

SectionProperties¶

A filter defined by an include list or and exclude list of the quantities and subsections.

name	type
visible	`Filter`	Defines the visible quantities and subsections. default: `1`
editable	`Filter`	Defines the editable quantities and subsections.
order	`List[str]`	To customize the order of the quantities and subsections.

Filter¶

A filter defined by an include list or and exclude list of the quantities or subsections.

name	type
include	`List[str]`	The list of quantity or subsection names to be included.
exclude	`List[str]`	The list of quantity or subsection names to be excluded.

Browser¶

The browser annotation allows to specify if the processed data browser needs to display a quantity differently. It can be applied to quantities. For example

    class Experiment(EntryData):
        description = Quantity(type=str, a_browser=dict(render_value='HtmlValue'))

or in yaml

Experiment:
  quantities:
    description:
      type: str
      m_annotations:
        browser:
          render_value: HtmlValue

name	type
adaptor	`str`	Allows to change the Adaptor implementation that is used to render the lane for this quantity. Possible values are: `RawFileAdaptor`: An adopter that is used to show files, including all file actions, like file preview. options: - `RawFileAdaptor`
render_value	`str`	Allows to change the Component used to render the value of the quantity. Possible values are: `HtmlValue`: Renders a string as HTML. `JsonValue`: Renders a dict or list in a collapsable tree. options: - `JsonValue` - `HtmlValue`

browser¶

The browser annotation allows to specify if the processed data browser needs to display a quantity differently. It can be applied to quantities. For example

    class Experiment(EntryData):
        description = Quantity(type=str, a_browser=dict(render_value='HtmlValue'))

or in yaml

Experiment:
  quantities:
    description:
      type: str
      m_annotations:
        browser:
          render_value: HtmlValue

name	type
adaptor	`str`	Allows to change the Adaptor implementation that is used to render the lane for this quantity. Possible values are: `RawFileAdaptor`: An adopter that is used to show files, including all file actions, like file preview. options: - `RawFileAdaptor`
render_value	`str`	Allows to change the Component used to render the value of the quantity. Possible values are: `HtmlValue`: Renders a string as HTML. `JsonValue`: Renders a dict or list in a collapsable tree. options: - `JsonValue` - `HtmlValue`

`label_quantity`¶

This annotation goes in the section that we want to be filled with tabular data, not in the single quantities. It is used to give a name to the instances that might be created by the parser. If it is not provided, the name of the section itself will be used as name. Many times it is useful because, i. e., one might want to create a bundle of instances of, say, a "Substrate" class, each instance filename not being "Substrate_1", "Substrate_2", etc., but being named after a quantity contained in the class that is, for example, the specific ID of that sample.

MySection:
  more:
    label_quantity: my_quantity
  quantities:
    my_quantity:
      type: np.float64
      shape: ['*']
      description: "my quantity to be filled from the tabular data file"
      unit: K
      m_annotations:
        tabular:
          name: "Sheet1/my header"
        plot:
          x: timestamp
          y: ./my_quantity

Important

The quantity designated as label_quantity should not be an array but a integer, float or string, to be set as the name of a file. If an array quantity is chosen, the parser would fall back to the use of the section as name.

Tabular data¶

`tabular`¶

Allows to map a quantity to a row or a column of a spreadsheet data-file. Should only be used in conjunction with tabular_parser.

name	type
name	`str`	The column name that should be mapped to the annotation quantity. Has to be the same string that is used in the header, i.e. first `.csv` line or first excel file `row`. For excel files with multiple sheets, the name can have the form `<sheet name>/<column name>`. Otherwise, only the first sheets is used. Has to be applied to the quantity that a column should be mapped to.
unit	`str`	The unit of the value in the file. Has to be compatible with the annotated quantity's unit. Will be used to automatically convert the value. If this is not defined, the values will not be converted. Has to be applied to the quantity that a column should be mapped to.

Each and every quantity to be filled with data from tabular data files should be annotated as the following example. A practical example is provided in How To section.

my_quantity:
  type: np.float64
  shape: ['*']
  description: "my quantity to be filled from the tabular data file"
  unit: K
  m_annotations:
    tabular:
      name: "Sheet1/my header"
    plot:
      x: timestamp
      y: ./my_quantity

`tabular_parser`¶

One special quantity will be dedicated to host the tabular data file. In the following examples it is called data_file, it contains the tabular_parser annotation, as shown below.

Instructs NOMAD to treat a string valued scalar quantity as a file path and interprets the contents of this file as tabular data. Supports both .csv and Excel files.

name	type
parsing_options	`TabularParsingOptions`	Options on how to extract the data from csv/xlsx file. Under the hood, NOMAD uses pandas `Dataframe` to parse the data from tabular files. These are the available options that can be passed down to the parser. The supported values are: `skiprows`: Number of rows to be skipped while reading the file. `sep`: The character used to separate cells (specific to csv files). `comment`: The character denoting the commented lines. `separator`: An alias for `sep`. default: Complex object, default value not displayed.
mapping_options	`List[TabularMappingOptions]`	A list of directives on how to map the extracted data from the csv/xlsx file to NOMAD. Each directive is a distinct directive, which allows for more modular definition of your tabular parser schema. If no item is provided, the entire schema is treated to be parsed under column mode. The supported values in each item of this list are: `mapping_mode`: A `list` of paths to the repeating sub-sections where the tabular quantities are to be filled from individual rows of the excel/csv file (i.e. in the row mode). Each path is a `/` separated list of nested sub-sections. The targeted sub-sections, will be considered when mapping table rows to quantities. Has to be used to annotate the quantity that holds the path to the `.csv` or excel file. `file_mode`: The character used to separate cells (specific to csv files). `sections`: The character denoting the commented lines. default: `[]`

TabularParsingOptions¶

name	type
skiprows	`Union[List[int], int]`	Number of rows to skip
sep	`str`	Character identifier of a separator
comment	`str`	Character identifier of a commented line
separator	`str`	Alias for `sep`

TabularMappingOptions¶

name	type
mapping_mode	`str`	This controls the behaviour of mapping of the extracted data onto NOMAD schema. The supported values are: `row`: A `list` of paths to the repeating sub-sections where the tabular quantities are to be filled from individual rows of the excel/csv file (i.e. in the row mode). Each path is a `/` separated list of nested sub-sections. The targeted sub-sections, will be considered when mapping table rows to quantities. Has to be used to annotate the quantity that holds the path to the `.csv` or excel file. `column`: A `list` of paths to the sub-sections where the tabular quantities are to be filled from the entire column of the excel/csv file (i.e. in the column mode). Each path is a `/` separated list of nested sub-sections. The targeted sub-sections, will be considered when mapping table columns to quantities. Has to be used to annotate the quantity that holds the path to the `.csv` or excel file. `enrty`: A `list` of paths to the (sub)sections where the tabular quantities are to be filled from individual rows of the excel/csv file, to create distinct entries. Each path is a `/` separated list of nested sub-sections. The targeted (sub)sections, will be considered when mapping table rows to quantities. The schema of the resultant entry follows the (sub)section's schema. In order to parse the entire schema using entry mode, then set the first item in this list to `root`. Has to be used to annotate the quantity that holds the path to the `.csv` or excel file. default: `column` options: - `row` - `column`
file_mode	`str`	This controls the behaviour of the parser towards working physical files in file system. The supported values are: `current_entry`: Processing the data into the same NOMAD entry. `single_new_entry`: Creating a new entry and processing the data into this new NOMAD entry. `multiple_new_entries`: Creating many new entries and processing the data into these new NOMAD entries. options: - `current_entry` - `single_new_entry` - `multiple_new_entries`
sections	`List[str]`	A `list` of paths to the (sub)sections where the tabular quantities are to be filled from the data extracted from the tabular file.

Available Combinations¶

Tutorial ref.	`file_mode`	`mapping_mode`	`sections`	How to ref.
1	`current_entry`	`column`	`root`	HowTo
2	`current_entry`	`column`	my path	HowTo
np1	`current_entry`	`row`	`root`	Not possible
3	`current_entry`	`row`	my path	HowTo
np2	`single_new_entry`	`column`	`root`	Not possible
4	`single_new_entry`	`column`	my path	HowTo
np3	`single_new_entry`	`row`	`root`	Not possible
5	`single_new_entry`	`row`	my path	HowTo
np4	`multiple_new_entries`	`column`	`root`	Not possible
np5	`multiple_new_entries`	`column`	my path	Not possible
6	`multiple_new_entries`	`row`	`root`	HowTo
7	`multiple_new_entries`	`row`	my path	HowTo

data_file:
  type: str
  description: "the tabular data file containing data"
  m_annotations:
    tabular_parser:
      parsing_options:
        comment: '#'
      mapping_options:
      - mapping_mode: column
        file_mode: single_new_entry
        sections:
        - my_section/my_quantity

Plot¶

The PlotSection base section serves as an additional functionality to your sections. This base section is designed to simplify the process of creating various types of plots, making it easy to use Plotly Express, Plotly Subplot, and the general Plotly graph objects.

Features:

Plotly Express: Create simple and quick plots with a high-level, expressive API.
Plotly Subplot: Organize multiple plots into subplots for more complex visualizations.
General Plotly Graph Objects: Fine-tune your plots by working directly with Plotly's graph objects.

Usage:

Inherit from this base section to leverage its plot functionality.
Customize your plots using the annotations plotly-express, plotly-subplots, or/and plotly-graph-object.

The PlotSection class makes it possible to define plots that are shown alongside your data. Underneath, we use the Plotly Open Source Graphing Libraries to control the creation of the plots, and you can find many useful examples in their documentation.

In Python schemas, the PlotSection class gives you full freedom to define plots programmatically. For example, you could use plotly.express and plotly.graph_objs to define plots like this:

from nomad.datamodel.metainfo.plot import PlotSection, PlotlyFigure
from nomad.datamodel.data import EntryData
import plotly.express as px
import plotly.graph_objs as go
from plotly.subplots import make_subplots

class CustomSection(PlotSection, EntryData):
    m_def = Section()
    time = Quantity(type=float, shape=['*'], unit='s', a_eln=dict(component='NumberEditQuantity'))
    substrate_temperature = Quantity(type=float, shape=['*'], unit='K', a_eln=dict(component='NumberEditQuantity'))
    chamber_pressure = Quantity(type=float, shape=['*'], unit='Pa', a_eln=dict(component='NumberEditQuantity'))

    def normalize(self, archive, logger):
        super(CustomSection, self).normalize(archive, logger)

        first_line = px.scatter(x=self.time, y=self.substrate_temperature)
        second_line = px.scatter(x=self.time, y=self.chamber_pressure)
        figure1 = make_subplots(rows=1, cols=2, shared_yaxes=True)
        figure1.add_trace(first_line.data[0], row=1, col=1)
        figure1.add_trace(second_line.data[0], row=1, col=2)
        figure1.update_layout(height=400, width=716, title_text="Creating Subplots in Plotly")
        self.figures.append(PlotlyFigure(label='figure 1', figure=figure1.to_plotly_json()))

        figure2 = px.scatter(x=self.substrate_temperature, y=self.chamber_pressure, color=self.chamber_pressure, title="Chamber as a function of Temperature")
        self.figures.append(PlotlyFigure(label='figure 2', index=1, figure=figure2.to_plotly_json()))

        heatmap_data = [[None, None, None, 12, 13, 14, 15, 16],
             [None, 1, None, 11, None, None, None, 17],
             [None, 2, 6, 7, None, None, None, 18],
             [None, 3, None, 8, None, None, None, 19],
             [5, 4, 10, 9, None, None, None, 20],
             [None, None, None, 27, None, None, None, 21],
             [None, None, None, 26, 25, 24, 23, 22]]

        heatmap = go.Heatmap(z=heatmap_data, showscale=False, connectgaps=True, zsmooth='best')
        figure3 = go.Figure(data=heatmap)
        self.figures.append(PlotlyFigure(label='figure 3', index=0, figure=figure3.to_plotly_json()))

In YAML schemas, plots can be defined by using the PlotSection as a base class, and additionally utilizing different flavours of plot annotations. The different annotation options are described below.

PlotlyGraphObjectAnnotation¶

Allows to plot figures using plotly graph object.

    Example:
      base_sections:
        - 'nomad.datamodel.metainfo.plot.PlotSection'
      m_annotations:
        plotly_graph_object:
        - data:
            x: '#xArr'
            y: '#xArr'
          layout:
            title:
              text: 'Plotly Graph Object'
          label: 'Plotly Graph Object'
          index: 1

name	type
label	`str`	Figure label
data	`Dict`	Plotly data
layout	`Dict`	Plotly layout
config	`Dict`	Plotly config

PlotlyExpressAnnotation¶

Allows to plot multi trace figures using plotly Express.

  sections:
    Example:
      base_sections:
        - 'nomad.datamodel.metainfo.plot.PlotSection'
      m_annotations:
        plotly_express:
          method: scatter
          x: '#xArr'
          y: '#yArr'
          label: 'Example Express Plot'
          index: 0
          layout:
            title:
              text: 'Example Express Plot'
            xaxis:
              title:
                text: 'x axis'
            yaxis:
              title:
                text: 'y axis'
          traces:
            - method: scatter
              x: '#xArr'
              y: '#zArr'

name	type
method	`str`	Plotly express plot method
layout	`Dict`	Plotly layout
x	`Union[List[float], List[str], str]`	Plotly express x
y	`Union[List[float], List[str], str]`	Plotly express y
z	`Union[List[float], List[str], str]`	Plotly express z
color	`Union[List[float], List[str], str]`	Plotly express color
symbol	`str`	Plotly express symbol
title	`str`	Plotly express title
label	`str`	Figure label
traces	`List[PlotlyExpressTraceAnnotation]`	List of traces added to the main trace defined by plotly_express method default: `[]`

PlotlyExpressTraceAnnotation¶

Allows to plot figures using plotly Express.

name	type
method	`str`	Plotly express plot method
layout	`Dict`	Plotly layout
x	`Union[List[float], List[str], str]`	Plotly express x
y	`Union[List[float], List[str], str]`	Plotly express y
z	`Union[List[float], List[str], str]`	Plotly express z
color	`Union[List[float], List[str], str]`	Plotly express color
symbol	`str`	Plotly express symbol
title	`str`	Plotly express title

PlotlySubplotsAnnotation¶

Allows to plot figures in subplots.

    Example:
      base_sections:
        - 'nomad.datamodel.metainfo.plot.PlotSection'
      m_annotations:
        plotly_subplots:
          parameters:
            rows: 2
            cols: 2
          layout:
            title:
              text: 'All plots'
          plotly_express:
            - method: scatter
              x: '#xArr'
              y: '#yArr'
              title: 'subplot 1'
            - method: scatter
              x: '#xArr'
              y: '#zArr'
              title: 'subplot 2'
            - method: scatter
              x: '#zArr'
              y: '#xArr'
              title: 'subplot 3'
            - method: scatter
              x: '#zArr'
              y: '#yArr'
              title: 'subplot 4'

name	type
label	`str`	Figure label
layout	`Dict`	Plotly layout
parameters	`Dict`	plotly.subplots.make_subplots parameters i.e. rows, cols, shared_xaxes, shared_xaxes, horizontal_spacing , ... See plotly make_subplots documentation for more information.
plotly_express	`List[PlotlyExpressAnnotation]`	List of subplots defined by plotly_express method default: `[]`

PlotAnnotation (Deprecated)¶

The PlotAnnotation is now deprecated and will be removed in future releases. We recommend transitioning to the use of PlotSection and PlotlyGraphObjectAnnotation for your plotting needs.

This annotation can be used to add a plot to a section or quantity. Example:

class Evaporation(MSection):
    m_def = Section(a_plot={
        'label': 'Temperature and Pressure',
        'x': 'process_time',
        'y': ['./substrate_temperature', './chamber_pressure'],
        'config': {
            'editable': True,
            'scrollZoom': False
        }
    })
    time = Quantity(type=float, shape=['*'], unit='s')
    substrate_temperature = Quantity(type=float, shape=['*'], unit='K')
    chamber_pressure = Quantity(type=float, shape=['*'], unit='Pa')

You can create multi-line plots by using lists of the properties y (and x). You either have multiple sets of y-values over a single set of x-values. Or you have pairs of x and y values. For this purpose the annotation properties x and y can reference a single quantity or a list of quantities. For repeating sub sections, the section instance can be selected with an index, e.g. "sub_section_name/2/parameter_name" or with a slice notation start:stop where negative values index from the end of the array, e.g. "sub_section_name/1:-5/parameter_name".

The interactive examples of the plot annotations can be found here.

name	type
label	`str`	Is passed to plotly to define the label of the plot.
x	`Union[List[str], str]`	A path or list of paths to the x-axes values. Each path is a `/` separated list of sub-section and quantity names that leads from the annotation section to the quantity. Repeating sub sections are indexed between two `/`s with an integer or a slice `start:stop`.
y	`Union[List[str], str]`	A path or list of paths to the y-axes values. list of sub-section and quantity names that leads from the annotation section to the quantity. Repeating sub sections are indexed between two `/`s with an integer or a slice `start:stop`.
lines	`List[dict]`	A list of dicts passed as `traces` to plotly to configure the lines of the plot. See https://plotly.com/javascript/reference/scatter/ for details.
layout	`dict`	A dict passed as `layout` to plotly to configure the plot layout. See https://plotly.com/javascript/reference/layout/ for details.
config	`dict`	A dict passed as `config` to plotly to configure the plot functionality. See https://plotly.com/javascript/configuration-options/ for details.