Schema annotations¶
Definitions in a schema can have annotations. These annotations provide additional information that NOMAD can use to alter its behavior around these definitions. Annotations are named blocks of key-value pairs:
Many annotations control the representation of data in the GUI. This can be for plots or data entry/editing capabilities.
ELN annotations¶
These annotations control how data can be entered and edited.
Use the key eln
to add this annotations. For example:
class Sample(EntryData):
sample_id = Quantity(type=str, a_eln=dict(component='StringEditQuantity'))`)
or in YAML schemas:
An eln
annotation can be added to section and quantity definitions to different
effects. In both cases, it controls how sections and quantities are represented in the GUI
with different parameters; see below.
The UI gives an overview about all ELN edit annotations and components here.
name | type | |
---|---|---|
component | str |
The form field component that is used to make the annotated quantity editable. If no component is given, the quantity won't be editable. This can be used on quantities only. The supported values are: StringEditQuantity : For editing simple short string values.URLEditQuantity : For editing strings that are validated to be URLs.EnumEditQuantity : For Editing enum values. Uses a dropdown list with enum values. This component may be used for short enumerates.RadioEnumEditQuantity : For Editing enum values. Uses radio buttons.AutocompleteEditQuantity : For editing enum values. Uses an autocomplete form with dropdown list. This component may be used for longer enumerates.FileEditQuantity : For editing a reference to a file. Will allow to choose a file or upload a file.BoolEditQuantity : For editing boolean choices.NumberEditQuantity : For editing numbers with our without unit.SliderEditQuantity : For editing numbers with a horizontal slider widget.DateTimeEditQuantity : For editing datetimes.RichTextEditQuantity : For editing long styled text with a rich text editor.ReferenceEditQuantity : For editing references to other sections.UserEditQuantity : For entering user information. Lets you choose a nomad user or enter information manually.AuthorEditQuantity : For entering author information manually.options:- StringEditQuantity - URLEditQuantity - EnumEditQuantity - RadioEnumEditQuantity - AutocompleteEditQuantity - FileEditQuantity - BoolEditQuantity - NumberEditQuantity - SliderEditQuantity - DateTimeEditQuantity - DateEditQuantity - TimeEditQuantity - RichTextEditQuantity - ReferenceEditQuantity - UserEditQuantity - AuthorEditQuantity - QueryEditQuantity - ActionEditQuantity |
label | str |
[Deprecated] ELN label annotation has been deprecated and it is advised to utilize display annotation instead. Custom label for the quantity shown on the form field. It is recommended to adhere to the convention of using lowercase letters for the label, except for abbreviations which could be capitalized. |
props | Dict[str, Any] |
A dictionary with additional props that are passed to the edit component. |
default | Any |
Prefills any set form field component with the given value. This is different from the quantities default property. The quantities default is not stored in the data; the default value is assumed if no other value is given. The ELN form field default value will be stored, even if not changed. |
defaultDisplayUnit | str |
This attribute is deprecated, use the unit attribute of display annotation instead. Allows to define a default unit to initialize a NumberEditQuantity with. The unit has to be compatible with the unit of the annotation quantity and the annotated quantity must have a unit. Only applies to quantities and with component=NumberEditQuantity .deprecated |
minValue | Union[int, float] |
Allows to specify a minimum value for quantity annotations with number type. Will show an error, if outside numbers are entered. Only works on quantities and in conjunction with component=NumberEditQuantity . |
maxValue | Union[int, float] |
Allows to specify a maximum value for quantity annotations with number type. Will show an error, if outside numbers are entered. Only works on quantities and in conjunction with component=NumberEditQuantity . |
showSectionLabel | int |
To customize the ReferenceEditQuantity behaviour. If true the section label will be shown instead of referenced file name and the path to the section. |
hide | List[str] |
This attribute is deprecated. Use visible attribute of display annotation instead. Allows you to hide certain quantities from a section editor. Give a list of quantity names. Quantities must exist in the section that this annotation is added to. Can only be used in section annotations.deprecated |
overview | int |
Shows the annotation section on the entry's overview page. Can only be used on section annotations. |
lane_width | Union[str, int] |
Value to overwrite the css width of the lane used to render the annotation section and its editor. |
properties | SectionProperties |
The value to customize the quantities and sub sections of the annotation section. The supported keys: visible : To determine the visible quantities and sub sections by their nameseditable : To render things visible but not editable, e.g. in inheritance situationsorder : # To order things, properties listed in that order first, then the rest |
SectionProperties¶
The display settings for quantities and subsections. (Deprecated)
name | type | |
---|---|---|
visible | Filter |
Defines the visible quantities and subsections. (Deprecated)default: 1 |
editable | Filter |
Defines the editable quantities and subsections. (Deprecated) |
order | List[str] |
To customize the order of the quantities and subsections. (Deprecated) |
Filter¶
A filter defined by an include list or and exclude list of the quantities or subsections.
name | type | |
---|---|---|
include | List[str] |
The list of quantity or subsection names to be included. |
exclude | List[str] |
The list of quantity or subsection names to be excluded. |
Browser¶
The browser
annotation allows to specify if the processed data browser needs to
display a quantity differently. It can be applied to quantities. For example
class Experiment(EntryData):
description = Quantity(type=str, a_browser=dict(render_value='HtmlValue'))
or in yaml
name | type | |
---|---|---|
adaptor | str |
Allows to change the Adaptor implementation that is used to render the lane for this quantity. Possible values are:RawFileAdaptor : An adopter that is used to show files, including all file actions, like file preview.options:- RawFileAdaptor |
render_value | str |
Allows to change the Component used to render the value of the quantity. Possible values are:HtmlValue : Renders a string as HTML.JsonValue : Renders a dict or list in a collapsable tree.options:- JsonValue - HtmlValue |
Display annotations¶
Display annotation for quantities¶
This annotations control how quantities are displayed in the GUI. Use the
key display
to add this annotation. For example in Python:
or in YAML:
name | type | |
---|---|---|
visible | Filter |
Defines the visible quantities and subsections.default: 1 |
editable | Filter |
Defines the editable quantities and subsections. |
unit | str |
To determine the default display unit for quantity. |
Display annotation for sections¶
This annotations control how sections are displayed in the GUI. Use the
key display
to add this annotation. For example in Python:
or in YAML:
name | type | |
---|---|---|
visible | Filter |
Defines the visible quantities and subsections.default: 1 |
editable | Filter |
Defines the editable quantities and subsections. |
order | List[str] |
To customize the order of the quantities and subsections. |
label_quantity
¶
This annotation goes in the section that we want to be filled with tabular data, not in the single quantities. It is used to give a name to the instances that might be created by the parser. If it is not provided, the name of the section itself will be used as name. Many times it is useful because, i. e., one might want to create a bundle of instances of, say, a "Substrate" class, each instance filename not being "Substrate_1", "Substrate_2", etc., but being named after a quantity contained in the class that is, for example, the specific ID of that sample.
MySection:
more:
label_quantity: my_quantity
quantities:
my_quantity:
type: np.float64
shape: ['*']
description: "my quantity to be filled from the tabular data file"
unit: K
m_annotations:
tabular:
name: "Sheet1/my header"
plot:
x: timestamp
y: ./my_quantity
Important
The quantity designated as label_quantity
should not be an array but a integer, float or string, to be set as the name of a file. If an array quantity is chosen, the parser would fall back to the use of the section as name.
Tabular data¶
tabular
¶
Allows to map a quantity to a row or a column of a spreadsheet data-file. Should only be used
in conjunction with tabular_parser
.
name | type | |
---|---|---|
name | str |
The column name that should be mapped to the annotation quantity. Has to be the same string that is used in the header, i.e. first .csv line or first excel file row . For excel files with multiple sheets, the name can have the form <sheet name>/<column name> . Otherwise, only the first sheets is used. Has to be applied to the quantity that a column should be mapped to. |
unit | str |
The unit of the value in the file. Has to be compatible with the annotated quantity's unit. Will be used to automatically convert the value. If this is not defined, the values will not be converted. Has to be applied to the quantity that a column should be mapped to. |
Each and every quantity to be filled with data from tabular data files should be annotated as the following example. A practical example is provided in How To section.
my_quantity:
type: np.float64
shape: ['*']
description: "my quantity to be filled from the tabular data file"
unit: K
m_annotations:
tabular:
name: "Sheet1/my header"
plot:
x: timestamp
y: ./my_quantity
tabular_parser
¶
One special quantity will be dedicated to host the tabular data file. In the following examples it is called data_file
, it contains the tabular_parser
annotation, as shown below.
Instructs NOMAD to treat a string valued scalar quantity as a file path and
interprets the contents of this file as tabular data. Supports both
.csv
and Excel files.
name | type | |
---|---|---|
parsing_options | TabularParsingOptions |
Options on how to extract the data from csv/xlsx file. Under the hood, NOMAD uses pandas Dataframe to parse the data from tabular files. These are the available options that can be passed down to the parser.The supported values are: skiprows : Number of rows to be skipped while reading the file.sep : The character used to separate cells (specific to csv files).comment : The character denoting the commented lines.separator : An alias for sep .default: Complex object, default value not displayed. |
mapping_options | List[TabularMappingOptions] |
A list of directives on how to map the extracted data from the csv/xlsx file to NOMAD. Each directive is a distinct directive, which allows for more modular definition of your tabular parser schema. If no item is provided, the entire schema is treated to be parsed under column mode. The supported values in each item of this list are: mapping_mode : A list of paths to the repeating sub-sections where the tabular quantities are to be filled from individual rows of the excel/csv file (i.e. in the row mode). Each path is a / separated list of nested sub-sections. The targeted sub-sections, will be considered when mapping table rows to quantities. Has to be used to annotate the quantity that holds the path to the .csv or excel file.file_mode : The character used to separate cells (specific to csv files).sections : The character denoting the commented lines.default: [] |
TabularParsingOptions¶
name | type | |
---|---|---|
skiprows | Union[List[int], int] |
Number of rows to skip |
sep | str |
Character identifier of a separator |
comment | str |
Character identifier of a commented line |
separator | str |
Alias for sep |
TabularMappingOptions¶
name | type | |
---|---|---|
mapping_mode | str |
This controls the behaviour of mapping of the extracted data onto NOMAD schema. The supported values are: row : A list of paths to the repeating sub-sections where the tabular quantities are to be filled from individual rows of the excel/csv file (i.e. in the row mode). Each path is a / separated list of nested sub-sections. The targeted sub-sections, will be considered when mapping table rows to quantities. Has to be used to annotate the quantity that holds the path to the .csv or excel file.column : A list of paths to the sub-sections where the tabular quantities are to be filled from the entire column of the excel/csv file (i.e. in the column mode). Each path is a / separated list of nested sub-sections. The targeted sub-sections, will be considered when mapping table columns to quantities. Has to be used to annotate the quantity that holds the path to the .csv or excel file.enrty : A list of paths to the (sub)sections where the tabular quantities are to be filled from individual rows of the excel/csv file, to create distinct entries. Each path is a / separated list of nested sub-sections. The targeted (sub)sections, will be considered when mapping table rows to quantities. The schema of the resultant entry follows the (sub)section's schema. In order to parse the entire schema using entry mode, then set the first item in this list to root . Has to be used to annotate the quantity that holds the path to the .csv or excel file.default: TabularMode.column options:- row - column |
file_mode | str |
This controls the behaviour of the parser towards working physical files in file system. The supported values are: current_entry : Processing the data into the same NOMAD entry.single_new_entry : Creating a new entry and processing the data into this new NOMAD entry.multiple_new_entries : Creating many new entries and processing the data into these new NOMAD entries.options: - current_entry - single_new_entry - multiple_new_entries |
sections | List[str] |
A list of paths to the (sub)sections where the tabular quantities are to be filled from the data extracted from the tabular file. |
Available Combinations¶
Tutorial ref. | file_mode |
mapping_mode |
sections |
How to ref. |
---|---|---|---|---|
1 | current_entry |
column |
root |
HowTo |
2 | current_entry |
column |
my path | HowTo |
np1 | current_entry |
row |
root |
Not possible |
3 | current_entry |
row |
my path | HowTo |
np2 | single_new_entry |
column |
root |
Not possible |
4 | single_new_entry |
column |
my path | HowTo |
np3 | single_new_entry |
row |
root |
Not possible |
5 | single_new_entry |
row |
my path | HowTo |
np4 | multiple_new_entries |
column |
root |
Not possible |
np5 | multiple_new_entries |
column |
my path | Not possible |
6 | multiple_new_entries |
row |
root |
HowTo |
7 | multiple_new_entries |
row |
my path | HowTo |
data_file:
type: str
description: "the tabular data file containing data"
m_annotations:
tabular_parser:
parsing_options:
comment: '#'
mapping_options:
- mapping_mode: column
file_mode: single_new_entry
sections:
- my_section/my_quantity
Plot¶
The PlotSection base section serves as an additional functionality to your sections. This base section is designed to simplify the process of creating various types of plots, making it easy to use Plotly Express, Plotly Subplot, and the general Plotly graph objects.
Features:
- Plotly Express: Create simple and quick plots with a high-level, expressive API.
- Plotly Subplot: Organize multiple plots into subplots for more complex visualizations.
- General Plotly Graph Objects: Fine-tune your plots by working directly with Plotly's graph objects.
Usage:
- Inherit from this base section to leverage its plot functionality.
- Customize your plots using the annotations plotly-express, plotly-subplots, or/and plotly-graph-object.
The PlotSection class makes it possible to define plots that are shown alongside your data. Underneath, we use the Plotly Open Source Graphing Libraries to control the creation of the plots, and you can find many useful examples in their documentation.
In Python schemas, the PlotSection class gives you full freedom to define plots programmatically. For example, you could use plotly.express and plotly.graph_objs to define plots like this:
from nomad.datamodel.metainfo.plot import PlotSection, PlotlyFigure
from nomad.datamodel.data import EntryData
import plotly.express as px
import plotly.graph_objs as go
from plotly.subplots import make_subplots
class CustomSection(PlotSection, EntryData):
m_def = Section()
time = Quantity(type=float, shape=['*'], unit='s', a_eln=dict(component='NumberEditQuantity'))
substrate_temperature = Quantity(type=float, shape=['*'], unit='K', a_eln=dict(component='NumberEditQuantity'))
chamber_pressure = Quantity(type=float, shape=['*'], unit='Pa', a_eln=dict(component='NumberEditQuantity'))
def normalize(self, archive, logger):
super(CustomSection, self).normalize(archive, logger)
first_line = px.scatter(x=self.time, y=self.substrate_temperature)
second_line = px.scatter(x=self.time, y=self.chamber_pressure)
figure1 = make_subplots(rows=1, cols=2, shared_yaxes=True)
figure1.add_trace(first_line.data[0], row=1, col=1)
figure1.add_trace(second_line.data[0], row=1, col=2)
figure1.update_layout(height=400, width=716, title_text="Creating Subplots in Plotly")
self.figures.append(PlotlyFigure(label='figure 1', figure=figure1.to_plotly_json()))
figure2 = px.scatter(x=self.substrate_temperature, y=self.chamber_pressure, color=self.chamber_pressure, title="Chamber as a function of Temperature")
self.figures.append(PlotlyFigure(label='figure 2', index=1, figure=figure2.to_plotly_json()))
heatmap_data = [[None, None, None, 12, 13, 14, 15, 16],
[None, 1, None, 11, None, None, None, 17],
[None, 2, 6, 7, None, None, None, 18],
[None, 3, None, 8, None, None, None, 19],
[5, 4, 10, 9, None, None, None, 20],
[None, None, None, 27, None, None, None, 21],
[None, None, None, 26, 25, 24, 23, 22]]
heatmap = go.Heatmap(z=heatmap_data, showscale=False, connectgaps=True, zsmooth='best')
figure3 = go.Figure(data=heatmap)
figure_json = figure3.to_plotly_json()
figure_json['config'] = {'staticPlot': True}
self.figures.append(PlotlyFigure(label='figure 3', index=0, figure=figure_json)
To customize the plot configuration in python one can add the config to the generated json by to_plotly_json().
In YAML schemas, plots can be defined by using the PlotSection as a base class, and additionally utilizing different flavours of plot annotations. The different annotation options are described below.
PlotlyGraphObjectAnnotation¶
Allows to plot figures using plotly graph object.
Example:
base_sections:
- 'nomad.datamodel.metainfo.plot.PlotSection'
m_annotations:
plotly_graph_object:
- data:
x: '#xArr'
y: '#xArr'
layout:
title:
text: 'Plotly Graph Object'
label: 'Plotly Graph Object'
index: 1
open: true
name | type | |
---|---|---|
label | str |
Figure label |
data | Dict |
Plotly data |
layout | Dict |
Plotly layout |
config | Dict |
Plotly config |
PlotlyExpressAnnotation¶
Allows to plot multi trace figures using plotly Express.
sections:
Example:
base_sections:
- 'nomad.datamodel.metainfo.plot.PlotSection'
m_annotations:
plotly_express:
method: scatter
x: '#xArr'
y: '#yArr'
label: 'Example Express Plot'
index: 0
open: true
layout:
title:
text: 'Example Express Plot'
xaxis:
title:
text: 'x axis'
yaxis:
title:
text: 'y axis'
traces:
- method: scatter
x: '#xArr'
y: '#zArr'
name | type | |
---|---|---|
method | str |
Plotly express plot method |
layout | Dict |
Plotly layout |
x | Union[List[float], List[str], str] |
Plotly express x |
y | Union[List[float], List[str], str] |
Plotly express y |
z | Union[List[float], List[str], str] |
Plotly express z |
color | Union[List[float], List[str], str] |
Plotly express color |
symbol | str |
Plotly express symbol |
title | str |
Plotly express title |
label | str |
Figure label |
traces | List[PlotlyExpressTraceAnnotation] |
List of traces added to the main trace defined by plotly_express methoddefault: [] |
PlotlyExpressTraceAnnotation¶
Allows to plot figures using plotly Express.
name | type | |
---|---|---|
method | str |
Plotly express plot method |
layout | Dict |
Plotly layout |
x | Union[List[float], List[str], str] |
Plotly express x |
y | Union[List[float], List[str], str] |
Plotly express y |
z | Union[List[float], List[str], str] |
Plotly express z |
color | Union[List[float], List[str], str] |
Plotly express color |
symbol | str |
Plotly express symbol |
title | str |
Plotly express title |
PlotlySubplotsAnnotation¶
Allows to plot figures in subplots.
Example:
base_sections:
- 'nomad.datamodel.metainfo.plot.PlotSection'
m_annotations:
plotly_subplots:
parameters:
rows: 2
cols: 2
layout:
title:
text: 'All plots'
plotly_express:
- method: scatter
x: '#xArr'
y: '#yArr'
title: 'subplot 1'
- method: scatter
x: '#xArr'
y: '#zArr'
title: 'subplot 2'
- method: scatter
x: '#zArr'
y: '#xArr'
title: 'subplot 3'
- method: scatter
x: '#zArr'
y: '#yArr'
title: 'subplot 4'
name | type | |
---|---|---|
label | str |
Figure label |
layout | Dict |
Plotly layout |
parameters | Dict |
plotly.subplots.make_subplots parameters i.e. rows, cols, shared_xaxes, shared_xaxes, horizontal_spacing , ... See plotly make_subplots documentation for more information. |
plotly_express | List[PlotlyExpressAnnotation] |
List of subplots defined by plotly_express methoddefault: [] |
plot annotations in python¶
For simple plots in Python schema one could use the annotations without normalizer:
from nomad.datamodel.metainfo.plot import PlotSection
from nomad.metainfo import Quantity, Section
from nomad.datamodel.data import EntryData
class CustomSection(PlotSection, EntryData):
m_def = Section(
a_plotly_graph_object=[
{
'label': 'graph object 1',
'data': {'x': '#time', 'y': '#chamber_pressure'},
'layout': {
'title': {
'text': 'Plot in section level'
},
'xaxis': {
'title': {
'text': 'x data'
}
},
'yaxis': {
'title': {
'text': 'y data'
}
}
}
}, {
'label': 'graph object 2',
'data': {'x': '#time', 'y': '#substrate_temperature'}
}
],
a_plotly_express={
'label': 'fig 2',
'index': 2,
'method': 'scatter',
'x': '#substrate_temperature',
'y': '#chamber_pressure',
'color': '#chamber_pressure'
},
a_plotly_subplots={
'label': 'fig 1',
'index': 1,
'parameters': {'rows': 2, 'cols': 2},
'layout': {
'title': {
'text': 'All plots'
}
},
'plotly_express': [
{
'method': 'scatter',
'x': '#time',
'y': '#chamber_pressure',
'color': '#chamber_pressure'
},
{
'method': 'scatter',
'x': '#time',
'y': '#substrate_temperature',
'color': '#substrate_temperature'
},
{
'method': 'scatter',
'x': '#substrate_temperature',
'y': '#chamber_pressure',
'color': '#chamber_pressure'
},
{
'method': 'scatter',
'x': '#substrate_temperature',
'y': '#chamber_pressure',
'color': '#substrate_temperature'
}
]
}
)
time = Quantity(type=float, shape=['*'], unit='s', a_eln=dict(component='NumberEditQuantity'))
substrate_temperature = Quantity(type=float, shape=['*'], unit='K', a_eln=dict(component='NumberEditQuantity'))
chamber_pressure = Quantity(type=float, shape=['*'], unit='Pa', a_eln=dict(component='NumberEditQuantity'))
PlotAnnotation (Deprecated)¶
The PlotAnnotation
is now deprecated and will be removed in future releases.
We recommend transitioning to the use of PlotSection
and PlotlyGraphObjectAnnotation
for your plotting needs.
This annotation can be used to add a plot to a section or quantity. Example:
class Evaporation(MSection):
m_def = Section(a_plot={
'label': 'Temperature and Pressure',
'x': 'process_time',
'y': ['./substrate_temperature', './chamber_pressure'],
'config': {
'editable': True,
'scrollZoom': False
}
})
time = Quantity(type=float, shape=['*'], unit='s')
substrate_temperature = Quantity(type=float, shape=['*'], unit='K')
chamber_pressure = Quantity(type=float, shape=['*'], unit='Pa')
You can create multi-line plots by using lists of the properties y
(and x
).
You either have multiple sets of y
-values over a single set of x
-values. Or
you have pairs of x
and y
values. For this purpose the annotation properties
x
and y
can reference a single quantity or a list of quantities.
For repeating sub sections, the section instance can be selected with an index, e.g.
"sub_section_name/2/parameter_name" or with a slice notation start:stop
where
negative values index from the end of the array, e.g.
"sub_section_name/1:-5/parameter_name".
The interactive examples of the plot annotations can be found here.
name | type | |
---|---|---|
label | str |
Is passed to plotly to define the label of the plot. |
x | Union[List[str], str] |
A path or list of paths to the x-axes values. Each path is a / separated list of sub-section and quantity names that leads from the annotation section to the quantity. Repeating sub sections are indexed between two / s with an integer or a slice start:stop . |
y | Union[List[str], str] |
A path or list of paths to the y-axes values. list of sub-section and quantity names that leads from the annotation section to the quantity. Repeating sub sections are indexed between two / s with an integer or a slice start:stop . |
lines | List[dict] |
A list of dicts passed as traces to plotly to configure the lines of the plot. See https://plotly.com/javascript/reference/scatter/ for details. |
layout | dict |
A dict passed as layout to plotly to configure the plot layout. See https://plotly.com/javascript/reference/layout/ for details. |
config | dict |
A dict passed as config to plotly to configure the plot functionality. See https://plotly.com/javascript/configuration-options/ for details. |