pyconform.flownodes¶
Data Flow Node Classes and Functions
This module contains the classes and functions needed to define nodes in Data Flows.
Copyright 2017-2020, University Corporation for Atmospheric Research LICENSE: See the LICENSE.rst file for details
-
class
pyconform.flownodes.DataNode(data)[source]¶ Bases:
pyconform.flownodes.FlowNodeFlowNode class to create data in memory
This is a “source” FlowNode.
-
exception
pyconform.flownodes.DateTimeAutoParseWarning[source]¶ Bases:
WarningWarning for not being able to autoparse new filename based on date-time in the file
-
class
pyconform.flownodes.EvalNode(label, func, *args, **kwds)[source]¶ Bases:
pyconform.flownodes.FlowNodeFlowNode class for evaluating a function on input from neighboring DataNodes
The EvalNode is constructed with a function reference and any number of arguments to that function. The number of arguments supplied must match the number of arguments accepted by the function. The arguments can be any type, and the order of the arguments will be preserved in the call signature of the function. If the arguments are of type FlowNode, then a reference to the FlowNode will be stored. If the arguments are of any other type, the argument will be stored by the EvalNode.
This is a “non-source”/”non-sink” FlowNode.
-
property
sumlike_dimensions¶ Return the set of sum-like dimensions registered by the node’s function
-
property
-
class
pyconform.flownodes.FlowNode(label, *inputs)[source]¶ Bases:
objectThe base class for objects that can appear in a data flow
The FlowNode object represents a point in the directed acyclic graph where multiple edges meet. It represents a functional operation on the DataArrays coming into it from its adjacent DataNodes. The FlowNode itself outputs the result of this operation through the __getitem__ interface (i.e., FlowNode[item]), returning a slice of a PhysArray.
-
property
inputs¶ Inputs into this FlowNode
-
property
label¶ The FlowNode’s label
-
property
-
class
pyconform.flownodes.MapNode(label, dnode, dmap={})[source]¶ Bases:
pyconform.flownodes.FlowNodeFlowNode class to map input data from a neighboring FlowNode to new dimension names and units
The MapNode can rename the dimensions of a FlowNode’s output data. It does not change the data itself, however. The input dimension names will be changed according to the dimension map given. If an input dimension name is not referenced by the map, then the input dimension name does not change.
This is a “non-source”/”non-sink” FlowNode.
-
class
pyconform.flownodes.ReadNode(variable, index=slice(None, None, None))[source]¶ Bases:
pyconform.flownodes.FlowNodeFlowNode class for reading data from a NetCDF file
This is a “source” FlowNode.
-
class
pyconform.flownodes.ValidateNode(vdesc, dnode)[source]¶ Bases:
pyconform.flownodes.FlowNodeFlowNode class to validate input data from a neighboring FlowNode
The ValidateNode takes additional attributes in its initializer that can effect the behavior of its __getitem__ method. The special attributes are:
‘valid_min’: The minimum value the data should have, if valid ‘valid_max’: The maximum value the data should have, if valid ‘min_mean_abs’: The minimum acceptable value of the mean of the absolute value of the data ‘max_mean_abs’: The maximum acceptable value of the mean of the absolute value of the data
If these attributes are supplied to the ValidateNode at construction time, then the associated validation checks will be made on the data when __getitem__ is called.
Additional attributes may be added to the ValidateNode that do not affect functionality. These attributes may be named however the user wishes and can be retrieved from the FlowNode as a dictionary with the ‘attributes’ property.
This is a “non-source”/”non-sink” FlowNode.
-
property
attributes¶ Attributes dictionary of the variable returned by the ValidateNode
-
property
dimensions¶ Dimensions tuple of the variable returned by the ValidateNode
-
property
-
exception
pyconform.flownodes.ValidationWarning[source]¶ Bases:
WarningWarning for validation errors
-
class
pyconform.flownodes.WriteNode(filedesc, inputs=())[source]¶ Bases:
pyconform.flownodes.FlowNodeFlowNode that writes validated data to a file.
This is a “sink” node, meaning that the __getitem__ (i.e., [index]) interface does not return anything. Rather, the data “retrieved” through the __getitem__ interface is sent directly to file.
For this reason, it is possible to “retrieve” data multiple times, resulting in writing and overwriting of data. To eliminate this inefficiency, it is advised that you use the ‘execute’ method to write data efficiently once (and only once).
-
execute(chunks={}, deflate=None)[source]¶ Execute the writing of the WriteNode file at once
This method efficiently writes all of the data for each file only once, chunking the data according to the ‘chunks’ parameter, as needed.
- Parameters
chunks (dict) – A dictionary of output dimension names and chunk sizes for each dimension given. Output dimensions not included in the dictionary will not be chunked. (Use OrderedDict to preserve order of dimensions, where the first dimension will be assumed to correspond to the fastest-varying index and the last dimension will be assumed to correspond to the slowest-varying index.)
deflate (int) – Override the output file deflate level with given value
-