pyconform.dataflow¶
Data Flow Classes and Functions
This module contains the classes and functions needed to define Data Flows.
A Data Flow is a directed acyclic graph (DAG) that describes the flow of data from one node in the graph to another. Each node in the flow represents a data action of some sort, such as reading data from file, transposing data, unit conversion, addition, subtraction, etc. The data transmitted along the graph edges is assumed to a Numpy.NDArray-like object.
The action associated with each node is not performed until the data is “requested” with the __getitem__ interface, via Node[key].
Copyright 2017-2020, University Corporation for Atmospheric Research LICENSE: See the LICENSE.rst file for details
-
class
pyconform.dataflow.DataFlow(inpds, outds)[source]¶ Bases:
objectAn object describing the flow of data from input to output
-
property
dimension_map¶ The internally generated input-to-output dimension name map
-
execute(chunks={}, serial=False, history=False, scomm=None, deflate=None, debug=False)[source]¶ Execute the Data Flow
- Parameters
chunks (dict) – A dictionary of output dimension names and chunk sizes for each dimension given. Output dimensions not included in the dictionary will not be chunked. (Use OrderedDict to preserve order of dimensions, where the first dimension will be assumed to correspond to the fastest-varying index and the last dimension will be assumed to correspond to the slowest-varying index.)
serial (bool) – Whether to run in serial (True) or parallel (False)
history (bool) – Whether to write a history attribute generated during execution for each variable in the file
scomm (SimpleComm) – An externally created SimpleComm object to use for managing parallel operation
deflate (int) – Override all output file deflate levels with given value
debug (bool) – Whether to enable some rudimentary debugging features
-
property