Deeploy.DeeployTypes.NetworkContainer

class Deeploy.DeeployTypes.NetworkContainer(graph: ~onnx_graphsurgeon.ir.graph.Graph, platform: ~Deeploy.DeeployTypes.DeploymentPlatform, inputTypes: ~typing.Dict[str, ~typing.Type[~Deeploy.AbstractDataTypes.Pointer]], scheduler: ~typing.Callable[[~onnx_graphsurgeon.ir.graph.Graph], ~typing.List[~typing.List[~onnx_graphsurgeon.ir.node.Node]] | ~typing.List[~onnx_graphsurgeon.ir.node.Node]] = <function NetworkContainer.<lambda>>, name: str = 'DeeployNetwork', deeployStateDir: str = 'DeeployState')

Bases: object

Deeploy abstraction for containing the information needed to describe a complete neural network to be deployed

Methods

__init__(graph: ~onnx_graphsurgeon.ir.graph.Graph, platform: ~Deeploy.DeeployTypes.DeploymentPlatform, inputTypes: ~typing.Dict[str, ~typing.Type[~Deeploy.AbstractDataTypes.Pointer]], scheduler: ~typing.Callable[[~onnx_graphsurgeon.ir.graph.Graph], ~typing.List[~typing.List[~onnx_graphsurgeon.ir.node.Node]] | ~typing.List[~onnx_graphsurgeon.ir.node.Node]] = <function NetworkContainer.<lambda>>, name: str = 'DeeployNetwork', deeployStateDir: str = 'DeeployState')

Initializes a new NetworkContainer and its NetworkContext

Parameters:

• graph (gs.Graph) – Neural network graph to be deployed

• platform (DeploymentPlatform) – DeploymentPlatform being targetted

• inputTypes (Dict[str, Type[Pointer]]) – DataType for each global network input

• scheduler (Callable[[gs.Graph], Schedule]) – Callable that ingests the graph and returns a list of operators to execute

• name (str) – Prefix to use in deployment to uniquify tensor names

• deeployStateDir (str) – Path to a directory to dump intermediate outputs

__init__(graph, platform, inputTypes[, ...])	Initializes a new NetworkContainer and its NetworkContext
bind()	Bind the entire network layer-by-layer
codeTransform([verbose])	Apply code transformations on every layer's execution block
exportDeeployState(folderPath, fileName)	Export compressed network context and neural network graph
generateBufferAllocationCode()	Generates code to allocate space for the global input and output buffer of the network
generateBufferDeAllocationCode()	Generates code to deallocate all global buffers
generateBufferInitializationCode()	Generates code for all forward-declaration of buffers used during inference
generateEngineInitializationCode()	Generate initialization code for all compute engines
generateGlobalDefinitionCode()	Generate all global definition code for inference
generateIOBufferInitializationCode()	Generate initialization code for global network inputs and outputs
generateIncludeString()	Generate code to include platform-dependent includes
generateInferenceCode()	Generate the actual inference function for the entire network
generateInferenceInitializationCode()	Generate initialization code, including static memory allocation and other setup tasks
getParameterSize()	Return the BYTE size of all static network parameters (weights, biases, parameters,...)
getTotalSize()	Returns total size of the network, consisting of all parameters and intermediate buffer size
importDeeployState(folderPath, fileName)	Override this container's graph and context with loaded compressed artifacts
inputs()	Return a list of all VariableBuffers that are also global inputs of the network
numberOfOps(verbose)	Returns the total number of operations per network inference
outputs()	Return a list of all VariableBuffers that are also global outputs of the network
parse([default_channels_first])	Parses the full network by iteratively exploring mapping and binding options with backtracking

Attributes

worstCaseBufferSize

Return the worst-case buffer size occupied by the network implementaiton

inputs() → List[VariableBuffer]

Return a list of all VariableBuffers that are also global inputs of the network

Returns:

Global inputs

Return type:

List[VariableBuffer]

outputs() → List[VariableBuffer]

Return a list of all VariableBuffers that are also global outputs of the network

Returns:

Global outputs

Return type:

List[VariableBuffer]

codeTransform(verbose: CodeGenVerbosity = CodeGenVerbosity(tilingProfiling=None))

Apply code transformations on every layer’s execution block

Parameters:

verbose (CodeGenVerbosity) – Control code generation verbosity

Raises:

RuntimeError – Raises a RuntimeError if the entire network is not bound

parse(default_channels_first: bool = True) → bool

Parses the full network by iteratively exploring mapping and binding options with backtracking

Parameters:

default_channels_first (bool) – Whether the default data layout is CxHxW or HxWxC

Returns:

Returns a boolean to indicate whether parsing was successful

Return type:

bool

Raises:

RuntimeError – Raises a RuntimeError if backtracking was exhausted without finding a mapping solution

bind() → bool

Bind the entire network layer-by-layer

Returns:

Return true if binding was successful

Return type:

bool

Raises:

RuntimeError – Raises a RuntimeError if the network has not been parsed of there exists no valid binding

generateInferenceCode() → str

Generate the actual inference function for the entire network

Returns:

The full inference method

Return type:

str

Raises:

ValueError – Raises a RuntimeError if network is not parsed and bound

generateGlobalDefinitionCode() → str

Generate all global definition code for inference

Returns:

Global Definition code

Return type:

str

Raises:

RuntimeError – Raises a RuntimeError if network is not parsed and bound

generateInferenceInitializationCode() → str

Generate initialization code, including static memory allocation and other setup tasks

Returns:

Initialization code

Return type:

str

Raises:

RuntimeError – Raises a RuntimeError if network is not parsed and bound

generateIOBufferInitializationCode() → str

Generate initialization code for global network inputs and outputs

Returns:

Initialization code

Return type:

str

Raises:

RuntimeError – Raises a RuntimeError if network is not parsed and bound

property worstCaseBufferSize

Return the worst-case buffer size occupied by the network implementaiton

generateBufferInitializationCode() → str

Generates code for all forward-declaration of buffers used during inference

Returns:

Returns forward-declaration code

Return type:

str

Raises:

RuntimeError – Raises a RuntimeError if network is not parsed and bound

generateBufferAllocationCode() → str

Generates code to allocate space for the global input and output buffer of the network

Returns:

Allocation code for global IO buffers

Return type:

str

Raises:

RuntimeError – Raises a RuntimeError if network is not parsed and bound

generateBufferDeAllocationCode() → str

Generates code to deallocate all global buffers

Returns:

Code to deallocate buffers

Return type:

str

Raises:

RuntimeError – Raises a RuntimeError if network is not parsed and bound

generateIncludeString() → str

Generate code to include platform-dependent includes

Returns:

Include code

Return type:

str

generateEngineInitializationCode() → str

Generate initialization code for all compute engines

Returns:

Initialization code for all engines

Return type:

str

getParameterSize() → int

Return the BYTE size of all static network parameters (weights, biases, parameters,…)

Returns:

Size of all network parameters

Return type:

int

Raises:

RuntimeError – Raises a RuntimeError if network is not parsed and bound

getTotalSize() → int

Returns total size of the network, consisting of all parameters and intermediate buffer size

Returns:

Total network size

Return type:

int

Raises:

RuntimeError – Raises a RuntimeError if network is not parsed and bound

numberOfOps(verbose: bool) → int

Returns the total number of operations per network inference

Parameters:

verbose (bool) – Control whether the number of operations are printed to STDOUT for each operator

Returns:

Number of operations (1 MAC = 2 Ops) per network inference

Return type:

int

Raises:

RuntimeError – Raises a RuntimeError if network is not parsed and bound

exportDeeployState(folderPath: str, fileName: str)

Export compressed network context and neural network graph

Parameters:

• folderPath (str) – path to directory where to save context and graph

• fileName (str) – prefix to use when saving artifacts

importDeeployState(folderPath: str, fileName: str)

Override this container’s graph and context with loaded compressed artifacts

Parameters:

• folderPath (str) – Path to the artifact directory

• fileName (str) – prefix of the saved artifacts