pyats.kleenex package

Module:

kleenex

Authors:

Myles Dear(mdear), CSG Test - Ottawa Siming Yuan (siyuan), CSG Test - Ottawa

Description:

This module provides a framework that allows users to:

• plug in own orchestrators that bring up dynamic device topologies on a variety of different backends.

• plug in their own cleaners that prepare a physical device for testing.

Subpackages

• pyats.kleenex.loader package
  • Submodules
• pyats.kleenex.reporter package
  • Submodules

Submodules

This module contains the pyATS clean and bringup base classes from which all clean and bringup implementations must inherit.

class pyats.kleenex.bases.BringUpBase(*args, **kwargs)

Bases: object

Base class for a dynamic topology orchestrator.

clean_config

Dictionary representation of the full clean configuration, if specified by the user.

Type

dict

bringup_clean_config

Dictionary representation of the subset of the clean configuration that specifies bringup worker configuration, if specified by the user.

Type

dict

autogenerated_clean_config

Dictionary representation of the autogenerated clean configuration, if populated by the bringup worker. All device names are actual and not logical. This configuration details the inputs used by a “mini-clean” done within the scope of bringup before the newly brought up topology is handed back for possible post-bringup cleaning. This “mini-clean” is often used to “babysit” virtual devices until they are up and ready for testing.

Type

dict

output_clean_config

Dictionary representation of the output clean configuration, if populated by the bringup worker, to be merged on top of the user-specified clean configuration.

Type

dict

logical_devices

Devices in the logical testbed config yaml file (if specified by the user) that are declared to be logical by use of logical: True.

Type

list

Create an instance of a dynamic topology orchestrator.

autogenerated_clean_config = None

bringup_clean_config = None

clean_config = None

launch_topology()

Launch the topology, wait for it to come up, leave it running.

logical_devices = []

output_clean_config = None

tear_down_topology()

Destroy the topology and reclaim all associated resources.

Note

Orchestrators (bringup workers) run in a worker subprocess.

Please ensure that you make a call to pyats.kleenex.request_bringup_worker_server_shutdown when you are done tearing down your topology. This ensures that the bringup worker subprocess gracefully terminates at the appropriate time.

The bringup worker is expected to install the appropriate signal handlers to guarantee all resources are properly cleaned up if the worker subprocess gets interrupted with a signal.

class pyats.kleenex.bases.BringUpWorkerBase(*args, topology_name=None, interactive=False, easypy=False, taskid=None, runinfo_dir=None, testbed_file=None, logical_testbed_file=None, clean_config=None, bringup_clean_config=None, bringup_log_level=None, skip_teardown=False, tb_yaml_output_file_name=None, **kwargs)

Bases: pyats.kleenex.bases.BringUpBase

Base class to be inherited by bringup worker (orchestrator) classes.

This class provides common infrastructure that any bringup implementation would need, including:

• asyncio framework (including exception handling model).

• merge an input (sparse) testbed config with the raw testbed config coming from the orchestrator.

• logical to actual device and interface name matching

• ability to pull in pluggable CLI help content from the inheriting worker class. This allows a consolidated help display to be provided to the decoupled bringup tool.

logical_virtual_devices

Devices in the testbed config yaml file (if specified by the user) that are declared to be logical and virtual by use of logical: True and device: type specifying a supported virtual device type.

Type

list

static_testbed_config

If the user specifies a testbed YAML, this variable contains the configuration details specified in that file.

Type

dict

logical_testbed_config

If the user specifies a logical testbed YAML, this variable contains the configuration details specified in that file. It is expected that if this file specifies logical virtual devices by containing logical: True and having a device: type declared whereby the type is a member of allowed_virtual_device_types. This file is considered sparse and not able to be validated until testbed configuration generated by the orchestration backend is merged in, the dynamic router placeholder sections are removed and logical-to-actual mapping is done.

Type

dict

Create an instance of a dynamic topology orchestrator.

Users inheriting from this class are expected to call super().__init__ (*args, **kwargs) as their very first line.

Parameters

• topology_name (str) – A name for this topology. If specified, a timestamp is appended to it to ensure uniqueness.

• interactive (bool) – Whether or not the testbed is being brought up in a standalone interactive tool. Internal parameter (not set via CLI). Defaults to False.

• easypy (bool) – If True, then the testbed is being launched from an easypy plugin. Internal parameter (not set via CLI). Defaults to False.

• taskid (str) – The unique taskid, used when launched via an easypy plugin. Internal parameter (not set via CLI). Defaults to None.

• runinfo_dir (str) – The directory to place orchestrator logs and the synthesized testbed YAML file into (passed in when invoked via easypy). Internal parameter (not set via CLI). Defaults to None.

• skip_teardown (bool) – If True, and easypy is True, then do not automatically tear down the topology when so requested and warn the user they are now responsible for tearing it down. If False, then tear down the topology when so requested. If easypy is False this option is ignored. Defaults to False.

• testbed_file (str) – The testbed config YAML filename (passed in when invoked via easypy). This testbed config may contain only static (non-logical) devices. If not specified, the testbed file is taken from the CLI input parameter -testbed_file. Defaults to None.

• logical_testbed_file (str) – The testbed config YAML filename (passed in when invoked via easypy). This testbed config may contain either static (non-logical) or logical devices. If not specified, the testbed file is taken from the CLI input parameter -logical_testbed_file. Defaults to None.

• bringup_log_level (str) – Logging level to use for bringup. One of the following : debug, info, warning, error, critical. If not specified, the value from bringup/log_level in the clean configuration YAML is used. Defaults to info.

• tb_yaml_output_file_name (str) – The name of the synthesized pyATS testbed YAML file that is automatically written after the topology has been launched. If not specified but easypy is True, a unique name is chosen to be written to the runinfo directory. Defaults to None, meaning no file is written.

• clean_config (dict) – A dictionary holding user-specified clean configuration content. Orchestrator configuration content is a subset of the clean configuration. This parameter is internal and cannot be specified via CLI. Defaults to {}.

• bringup_clean_config (dict) – A dictionary holding the subset of the user-specified clean configuration content under the orchestrator class. This parameter is internal and cannot be specified via CLI. Defaults to {}.

dev_name_xref

This dict must be populated by the inheriting class. It is indexed by actual device name and contains the corresponding logical device name.

Type

dict

dev_intf_xref

This dict is populated as part of the just-in-time testbed topology processing logic. It is indexed by actual device name and actual interface name and contains the corresponding logical interface name.

Type

dict

event_loop

Event loop for running asyncio tasks. This loop is used to coordinate all the async events occuring during testbed launch and teardown.

Type

asyncio.unix_events._UnixSelectorEventLoop

help

This gets set to true when the user specifies -help via the command line (as is the case when they type bringup -help).

Type

bool

shutting_down

Set to True when the topology is shutting down. This is used by the signal handling model to ensure the topology is not torn down if it is already shutting down.

Type

bool

testbed_config

Contains the testbed YAML content emitted by the orchestrator using information only available at run-time (such as console connection details). This content is merged with logical_testbed_config and static_testbed_config (if specified) and saved in this variable. This testbed config is meant to be loaded into a pyATS topology object.

Type

dict

topology_name

A unique name for this topology.

Type

str

logical_devices

A list of logical devices in the provided topology.

Type

list

logical_virtual_devices

A list of logical virtual devices in the provided topology. A virtual device is one whose type is one of the allowed virtual device types.

Type

list

logical_physical_devices

A list of logical physical devices in the provided topology.

Type

list

actual_devices

A list of actual devices populated by the orchestrator.

Type

list

launch_topology()

Schedule a virtual topology to be launched.

logical_physical_devices = []

logical_testbed_config = {}

logical_virtual_devices = []

set_interactive(interactive)

Setter to allow member to be set by xmlrpc client.

static_testbed_config = {}

tear_down_topology(raise_stored_exception=True)

Schedule a virtual topology to be gracefully torn down.

update_help()

Update help from my CLI parser.

Implementors inheriting from this class are expected to first call this parent implementation, then update the help from their own parser.

A CLI parser for BringUpWorkerBase

pyats.kleenex.bringup_base_cli_parser.populate_base_bringup_parser(parser)

Add bringup options to a parser.

Module responsible for bringing up and tearing down dynamic topologies in their own subprocesses, to ensure graceful teardown always occurs.

class pyats.kleenex.bringup_manager.BringUp(*args, orchestrator=None, clean_files=None, bringup_log_level=None, clean_config=None, bringup_clean_config=None, debug=False, **kwargs)

Bases: pyats.kleenex.bases.BringUpBase

Bring up and tear down a dynamic topology.

Parameters may be specified either directly or via the command line.

This class causes the bringup worker object to be instantiated in a subprocess and manages it thereafter via a proxy.

Instantiate a bringup proxy instance.

This object follows the same base API as a bringup worker, but causes the actual worker object to be instantiated in a subprocess.

Requests to this object are proxied to the worker object.

Parameters

• clean_files (list) – The loadables containing the clean configuration dict or files containing YAML text.

• orchestrator (str) – The bringup worker class to instantiate. If not specified, the value from bringup/class in the clean configuration YAML is used. Otherwise, an AssertionError is thrown.

• bringup_log_level (str) – Logging level to use for bringup. One of the following : debug, info, warning, error, critical. If not specified, the value from bringup/log_level in the clean configuration YAML is used. Defaults to info.

• clean_config (dict) – Clean configuration (which contains the bringup worker configuration) if not specified via clean_file. This parameter is internal and cannot be specified via CLI.

• bringup_clean_config (dict) – Bringup worker configuration (which is a subset of clean configuration) if not specified implicitly via clean_file. This parameter is internal and cannot be specified via CLI.

• debug (bool) – Whether or not to enable debug mode. If specified as True and a bringup worker fails to instantiate or throws an exception an interactive debugger is launched at the point of failure. It also becomes possible for a user to insert pdb.set_trace() calls in their worker and perform debugging on the worker as it runs in its forked subprocess. If specified as False the worker cannot be debugged. Defaults to False. This parameter is internal and cannot be specified via CLI.

bringup_worker

This object is created when the user enters a context manager block and represents by proxy the actual bringup instance running in a forked subprocess.

Type

xmlrpc.client.ServerProxy

manager_cli_help_details

CLI parser help details populated by pyats.kleenex.bases.BringUpWorkerBase.update_help. Used to ensure a consistent display when an orchestrator’s xxxxbringup -help is called.

Type

list

manager_cli_usage_details

CLI parser usage details populated by pyats.kleenex.bases.BringUpWorkerBase.update_help. Used to ensure a consistent display when bringup -help is called.

Type

list

Raises

AssertionError – If the bringup worker class is not specified.

property cli_help_details

Retrieve the CLI help details for the bringup -help command.

property cli_usage_details

Retrieve the CLI usage details for the bringup -help command.

property help

Retrieve whether or not the topo was launched in help mode.

property interactive

Retrieve whether or not the topo was launched in interactive mode.

launch_topology()

Command the bringup worker to launch the topology.

property logical_devices

Retrieve list of logical devices.

property output_clean_config

Retrieve the output clean configuration. The user may use this property to merge just-in-time clean configuration on top of static clean configuration originating from the clean YAML file.

tear_down_topology()

Command the bringup worker to tear down the topology.

property topology_config

Retrieve the post-bind testbed topology. The user uses this property to retrieve the testbed configuration when the orchestrator has successfully bound the logical topology to an actual topology.

property topology_name

Retrieve the topology name. The user uses this property to retrieve the topology name.

update_help()

Update help from my CLI parser.

pyats.kleenex.bringup_manager.request_bringup_worker_server_shutdown()

Request XMLRPC server shutdown. The last thing the bringup worker must do before returning after tearing down the topology is to request its server to shut itself down. This is necessary to return control back to the calling user.

A CLI parser for BringUp

pyats.kleenex.bringup_manager_cli_parser.populate_bringup_manager_parser(parser)

Add bringup manager options to a parser.

Signal handling to ensure graceful topology termination.

pyats.kleenex.bringup_signals.install_signal_handlers()

Set up handlers for all possible signals to ensure the virtual testbed gets taken down correctly.

class pyats.kleenex.engine.KleenexEngine(reporter, debug_=False, **clean_mappings)

Bases: object

KleenexEngine effectively allows users to clean multiple devices at the same time, using various clean implementations for each device, without having to worry about about forks, processes & etc.

Internally this class is just about the same as async_.pcall, however, as clean definitions are designed to be mostly reused, there are some overlaps in code…

clean(devices, timeout=None, task=None, lcl_logger=<Logger pyats.kleenex.engine (WARNING)>)

Clean one or more sets of devices. Devices within each set are cleaned in parallel (if not running in debug mode).

Each set is cleaned sequentially.

clean_testbed(testbed, *, timeout=None, task=None, lcl_logger=<Logger pyats.kleenex.engine (WARNING)>)

Performs best-attempt cleaning of the testbed, assuming there is clean definition for each device in this testbed. If none is found - that device is skipped

classmethod from_cleandefs(*cleandefs, reporter, debug_=False)

has_cleandef(device)

Test whether KleenexEngine has clean definitions for a given device.

Parameters

device (device object or device name string) –

Returns

Return type

True or False

static update_testbed(testbed, **cleaninfo)

exception pyats.kleenex.exceptions.ArgsParseError

Bases: Exception

Exception when there is an exception with args parse argument

class pyats.kleenex.exceptions.BringupExceptionWithTraceback(exc)

Bases: pyats.kleenex.exceptions.ExceptionWithTraceback

get_user_exception()

class pyats.kleenex.exceptions.CleanExceptionWithTraceback(exc, device_name)

Bases: pyats.kleenex.exceptions.ExceptionWithTraceback

exception pyats.kleenex.exceptions.FailedToCreateBringupWorker(*args)

Bases: pyats.kleenex.exceptions.BringupExceptionBase

Exception class for bringup worker that cannot be instantiated

exception pyats.kleenex.exceptions.NoConnectionError(port, client_pid, *args)

Bases: pyats.kleenex.exceptions.BringupExceptionBase

Exception class for clients that tries to connect or transfer information to the server but are not connected.

exception pyats.kleenex.exceptions.NoOrchestratorSpecified(*args)

Bases: pyats.kleenex.exceptions.BringupExceptionBase

Exception class for unspecified bringup class

exception pyats.kleenex.exceptions.ServerNotStarted(*args)

Bases: pyats.kleenex.exceptions.BringupExceptionBase

Exception class for server that cannot be started

exception pyats.kleenex.exceptions.TopologyDidntComeUpInTime(*args)

Bases: pyats.kleenex.exceptions.BringupExceptionBase

The topology took too long to come up.

exception pyats.kleenex.exceptions.TopologyTerminatedDueToSignal

Bases: pyats.kleenex.exceptions.BringupExceptionBase

A received signal has forced the topology to terminate.

exception pyats.kleenex.exceptions.YamlConfigError

Bases: Exception

pyats.kleenex.exceptions.format_exception(etype, value, tb, limit=None, chain=True)

A decoupled tool to perform bringup and/or clean. If bringup is selected, launch the topology, optionally clean the topology, wait for the user to press <Ctrl><C>, then tear down the topology.

returns

• Returns a Linux exit code of 0 if all actions succeeded.

• Otherwise, returns a Linux exit code if 1 if one or more actions failed.

class pyats.kleenex.kleenex_main.KleenexMain(testbed_file=None, clean_files=None, clean_devices=None, loglevel=20, logdir=None, no_logdir=None, orchestrator=None, debug=False, no_mail=False, commandline=False, device_image_deprecated=None, os_image_deprecated=None, clean_separator=':', device_image=None, os_image=None, group_image=None, platform_image=None, legacy=True, parser=None, no_upload=False, **kwargs)

Bases: object

This class implements bringup/clean behavior.

Bringup describes how a logical topology is converted into an actual topology consisting of physical and/or software-based routers.

Clean describes how one or more physical devices are prepared for testing by placing user-specified images on them. It may also include such steps as netbooting, bringing up a device from rommon mode or power cycling a device in order to recover it.

Create an instance of a bringup/clean orchestrator.

If an exception is thrown or a CLI argument fails validation, sys.exit(1) is called. This allows proper integration with test runners such as Jenkins.

Parameters

• testbed_file (str) – A YAML-formatted file describing the devices in the topology and how they are interconnected.

• clean_files (list) – YAML-formatted files describing how the devices in the topology are to be brought up and / or cleaned.

• clean_devices (list) – A list of devices to clean. If not specified, all devices in the topology are cleaned.

• loglevel (str) – The Kleenex log level. One of the following : DEBUG, INFO, WARNING, ERROR, CRITICAL.

• logdir (str) – The directory to store Kleenex logs and console output to. If not specified, defaults to a time-stamped directory under the current working directory, with a prefix of “Kleenex”.

• no_logdir (bool) – If True, no log directory is produced and all output goes to the console. If False, a log directory is produced as described by the logdir parameter. Defaults to False.

• orchestrator (pyats.kleenex.bases.BringUpWorkerBase) – The bringup orchestrator to use. If not specified, then bringup is skipped.

• no_mail (bool) – If True, no email is sent on failure. If False, an email is sent on failure Defaults to False.

• commandline (bool) –

  If True, the object is being instantiated as part of a standalone tool and all arguments appear in the -help display.

  If False, the object is being instantiated programmatically and the user is expected to pass arguments to the constructor instead of via the command line. Defaults to False.

• legacy (bool) –

  If True, the commandLine arguments use the legacy version such as -testbed_file

  If False, the commandLine arguments use the current version such as --testbed-file

• parser (obj) – Provide a custom parser object

• no_upload (bool) – If True, log archive uploading is disabled. If False, log archives are uploaded to TaaS Defaults to False.

parser

CLI parser that does help processing and contains help/usage details from other CLI parsers contained by the module.

Type

pyats.kleenex.utils.ArgvQuotingParser

local_parser

CLI parser for local module-scope arguments.

Type

pyats.kleenex.utils.ArgvQuotingParser

bringup

Bringup instance containing usage and help text.

Type

pyats.kleenex.bringup_manager.BringUp

help_requested

Set to True then this class has been instantiated in commandline mode and the user has specified -h on the command line. Otherwise, defaults to False.

Type

bool

cleanup()

configure_local_parser(commandline, legacy)

Create parser for all local module-scope arguments except help.

configure_logging()

configure logging Configure logging level based on the values from CLI args parsing and values passed to the method.

Parameters

None –

configure_parser(commandline, legacy)

Create a parser integrating help content with contained parsers.

do_clean_if_required()

generate_archive_file()

Generate archive file for clean

get_results_zipfile_content()

Produce zipfile content for all results.

Returns

Return type

Binary representation of the zipfile content.

instantiate_bringup_worker_if_required()

launch_dynamic_topology_if_required()

Lauch a dynamic topology if logical devices have been specified.

run(**kwargs)

Execute bringup and / or clean behavior as required.

start_liveview()

stop_liveview()

teardown_dynamic_topology_if_required()

update_contained_help()

Consolidate a parser’s usage and help text from contained objs.

Updates the help output of a provided parser. Automatically aggregates its current help information with those of the plugins. this allows an orchestrator’s xxxbringup -help to show all available arguments from every class along the chain.

upload_results_to_taas()

upload_results_to_xpresso_if_required()

write_env_file()

Utilities to filter and format exceptions.

This module allows stack traces to be simplified for user consumption, ensuring parts of the traceback that are either internal to the pyATS core or otherwise considered noisy, confusing or not adding value for support purposes to be omitted from the final customer-facing exception output.

The format_exception API from the core traceback package is overloaded here and given a new parameter, skip_pred.

A skip predicate is evaluated for each stack frame in an exception, and in all associated cause/context stack frames as well. If the predicate evaluates to False, it is not included in the traceback output.

pyats.kleenex.kleenex_traceback.format_exception(etype, value, tb, limit=None, chain=True, skip_pred=None)

Format a stack trace and the exception information.

The arguments have the same meaning as the corresponding arguments to print_exception(). The return value is a list of strings, each ending in a newline and some containing internal newlines. When these lines are concatenated and printed, exactly the same text is printed as does print_exception().

This module contains the Kleenex YAML schema used by clean and bringup.

pyats.kleenex.schema.allowed_virtual_device_types

The list of allowed virtual device types.

Type

list of str

This module contains common utilities required by bringup.

pyats.kleenex.utils.help_suppress_kleenex(msg)

Suppress this argument if called as a decoupled kleenex tool. Since kleenex requires clean and testbed YAML files to be specified, all DE-scope arguments are to be suppressed. A DE-scope argument always has an equivalent in the clean YAML schema. The DE-scope arguments may be accessed via the normal orchestrator- specific bringup tool.

Parameters

msg (str) – The help text that is suppressed in the kleenex -help display and otherwise appears in the decoupled bringup tool’s xxxbringup -help display.

pyats.kleenex.utils.parse_cli_args(parser)

Parse known CLI arguments found in sys.argv. After parsing the known arguments remove them from sys.argv and keep the rest.

Implementation copied from pyats.aetest.main.parseArgs.

Returns

Return type

dictionary of argument name/value where value is not None

pyats.kleenex.utils.tear_down_all_topologies()

Tear down all topologies. This is typically called when an exception is caught after topologies have already been brought up.

pyats.kleenex.utils.testbed_config_contains_logical_routers(testbed_config)

testbed_config: file or dict

The loadable containing testbed configuration details as passed in by the user via the testbed_config CLI parameter. The loadable may be in one of the following forms:

• dictionary respecting the yaml schema format

• full path/name of the yaml testbed file

• any yaml python stream readable

Returns

True if one or more routers in the testbed config YAML file contain the key logical: True, False otherwise.

Return type

bool

pyats.kleenex.utils.validate_clean_devices(clean_devices)

Validate clean devices.

Returns

• None if no devices to clean.

• Otherwise, the devices to clean in a two-deep list format

• [[‘dev1’, [‘dev2’, ‘dev3’], [‘dev4’]]

• This input causes the following actions to be taken (preceding

• step must be complete for the next step to start) –

  1. dev1 is cleaned.

  2. dev2/dev3 are cleaned in parallel.

  3. dev4 is cleaned

• A single-deep list format is also supported. In the following

• example, dev1, dev2, dev3 and dev4 are cleaned in parallel

• [‘dev1’, ‘dev2’, ‘dev3’, ‘dev4’]

The following formats are allowed:

1. A string containing a space-separated list of devices. devices specified in quotes are made part of the list. All devices in this list are cleaned in parallel. dev1 dev2 dev3 “dev4 dev5”

2. A YAML-formatted string “[dev1, [dev2, dev3], dev4]”

   Validation fails if the user nests the lists more than one level deep, for example : [a, b, [c, d, [e, f]]]

3. A list of device names.

class pyats.kleenex.worker.KleenexSteps(parent=None)

Bases: pyats.aetest.steps.implementation.Steps

built-in __init__

instantiates the Steps class by creating an internal list that houses number of steps currently taken.

Note

step index is the same as the list index of current step.

start()

Creates a new Step instance and return it to the user scope. This is the main API called within the user realm to create and start new steps within the current scope.

Parameters

• args (all arguments to this api propagates directly to Step) –

• kwargs (all arguments to this api propagates directly to Step) –

Returns

Return type

new Step() instance.

class pyats.kleenex.worker.KleenexWorker(cleaner, device, reporter, logfile, queue, timeout, debug_=False)

Bases: multiprocessing.context.ForkProcess

property result

run()

Method to be run in sub-process; can be overridden in sub-class