Glossary¶

This document defines the formal lexicon for Huginn. These terms have specific meanings within the framework and documentation.

Infrastructure¶

Device¶

A unit of infrastructure within a testbed. Devices can be network equipment (routers, switches, firewalls), servers, appliances, or any system that can be connected to and queried. Each device has a hostname, operating system identifier, optional device group memberships, and one or more connection configurations.

Device Group¶

A logical grouping of devices in a testbed. Device groups are arbitrary labels assigned to devices for organizational and targeting purposes. A device can belong to multiple device groups. Common patterns include role-based groups (spine, leaf, border), location-based groups (datacenter-1, building-a), and environment-based groups (production, staging).

Testbed¶

A collection of devices making up a production-like, scaled-down version of an environment. The testbed defines the infrastructure inventory including device connection parameters, device group memberships, and metadata. Defined in YAML format.

Test Definitions¶

Job¶

A unit of test automation that can be executed against one or more devices in a testbed. A job is implemented as a Python module containing a class that inherits from the TestCase base class. Jobs define reusable test logic independent of specific targets or parameters. The same job can be referenced by multiple test cases, each with different parameters.

Parameters¶

Expected state data associated with a test case, used for validation during test execution. Parameters can be sourced from:

File-based: JSON files following the convention parameters/{test_case_id}.json. In learning mode, parameters are captured from live infrastructure and persisted. In testing mode, parameters are loaded and compared against current state.
Data model-based: Derived from an external data model (e.g., Network as Code YAML) representing intended infrastructure state. The job queries the data model for expected values.

If a test case has no parameters file and no data model, it requires execution in learning mode to establish baseline parameters.

Data Model¶

An external source of truth representing intended infrastructure state, typically a YAML file conforming to a defined schema. Data models can serve as parameters for test cases, enabling validation of actual device state against declared intent. This pattern is commonly used with Infrastructure as Code approaches like Cisco's Network as Code.

Target¶

The specification of which devices a test case applies to. Targets can be defined by:

Explicit devices: A list of device names.
Device groups: All devices belonging to specified device groups.
Operating system: All devices running specified operating systems.

Multiple selectors are intersected (AND logic). Targets can be specified at the phase level, test case group level, test case level, or any combination (intersected).

Test Case¶

A first-class entity in the test plan that instantiates a job with a specific identity. Test cases are defined once in the test_cases section of a test plan and referenced by ID in test case groups. Each test case has:

A unique identifier
A reference to a job (module path)
Optional target specification
Optional tags for filtering
Parameters sourced by convention (parameters/{id}.json) or from a data model

The same test case can be referenced in multiple test case groups, enabling reuse across phases (e.g., pre-change and post-change validation).

Test Case Group¶

A logical grouping of test cases and/or other test case groups within a phase. Test case groups reference test cases by ID and can include other groups by name, enabling hierarchical organization. Groups can specify targets that apply to all contained test cases, and the framework executes test cases within a group potentially in parallel.

Nesting: Test case groups can be nested to create reusable, feature-specific groupings. For example, an "ospf-tests" group containing OSPF-related test cases can be included in both "pre-change-validation" and "post-change-validation" groups. This promotes reuse and keeps feature-specific tests organized together.

Phase¶

A high-level organizational unit in a test plan representing a stage of test execution. Phases contain one or more test case groups and can declare dependencies on other phases. Common patterns include:

Pre-change: Validate state before making changes
Change: Apply configuration or operational changes
Post-change: Validate state after changes

Phases provide structure for reporting (collapse/expand, filtering) and establish execution order through dependencies.

Test Plan¶

A collection of test cases, test case groups, phases, and associated metadata defining how testing outcomes should be achieved. The test plan specifies:

Test cases: First-class definitions of what to test
Test case groups: Logical groupings of test case references
Phases: High-level stages with dependencies defining execution order

Defined in YAML format.

Execution¶

Mode¶

The execution mode for a test run. Huginn supports two modes:

Learning: Execute against live infrastructure, capture current state, and persist it as parameters for future comparison.
Testing: Execute against live infrastructure, compare current state against previously learned parameters (or data model), and report deviations.

Run¶

A single execution of a test plan against a testbed. A run establishes connections to all devices, executes phases in dependency order, executes test case groups within each phase, collects results, and generates reports. Test cases filtered out by tags or other criteria do not appear in run results.

Command Support¶

The determination of whether a target device supports the CLI command(s) required by a test case. Command support can be:

Static: Declared in the test plan via target specifications (devices, device groups, operating systems). Resolved before test execution.
Dynamic: Determined at runtime by the test case itself via the check_command_support() method or during gather_state(). Enables tests to introspect their assigned targets and filter based on device capabilities, running features, or other runtime conditions.

Dynamic non-applicability arises from two distinct situations:

Command not supported: The device does not recognize the show command the job requires. Detected in check_command_support().
Attribute absent: The command succeeds but the specific attribute the job validates does not exist in the parsed output - either because the platform does not report it or because the feature is not configured. Detected in gather_state() when per-item extraction produces an empty result for a device.

A device that is statically targeted but dynamically determined to lack command support is recorded with a NOT_APPLICABLE or LOST_APPLICABILITY result (depending on whether learned parameters exist) and the reason for non-support.

CommandSupportResult¶

The return type of a test case's check_command_support() method. Contains:

applicable: List of devices that support the required command(s).
not_applicable: Dictionary mapping device names to reasons why the device does not support the required command(s).

The framework uses this result to update context.targets before calling setup() and test().

Context¶

The object passed to jobs during execution. Contains access to the connection broker, target devices, results collector, parameters (file-based or data model), and execution metadata. The context is the primary interface between a job and the framework.

Result¶

The outcome of a test case execution. Possible values:

Passed: All assertions succeeded.
Failed: One or more assertions did not match expected state.
Errored: An exception occurred during execution.
Not Applicable: The test case was in scope but determined at runtime to be not applicable, and no learned parameters exist for the device. This is the expected outcome when a test is assigned to a broad target group but only applies to a subset of devices.
Lost Applicability: The test case was applicable when parameters were learned but is no longer applicable during testing. This indicates that something changed between learning and testing that caused a previously testable device to become untestable. This status contributes to test failure by default, as it typically indicates an unexpected change that warrants investigation.
Blocked: The test case could not run because a dependency (phase or group) failed.

The distinction between NOT_APPLICABLE and LOST_APPLICABILITY is important:

Scenario	Learned Parameters Exist?	Result
Device never applicable	No	NOT_APPLICABLE
Device was applicable, now isn't	Yes	LOST_APPLICABILITY

Test cases filtered out before execution (e.g., by tags) do not appear in results at all. This is distinct from NOT_APPLICABLE and LOST_APPLICABILITY, which appear in results with reasons.

Aggregate Result¶

The computed outcome for a test case group or phase, derived from the results of contained test cases. Possible values:

Passed: 100% of test cases passed (no failures, errors, or lost applicability results).
Partial: Some test cases passed, some did not (mixed results including failures, errors, or lost applicability).
Failed: 0% of test cases passed (catastrophic failure).
Blocked: Could not execute because a dependency failed.
Not Applicable: All contained test cases were not applicable (none were applicable and none had prior learned parameters).

Aggregate results include counts (e.g., "1995/2000 passed") to provide visibility into the scope and nature of any failures. LOST_APPLICABILITY results are counted separately from NOT_APPLICABLE in aggregate reporting to highlight unexpected applicability changes.