Skip to content

Future Considerations

This document captures use cases, architectural questions, and enhancement ideas that are out of scope for the initial MVP but warrant future exploration.

Production Migration Testing

Use Case

While Huginn primarily targets pre-production testing workflows, integration with production data models opens the door to production validation scenarios. A key use case is hardware migration validation - verifying network state before and after replacing physical devices.

Example scenario:

Catalyst 6500 (192.0.2.10, hostname: core-sw-01)
    ↓ migration
Catalyst 9600 (192.0.2.20, hostname: core-sw-01-new)

During hardware migrations:

  • The device hostname may change (temporarily or permanently)
  • The management IP address may change
  • Some state changes are expected (new hardware capabilities, different interface naming)
  • Other state should remain consistent (BGP neighbors, OSPF adjacencies, VLAN assignments)

Problem Statement

The current parameter storage convention keys on test case ID, which implicitly includes per-device state keyed by device name (the testbed key). If the device name changes between pre-migration and post-migration test runs, the framework cannot correlate the "before" and "after" states for comparison.

parameters/
  3.0.0-pre.json    # Contains state for "core-sw-01"
  3.0.0-post.json   # Contains state for "core-sw-01-new" - no correlation!

Questions to Explore

  1. Device Identity Abstraction

  2. Should devices have a stable logical identifier separate from their testbed key/hostname?

  3. How would this affect testbed YAML authoring complexity?

  4. Could this be opt-in for migration scenarios only?

  5. Migration Mapping

  6. Could the test plan define explicit device mappings for migration scenarios?

  7. Example: device_mapping: { "core-sw-01": "core-sw-01-new" }

  8. How would this interact with the data model?

  9. Expected State Transformations

  10. Some state changes are expected during migration (interface naming, platform-specific features)

  11. How do we express "these differences are acceptable" vs "these differences are failures"?

  12. Could transformation rules be defined per-migration type?

  13. Parameter Migration Tooling

  14. Should there be CLI tooling to "migrate" parameters from one device identity to another?

  15. huginn parameters migrate --from core-sw-01 --to core-sw-01-new

  16. Testbed Versioning

  17. Should the testbed support "before" and "after" snapshots for migration scenarios?

  18. How does this interact with inventory plugins that pull live state?

Potential Approaches

Approach A: Logical Device Names Introduce an optional logical_name field that remains stable across migrations:

devices:
  core-sw-01-new:
    os: iosxe
    logical_name: core-switch-1  # Stable identifier for parameter correlation
    metadata:
      replaces: core-sw-01       # Documents migration lineage

Approach B: Migration-Aware Test Plans Add migration context to test plans:

migration:
  mappings:
    core-sw-01: core-sw-01-new
  expected_changes:
    - interface_naming    # Platform-specific interface names may differ
    - hardware_inventory  # New hardware, different modules

Approach C: External Correlation Keep the framework simple; handle device correlation externally via:

  • Pre-processing testbed files
  • Post-processing parameter files
  • Custom job logic that understands migration context

Scale Considerations (Noted, Not Addressed)

Production testing at scale introduces additional concerns:

  • Connection pooling limits against production infrastructure
  • Rate limiting to avoid impacting production traffic
  • Rollback/abort mechanisms if tests detect critical issues
  • Integration with change management systems

Additional Future Items

(Placeholder for additional considerations as they arise)

Authentication Mechanisms

Huginn already supports username/password authentication, SSH key authentication (private_key for the SSH and NETCONF brokers), and API token authentication (token for the HTTP broker); see Credential Fields. Future releases should consider:

  • Certificate-based authentication
  • Integration with secrets management systems (HashiCorp Vault, AWS Secrets Manager)

Custom Report Templates

Support for user-defined report templates was descoped from MVP. Integration with external reporting systems is already possible by writing a reporter plugin registered under the huginn.reporters entry point group, which receives the full RunResult. What remains is customizing the built-in HTML report without writing a plugin:

  • Jinja2 template support for HTML reports
  • Custom data transformations for report output

Inventory Plugin Merging

Inventory plugins are selected per run with --inventory-plugin <name>:<config> (for example --inventory-plugin netbox:netbox.yaml) and completely replace the static testbed YAML file: --testbed and --inventory-plugin are mutually exclusive. Future releases could support merging plugin inventory with a testbed file, for example with a hypothetical flag:

# Future: not implemented
huginn run --inventory-plugin netbox:netbox.yaml --testbed lab-overrides.yaml --merge-inventory

Use cases:

  • Plugin provides base device inventory from NetBox
  • Testbed file adds lab-specific devices not in NetBox
  • Testbed file overrides specific connection parameters for certain devices

Questions to explore:

  • Conflict resolution strategy (plugin wins? testbed wins? error?)
  • Merge granularity (device-level? field-level?)
  • Credential merging behavior
  • Group membership merging (union? intersection?)

Job Packages

Test cases can reference jobs in installed Python packages by module path (see Package-Based Job References). These questions will be resolved as the first job package is built:

  • Formal package interface: Does the framework need a registration mechanism (e.g., entry points) for job packages, or is any importable module with LearningTestCase subclasses sufficient?
  • Job discovery / browsing: Should the framework provide tooling to list available jobs from installed packages (e.g., huginn jobs list --package huginn-jobs-network)?
  • Parser bundling: Should job packages bundle their own Muninn parsers, or depend on a separate parser package?
  • Unit test conventions: Should job packages follow the same spec-driven test harness pattern documented in the Unit Testing Automation guide?