opentide generate

Builds client-visible outputs first, then framework internals from Pydantic models and bundled vocabulary.

opentide generate                    # full pipeline (no extract)
opentide generate schemas            # single phase
opentide generate schemas templates vocabs snippets
opentide generate docs --output docs
opentide generate exports navigator
opentide generate inflight
opentide generate inflight prune
opentide generate extract sentinel   # opt-in platform import

Default pipeline order

When run without a subcommand, phases execute in this order:

extract is not part of the default run — it calls live platform APIs and writes Imported/ in the working directory. Use opentide generate extract sentinel, opentide generate extract defender or opentide generate extract elastic explicitly when importing rules.

A freshly scaffolded repository (no objects yet) is a valid generate target. Docs and exports write empty artifacts. Snippet conversion requires the YAML templates from the templates phase and fails if a core or enabled-platform template is missing. Snippet prefixes are tide-threat, tide-objective, tide-rule, and tide-<platform> for enabled platforms, with scope: yaml and tabstops on empty values.

When a phase fails

A phase fails the run when it raises, and also when it logs an error and carries on, for example after skipping an object file it could not parse. The run stops at that phase and exits 1. The result names the phase in phase and at the start of message. The message keeps the exception type. When the phase failed on one object's docs page, index row or export entry, the message names that object as while rendering <uuid>. With --json, a failed phase prints this one document instead of a traceback:

opentide --json generate docs

{
  "phase": "docs",
  "ok": false,
  "status": "failed",
  "message": "docs: Exception: detection_model 00000000-0000-4000-8002-DEADBEEF0000 does not exist in the catalogue while rendering 00000000-0000-4000-8003-000000000001"
}

A phase that logged errors reports the first one, as docs: logged an error: malformed_object_yaml (path=…, error=…) or logged 3 errors, first: …. The same error logged twice counts once. A phase that logged an error and then raised reports both, the logged error after a ;, because the skipped object is often what broke the later one. Every error is on stderr. Run opentide validate to list every broken object at once.

Every phase that loads the registry parses every object file, and the schemas and templates phases load it again. A malformed object file therefore fails the first such phase: opentide generate schemas templates vocabs snippets stops at schemas.

generate extract is not a phase, and generate docs --changed with no git baseline fails before it renders anything. Both print their own failed document, with no phase field.

generate templates

opentide generate templates walks Pydantic model_fields (not JSON Schema $defs). Output is line-oriented YAML:

• Uncommented keys are required FieldInfo on that model (aliases honoured: att&ck, schema).
• Optional metadata keys that opentide lint requires are uncommented too: author and organisation by default, or the [metadata].required list in lint.toml.
• Optional fields are the same YAML with # hugging each key (#references: / #public:). Nested optionals inside that block are not commented again. Spacer blank lines stay blank (they are not turned into a lone #). Uncomment a block and it is valid YAML.
• Placeholders are typed: empty strings, YYYY-MM-DD, multiline | + ..., https://, 3, false. There are no YAML null tokens.
• Vocabulary fields are empty scalars (no enum dump).
• DetectionRule.file and other Path / hidden fields are omitted.
• Rule configurations is a live key whose platform slots (#sentinel:, #splunk:, …) are commented stubs including nested query: |. It is never configurations: {}.

JSON Schema generation (generate schemas) stays a separate pipeline. It writes one schema per registered revision (rule.1.0.schema.json, rule.1.1.schema.json, …) and the router opentide.schema.json, which applies the revision named in each object's metadata.schema. Map the router in your editor; see Schema revision. VS Code snippets copy the YAML templates and add tabstops; they do not re-walk models.

Generated schemas are written for editor hovers and completions. Every field carries a title and a description, taken from the docstring under the model field; a description that uses Markdown also gets a markdownDescription. Vocabulary enums carry one description per value: the full entry in markdownEnumDescriptions, and its name and first sentence in enumDescriptions (VS Code) and x-enumDescriptions (the same list, for other editors). Configuration enums (statuses, log sources, detectors, tenants) carry their whole description as plain text in the same two keys. Each schema's $comment names opentide and its tagline. These are annotation keywords: they change what an editor shows, never what validates. Each snippet carries a description naming the object or platform block it inserts.

generate docs

The only supported entry point for markdown documentation.

opentide generate docs
opentide generate docs --changed
opentide generate docs --rules --threats --objectives
opentide generate docs --output docs --flavor github
opentide generate docs rules
opentide generate docs index

--changed is only supported on full opentide generate docs runs and cannot be combined with --rules, --threats, or --objectives.

Changed-object baseline

--changed and generate inflight resolve the same baseline, in order: the current branch's upstream, then origin/development, origin/main, origin/master, then the local development, main, master. The current branch is never its own baseline. A single-branch repository with no upstream falls back to HEAD, so uncommitted and untracked object files still count as changed.

Without a git worktree or without a first commit, generate docs --changed fails with a structured result (status: failed, exit 1) instead of raising a CalledProcessError; generate inflight logs a warning and writes no shards.

Subcommands rules, objectives, threats, and index accept the same --output and --flavor flags.

Rendered sections by object type

generate docs now renders shared metadata and references sections for all object families, plus richer object-specific sections:

• Shared (rules, objectives, threats):
  • Metadata (UUID, schema, version, created/modified, TLP, optional author/contributors/organisation)
  • References (Public, Internal, Reports buckets when present)
• Rules:
  • Status, ATT&CK Techniques, Detection model (relative backlink), Response, Platform configurations, Coverage, Related objects
• Objectives:
  • Objective metadata (priority is the author string — not looked up in criticality::1.0), Signals, Signal rule coverage (rule backlinks), Coverage, Related objects
• Threats:
  • Criticality, Terrain, Threat Assessment, Actors, CVE (Vulnerability-Lookup links; optional API enrichment), ATT&CK Techniques (when present), Chaining, Chaining details, Coverage, Related objects

Metadata is a Field/Value table. Folder index tables add Status/Severity (rules), Priority (objectives), or Criticality (threats).

Internal wiki links are relative to the current folder (../objectives/…, ./slug.md on indexes). Signals are not separate pages — they link to the parent objective heading; on the github flavor the link uses GitHub's own heading anchor.

A page is named after its object's slugified name. When two objects in one folder slugify alike, the one with the lowest UUID keeps the plain name and the others get a -<uuid[:8]> suffix, so existing page names change only on a collision.

Every run (full, scoped and --changed) removes generated pages that no object writes any more, such as the old page of a renamed object, and lists them under removed in the result. Only files directly inside an object folder that carry the generator's metadata table are removed; README.md and hand-written pages are never touched.

Mermaid behavior and relations direction

Threat chaining and coverage are rendered with Mermaid across flavors:

• Coverage diagram: 2-hop threat → objective → signal → rule flowchart with typed node shapes (hexagon / stadium / circle / rectangle) and family subgraphs. The section is titled Coverage only when this graph is present; a generic relations flowchart uses a Relations heading instead.
• Chaining diagram: inbound + outbound threat-to-threat flow, with kill-chain subgraphs when present and bidirectional arrows for synergistic links.
• Related objects table: clickable backlinks (GitHub does not honor Mermaid click). relations_direction still controls which rows appear (upstream, downstream, or both).

[diagrams]
relations_direction = "both" # upstream | downstream | both

Flavor notes:

• github / generic write lowercase docs/{rules,objectives,threats}/<slug>.md. [gitlab] uuid_permalinks does not apply.
• gitlab keeps UUID filenames when [gitlab] uuid_permalinks = true.
• github / gitlab / generic render flowchart-style relations and chaining.
• azure-devops downgrades to graph syntax and disables subgraphs for compatibility.

Index enrichment options

Folder and root index pages can be enriched from .opentide/configurations/documentation.toml:

folder_index_pages = true

[index]
relation_counts = true
icons = false

• folder_index_pages: write README.md index pages for rules/objectives/threats and root. Empty sections are still written, so the root index does not link to a missing file.
• index.relation_counts: include relation counts (or object counts on root index table).
• index.icons: prefix section/object labels with emoji markers.

CVE sections on threat pages are controlled by the same file:

[cve]
base_url = "https://vulnerability.circl.lu"
default_db_link = "https://vulnerability.circl.lu/vuln/"
retrieve_details = true  # fetch title, published date, severity, aliases

Identifiers always link to Vulnerability-Lookup (/vuln/{id}). CVE, GHSA, and GCVE identifiers are accepted. GNA 0 GCVE ids map to the reserved CVE form (GCVE-0-2024-3094 → CVE-2024-3094) for both API lookup and page links. Legacy NVD default_db_link values from older configs are remapped to CIRCL. Set retrieve_details = false to emit links only (no API calls).

generate exports

opentide generate exports              # all export targets
opentide generate exports navigator
opentide generate exports objects
opentide generate exports revisions

generate exports navigator writes .opentide/exports/attack-navigator.json. Techniques are sorted by ID, and each comment lists threats, then rules, by name. The file changes only when coverage or object names change, not when object files are renamed or moved.

generate exports objects writes .opentide/exports/objects.export.json. Threat actors are objects whose name is a scoped vocabulary ID (att&ck::G0006); the export enriches those names to display names.

generate inflight

Write preview shards for changed objects, or remove shards superseded by committed objects:

opentide generate inflight
opentide generate inflight prune

The pipelines from opentide setup ci run this on every pull request and commit only .opentide/inflight/ to the default branch, from a worktree of that branch, never the pull request itself.

generate extract

Import detection rules from external platforms (credential-heavy, experimental for Defender):

opentide generate extract sentinel
opentide generate extract defender
opentide generate extract elastic
opentide generate extract elastic --space staging --include-prebuilt

Configure tenant credentials under .opentide/configurations/platforms/ before running.

• Sentinel and Defender imports never overwrite one another: when two rules share a display name, the second file name gets the first eight characters of its UUID.
• A Defender detection found in several tenants (one detectorId) is imported once, with every one of those tenants under tenants; each tenant's Graph id is recorded as deployment state. When the tenants' copies differ, the first tenant's is imported and defender_import_tenants_differ names the others.
• Every Sentinel file is parsed before it is written; one that would not parse is skipped with sentinel_import_rule_skipped.
• The Defender listing follows @odata.nextLink, times out after 60 s per page and reports an HTTP error with the tenant and status.

generate extract elastic

Exports the detection rules of one Kibana space and writes each one to Imported/ as a rule::1.0 file with a platform::elastic::1.0 block.

• A rule's Kibana rule_id becomes metadata.uuid. A custom rule whose rule_id is not a UUID is skipped, because deploying it under a new id would create a duplicate.
• An enabled rule gets status: PRODUCTION, a disabled one status: DISABLED, so a deploy keeps it disabled.
• Imported rules are tlp: amber, as Sentinel's are. Review the TLP before a share push: PRODUCTION rules are shared by default.
• The rule's note becomes guide.investigation.
• Every file is loaded as an opentide rule before it is written. A rule that would not load (for example a saved_query rule) is skipped with elastic_import_rule_skipped and listed under skipped in the JSON result, with the reason.
• Two rules with the same name do not overwrite each other: the second file name gets the first eight characters of its UUID.
• A look-back window that RFC 0007 cannot express is logged as elastic_import_window_widened or elastic_import_schedule_defaulted.
• No deployment state is written: Elastic addresses rules by metadata.uuid.

When to run

• After upgrading the opentide package version.
• When .opentide/schemas/ is missing or stale.
• After vocabulary or model changes in the package (client repos inherit on upgrade).

Side effects

Schema and template phases reload IndexManager and Opentide registry caches.

Deprecated shims

Source

src/opentide/cli/commands/generate.py, src/opentide/cli/services/generation.py, src/opentide/generation/errors.py