RFC 0010: Deployment external ids

The shard

<uuid> is metadata.uuid in lowercase canonical form. One file per rule that has at least one stored external id. A rule that has never been deployed to Defender, SentinelOne, or CrowdStrike has no file. Deploying that rule to Sentinel, Splunk, HarfangLab, Carbon Black, or Elastic does not create one.

The file is UTF-8 JSON, two-space indent, one trailing newline. Object keys are sorted at every level so a rewrite of one tenant does not reshuffle the others.

{
  "object_uuid": "00000000-0000-4000-8003-000000000010",
  "platforms": {
    "crowdstrike": {
      "falcon-prod": "c5e1b2a0e4c84f0a9d3b7e6a1c8f0d22"
    },
    "defender_for_endpoint": {
      "contoso": "12345",
      "fabrikam": "67890"
    },
    "sentinel_one": {
      "s1-prod": "4242"
    }
  },
  "schema": "deployment.state::1.0"
}

The tenant name is the platform tenant name, with surrounding whitespace stripped. Lookup strips the same way. A stored tenant key MUST be non-empty and MUST equal its own stripped form; " contoso " fails deployment_state_shape instead of silently missing the lookup and creating a second rule. One binding per pair of platform and tenant. The external id is a JSON string, never a JSON number.

A later platform that needs a vendor id is added to this table. It does not gain a field on its block. A platform key other than the three in the table fails deployment_state_platform.

Root keys other than the three above are preserved when a writer rewrites the file, so a newer field can survive an older CLI. Inside a platform object every key is a tenant name and every value is a string. A non-string tenant value fails deployment_state_shape.

Writers are opentide deploy, opentide deploy migrate-state, and the CI replay in §6. A hand edit is possible and the next deploy overwrites the tenant entry it touches. Other tenants and other platforms in the same file stay as they were. A writer MUST read the file, change one tenant entry, and write the rest back. Create writes the entry only after the vendor returns an id. A failed create writes nothing. A successful delete removes that tenant entry, then the platform key when it has no tenants, then the file when platforms is empty.

Canonical form

A non-canonical shard is the quietest way to lose an id: an uppercase filename is never found by a lowercase lookup, so deploy would create a new lowercase shard and a duplicate remote rule. So writers MUST emit, and readers MUST enforce:

• The filename stem and object_uuid are the lowercase canonical uuid and are equal. Anything else is deployment_state_uuid_mismatch.
• Two shard files whose names differ only by case, or canonicalise to the same uuid, are deployment_state_uuid_mismatch. The reader fails closed: no create for that rule. Before treating a shard as absent, a reader looks for a case-insensitive match.
• Tenant keys are non-empty with no surrounding whitespace, or deployment_state_shape.
• Keys are sorted at every level, with a two-space indent and one trailing newline.

Readers MAY treat an empty platforms and an empty platform map as no bindings. Writers MUST remove an empty platform map and MUST delete a shard with no binding. The JSON Schema follows the reader: it accepts the empty maps, and the writer form is checked separately.

A shard whose object_uuid is no longer in the registry (an orphan) is kept. It is the only handle on a remote rule whose YAML was removed while the rule may still run.

Schema changes

No new metadata.schema. No new platform::<identifier>::1.1. The platform schema id stays platform::<identifier>::1.0 on the existing spec file. Only the spec documents' version advances (to 1.1, with a history row). The break is that the keys below become invalid keys.

Minting ::1.1 would make every rule change its schema: line on top of the id migration. The identifier is already not a closed enum (legacy values such as defender_for_endpoint::2.3 still validate). A new id would not tell an old file what to do with rule_id::Contoso. Removing the key does.

rule::1.0 is unchanged. configurations stays a map of platform blocks.

Keys removed

Keys that stay

A legacy key on any of these platforms is invalid too (external_id_in_rule). Migration removes it without storing it (§3).

Edits inside each platform spec

Acceptance edits the ::1.0 files in place, bumps each document's version to 1.1, and adds a history row. The field row, the requirement that names the field, and the deploy sentences that read or write it all go.

defender-for-endpoint-1.0.md:

• Summary no longer says the id is stored on the block.
• The requirement that rule_id may be omitted goes away with the field.
• The deploy table's DISABLEMENT row updates the bound rule with isEnabled: false; without a binding, deploy reports external_id_missing (warning) and makes no HTTP call.
• The deploy table's DELETION row deletes the bound rule and removes the binding after the delete succeeds; without a binding, external_id_missing.
• The other row updates when a binding exists, and otherwise creates and durably records the returned id in the shard. It does not write the rule file.

sentinel-one-1.0.md and crowdstrike-1.0.md: the same replacement of rule_id_bundle with the shard and the same three rows (STAR status Disabled; Falcon inactive). The CrowdStrike paragraph that recorded the deployer's reversed active/inactive mapping goes: opentide#555 fixes the mapping, and the spec states the shared strategy.

harfanglab-1.0.md and carbon-black-cloud-1.0.md: delete the field row and the requirements that say the bundle may be present and is not read. State that legacy keys are invalid and that migration removes them as external_id_unused. Deploy behaviour is otherwise the same text.

platforms/index.md: replace External rule ids with the rule that no block carries a vendor id, a table of how each platform (Elastic included) is addressed, and a pointer at the deployment-state spec. The sentence that says loaders fold rule_id::<tenant> before the unknown-key check goes away with the alias. The Splunk loader aliases and the Defender quarantine_files alias stay.

workspace.md: add the path to the .opentide/ table and the example tree. State that .opentide/states/deployment/ is generated and committed, that .opentide/states/sharing.jsonl stays a gitignored cache, and that .opentide/states/ is never ignored whole. Add the production write-back to the CI jobs table. The inflight section stays on .opentide/inflight/.

sharing.md: the sentence that a later subsystem gets its own file under .opentide/states/ becomes: sharing owns the regenerable cache sharing.jsonl; deployment owns the committed shard directory; a further subsystem MUST say whether its state is a cache or an authoritative record and whether it is committed. Sharing's ledger fields are unchanged.

deployment.md: one section. Update, disablement, and deletion for Defender, SentinelOne, and CrowdStrike require the binding; a create needs a destination that is not ignored and a run that can persist state. Strategies themselves do not change.

validation.md: the schema check reports external_id_in_rule and suppresses the generic unknown-key error for the same field. Full-workspace validation validates every shard, orphans included. A missing shard is not an error. Every --file target must match an object.

New specs/deployment-state.md, schema_id: deployment.state::1.0, holding the shard format, canonical form, migration, git policy, deploy decisions, CI requirements, the diagnostic catalogue, and exit semantics. schemas/deployment.state.1.0.schema.json is the JSON Schema for the shard, on the same footing as schemas/inflight.shard.1.0.schema.json: a checker artifact, not a second normative source.

SPECS.md, CHANGELOG.md, llms.txt, and the site navigation (site/meta.json) gain the new spec and the new versions in the same change.

Fixtures

scripts/deployment_state_fixtures.py checks them from scripts/validate.py, and scripts/test_deployment_state_fixtures.py unit-tests the checker, including the cases no fixture can hold on a case-insensitive file system (two shard names that differ only by case).

No vocabulary change. No change under specs/objects/ or schemas/pins/.

Existing rules must migrate

This section is the upgrade. A catalogue that skips it and deploys with the new CLI does not "keep working".

Defender, SentinelOne, and CrowdStrike will not look up a remote rule by metadata.uuid, by display name, or by walking the tenant and guessing. The id written into the rule on the last create is the only handle. After the keys are illegal, that handle exists only in the shard. A deploy with neither the key nor the shard creates a new remote rule and leaves the previous one in place, still firing. DELETION of the original then reports external_id_missing, because the id was thrown away.

There is no dual-read. Deploy does not open the rule looking for rule_id, rule_id_bundle, or rule_id::<tenant>. If any of those keys is present, on any platform block, deploy fails that rule with external_id_in_rule before any platform client is built or HTTP call made, whether or not a shard entry also exists. That holds on every path that can reach a vendor: deploy, deploy --dry-run, a plan narrowed by --platform, --skip-unconfigured, validate query --live, and direct deployer or MCP calls. Validate fails the same rule with the same code. The only command that reads the old keys is the migration.

flowchart TD
  start["opentide deploy, one rule, one tenant"] --> legacy{"rule still has rule_id, rule_id_bundle, or rule_id:: ?"}
  legacy -->|yes| stop["external_id_in_rule\nno HTTP, no shard write"]
  legacy -->|no| shard{"shard has this platform and tenant?"}
  shard -->|yes| upd["update, disable, or delete and drop the entry"]
  shard -->|no, deletion or disablement| missing["external_id_missing (warning)\nno HTTP"]
  shard -->|no, create| ignored{"deployment state gitignored?"}
  ignored -->|yes| refuse1["deployment_state_ignored\nno HTTP"]
  ignored -->|no| persist{"CI run under a plan other than PRODUCTION?"}
  persist -->|yes| refuse2["deployment_state_unpersisted\nno HTTP"]
  persist -->|no| create["create, then write the returned id into the shard"]

DELETION or DISABLEMENT with no shard entry does not create. It reports external_id_missing and does not call the vendor, which is today's missing-id path. It is a warning, not an error: nothing is known to exist on that tenant, so there is nothing opentide can act on, and a rule retired before it was ever deployed must not fail every later run. The message says that a remote copy, if one exists, has to be removed by hand. §5 has the full table.

What has to be migrated

On every platform block, at the block root only (not inside query, exclusions, alert, or any other nested key):

Both of these Defender shapes are the same binding and both have to move:

configurations:
  defender_for_endpoint:
    rule_id::Contoso: 12345
    enabled: true
    schema: platform::defender_for_endpoint::1.0
    query: |
      DeviceProcessEvents
      | where FileName =~ "cmd.exe"

configurations:
  defender_for_endpoint:
    rule_id:
      Contoso: 12345
      Fabrikam: 67890

SentinelOne and CrowdStrike use rule_id_bundle or the same rule_id::<tenant> line. CrowdStrike values may be strings.

The command

opentide deploy migrate-state does the move. It does not call a vendor API and it does not deploy.

1. Refuse a gitignored destination. If Git ignores .opentide/states/deployment/ (for example through a .opentide/states/ line), the command exits 1 with deployment_state_ignored and does not edit any rule. "Ignored" is decided with Git's complete ignore semantics (see §4). The message names the ignore source and the replacement: ignore .opentide/states/sharing.jsonl only. Setup today writes the directory ignore, and it does not rewrite an existing .gitignore, so every existing workspace hits this until someone changes that line. The command does not edit .gitignore itself.
2. Scan rule files. Every .yaml and .yml file under the configured rule directory, recursively, including files in subfolders that deploy itself skips. An id in a nested file still has to move.
3. Collect bindings from the three keys above.
4. Write the shard first, for Defender, SentinelOne, and CrowdStrike only, and make that write durable before the YAML changes. A crash after the shard write and before the YAML edit leaves both copies. The next run sees that they agree and strips the YAML.
5. Then remove the keys from the YAML without rewriting the rest of the document. Comments, key order, and unrelated whitespace stay. The migration deletes the external-id lines and the rule_id / rule_id_bundle mapping blocks. It does not round-trip the file through a parser that drops comments.
6. Leave a failing rule untouched. The same tenant mapped twice, or mapped to a different id than the shard already holds, is external_id_conflict for that rule. A bad value is external_id_shape; an unparseable file is yaml_parse; a missing or invalid metadata.uuid is invalid_uuid. In every case that rule's YAML and that rule's shard are not modified. Other rules still migrate. The command exits 1.
7. Drop every other platform's keys loudly. For every platform other than Defender, SentinelOne, and CrowdStrike, including Sentinel, Splunk, HarfangLab, Carbon Black Cloud, and Elastic, a rule_id, rule_id_bundle, or rule_id::<tenant> is removed from the YAML and is not written to a shard. Those deployers never read the value; copying it into the shard would pretend it is a live address. Each discarded binding is printed as an external_id_unused warning (platform, tenant, value, source file) before the YAML is edited. The value remains in git history of the rule file.
8. A clean second run is a success. No legacy keys, shards already in place, exit 0. external_id_unused warnings alone also exit 0.

A value that is not a scalar string or int, a rule_id / rule_id_bundle that is not a mapping, an empty tenant name, an empty id, or a Defender or SentinelOne value that is not a canonical decimal integer fails that rule with external_id_shape and leaves it untouched.

Integer scalars are stored as canonical decimal strings. 12345 and "12345" become "12345". A string with a leading zero fails external_id_shape on Defender and SentinelOne rather than being rewritten to a different integer. CrowdStrike strings are copied verbatim.

The commit

The migration is not finished when the command exits 0. It is finished when the default branch contains, in the same commit:

• every rule with the legacy keys removed, and
• every new or updated file under .opentide/states/deployment/, and
• a .gitignore that does not exclude that directory.

A commit that strips the YAML and forgets the shards, or a shard directory that git never sees, is the duplicate-rule failure. opentide deploy MUST refuse to create a remote rule when the deployment-state directory is gitignored (deployment_state_ignored, error), including when the ignore is the existing .opentide/states/ line. An update or delete that can read an existing binding still runs, and warns with the same code, so a tree that already has the files is not stuck while gitignore is fixed.

If the shard write fails after the vendor has created a rule, deploy reports deployment_state_write_failed and includes the platform, tenant, and id in that error. It does not delete the remote rule to roll the create back. The same code covers a binding that cannot be removed after a successful delete.

Worked example

Before, a file deploy has already spliced:

configurations:
  defender_for_endpoint:
    rule_id::Contoso: 12345
    enabled: true
    schema: platform::defender_for_endpoint::1.0
    status: PRODUCTION
    query: |
      DeviceProcessEvents
      | where FileName =~ "cmd.exe"

After migrate-state, that line is gone and the rest of the file is unchanged. The new file .opentide/states/deployment/00000000-0000-4000-8003-000000000010.json holds:

{
  "object_uuid": "00000000-0000-4000-8003-000000000010",
  "platforms": {
    "defender_for_endpoint": {
      "Contoso": "12345"
    }
  },
  "schema": "deployment.state::1.0"
}

The next deploy to tenant Contoso patches detection rule 12345. It does not post a second rule. A deploy that runs on a checkout where this JSON is absent, and where the YAML line is already gone, posts a second rule.

Git policy

opentide setup MUST stop writing .opentide/states/ as an ignore rule. The ignore it writes for a new repository is .opentide/states/sharing.jsonl. Setup does not rewrite an existing .gitignore; migrate-state and the create guard are what stop an old ignore from hiding the shards.

Whether the shard directory is ignored MUST be decided with the repository's complete Git ignore semantics: nested .gitignore files, parent-directory patterns, wildcards, negations, .git/info/exclude, and core.excludesFile. Implementations MUST NOT decide it by searching .gitignore text, which misses **/states/, a global excludes file, and a negation that re-includes the directory. A fail-safe false positive (reporting ignored when Git would not) is acceptable; a false negative is the duplicate-rule failure.

What deploy does with the shard

For Defender, SentinelOne, and CrowdStrike, per tenant the deployment plan already selected, in this order:

Disablement is an update that sets the vendor's disabled state, using the shard id. It does not remove the entry.

The run cannot persist state when it runs in CI (Azure Pipelines, GitHub Actions, or GitLab CI is detected) under any deployment plan other than PRODUCTION: STAGING, FULL, ALWAYS, MANUAL, DEBUG. Only the generated production job commits shards (§6). Before this rule, every pull-request rerun of deploy --plan STAGING created another Defender, SentinelOne, or CrowdStrike rule on the staging tenant, and a tenant with deployment = "ALWAYS" received one untracked staging copy and then a production copy after merge (opentide#554). Documenting "use separate tenants and clean up by hand" would have kept the very duplicate this RFC exists to remove.

• A local run (no CI detected) may create. The operator commits the shard.
• opentide deploy --persist-state-externally lifts the refusal for an operator who persists .opentide/states/deployment/ some other way. It is documented, opt-in, and never emitted by generated CI. It does not lift deployment_state_ignored.
• Updates, disablements, and deletes that already have a binding are always allowed.

Deploy continues with independent rules after a per-rule error and exits 1 if any selected rule reported an error-level code. Warnings alone (external_id_missing, deployment_state_ignored on an update or delete) exit 0.

The shard is the record of what opentide created. It is not rebuilt by listing the tenant's rules and matching titles. A title match is how a renamed rule, or two rules with one name, would update the wrong remote object.

CI persistence

A created id that never reaches the default branch is a future duplicate. So the production job that opentide setup generates for GitHub Actions, GitLab CI, and Azure Pipelines (opentide deploy --plan PRODUCTION on the default branch) commits deployment state itself. It MUST:

1. Serialise production jobs. GitHub: job-level concurrency: {group: opentide-production-<branch>, cancel-in-progress: false} (superseded by deployment-state 1.1, specifications#43: GitHub cancels pending runs, so the production job has no concurrency group). GitLab: resource_group: opentide-production. Azure: the strongest serialisation available from YAML (lockBehavior: sequential), with the generated file stating that it only takes effect when an Exclusive Lock check is configured on the resource the stage uses.
2. Overlay the default-branch tip's state before deploying. Before the dry run and the deploy, fetch the default branch and replace .opentide/states/deployment/ in the working tree with the tip's copy: remove the directory first so the tip's deletions hold, and leave it absent if the tip has none. The tip descends from the triggering commit, so it already contains that commit's state and every later production write-back. Without this, a job that starts on an older commit cannot see the ids a newer job recorded and creates them again.
3. Write back whenever the deploy started, including after a partial failure, a cancellation, or a timeout. GitHub: if: ${{ always() && steps.deploy.outcome != 'skipped' }}. Azure: condition: and(always(), eq(variables['OPENTIDE_DEPLOY_STARTED'], 'true')). GitLab: the equivalent, preferably from after_script gated on a marker the deploy writes; where the job cannot run the write-back on cancellation or timeout, the generated file says so.
4. Stage shards independently of rule files. Every shard change is staged unconditionally. A rule file the deploy changed (promotion rewrites status) is committed only if opentide validate --strict passes on exactly those files. The file list is read NUL-delimited (git diff -z --name-only, read -r -d ''), and opentide validate --file fails when any --file target matches no object, so a path with a newline or an unmatched path cannot slip past validation. A rule file that fails is left out and the job fails after the state is pushed; it never costs a shard.
5. Replay on conflict. When the push is rejected and the rebase conflicts, abort the rebase, reset to the new tip, and replay this run's delta per platform and tenant (opentide deploy replay-state <state-commit>), never by copying shard files, which would revert other runs' bindings and resurrect deleted shards. When the same binding changed on both sides, keep this run's binding (the newest record of what it created), report deployment_state_conflict with both values, publish, and fail the job so an operator removes the duplicate remote rule.
6. Recover on every terminal failure. Once the state commit exists, every terminal failure (push retries exhausted, fetch failure, replay failure) prints the state diff and makes a best-effort push of the commit to refs/opentide/deployment-state/<commit> (a shell trap on EXIT), then fails. One ref per commit, so a later failure never overwrites an earlier one.
7. Keep the ids in the log. external_id_recorded (platform, tenant, id, rule) is logged at WARNING, visible at the default console level, so the CI log holds every id even if the write-back never ran. It is a log record, not a diagnostic, and does not change the exit code.

Staging, pull-request, and merge-request jobs do not commit state; §5 refuses their creates instead.

Other surfaces

• Validation. The schema check reports external_id_in_rule for a legacy key at the root of any platform block and suppresses the generic unknown-key error for the same field, so the author sees one actionable code. Full-workspace validation validates every *.json under .opentide/states/deployment/, orphans included, because an orphan shard is the only handle on a remote rule and a malformed one would otherwise go unnoticed. A missing shard is not an error.
• MCP deploy_rule. Resets the deployment outcome flags before it runs and reports status: failed with the error when the deployment outcome failed, instead of reporting success for a refused rule.
• Status and reporting. CLI, MCP, and exports read ids from deployment state only, never from YAML. An unreadable shard is reported with its deployment_state_* code.

Diagnostics and exit codes

opentide deploy migrate-state exits 0 after a successful migration, a clean idempotent run, or external_id_unused warnings only, and 1 if any rule fails or the deployment state is ignored. opentide deploy continues with independent rules and exits 1 if any selected rule reported an error-level code above.