Elastic Security (`platform::elastic::1.0`)

Summary

configurations.elastic is an Elastic Security detection rule. Deploy creates it through the Kibana detection engine API (/api/detection_engine/rules, API version 2023-10-31). One platform serves self-managed, Elastic Cloud Hosted, and Elastic Cloud Serverless Security: the deployment type changes only the tenant URLs, the credentials, and which features the licence or project tier enables.

type selects one of seven rule types, every type the Security create-rule UI offers except a saved query. The detection language comes from the block, not from the platform: KQL or Lucene for query, threshold, new_terms, and threat_match; EQL for eql; ES|QL for esql; none for machine_learning. Query validation is supported and live only: Kibana rule preview plus the Elasticsearch checks, run against a tenant. There is no offline syntax check.

The Kibana rule_id is the rule's metadata.uuid, so one UUID addresses the rule in every tenant and space. Elastic never stores an external id: Kibana's internal id is not recorded in the block, in the rule file, or in deployment state.

Shared fields (enabled, name, schema, status, flags, tenants, contributors) are defined in platforms. On this block status is required, and it alone decides whether the deployed rule is enabled. The design, the research behind it, and a reference test against Kibana 8.19 and 9.5 are in RFC 0007.

Requirements

Static validation (opentide validate) and registry load:

• type and status MUST be present. type MUST be one of query, eql, esql, threshold, new_terms, threat_match, or machine_learning.
• query MUST be present for every type except machine_learning. When present it MUST be a non-blank string.
• A block MUST satisfy every constraint. A violation MUST be reported with its constraint code.
• Unknown keys MUST fail, at the block root and inside every object in the field tables. The open objects are filters[].meta, filters[].query, the entries of threat.filters, the entries of actions.notify, and the osquery object of an actions.respond entry.
• Legacy external-id keys (rule_id, rule_id_bundle, rule_id::<tenant>) at the block root MUST fail with external_id_in_rule (External rule ids). Registry load drops those keys and passes every other key to the model, so registry load accepts exactly the blocks static validation accepts, apart from the legacy keys.
• Field names (suppression.group_by, threshold.field, threshold.cardinality.field, new_terms.fields, highlighted_fields, required_fields[].name, threat.mapping[].entries[].field and value) name fields in the tenant's data. They MUST NOT be checked against a closed list. Deploy checks the aggregation, timestamp, and indicator fields against the tenant instead (Elasticsearch checks).
• Durations MUST follow Durations.
• An advisory SHOULD be reported as a warning. A reported advisory MUST carry its advisory code. An advisory never rejects a rule.

Deploy:

• Every Kibana request MUST carry elastic-api-version: 2023-10-31 and Authorization: ApiKey <api_key>. Every Kibana request other than GET MUST also carry kbn-xsrf: true. Elasticsearch requests use the same API key.
• The compiled body MUST NOT contain id or version. Deploy MUST NOT call _bulk_create, _bulk_update, or _bulk_delete, which Kibana 9.0 removed.
• Deploy MUST run the Elasticsearch checks before every create or update of a rule. A failed check fails that rule: it is not created, updated, or replaced.
• Deploy MUST NOT modify an Elastic prebuilt rule (Deploy).
• Deploy MUST NOT write to the rule file and MUST NOT create a deployment-state shard.
• Deploy MUST NOT delete a remote rule that has no MDR. Orphan pruning is not part of this revision.
• Deploy MUST NOT call Kibana Stack alerting (/api/alerting/rule), Watcher, Cases, Fleet, Elastic Defend policy or artifact APIs, or the prebuilt-rule installer. It MUST NOT create or edit data views and MUST NOT write the space settings securitySolution:defaultIndex or securitySolution:defaultThreatIndex.
• When a tenant sets suppression = false, deploy MUST omit alert_suppression and MUST warn once per affected rule.

Query validation:

• opentide validate query --platform elastic --live MUST preview each rule and run the Elasticsearch checks (Query validation). Without --live the command refuses and names --live, as on every platform.

Definition

Block

Not authored, because the Security create-rule screens do not ask for them: saved queries (saved_id), the prebuilt-rule license string, concurrent_searches, items_per_search, meta, output_index, namespace, and throttle. The last six are kept from the remote rule on update (Deploy).

Rule types

A new terms rule alerts when a value, or a combination of up to three values, appears that the rule has not seen in the history window. It does not count events; a threshold rule does.

Data source

On query, eql, threshold, new_terms, and the event side of threat_match, the compiled index is the block index when set, otherwise the tenant setup.index, otherwise absent. When it is absent, Kibana applies the space setting securitySolution:defaultIndex. data_view_id names one existing data view and replaces that chain. esql and machine_learning send neither index nor data_view_id: an ES|QL FROM clause and a job's datafeed select their data. threat.index is always required; the indicator UI pre-fills logs-ti_*, but the API does not.

Durations

A duration is a string in one of two equivalent forms:

; literals are case-sensitive
duration  = shorthand / iso8601
shorthand = [n "d"] [n "h"] [n "m"] [n "s"]                 ; at least one part
iso8601   = "P" [n "D"] ["T" [n "H"] [n "M"] [n "S"]]       ; at least one part
n         = 1*DIGIT

• Shorthand is single-unit (90s, 5m, 1h, 14d) or compound (1h10m, 1d2h, 2m30s). Units are lower case, appear in the order d, h, m, s, and each appears at most once.
• ISO 8601 is the same set of whole days, hours, minutes, and seconds (PT1H, P14D, P1DT2H), with upper-case designators. After T at least one time part MUST follow.
• The value is the sum of its parts, so 1h10m and PT1H10M are the same duration. Weeks, fractions, signs, and spaces are invalid.
• Every duration MUST be greater than zero, except scheduling.lookback, which MAY be zero (0m).

Kibana accepts only a single-unit value: it rejects ISO 8601 and weeks in interval, and alert_suppression.duration has no d unit. The compiler therefore sends every duration as one whole number in the largest unit that divides it exactly, choosing from d, h, m, s, and from h, m, s for alert_suppression.duration:

The from column shows the duration alone. A rule's from is the sum of interval and lookback (Schedule).

Constraints

Static validation MUST reject a block that breaks any of these, and report the code:

A type object (threshold, new_terms, eql, threat, machine_learning) whose value is null counts as absent. A retired key in the type_block list is invalid even when its value is null. A block that omits status fails as a missing required field; implementations MAY report the constraint codes only for blocks that set status.

scheduling (ElasticScheduling)

Schedule

lookback is only the overlap added to interval. The compiler sends interval and sets from to now- plus the sum, so the defaults become interval: 5m and from: now-6m, and lookback: 0m sends from equal to interval. The detection engine has no separate field for the overlap; the UI shows from minus interval. Sentinel's scheduling.lookback has a different meaning (the whole query window): an Elastic lookback of 1m with an interval of 5m covers six minutes, not one. to is always now.

severity (ElasticSeverity)

mapping[]:

risk (ElasticRisk)

mapping[]:

suppression (ElasticSuppression)

Alert suppression needs a Platinum licence or higher on self-managed and Hosted deployments, and a project tier that includes it on Serverless. On a Basic licence Kibana stores, previews, and runs alert_suppression without error but writes one alert per match. A least-privilege API key cannot read the licence, so the tenant declares it with setup.suppression (Tenant setup).

threshold (ElasticThreshold)

cardinality:

new_terms (ElasticNewTerms)

eql (ElasticEql)

threat (ElasticThreatMatch)

The indicator side of an indicator match rule. The event query, language, index, and filters stay on the block. This object is not the detection engine's ATT&CK threat array, and the compiler MUST NOT copy it to that body key.

mapping[] has one key, entries (list, required). entries[]:

machine_learning (ElasticMachineLearning)

machine_learning.job_id and threat.index name objects in one tenant. The block carries them as written.

filters[] (ElasticFilter)

A shorthand entry compiles to a phrase filter:

{
  "meta": {"key": "<field>", "negate": false, "disabled": false, "type": "phrase", "params": {"query": "<value>"}},
  "query": {"match_phrase": {"<field>": "<value>"}}
}

A raw entry is copied unchanged.

integration[] (ElasticIntegration)

An entry is a package name (endpoint) or an object:

required_fields[] (ElasticRequiredField)

Kibana adds ecs to each entry on create. It is true only when both the name and the type match ECS.

guide (ElasticGuide)

exceptions (ElasticExceptions)

lists[]:

overrides (ElasticOverrides)

actions (ElasticActions)

A respond entry compiles as follows:

field is the custom process field that kill-process and suspend-process act on. Kibana rejects those two commands without params.config, so when field is omitted the compiler sends Kibana's default, the process.entity_id field with overwrite: true (opentide#558, opentide#562). isolate and runscript without field send no config.

Tenant setup

Tenants are workspace configuration in .opentide/configurations/platforms/elastic.toml (platforms), not part of the block. Three setup keys change the compiled body.

A tenant that sets proxy is refused; the proxy is the global opentide --proxy option. One API key serves Kibana and Elasticsearch. Rules run with the privileges of the key that last created or updated them, so each tenant SHOULD use a dedicated, non-personal key with Security rules management All in its space, read and view_index_metadata on every source index the rules query, and, for live validation, read on .preview.alerts-security.alerts-<space> and .internal.preview.alerts-security.alerts-<space>-*. Kibana renames the feature privileges between versions, so this spec does not pin their names.

The supported floor is Kibana 8.19 and 9.x. ES|QL rules need 8.14 or later. The detection engine and the query, eql, esql, threshold, and new_terms types work on the Basic licence; alert suppression does not (suppression).

Stack version

Query syntax differs across the supported range: ES|QL commands that shipped in 9.2 or later, such as INLINE STATS, are missing on 8.19, and so is FORK. setup.version lets a tenant declare the lowest stack version among the deployments it reaches, as a string MAJOR.MINOR ("8.19", "9.5") or "serverless" for Elastic Cloud Serverless. Any other value, a TOML number included (9.10 would read as 9.1), is a configuration error. Versions compare by major, then minor, as integers; serverless is later than every MAJOR.MINOR.

• Static validation and editors MAY use the lowest version declared across a rule's target tenants (the block's tenants, or every tenant when it is empty or omitted) to report syntax that version lacks. A target tenant without version counts as the floor.
• A version older than 8.19 is outside the supported range and is checked as 8.19.
• Absent means unknown: report against the spec floor, Kibana 8.19.
• Deploy MUST NOT read the version from the cluster on the author's behalf. A least-privilege key may not see it, as with suppression.
• When the lowest version is 9.4 or later, or serverless, validation MAY skip esql_metadata_id (Advisories).

version does not change the compiled body.

Compiled body

Deploy compiles each rule into one detection-engine create or update body. Kibana's rule_id is metadata.uuid, verbatim. Every other key:

An absent optional source produces no key. Kibana's concurrent_searches, items_per_search, meta, output_index, namespace, throttle, saved_id, and license are never compiled.

Severity and risk

The alert severity is severity (a string), else severity.default, else response.alert_severity, else Informational, matched case-insensitively:

risk (an integer) or risk.score replaces the table score. The table follows Elastic's prebuilt-rule convention and sits inside Kibana's risk bands (0–21 low, 22–47 medium, 48–73 high, 74–100 critical). An explicit score SHOULD stay inside its severity's band (risk_band).

ATT&CK

The body key threat is the ATT&CK array. It is derived from the rule's techniques plus the techniques inherited through detection_model (objective, then its threats), de-duplicated. Sentinel uses only the inherited set; Elastic takes the union, so a rule's own techniques are never dropped.

1. Keep Enterprise techniques only. Mobile and ICS entries in att&ck.vocab.toml carry a Mobile : or Industrial : name prefix. Deploy MUST skip them with a warning (technique_not_enterprise).
2. Map each technique's tide.vocab.stages to tactics with the table below. A sub-technique uses its own stages. A legacy stage is first renamed through the legacy-stage table; any other stage outside the table emits no tactic.
3. Emit one entry per tactic, in table order, with framework: "MITRE ATT&CK". Techniques are sorted by id. A sub-technique appears under its parent technique, named without the <parent name>: prefix. An empty subtechnique list is omitted.
4. reference is the vocabulary link for a technique and https://attack.mitre.org/tactics/<id>/ for a tactic.

ATT&CK v19 renamed Defense Evasion (TA0005) to Stealth and split Defense Impairment (TA0112) out of it. Keys that ATT&CK has since revoked stay in att&ck::1.x with removed = "2.0" and keep the stage they were last published with. A legacy stage maps to the stage that now holds its tactic id:

Deploy

The base path is <url>/api/detection_engine/rules in space default, and <url>/s/<space>/api/detection_engine/rules in any other space.

PUT replaces the whole rule, so a key the block omits returns to Kibana's default. These remote fields are the exception: when the compiled body does not set one, deploy copies it from the GET response.

Type changes. Kibana cannot change type with PUT. When the remote type differs, deploy MUST DELETE the rule and POST the compiled body plus the preserved fields. The rule_id stays the same; Kibana assigns a new internal id, so execution history restarts.

Prebuilt rules. A remote rule with rule_source.type: external (customised prebuilt rules included), or with immutable: true on versions without rule_source, is an Elastic prebuilt rule. Deploy MUST report an error for it and MUST NOT change it.

Drift. GET also returns server-owned fields (id, revision, created_at, updated_at, execution_summary, rule_source, immutable) and defaults such as max_signals and risk_score_mapping, and Kibana adds ecs to each required_fields entry. A deployer that compares remote and compiled rules MUST compare only the compiled keys and MUST ignore required_fields[].ecs.

Errors. HTTP 400 or 409 on one rule, or a failed Elasticsearch check, fails that rule with the server's message, and the batch continues. HTTP 401 or 403 fails the tenant, and deploy moves to the next tenant. HTTP 429 and 5xx are retried up to three times with exponential backoff.

A deployer MAY list managed rules with GET _find?filter=alert.attributes.tags:"opentide" instead of one GET per rule.

Query validation

elastic is in the query-validation allowlist. The language comes from each rule's block: its type, and language for the types that take one (Rule types).

Preview runs the rule once against live data and writes only to the preview alerts index. It catches KQL, Lucene, and EQL syntax errors, unknown EQL fields, ES|QL errors inside a command, unknown ES|QL columns, and rule-level schema errors. It does not check licence-gated behaviour, on 9.x it does not catch an unknown ES|QL command, and it cannot tell a misspelt aggregation field from one with no matches. The Elasticsearch checks cover those gaps. CI SHOULD point --live at a staging tenant, because preview costs a real query.

Elasticsearch checks

Kibana accepts some rules that then fail silently. Before every create or update, and during live validation, deploy MUST run these checks against the tenant's elasticsearch_url with its API key. HTTP 401, 403, 429, and 5xx follow the deploy error rules. A check that could not run is not a pass.

ES|QL pre-flight. Kibana does not parse ES|QL on create: on 8.19 a mistyped command makes every scheduled run fail, and on 9.5 Kibana drops the unknown command and the rule matches everything in its window. For every esql rule, deploy sends POST <elasticsearch_url>/_query with the query plus \n| LIMIT 0. HTTP 200 passes and returns the result columns. A parsing_exception fails the rule. On a verification_exception, deploy calls GET <elasticsearch_url>/_resolve/index/<FROM sources>?ignore_unavailable=true&allow_no_indices=true: when indices, aliases, and data_streams are all empty, the sources are not onboarded yet and the rule deploys with a warning; otherwise the rule fails.

Aggregation fields. suppression.group_by, threshold.field, threshold.cardinality.field, and new_terms.fields MUST each be present with aggregatable: true in GET <elasticsearch_url>/<sources>/_field_caps?fields=<fields>&ignore_unavailable=true&allow_no_indices=true. <sources> is the compiled index, or the title of the data view from GET /api/data_views/data_view/<data_view_id> in the tenant's space. Otherwise the rule fails. A data view lookup or _field_caps call that does not return 200 also fails the rule. An empty indices list in the response means the sources are not onboarded yet: the rule deploys with a warning. A rule with no explicit source uses the space default index, and the check is skipped with a warning. For esql, the group_by fields MUST instead be among the pre-flight columns.

Timestamp fields. When overrides.timestamp or eql.timestamp_field is set to a field other than @timestamp, that field MUST be mapped as date or date_nanos in the same <sources>; any other mapping fails the rule. Sources not onboarded yet, or a field not mapped yet, are a warning.

Indicator match fields. For threat_match, each threat.mapping entry's field MUST be searchable in the event <sources>, and its value searchable in threat.index. A field mapped as not searchable fails the rule. Sources not onboarded yet, or a field not mapped yet, are a warning.

Advisories

Validation SHOULD warn, without rejecting the rule, when a condition below holds. A warning MUST carry its code, so the CLI and an editor name the same finding the same way.

Each condition has one code, and a rule that meets two conditions gets both warnings. The two EQL conditions do not overlap: a maxspan above interval + lookback reports only eql_maxspan_window.

An ES|QL query is non-aggregating as Kibana decides it: no pipeline step is STATS. INLINE STATS keeps every row, so it does not make a query aggregating. Kibana 9.4 and later add METADATA _id to a non-aggregating query themselves; esql_metadata_id matters for Kibana 8.19 through 9.3.

Relationships

• platforms — shared fields, capability row, external ids
• rule-1.0.md, rule-1.1.md — parent object
• deployment.md — status strategies
• deployment-state.md — Elastic never writes a shard
• validation.md — schema validation and live query validation
• configuration.md — platform TOML merge
• sentinel-1.0.md — the same shorthand duration grammar (Sentinel's ISO 8601 form also accepts weeks), with a different lookback meaning
• RFC 0007 — design, research, and reference test

Defaults & overrides

Examples

The smallest useful block, a custom query:

configurations:
  elastic:
    enabled: true
    schema: platform::elastic::1.0
    status: STAGING
    type: query
    language: kuery
    query: 'event.category : "process"'
    index: [logs-*]
    scheduling:
      interval: 5m
      lookback: 1m

With rule name: Rule name, metadata.uuid: 00000000-0000-4000-8000-000000000000, metadata.author: Detection Engineering, description: What this rule detects., response.alert_severity: Medium, and no techniques, it compiles for a tenant with no index or tags to:

{
  "rule_id": "00000000-0000-4000-8000-000000000000",
  "type": "query",
  "name": "Rule name",
  "description": "What this rule detects.",
  "enabled": true,
  "severity": "medium",
  "risk_score": 47,
  "interval": "5m",
  "from": "now-6m",
  "to": "now",
  "language": "kuery",
  "query": "event.category : \"process\"",
  "index": ["logs-*"],
  "tags": ["opentide"],
  "author": ["Detection Engineering"]
}

The keys that change with the rule type:

# eql: no language
type: eql
query: |
  sequence by host.id with maxspan=1m
    [process where event.type == "start" and process.name == "x.exe"]
index: [logs-endpoint.events.process-*]
eql:
  timestamp_field: "@timestamp"

# esql: no index, data_view_id, filters, or language
type: esql
query: |
  FROM logs-*
  | STATS count = COUNT(*) BY host.name
  | WHERE count > 10

# threshold: suppression is a duration, with no group_by
type: threshold
query: 'event.category : "authentication" and event.outcome : "failure"'
threshold:
  field: [source.ip]
  value: 20
  cardinality: {field: user.name, value: 5}
suppression:
  duration: 1h

# new_terms
type: new_terms
query: 'event.category : "iam"'
new_terms:
  fields: [user.name]
  history_window_start: 14d

# threat_match: the event query stays flat; the indicator side is threat
type: threat_match
query: 'event.category : "file"'
threat:
  index: [logs-ti_*]
  query: '@timestamp >= "now-30d/d"'
  mapping:
    - entries:
        - field: file.hash.sha256
          value: threat.indicator.file.hash.sha256

# machine_learning: no query, index, or language
type: machine_learning
machine_learning:
  job_id: rare_process_by_host
  anomaly_threshold: 50

This block breaks three constraints and reports esql_source, language, and suppression_shape:

elastic:
  enabled: true
  schema: platform::elastic::1.0
  status: STAGING
  type: esql
  language: kuery
  query: FROM logs-* | LIMIT 10
  index: [logs-*]
  suppression: {duration: 1h}

Fixtures:

• fixtures/platforms/elastic.yaml — the custom query block above on a full rule
• fixtures/valid/rule-elastic-1.0.yaml — every shared key set (RFC 0007 Example A)
• fixtures/invalid/rule-elastic-esql-index.yaml — the invalid block above on a full rule

RFC 0007 has a complete rule for every rule type, and the compiled body of its Example A.

History