Exploring Azure Data Explorer and Best Practices
A self-sufficient deep-dive on Azure Data Explorer (ADX/Kusto) — architecture, the KQL language from zero to advanced, ingestion patterns, performance/cost levers, and operational best practices.
Published Updated
In this article
- Choose KQL join, lookup and window semantics without changing row grain
- Distinguish retention, hot caching, ingestion and query visibility
- Design measured performance, anomaly, access and recovery checks
Background & source details
Level: intermediate
Bring with you
Sources checked: . A source check does not change the publication or update date.
Editorial review: · What review means
In this article · 29 sections
Azure Data Explorer — Deep Dive
What you'll leave with: a correct mental model of how ADX stores and queries data, enough KQL to get real work done (including time-series, joins, and advanced analytics), production-oriented ingestion patterns, and a checklist of performance & cost levers you can pull before you ever ask support for help.
Try the read-only StormEvents examples on the public help cluster. Perf/SecurityEvent belong to other schemas; management commands, stored functions, ingestion and access-policy exercises need an independently authorized environment, not the shared sample cluster: https://dataexplorer.azure.com/clusters/help/.
1. What ADX actually is (and what it isn't)
Azure Data Explorer (internal codename Kusto) is a columnar, append-oriented analytics store optimized for telemetry and time-series analysis with its own query language (KQL). Think "log analytics engine" — in fact, Azure Monitor logs, Sentinel and Fabric expose KQL-based analytical experiences. Their table schemas, supported operators, management surfaces and limits are not identical to an ADX cluster.
What it's great at:
- Ingesting high-volume events with throughput determined by SKU, batching, schema and workload.
- Running ad-hoc analytical queries with filtering, indexing and distributed execution; measure latency on representative data.
- Time-series analytics (sessionisation, sliding windows, anomaly detection).
- Text search (full-text
has,contains, regex). - Geo, IP, and graph operators out of the box.
What it's not:
- Not an OLTP database. Prefer immutable events and versioned corrections. Management commands can delete/update data under their documented constraints, but this is not a general SQL row-transaction workflow.
- Not a data warehouse for complex star-schema OLAP with joins across huge fact tables. It can do joins, but a cluster tuned for ADX workloads shines on one-big-table plus small dimension tables.
- Not a replacement for Azure SQL / Cosmos DB for operational workloads.
KQL skills transfer across ADX, Fabric and Azure Monitor/Sentinel, but verify the service’s supported syntax, schemas and management permissions before copying a query.
2. Architecture — the mental model that explains performance
Consider an incident query over the last hour of API logs. The logs first become durable extents; the query then runs on engine nodes that may already have those extents in their hot cache. Follow those two paths separately in the diagram: retaining an event and keeping it fast to query are different decisions, which is why retention and cache policies appear separately below.
Figure text equivalent — labels and relationships
- Node “Event Hubs / Kafka / IoT Hub / ADF / Fluent-bit” (square).
- Node “Ingestion nodes” (square).
- Node “Extents: compressed, columnar immutable shards on Azure Storage” (square).
- Node “Engine nodes (subGraph1) · Hot cache NVMe SSD (C1)” (square).
- Node “Engine nodes (subGraph1) · Hot cache NVMe SSD (C2)” (square).
- Node “KQL Query” (square).
- Node “Admin node” (square).
- Node “Engine nodes” (square).
- Node “Result set” (square).
- Group “Ingest” contains “Ingestion nodes”, “Event Hubs / Kafka / IoT Hub / ADF / Fluent-bit”.
- Group “Engine nodes (subGraph1)” contains “Engine nodes (subGraph1) · Hot cache NVMe SSD (C2)”, “Engine nodes (subGraph1) · Hot cache NVMe SSD (C1)”.
- Group “Cluster” contains “Engine nodes (subGraph1)”.
- “Event Hubs / Kafka / IoT Hub / ADF / Fluent-bit” leads to “Ingestion nodes”. Line: normal; ends: arrow point.
- “Ingestion nodes” leads to “Extents: compressed, columnar immutable shards on Azure Storage”. Line: normal; ends: arrow point.
- “Engine nodes (subGraph1) · Hot cache NVMe SSD (C1)” is connected to “Engine nodes (subGraph1) · Hot cache NVMe SSD (C2)”. Line: normal; ends: arrow open.
- “Extents: compressed, columnar immutable shards on Azure Storage” leads to “Engine nodes (subGraph1) · Hot cache NVMe SSD (C1)”. Line: normal; ends: arrow point.
- “Extents: compressed, columnar immutable shards on Azure Storage” leads to “Engine nodes (subGraph1) · Hot cache NVMe SSD (C2)”. Line: normal; ends: arrow point.
- “KQL Query” leads to “Admin node”. Line: normal; ends: arrow point.
- “Admin node” leads to “Engine nodes”. Line: normal; ends: arrow point.
- “Engine nodes” leads to “Result set”. Line: normal; ends: arrow point.
Key pieces to internalise:
- Extent (a.k.a. shard) — the unit of storage. Each extent is columnar, compressed, immutable, and has metadata including creation/ingestion times and column statistics; late event timestamps need not equal extent creation times. Extents live in Azure Blob Storage; the hot-cache policy selects data for local SSD/RAM caching on engine nodes, subject to capacity.
- Columnar storage + encoding — strings are dictionary-encoded, numbers are delta-encoded, etc. Column pruning, encoding and indexes can avoid reading unrelated data, but selectivity and cache state determine how much work remains.
- Hot cache vs cold storage — your cache policy per table decides how much history is kept on local NVMe. Hot-cache queries avoid some remote-storage work; no universal speedup factor follows.
- Merge policy — small extents are periodically merged into bigger ones (compaction). Lots of tiny ingestions → lots of tiny extents → slow queries until merge catches up. Prefer batched ingestion.
- Update policy — a server-side function that runs on ingestion into a source table and writes the query result into a target table. Filtering, expansion or aggregation can yield zero, one or many target rows per source row; do not assume row-for-row delivery. This implements ingestion-time transformation inside ADX, with failure behavior governed by the policy’s transaction setting.
- Materialized views — pre-aggregated, continuously maintained queries. Think "indexed view". Great for rollups ("events per minute per tenant") you query constantly.
The two policies you will touch the most:
| Policy | What it controls | Inspect effective policy | Typical tweak |
|---|---|---|---|
| Retention | How long data is kept before deletion | Not a universal 365-day default | Shorten aggressively for high-volume raw telemetry |
| Cache | How much of retention is on NVMe | Inspect database/table inheritance | Extend for frequently-queried tables; shrink for archival |
Choose retained history from business/legal requirements and hot history from the working set. Hot duration can equal retained duration; cache < retention is not a mandatory invariant. Paying to keep cold data is cheap; paying to keep everything hot is not.
3. KQL — from zero to useful in one page
KQL reads top-to-bottom, left-to-right, like a Unix pipeline. You start with a table and pipe (|) it through operators. Unlike SQL you never write SELECT before knowing the source.
3.1 The operators you will use 90% of the time
Read the query as a sequence of shrinking questions: which Florida events are recent, which columns matter, and which event types caused the most damage? After aggregation, the result has one row per event type rather than one row per storm. On a historical sample dataset, the relative seven-day filter may return no rows; choose a time window represented in the data when trying the pattern.
// Canonical shape
StormEvents
| where StartTime > ago(7d) and State == "FLORIDA"
| project StartTime, EventType, DamageProperty, DeathsDirect
| summarize Events = count(), Damage = sum(DamageProperty) by EventType
| top 5 by Damage desc
| order by Damage descwhere— filter rows early where semantics permit; a predicate on an aggregate cannot simply be moved before aggregation.project/project-away/project-rename— choose or rename columns.extend— add computed columns.summarize ... by ...— GROUP BY with aggregations (count(),sum(),avg(),dcount(),percentile(),make_set(),make_list()).top N by col— k-largest; equivalent in intent to sorting then taking N. Do not assume the optimizer leaves the verbose spelling slower.join kind=inner|leftouter|innerunique ...— like SQL joins, but the default kind is innerunique; broadcast is an explicit strategy choice, not a universal default.union Table1, Table2— vertical stack.lookup— dimension lookup with a small right side; the default kind is leftouter, so specify the intended kind.render— ask the UI to chart (timechart,columnchart,piechart).
3.2 Time-series — the real superpower
Perf
| where TimeGenerated > ago(1d) and CounterName == "% Processor Time"
| summarize avg(CounterValue) by bin(TimeGenerated, 5m), Computer
| render timechartbin(TimeGenerated, 5m) buckets rows into 5-minute windows. Swap operators:
make-series— gap-fills missing buckets, returns arrays per series:Perf | where TimeGenerated >= ago(1d) and CounterName == "% Processor Time" | make-series Cpu=avg(CounterValue) default=double(null) on TimeGenerated from ago(1d) to now() step 5m by Computer | render timechartseries_decompose/series_decompose_anomalies/series_decompose_forecast— built-in seasonal decomposition and anomaly detection. No external model-training pipeline is needed for these functions, but missing-value treatment, baseline length and false-alert evaluation still matter.- For overlapping windows expand each event into the intended windows before aggregation; for sessions use
row_window_sessionon a correctly ordered serialized row set.hopping_windowandsession_windoware not KQL operators here.
For a uniformly sampled metric after an explicit missing-data decision, decompose the series and inspect flags, scores and baseline together:
range i from 0 to 47 step 1
| extend T=datetime(2026-01-01) + i * 1h,
Value=10.0 + sin(toreal(i)) + iff(i == 36, 20.0, 0.0)
| make-series Observed=avg(Value) default=0
on T from datetime(2026-01-01) to datetime(2026-01-03) step 1h
| extend (Flags, Scores, Baseline)=series_decompose_anomalies(Observed, 1.5, 0, "avg")
| project T, Observed, Flags, Scores, BaselineHere seasonality is deliberately disabled (0) and the trend model is an average. Compare the injected spike with the returned score rather than treating the threshold as a probability or sigma count. There are no missing hours in this synthetic fixture; zero-filling real missing observations would be a separate modeling choice. No Kusto output is claimed from this unexecuted query.
3.3 Joins and the hint.strategy
Kusto’s broadcast join broadcasts the left input, and that input should be small (documentation describes up to several tens of MB). lookup instead assumes a large left fact table and a small right dimension, broadcasting the right. Default regular join is not “broadcast right.” Preserve outer-join meaning when changing orientation; swapping inputs is not a free optimization.
Events
| lookup kind=leftouter (
Users | project UserId, Tier
) on UserIdFor large inputs and a sufficiently high-cardinality key, evaluate join hint.strategy=shuffle or a shuffle key. Measure skew, memory and output multiplicity; neither a table being “partitioned” nor adding a hint guarantees improvement. Ensure one dimension row per UserId before lookup or the output can still multiply.
3.4 User-defined functions
let topN = 10;
let errorsFor = (app:string) {
Logs
| where AppName == app and Level == "Error"
| summarize Count = count() by bin(Timestamp, 1h)
};
errorsFor("payments")
| top topN by Count descSave reusable logic as stored functions on the cluster: they appear alongside tables and show up in IntelliSense for consumers.
3.5 Dynamic (JSON) columns
ADX has first-class JSON:
AuditLog
| extend userId = tostring(Properties.userId),
roles = Properties.roles // dynamic array
| mv-expand role = roles to typeof(string) // explode array to rows
| summarize count() by userId, roleDynamic values are also indexed during ingestion. Extracting a repeatedly used property into a typed column via an update policy can reduce repeated parsing/casting work; measure the tradeoff against extra storage and write-time processing.
3.6 Solved local-data queries: joins, windows and sessions
These KQL examples use inline datatable values, not writes. Expected rows can be checked by hand; local Python reference checks retained with the review are not an ADX engine test.
let L = datatable(Key:string, Event:string) ["a", "e1", "a", "e2"];
let R = datatable(Key:string, Tier:string) ["a", "gold"];
L | join kind=inner R on Key | project Event, Tier | order by Event asc
// e1 gold; e2 gold. Default innerunique would keep only one left row for key a.Which duplicate survives innerunique is not a stable “first event” contract without a separately defined selection. Specify inner when both events matter.
let E = datatable(Timestamp:datetime) [datetime(2026-01-01 00:01:00), datetime(2026-01-01 00:06:00)];
E
| mv-expand WindowEnd = range(bin(Timestamp, 5m) + 5m, bin(Timestamp, 5m) + 10m, 5m) to typeof(datetime)
| summarize Events=count() by WindowEnd
| order by WindowEnd asc
// Ends 00:05, 00:10, 00:15 have counts 1, 2, 1.Each window is [end-10m, end) and advances five minutes. Expanding one event into two windows is intentional, unlike an accidental many-to-many join. Filter the final window range and source lookback consistently. make-series itself creates disjoint step bins, not these overlapping windows. For missing CPU readings use an explicit missing-data policy; zero imputation changes the metric and can create false anomalies.
datatable(User:string, T:datetime) [
"u", datetime(2026-01-01 00:00:00),
"u", datetime(2026-01-01 00:04:00),
"u", datetime(2026-01-01 00:10:00),
"v", datetime(2026-01-01 00:11:00)
]
| sort by User asc, T asc
| serialize
| extend SessionStart = row_window_session(T, 30m, 5m, User != prev(User))
| summarize Start=min(T), End=max(T), Events=count() by User, SessionStart
| extend Duration=End-Start
// u: 00:00–00:04 (2 events), u: 00:10 (1), v: 00:11 (1).Sort keys must keep users contiguous and establish event order; equal timestamps need a tie policy if additional ordering affects your question. A neighbor gap greater than or equal to five minutes, or distance greater than or equal to thirty minutes from the session’s first event, triggers a new session. Equality is a useful negative control: events at 00:00 and 00:05 are two sessions under these settings, not one. The fixture’s durations are four minutes, zero and zero; their median is zero. Late events may change historical session boundaries, so incremental materialization needs a lateness/recomputation strategy.
4. Ingestion — the patterns that actually scale
Ingestion is where most ADX deployments go wrong. Rules:
- Batch appropriately. Queued ingestion batches source items according to configured size/count/time triggers, not one extent per call or connection. The documented default sealing triggers are five minutes, 500 items or about 1 GB uncompressed, whichever is reached first. These are batching-policy defaults, not a universal event-to-query latency SLA.
- Prefer the managed pipelines to writing your own. Event Hubs → ADX data connection, IoT Hub → ADX, Event Grid → ADX blob ingestion. Managed connections reduce custom transport code, but still need identity, schema/mapping, failure monitoring and duplicate-handling design.
- Evaluate streaming ingestion for lower-latency workloads. It has service limits and different tradeoffs; there is no universal p99-under-10-seconds promise for an arbitrary pipeline.
- Choose a supported format/compression pair. Parquet is a file format that may contain compressed pages, not itself a compression algorithm. Check the ingest API’s exact supported encodings.
- Schema-on-write — define the table schema; ADX is not schema-on-read. Use mapping schemas for JSON/CSV/Parquet.
- Distinguish extent organization from a custom partitioning policy. Ingestion naturally groups data into extents; a custom datetime/hash partition policy is a separate management choice requiring measurement. Event-time filters and ingestion-time extents are not automatically the same partition key.
For repeated dashboard queries, move stable parsing work out of each query and into ingestion. In the pipeline below, raw events preserve the incoming shape, the update policy produces typed clean rows, and the materialized view serves a frequently requested rollup. Each arrow commits to a data shape, so choose the rollup grain from the questions the dashboard must answer.
Figure text equivalent — labels and relationships
- Node “Kafka/Event Hubs” (square).
- Node “Data connection Event Hubs → ADX” (square).
- Node “Raw_events table” (square).
- Node “Clean_events table” (square).
- Node “Hourly_rollup MV” (square).
- “Kafka/Event Hubs” leads to “Data connection Event Hubs → ADX”. Line: normal; ends: arrow point.
- “Data connection Event Hubs → ADX” leads to “Raw_events table”. Line: normal; ends: arrow point.
- “Raw_events table” leads to “Clean_events table”: update policy. Line: normal; ends: arrow point.
- “Clean_events table” leads to “Hourly_rollup MV”: materialized view. Line: normal; ends: arrow point.
The update policy moves parsing/flattening to ingestion processing; retries and source duplication still require an explicit identity/deduplication design, so this is not an end-to-end exactly-once claim; downstream queries hit the clean table. The materialized view pre-computes hot rollups so dashboards don't re-aggregate billions of rows on every refresh.
5. Performance — measure the levers without changing results
Before you scale up, pull these levers:
- Push filters down. Filter selective source columns/time early where semantically valid. The engine uses extent metadata to skip entire shards.
- Avoid
contains/regexwhen you can.hasuses term semantics (the term index normally indexes terms of at least three characters);containsuses substring semantics. They are not interchangeable predicates. For substring matching, consider storing a tokenised column. - Project early.
projectright afterwherereduces the columns dragged through subsequent operators. - Choose a distinct strategy from cardinality:
distinctis appropriate for deduplication; high-cardinality grouping can benefit fromsummarize bywith a shuffle strategy. Do not promise one spelling is always cheaper. - Watch your joins. Inspect query details/client statistics, memory and result multiplicity. Compare regular, broadcast, lookup or shuffle only with their size/semantic preconditions;
.show queryplanis not the documented inspection command taught here. - Materialized views for hot dashboards. A 30-day
summarizeover a trillion rows needs representative latency/cost measurement even when a materialized view helps. - Cache policy tuning. Inspect effective cache policy, extent sizes and cache-utilization metrics. Extent metadata alone is not proof that every byte is resident; compare warm/cold runs and memory pressure.
- Follower databases. For read-heavy scenarios, attach a follower cluster that reads the same storage — isolates query load from ingestion load.
- Compute choices. Choose from currently supported ADX SKUs and benchmark memory/storage/CPU needs. Configure autoscale with capacity limits; no single VM family is universally best.
Diagnostics you should know:
.show queries
| where StartedOn > ago(1h) and State == "Failed"
// .show queries lists visible completed queries; .show running queries lists active work.
.show table MyTable details // table metadata; permission required
.show table MyTable extents // extent metadata; not a cache-residency proof
.show commands // management-command history, not all data queries6. Cost model — what you're actually paying for
Two cost axes:
- Compute — cluster SKU × number of instances × hours. Its share depends on workload and duty cycle; it is not necessarily the bulk of the bill.
- Storage — durable data plus applicable transactions/network/features. The fraction of the bill depends on retained volume and compute duty cycle; do not assume a fixed percentage. ADX compute pricing also includes service charges beyond raw VM arithmetic.
Savings playbook:
- Consider stopping nonessential dev/test clusters. Stopping removes query/ingest availability, retained storage still costs money, and pending sources need a backpressure/replay plan.
- Autoscale based on cache utilisation or CPU.
- Shorten cache policy on high-volume tables you rarely query beyond 7 days.
- Use a supported SKU for the measured workload. Region and service availability determine choices; do not infer ADX SKU meaning from an unrelated VM-family abbreviation.
- Partition / rollups. Keeping raw telemetry 7 days + daily rollups 2 years is often far cheaper than keeping raw telemetry 2 years.
7. Operations — the non-negotiables
- Principals & RBAC — distinguish Azure resource-control roles from Kusto data-plane roles. Available roles and scopes differ; use the documented database/table/function grant rather than assuming every role exists at every level. Prefer Entra groups over individual principals.
- Row Level Security (RLS) — function on the table that returns the rows a caller can see; ADX applies it transparently.
- Private endpoints + managed identity — establish and test the private client/DNS path before disabling public access. Use supported managed-identity authentication and explicitly grant the required source/destination roles; an identity alone grants no access.
- Recovery — followers read shared data and are not independent backups of deleted/corrupted logical data. Design continuity around the documented deletion-recovery window, source replay/export and, when required, separate regional ingestion. Test restore/replay against the required RPO/RTO; durable storage alone is not a recovery plan.
- Deployment as code —
.kqlscripts in a repo, applied via the documented Kusto.Cli tool and supported Bicep/Terraform resources. Treat schema, functions, update policies, and materialized views like source code. - Observability on the cluster itself — Azure Monitor metrics and diagnostic logs for ingestion, query outcomes and cache utilization; use documented
.show queries/.show ingestion failurescommands for their specific scopes rather than invented general.show diagnosticsor.show metricscommands.
8. A realistic end-to-end mini-project
Building a "who is abusing our public API" detector on top of ADX:
- Ingestion: API gateway access logs → Event Hubs → ADX
raw_api_logs(append only). - Update policy: parse URL, IP, user-agent, extract tenant & route →
clean_api_logs. - Materialized view:
requests_per_min_per_ipgrouped byTenant, bin(Timestamp, 1m), Ip, Route. Preserve tenant in the rollup key and enforce tenant authorization on the queried surface; aggregating it away cannot be repaired by filtering later. - Detection KQL:
make-seriesover the MV +series_decompose_anomalies→ positive anomalies above a deliberately chosen decomposition residual threshold. The default ctukey score is based on a 10th–90th percentile range, not a Gaussian sigma score. - Notebook / Grafana: chart the anomalies; the on-call gets a Slack alert via an Azure Function that runs the KQL on a schedule.
The project ties the earlier choices together: ingestion supplies the events, parsing gives the detector typed fields, and the per-minute rollup makes repeated checks cheaper. A traffic spike is a candidate for investigation, not proof of abuse—legitimate launches and retries can also change volume. Keep enough raw history to inspect flagged windows, and supply the producer, dashboard, and scheduled alert integration around the KQL.
9. Corrections to common misunderstandings
- ❌ "KQL is like T-SQL." → It reads top-down like a pipeline, has different scoping rules, and
letis not a CTE — it binds a single expression for the rest of the query. - ❌ "More ingestion connections = faster ingestion." → Not necessarily. Batching, capacity and source throughput matter; a connection does not map one-to-one to an extent.
- ❌ "Kusto is an OLTP update store." → Prefer append/version semantics for events;
arg_max(Version, *) by Idcan select the latest version only with a deterministic version/tie policy. Management data-update commands have distinct costs and constraints. - ❌ "Joins are free." → Regular joins, left-broadcast joins, right-broadcast lookup and shuffle have different execution strategies. Check size, skew and row multiplicity before choosing.
- ❌ "Cache policy = retention." → They are independent. You can keep 2 years of retention with only 30 days in cache.
10. TODO — self-sufficient action list
Read-only fluency fixtures below are self-contained; the operational checklist is an optional, separately authorized deployment plan requiring your own resources, costs and permissions. No checklist box is claimed complete here.
Fluency
- On the
helpcluster, solve 10 KQL exercises on availableStormEventssample data or the inline fixtures below;SecurityEventandPerfrequire their own populated schema: top-N, time-bucketing, joins,make-series+ anomalies,mv-expand. - Write three query-scoped
letfunctions and invoke them. Storing functions is an optional management operation outside the shared read-only lab.
Your own cluster
Start with a deployment specification, not portal clicks: subscription/resource group, region, currently supported development SKU/capacity, database name, explicit retention/hot-cache settings, identity and network ingress. Azure CLI expresses imperative create/update operations; Bicep declares Azure resources and dependencies. Use the same specification for both approaches, inspect the proposed resource diff and resulting IDs, and compare effective settings rather than assuming two successful commands produced identical resources. Neither approach creates Kusto data-plane permissions merely because the resource exists.
For private connectivity, plan the client network path and private DNS as well as the private endpoint; disabling public access before a working private path can lock out the intended client. Managed identity authenticates a workload, but it still needs the destination’s scoped grants. Stop/start automation needs its own narrow control-plane permission, schedule and a backlog/replay plan. A stopped development cluster is intentionally unavailable; restart readiness and retained storage charges belong in the acceptance criteria.
- Provision a Dev SKU ADX cluster + database via Azure CLI and via Bicep. Commit the Bicep.
- Enable stop/start automation (Azure Automation runbook or simple Function).
- Enforce private endpoint + system-assigned managed identity.
Ingestion pipeline
Declare the event schema before a producer: event_id, tenant, event_time, route and numeric status/duration. A deterministic local fixture is more useful than claiming a five-line producer proves delivery. The service connection maps source JSON to a raw table; an update-policy function validates/casts into the clean table. Compare input IDs, accepted/rejected counts, duplicate behavior and query-visible timestamps. Authentication, ingestion acceptance and data correctness are three independent checks. Do not put source credentials in the producer code.
- Create an Event Hubs namespace + hub and a producer (5 lines of Python) sending 1 event / second.
- Wire an Event Hubs data connection into an ADX table. Verify rows appear without code.
- Add an update policy to parse a JSON payload into typed columns in a cleaned table. Inspect failures with
.show ingestion failures; measure successful end-to-end visibility separately using event timestamps/correlation IDs. Failure records do not measure every successful row’s latency.
Modelling + perf
- Set a realistic retention and cache policy on each table and record rationale.
- Build a materialized view for a hot rollup. Compare its query latency vs the raw query. Put the numbers in the post.
- Capture query details/statistics for a slow query, preserve expected results and compare latency/memory with the same data and cache conditions.
Analytics
- Use
make-series+series_decompose_anomalieson a real metric; produce a chart with at least one labelled anomaly. - Write a sessionisation query (session window by user) and extract median session length.
- Implement a simple RLS policy by tenant, verify with a second principal.
Ship
A dashboard consumer needs an authenticated least-privilege query identity, a fixed query/parameter contract, bounded time range and refresh frequency, and explicit empty/error states. A scheduled alert must persist the last evaluated window or deduplication key so retries do not notify repeatedly; use ingestion lateness allowances rather than always evaluating only the current minute. Keep notification recipients and credentials in the supported service configuration, not in KQL text. Version management scripts and apply them through an authorized release workflow only after compatibility, access and rollback/replay reviews. These are conceptual release requirements, not executed operations.
- Build a tiny Grafana (or Fabric) dashboard driven entirely by ADX queries.
- Schedule a KQL-based alert via Azure Monitor that emails on anomaly.
- Commit all
.kql(schemas, functions, policies, views) to the repo; apply via CLI on PR merge.
Worked operational assessment
Ingestion acceptance: give every event a producer identity and stable event ID, retain raw payload/event time, validate a typed clean schema, and reconcile counts plus rejected events after a bounded visibility delay. Decide whether duplicate events should count or be collapsed before building the rollup. An update policy is not a substitute for that business decision; transactional policy settings and failure handling determine whether target failure blocks source ingestion.
Anomaly answer: a ctukey threshold of 5 is not five standard deviations. Establish seasonality/trend and missing-bin handling, measure precision/recall on labeled historical incidents, then rate-limit alerts. A spike alone is not authorization to block a tenant.
RLS design answer: a trusted function maps the authenticated principal to allowed tenant IDs and filters the table under a supported row-level-security policy. Test with a separate allowed user, a denied user and any broad admin/service identity through the real query surface. Parameterizing tenant prevents syntax injection but does not make the caller entitled to that tenant.
Performance answer: first preserve row grain and expected results. Compare same data, time range, query version, cache state and concurrency. A faster innerunique result that discarded half the events is not a successful inner-join optimization. Materialize only reused subexpressions that fit its bounded cache, push valid filters/projections into it and record memory behavior.
Cost/recovery answer: inventory ingestion/query duty cycle, SKU/service charges, retained/hot bytes and data transfer. Stopping a cluster cannot stop storage charges and must not strand an unbounded producer queue. A follower is query isolation, not a backup; demonstrate recovery from a source replay or independent retained copy under a declared RPO/RTO before calling the system resilient.
Corrected contracts and failure analysis
ADX is optimized for analytical append-heavy telemetry, not arbitrary transactional updates. Ingestion success, query visibility and retention are separate stages. Keep event time and ingestion time distinct when reasoning about delayed arrivals. Extent/partition/cache policy choices affect query cost and freshness; do not copy a cluster sizing claim without a representative workload.
Use explicit join kind and grain. The default innerunique deduplicates the left input; it is not simply SQL inner join. Broadcast/shuffle hints depend on actual sizes and key distribution; skew can defeat a plausible plan. make-series step defines disjoint bins, not overlapping rolling percentiles. Percentiles are approximate nearest-rank estimates; aggregate raw values or suitable mergeable sketches rather than averaging percentile outputs.
Parameters prevent value injection but do not authorize which database/table/tenant may be queried. Materialize is a bounded caching decision within query execution, not a free optimization for all subexpressions. Benchmark with consistent data, cache state and result semantics.
Boundary exercise with solution
A query becomes faster after switching from inner to the default join, but row counts halve. Is the optimization valid?
Solution and reasoning
Not until the changed semantics are justified. Default innerunique removes duplicate left keys, potentially discarding meaningful events. Compare expected grain and multiplicity, specify inner when needed, and optimize filters/projections/plans without silently changing the question.
Source-backed review notes
- innerunique join - Kusto | Microsoft Learn — accessed 2026-10-07. Exact supporting passage: “The query executed the default join, which is an inner join after deduplicating the left side based on the join key. The deduplication keeps only the first record. The resulting left side of the join after deduplication is:”
- bin() - Kusto | Microsoft Learn — accessed 2026-10-07. Exact supporting passage: “The nearest multiple of roundTo below value. Null values, a null bin size, or a negative bin size will result in null.”
- percentile(), percentiles() - Kusto | Microsoft Learn — accessed 2026-10-07. Exact supporting passage: “The percentile() function calculates an estimate for the specified nearest-rank percentile of the population defined by expr.”
- Query parameters declaration statement - Kusto | Microsoft Learn — accessed 2026-10-07. Exact supporting passage: “In particular, client applications that combine user-provided input in queries that they then send to Kusto should use the mechanism to protect against the Kusto equivalent of SQL Injection attacks.”
Version-specific reference contracts
Documentation snapshots accessed 7 October 2026; mutable service/browser documentation is timestamped evidence, not an immutable product-version pin. Exact supporting passages and hashes accompany the review ledger.
- Broadcast join - Kusto | Microsoft Learn — Broadcast join broadcasts the small left input, not the right.
- lookup operator - Kusto | Microsoft Learn — lookup assumes a small right dimension and broadcasts it.
- row_window_session() - Kusto | Microsoft Learn — row_window_session operates on a serialized row set with maximum durations and restart conditions.
- series_decompose_anomalies() - Kusto | Microsoft Learn — Default ctukey anomaly scoring is a percentile-range rule, not standard deviations.
- IngestionBatching policy - Kusto | Microsoft Learn — Queued-ingestion batching seals on size/count/time, not one extent per client call.
- Retention policy - Kusto | Microsoft Learn — A default policy object is not the previously claimed 365-day default; effective inheritance must be inspected.
- Caching policy (hot and cold cache) - Kusto | Microsoft Learn — Null cache policy can mean all data is hot; 31 days is not universal.
- .show queries command - Kusto | Microsoft Learn — show queries returns visible completed-query statistics; show running queries is separate.
- Best practices for Kusto Query Language queries - Kusto | Microsoft Learn — High-cardinality joins/aggregations can use shuffle with measured semantics and skew.
- Update policy overview - Kusto | Microsoft Learn — Update-policy failure and transaction settings must be considered; not an end-to-end exactly-once claim.
- Materialized views - Kusto | Microsoft Learn — Materialized views maintain supported aggregations; they are not arbitrary indexed query results.
- Azure Data Explorer Business Continuity Overview - Azure Data Explorer | Microsoft Learn — Continuity needs a recovery design, not merely durable storage or a follower.
Review and execution boundary
The 9 October 2026 body review also checked the current batching policy, update policy, and session boundary contract. The tenant-grain correction is a data-model requirement, not a claim that this mini-project has been deployed.
ADX/KQL snapshot; performance hints are hypotheses, not measured 10× speedups.
Reviewed on 7 October 2026 against the official source snapshots linked below. The complete original/current lesson was reviewed against official ADX/KQL snapshots. Inline query results are reasoned and checked with a local reference model, not executed on Kusto. All deployment/ingestion/security tasks remain optional unexecuted exercises, not reasons to retire their teaching objectives. Historical setup commands and optional exercises were not executed. No cloud resources, third-party packages or external side effects were created. Old screenshots and unavailable private assets remain in the private recovery archive, not prerequisites for this lesson.
Pause / Recall / Apply
Can you explain it without the page?
Close the example. Reconstruct the core idea, then change one assumption. Mark complete when you’re ready; you can always undo it.
Stored in this browser only. No account, no sync. Clearing browser data removes your record.