Dataset Row-Cap Warning

Every dataset provider caps its read at AZURE_MAX_PAGE_SIZE (1000) rows. Before this, that cap was invisible: a survey with 1400 responses charted, exported, and tabulated as though it had 1000, and nothing on screen said otherwise. A bound dashboard quietly under-reported, and a personalized email export quietly skipped 400 people.

The fix is metadata, not pagination. A Dataset now carries the uncapped total alongside its capped rows, and every consumer surfaces the gap in the shape that fits it — a footnote on a chart, a banner over a table, a confirm before an export you cannot take back. Actually loading more rows stays deferred; this warning is that page's explicit revisit trigger.

How it works

Dataset gains one optional field, totalRows. getDatasetTruncation is the single check every consumer shares: given rows.length < totalRows, it returns what was shown, what is hidden, the total, and whether that total is itself a floor — otherwise nothing. A provider that cannot cheaply count omits totalRows, and its consumers simply never warn.

flowchart TD
  SHEET["readSheetDataset<br/>totalRows = data.rows.length"] --> DS[("Dataset<br/>columns · rows · totalRows?")]
  SURVEY["readSurveyResponsesDataset<br/>totalRows = bounded countEntities when capped"] --> DS
  PROGRAM["readProgramStatusDataset<br/>totalRows = bounded countEntities when capped"] --> DS
  DS --> CHECK["getDatasetTruncation<br/>rows.length < totalRows?"]
  CHECK -->|"no"| SILENT["render as complete"]
  CHECK -->|"yes"| FOOT["Dashboard visual — TruncationFootnote"]
  CHECK -->|"yes"| BANNER["Survey Responses — TruncationAlert"]
  CHECK -->|"yes"| CONFIRM["Email export — confirm before sending partial"]
  CHECK -->|"yes"| IMPORT["Sheet dataset import — warning alert"]

The providers

• Sheet parses the whole content blob before slicing to the cap, so totalRows costs nothing — it is already holding every row.
• SurveyResponses and ProgramStatus cannot count for free: Azure Table Storage has no count API, so a total means walking every matching row. They therefore only pay when they have to, and never more than a bounded walk. A read that proved itself complete answers for itself (totalRows = rows.length) — the survey read fetches one entity past the cap so a partition holding exactly the cap never pays for a count — and only a read known to be truncated runs countEntities: a keys-only page walk, bounded at DATASET_MAX_COUNTED_ROWS so a huge partition cannot stall the read just to render a banner number. A count that hit the bound is a floor: getDatasetTruncation marks it isCountCapped, and every surface renders it as "M+" via formatTruncationCount rather than as an exact total.

The consumers

Each surface is chosen by what an unnoticed truncation would cost there:

• Dashboard visual — a footnote under the chart (warning icon, tooltip explaining the cap). A chart still reads correctly as "the first 1000"; it just must not read as "all". A published snapshot froze whatever the read gave it, so a snapshotted visual warns exactly like a live one.
• Survey Responses blade — a banner above the table. Responses are the one dataset an owner reads as a record of truth, so a silent cut is never acceptable here.
• Email personalized export — a confirm dialog, not a footnote. Mailing a truncated audience is the one failure the sender cannot take back, so a capped read hands the decision over ("{M − N} rows will not get an email") instead of exporting. The command stages the dataset on a singleton dialog store and the Editor blade renders the choice; confirming and the uncapped path both run one export runner.
• Sheet dataset import — a warning alert after the copy. The sheet now looks like the whole survey, so a capped copy has to say so on the way in.

Key files

Notes

• Deliberately metadata-only. The fix for actually needing more rows is pagination, and this warning is what gives that page a consumer: once users see "of M" and complain, it has earned its build.
• One phrasing service backs every surface, so a chart footnote and a table banner can never quote different numbers for the same read.
• countEntities is never free and never on the happy path — the guard that only calls it on a filled page is load-bearing, not an optimization. Its maxCount bound is load-bearing too: without it, every open of a huge partition's blade pays a full key-walk to render one number.