{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0bc12c62",
   "metadata": {},
   "source": [
    "# Streaming large files\n",
    "\n",
    "`ecv.load` reads a whole recording into memory. For multi-gigabyte files, `ecv.open` returns a lazy `EventReader` that slices the file **on demand** — for HDF5 it binary-searches the on-disk timestamps, so a slice costs a handful of reads and the file is never fully materialised (rosbag and txt have their own in-place backends).\n",
    "\n",
    "We demonstrate the API on the small bundled fixture; the same calls work unchanged on a 707-million-event file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "a6f216cc",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-03T23:59:47.429871Z",
     "iopub.status.busy": "2026-07-03T23:59:47.429498Z",
     "iopub.status.idle": "2026-07-03T23:59:47.548364Z",
     "shell.execute_reply": "2026-07-03T23:59:47.532375Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "frames: 3\n",
      "span (ms): (0.0, 49.718) | duration: 49.718\n"
     ]
    }
   ],
   "source": [
    "import eventcv as ecv\n",
    "import numpy as np\n",
    "from pathlib import Path\n",
    "\n",
    "# Point PATH at your own recording (.npz / .hdf5 / .bag / .txt / .aedat / .dat).\n",
    "# Here we locate the small test fixture bundled with the EventCV source.\n",
    "def _fixture(rel=\"data/test/example.npz\"):\n",
    "    for base in (Path.cwd(), *Path.cwd().parents):\n",
    "        if (base / rel).exists():\n",
    "            return str(base / rel)\n",
    "    raise FileNotFoundError(rel)\n",
    "\n",
    "PATH = _fixture()\n",
    "\n",
    "reader = ecv.open(PATH, dt_ms=20)     # treat the recording as 20 ms frames\n",
    "print(\"frames:\", reader.n_slices)\n",
    "print(\"span (ms):\", reader.time_span_ms, \"| duration:\", reader.duration_ms)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d82522f",
   "metadata": {},
   "source": [
    "## Frame indexing\n",
    "\n",
    "`slice(n)` returns the n-th `dt_ms` frame from the recording start, as an `EventStream`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "a77f88d7",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-03T23:59:47.564165Z",
     "iopub.status.busy": "2026-07-03T23:59:47.562753Z",
     "iopub.status.idle": "2026-07-03T23:59:47.585591Z",
     "shell.execute_reply": "2026-07-03T23:59:47.577004Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EventStream with 54231 events\n"
     ]
    }
   ],
   "source": [
    "frame0 = reader.slice(0)\n",
    "print(type(frame0).__name__, \"with\", len(frame0), \"events\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3aa64434",
   "metadata": {},
   "source": [
    "## Walk every window\n",
    "\n",
    "`windows()` is a lazy iterator; it streams a huge file one window at a time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "4a1f31a4",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-03T23:59:47.614657Z",
     "iopub.status.busy": "2026-07-03T23:59:47.614154Z",
     "iopub.status.idle": "2026-07-03T23:59:47.651053Z",
     "shell.execute_reply": "2026-07-03T23:59:47.633833Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "window 0: 54231 events\n",
      "window 1: 37127 events\n",
      "window 2: 14937 events\n"
     ]
    }
   ],
   "source": [
    "for i, w in enumerate(reader.windows()):\n",
    "    print(f\"window {i}: {len(w)} events\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c1c08a6",
   "metadata": {},
   "source": [
    "## Slice by time or count\n",
    "\n",
    "Open without `dt_ms` to take arbitrary windows, relative to the recording start."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "49e196b5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-03T23:59:47.663616Z",
     "iopub.status.busy": "2026-07-03T23:59:47.663158Z",
     "iopub.status.idle": "2026-07-03T23:59:47.680583Z",
     "shell.execute_reply": "2026-07-03T23:59:47.679010Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "first 10 ms: 27044 events\n",
      "events [0, 1000): 1000\n"
     ]
    }
   ],
   "source": [
    "r = ecv.open(PATH)\n",
    "print(\"first 10 ms:\", len(r.slice(t0_ms=0.0, t1_ms=10.0)), \"events\")\n",
    "print(\"events [0, 1000):\", len(r.slice_count(0, 1000)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0a4aa01",
   "metadata": {},
   "source": [
    "## On a real gigabyte file\n",
    "\n",
    "The identical API scales to huge recordings without loading them:\n",
    "\n",
    "```python\n",
    "reader = ecv.open(\"brisbane_707M.hdf5\", dt_ms=30)  # opens instantly — nothing read yet\n",
    "reader.n_slices                                     # frame count from the timestamp range\n",
    "frame = reader.slice(5000)                          # ~32 ms: one binary search + read\n",
    "```\n",
    "\n",
    "Only the events in the requested window are decoded, so memory stays flat regardless of file size. To build a training loop or render a video, combine `windows()` with a representation and `export_png` — see the [transforms tutorial](02-transforms-and-representations.ipynb) and the [`open` reference](../api.md)."
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}