Getting Started

Getting Started#

This guide walks you through setting up and using the Camel Farrier repository to work with hydrometric station data and catchment polygons to automatically generate interactive documentation.

This guide covers environment configuration and dependencies, data initialization workflow, polygon version integration, diagnostic page generation and validation, and documentation building and deployment.

Environment Setup#

Clone Repository#

git clone https://github.com/dankovacek/camel_farrier.git
cd camel_farrier

Create and Activate Virtual Environment#

python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

Install Dependencies#

pip install -r book_docs/requirements.txt

Configure External Data Path#

External geospatial datasets (HYDAT database, WSC basins) are accessed via the COMMON_DATA_DIR environment variable.

export COMMON_DATA_DIR=/path/to/your/geospatial/data

# Persist in shell profile
echo 'export COMMON_DATA_DIR=/path/to/your/geospatial/data' >> ~/.bashrc
source ~/.bashrc

Download Required Datasets#

See Data Sources for download links and provenance. Required datasets include HYDAT database (Canadian hydrometric SQLite), WSC catchment polygons (Water Survey basin boundaries), and optionally HYDAT drainage basins for polygon version comparisons.

Expected directory structure:

$COMMON_DATA_DIR/
└── HYDAT/
    ├── Hydat.sqlite3
    ├── WSC_basin_polygons/
    └── polygons/              # Optional: For HYDAT polygon integration
        ├── MDA_ADP_07/
        ├── MDA_ADP_08/
        └── MDA_ADP_09/

Verify Setup#

python -c "
from scripts.config.paths import HYDAT_DB_PATH, COMMON_DATA_DIR
print(f'COMMON_DATA_DIR: {COMMON_DATA_DIR}')
print(f'HYDAT path: {HYDAT_DB_PATH}')
print(f'HYDAT exists: {HYDAT_DB_PATH.exists()}')
"

Expected output:

COMMON_DATA_DIR: /your/path/to/common_data
HYDAT path: /your/path/to/common_data/HYDAT/Hydat_sqlite3_20260116/Hydat.sqlite3
HYDAT exists: True

If HYDAT exists: False, check environment variables and download paths.

Quick Start Workflow#

# Initialize random set of stations (n=number, m=min per region)
python demo_data/generate_demo_stn_list.py -n 25 -m 2

# Initialize demo stations
python scripts/demo_setup/populate_demo_data.py

# (Optional) Integrate HYDAT polygon versions
python scripts/demo_setup/integrate_hydat_polygons.py --dry-run
python scripts/demo_setup/integrate_hydat_polygons.py

# Generate diagnostic pages
python scripts/demo_setup/process_station_pages.py

# Build documentation
cd book_docs/
jupyter book build .
open _build/html/index.html

# Deploy to GitHub Pages
ghp-import -n -p -f book_docs/_build/html

Station Data Initialization#

Demo Stations#

The repository includes a number of demonstration stations from across Canada:

Station IDs are defined in demo_data/DEMO_STATIONS.txt.

Running Initialization#

python scripts/demo_setup/populate_demo_data.py

This script:

  1. Extracts Water Office export (corrections, field visits, rating curves)

  2. Queries HYDAT for station metadata

  3. Extracts catchment polygons from WSC basins

  4. Generates JSON metadata files

  5. Retrieves Daymet timeseries meteorology

Output files per station:

book_docs/stations/{STATION_ID}/
├── _metadata.json                 # Station info from HYDAT (tracked in git)
├── _versions.json                 # Polygon version history (tracked in git)
├── _comparison_stats.json         # Pre-computed comparison metrics (tracked in git)
├── _qc_status.json                # Quality control status (tracked in git)
├── _data_sources.json             # Data provenance tracking (tracked in git)
├── {STATION_ID}_corrections.csv   # Data corrections (tracked in git)
├── {STATION_ID}_field_visits.csv  # Field visit records (tracked in git)
├── {STATION_ID}_RCs.csv           # Rating curves (tracked in git)
├── {STATION_ID}_daily_flows.csv   # Daily flow timeseries (local only)
├── {STATION_ID}_daily_levels.csv  # Daily level timeseries (local only)
├── {STATION_ID}_annual_peaks.csv  # Annual peak flows (local only)
├── {STATION_ID}_daymet.csv         # Daymet meteorology timeseries (local only)
└── {STATION_ID}_polygon_v*.geojson # Catchment polygons (local only, not in git)

Git Tracking Philosophy#

This repository tracks metadata only. CSV files and JSON metadata are tracked in git, while polygon GeoJSON files (large geospatial data) remain local only. This approach maintains data integrity by keeping polygons unmodified from source, keeps repository size lightweight with streamlined git history, ensures complete provenance tracking via metadata, and follows production patterns where large files stay external while metadata lives in git.

Integrating HYDAT Catchment Polygons#

To create versioned polygon histories, integrate alternative catchment polygons from HYDAT’s drainage basin dataset.

Download HYDAT Polygons#

Download from Hydrometric Network Basin Polygons.

cd $COMMON_DATA_DIR/HYDAT/
mkdir -p polygons

# Download regional files (example: regions 07, 08, 09)
wget https://collaboration.cmc.ec.gc.ca/cmc/hydrometrics/www/HydrometricNetworkBasinPolygons/geojson/MDA_ADP_07.zip
wget https://collaboration.cmc.ec.gc.ca/cmc/hydrometrics/www/HydrometricNetworkBasinPolygons/geojson/MDA_ADP_08.zip
wget https://collaboration.cmc.ec.gc.ca/cmc/hydrometrics/www/HydrometricNetworkBasinPolygons/geojson/MDA_ADP_09.zip

# Extract
unzip MDA_ADP_07.zip -d polygons/
unzip MDA_ADP_08.zip -d polygons/
unzip MDA_ADP_09.zip -d polygons/

Integration Workflow#

# Preview changes (dry-run mode)
python scripts/demo_setup/integrate_hydat_polygons.py --dry-run

# Integrate all demo stations
python scripts/demo_setup/integrate_hydat_polygons.py

# Integrate specific stations
echo -e "07DB005\n08NM134" > /tmp/my_stations.txt
python scripts/demo_setup/integrate_hydat_polygons.py --stations-file /tmp/my_stations.txt

What it does:

  1. Find HYDAT Polygon: Searches regional directories for matching StationNum

  2. Compare Geometries: Computes Jaccard similarity and area differences (equal-area projection)

  3. Determine Version: Assigns semantic version (major vs minor) based on keywords

  4. Extract Polygon: Transforms to EPSG:3857, saves as versioned GeoJSON

  5. Update Metadata: Adds version entry and comparison metrics to _versions.json

Comparison Metrics#

Jaccard Similarity Index (JSI) measures overlap as the ratio of intersection to union area, ranging from 0 (no overlap) to 1 (identical), with values above 0.95 indicating high similarity. Area difference expresses percent change as ((new_area - old_area) / old_area) × 100, with positive values indicating the new polygon is larger and negative values showing it is smaller.

Projection: Uses Lambert Azimuthal Equal Area (LAEA) centered at polygon centroid for accurate area calculations.

Version Graph Structure#

_versions.json uses a graph-based structure allowing any-to-any version comparisons:

{
  "station_id": "07DB005",
  "current_version": "v1.1.0",
  "versions": [
    {
      "semantic_tag": "v1.0.0",
      "timestamp": "2026-02-05T03:19:25.266031",
      "data_source": "WSC_basins",
      "polygon_file": "07DB005_polygon_v1.0.0.geojson"
    },
    {
      "semantic_tag": "v1.1.0",
      "timestamp": "2026-02-05T04:11:56.142026",
      "data_source": "HYDAT_polygons",
      "polygon_file": "07DB005_polygon_v1.1.0.geojson"
    }
  ],
  "comparisons": {
    "v1.0.0_vs_v1.1.0": {
      "computed_at": "2026-02-05T04:11:56.142190",
      "jaccard_index": 0.9945,
      "old_area_km2": 1823.45,
      "new_area_km2": 1833.12,
      "absolute_area_diff_km2": 9.67,
      "percent_area_diff": 0.5
    }
  }
}

Diagnostic Page Generation#

Generate Station Pages#

After initializing data, run:

python scripts/demo_setup/process_station_pages.py

This creates Markdown pages for each station in book_docs/station_pages/stations/ containing station metadata table (coordinates, drainage area, source, retrieval date), version history with polygon provenance and comparison metrics, data quality status from QC validation results, and interactive visualizations showing catchment maps, polygon comparisons, and flow plots.

Plot Generation#

Interactive visualizations use Bokeh plots generated by scripts/catchment_page_utils.py. The catchment boundary map provides interactive polygon visualization with basemap tiles. When two or more versions exist, polygon comparison plots show spatial differences using light green for agreement (intersection), red for old area not in new (false positive), and purple for new area not in old (false negative). Timeseries charts display flow and level data over time.

Plots execute at build time using MyST bokeh-plot directives.

Validation Rules#

Automated checks include:

  • Data Completeness - Required CSV/JSON files present

  • CSV Format - Valid column structure and data types

  • Discharge Values - No negative streamflow (≥0)

  • Timestamps - Valid ISO 8601 format

  • Polygon Geometry - Valid GeoJSON, no self-intersections

  • Metadata Quality - Required fields populated

Results stored in _qc_status.json per station.

Caravan vs WSC Polygon Comparison#

Benchmark Caravan’s inherited HYSETS polygons against the current WSC drainage basins. Start with the test region so you can validate dependencies and CRS handling before launching the full computation.

# Fast validation (39 stations, MDA_ADP_06 only)
python scripts/generation/compare_caravan_polygons.py --test

# Full national run (1,432 stations)
python scripts/generation/compare_caravan_polygons.py

Outputs land in book_docs/data/polygon_comparisons/caravan_vs_wsc2024/ and drive the “Broken Telephone” case study page. Rebuild the Jupyter Book after a successful run to refresh the plots.

Build and Deploy Documentation#

Build Jupyter Book#

cd book_docs/
jupyter book build .

View locally:

open _build/html/index.html

Deploy to GitHub Pages#

ghp-import -n -p -f book_docs/_build/html

Scripts Architecture#

The workflow uses modular, reusable scripts organized into layers:

Configuration Layer (scripts/config/)#

  • paths.py - Centralized path resolution (single source of truth)

Utilities Layer (scripts/utils/)#

  • bokeh.py - Bokeh visualization utilities

  • geometry.py - Geospatial operations (Jaccard, projections)

  • markdown.py - Template rendering and table generation

  • data_loaders.py - Unified data loading with caching

Generation Layer (scripts/generation/)#

The generation layer contains page_generator.py for unified page generation.

Workflow Scripts (scripts/demo_setup/)#

Workflow scripts include populate_demo_data.py to initialize station data, integrate_hydat_polygons.py to add polygon versions, and process_station_pages.py to generate diagnostic pages.

This design provides single source of truth where path changes only require editing paths.py, reusability enabling multiple workflows without duplication, testability with each layer tested independently, and maintainability through clear separation of concerns.

Troubleshooting#

“WaterOffice export not found”#

ls data/WaterOffice_RC_Export_20260203.zip

Download from Zenodo using DOI (see Data Sources).

“HYDAT database not found”#

Complete environment setup above. Script creates placeholder metadata if HYDAT unavailable.

“Import error: supporting_figure_functions”#

Run from repository root:

cd /path/to/camel_farrier
python scripts/demo_setup/populate_demo_data.py

CRS/Projection Warnings#

Non-fatal warnings indicate potential data updates are needed. See the DATA_SOURCES.md guide for instructions on updating geospatial datasets and regenerating station data.

Full Rebuild Workflow#

To clean legacy data and rebuild with latest polygons:

# 1. Clean existing station data
rm -rf book_docs/stations/*

# 2. Repopulate from scratch (creates v1.0.0 from WSC_basins)
python scripts/demo_setup/populate_demo_data.py

# 3. Integrate HYDAT polygons (creates v1.1.0 or higher)
python scripts/demo_setup/integrate_hydat_polygons.py

# 4. Generate diagnostic pages
python scripts/demo_setup/process_station_pages.py

# 5. Build documentation
cd book_docs/
jupyter book build .

Next Steps#