Data Sources and Provenance

Data Sources and Provenance#

This document records all data sources used in the Camel Farrier demonstration repository, including local hydrometric data and external polygon datasets.

Local Hydrometric Data

Hydrometric data comes from two distinct Water Survey of Canada sources with different access methods and update frequencies.

Directory Structure

data/
├── RCs/                    # Rating curves (1,147 stations) - from Water Office portal
│   └── {STATION_ID}_RCs.csv
├── corrections/            # Data corrections (1,147 stations) - from Water Office portal
│   └── {STATION_ID}_corrections.csv
├── field_visits/           # Field visit metadata (1,147 stations) - from Water Office portal
│   └── {STATION_ID}_field_visits.csv
├── HYSETS_watershed_boundaries/  # HYSETS polygon shapefiles
│   ├── HYSETS_watershed_boundaries_20200730.shp
│   └── HYSETS_watershed_boundaries_filtered.shp
└── WaterOffice_RC_Export_20260203.zip  # Archived Water Office portal extraction

1. HYDAT Database (Core Hydrometric Records)

  • Source: Water Survey of Canada - National Archive (HYDAT)

  • Database: Hydat.sqlite3

  • Location: External (not in repository)

  • Version: 2026-01-16

  • Access: Publicly available through Environment and Climate Change Canada

  • URL: https://www.canada.ca/en/environment-climate-change/services/water-overview/quantity/monitoring/survey/data-products-services/national-archive-hydat.html

  • Content:

    • Daily discharge and water level records

    • Station metadata (location, drainage area, status)

    • Data quality flags and symbols

    • Historical records dating back to station establishment

  • Update Frequency: Quarterly releases

  • License: Open Government License - Canada

2. Water Office Portal Data (Operational Metadata)

  • Source: Water Survey of Canada - Water Office Portal

  • Extraction Date: February 3, 2026

  • Access: Requires authenticated credentials (not publicly accessible)

  • Extraction Script: /home/danbot/code/2025/WSC_rating_curves/retrieve_RC_correction_data.py

  • Archived Dataset: WaterOffice_RC_Export_20260203.zip

  • DOI: <add_DOI> (to be assigned upon Zenodo publication)

  • Content:

    • Rating Curves (2,375 stations): Stage-discharge relationships with validity periods

    • Data Corrections (2,375 stations): Applied adjustments and correction rationale

    • Field Visit Metadata (2,375 stations): Site inspection records, measurement conditions

  • Update Frequency: n/a

  • Stations Covered: n/a

  • File Format: CSV per station with standardized column structure

Note: The Water Office portal data is not included directly in this repository due to access restrictions. For demo replication, download the archived extraction from Zenodo using the DOI above and extract to the data/ directory.

External Polygon Datasets

External watershed polygon datasets are used for cross-dataset comparisons and validation. These datasets are NOT included in this repository but are referenced for reproducibility.

1. WSC Basin Polygons (2021-12 Version)

  • Description: Water Survey of Canada drainage basin polygons from December 2021

  • Version: 2021-12

  • Format: GeoJSON

  • File: WSC_basins_2021-12.geojson

  • Location: External (will be archived with DOI)

  • Use: Baseline polygon version for v1_initial_state demonstration

  • DOI: <placeholder doi URL> (to be assigned upon Zenodo publication)

2. WSC Basin Polygons (2024 Version)

  • Description: Current Water Survey of Canada drainage basin polygons

  • Version: 2024

  • Format: GeoJSON (organized by region)

  • Structure: MDA_ADP_{region}/MDA_ADP_{region}_DrainageBasin_BassinDeDrainage.geojson

  • Location: External (will be archived with DOI)

  • Source URL: https://collaboration.cmc.ec.gc.ca/cmc/hydrometrics/www/HydrometricNetworkBasinPolygons/

  • Use: Updated polygon version for v2_revised_state demonstration

  • DOI: <placeholder doi URL> (to be assigned upon Zenodo publication)

3. HYSETS Watershed Boundaries

  • Description: HYSETS (Hydrometeorological Sandbox) watershed boundaries

  • Citation: Arsenault, R., Brissette, F., Martel, J. L., Troin, M., Lévesque, G., Davidson-Chaput, J., … & Poulin, A. (2020). A comprehensive, multisource database for hydrometeorological modeling of 14,425 North American watersheds. Scientific Data, 7(1), 243.

  • DOI: https://doi.org/10.17605/OSF.IO/RPC3W

  • Version: 2020-07-30

  • Format: Shapefile

  • File: HYSETS_watershed_boundaries_20200730.shp

  • Location: Included in repository at data/HYSETS_watershed_boundaries/

  • Stations: 671 Canadian stations with HYDAT IDs

  • Use: Cross-dataset comparison for Scenario 4 (HYDAT-HYSETS polygon discrepancies)

4. Caravan Dataset (HYSETS Subset)

  • Description: Caravan large-sample hydrology dataset (HYSETS subset polygons)

  • Citation: Kratzert, F., Nearing, G., Addor, N., Erickson, T., Gauch, M., Gilon, O., … & Matias, Y. (2023). Caravan - A global community dataset for large-sample hydrology. Scientific Data, 10(1), 61.

  • DOI: https://doi.org/10.5281/zenodo.15529786

  • Version: 2023 (HYSETS subset)

  • Format: Shapefile

  • File: hysets_basin_shapes.shp

  • Location: External (referenced for comparison)

  • Stations: 638 Canadian stations with HYDAT IDs

  • Use: Cross-dataset comparison for Scenario 4 (LSH dataset polygon differences)

  • Note: Caravan polygons for Canadian stations are identical to HYSETS polygons

Data Processing and Quality Control

Station Selection Criteria

For the demonstration repository, 10 stations were selected based on:

  1. User-specified for case studies (7 stations): 05MA022, 07DB005, 05LL013, 05NG027, 07GH004, 05QA002, 03OD007

  2. Geographic diversity (3 stations): 02HC038, 08NM134, 01FA001

  3. Data completeness: All stations have rating curves, corrections, and field visit records

  4. Polygon availability: All stations present in WSC 2021-12, WSC 2024, HYSETS, and Caravan datasets

Coordinate Reference Systems

All polygon datasets are standardized to:

  • Storage CRS: EPSG:4326 (WGS84 lat/lon)

  • Area calculations: Local Lambert Azimuthal Equal Area (LAEA) projection centered on each polygon centroid to minimize distortion

Version Control and Provenance

Each station maintains version metadata tracking:

  • Commit SHA: Immutable git identifier

  • Semantic tag: Human-readable version (e.g., WSC-2024-v1)

  • Source dataset: Which polygon source was used

  • Quality tier: Canonical, reviewed, experimental, or superseded

  • External comparisons: Jaccard Index and area differences with HYSETS/Caravan

Data Acquisition for Demo Replication

Required External Data

To replicate this demonstration, you will need:

  1. Water Office Portal Data (rating curves, corrections, field visits)

    • Download: WaterOffice_RC_Export_20260203.zip from Zenodo

    • DOI: <add_DOI>

    • Extract to: data/ directory in repository root

    • Verification: Run scripts/demo_setup/verify_data_integrity.py

  2. WSC Polygon Datasets (for version comparison demonstrations)

    • WSC 2021-12: DOI <placeholder doi URL>

    • WSC 2024: DOI <placeholder doi URL>

    • Extract to designated external directory (paths configurable in setup)

  3. HYDAT Database (core hydrometric records)

    • Download from: https://www.canada.ca/en/environment-climate-change/services/water-overview/quantity/monitoring/survey/data-products-services/national-archive-hydat.html

    • Place: External location (not in repository)

    • Version: 2026-01-16 or later

Automated Setup

# After downloading required archives:
python scripts/demo_setup/init_demo_repo.py \
  --wateroffice-archive data/WaterOffice_RC_Export_20260203.zip \
  --verify-checksums

Reproducibility

To ensure reproducibility:

  1. External datasets archived with persistent DOIs (Zenodo)

  2. Extraction scripts preserved in archived datasets

  3. Processing scripts document all transformations

  4. Version metadata links data versions to specific git commits

  5. Checksums enable verification of data integrity

  6. Extraction dates documented for time-sensitive data

Future Updates

When external polygon datasets are published to Zenodo or similar repository:

  1. Replace <placeholder doi URL> with actual DOI

  2. Update citation information

  3. Verify all links and checksums

  4. Document any versioning changes

Contact

For questions about data sources or access:

  • HYDAT Database: Environment and Climate Change Canada

    • URL: https://www.canada.ca/en/environment-climate-change/services/water-overview/quantity/monitoring/survey.html

  • Water Office Portal: Environment and Climate Change Canada - Water Office

    • URL: https://wateroffice.ec.gc.ca/ (requires credentials)

    • For access: Contact regional WSC offices

  • HYSETS: Richard Arsenault (École de technologie supérieure)

    • Email: richard.arsenault@etsmtl.ca

  • Caravan: Frederik Kratzert (Google Research)

    • GitHub: https://github.com/kratzert/Caravan

  • This Repository: See CONTRIBUTING

Data Citation

When using data from this repository in publications, please cite:

For HYDAT data:

Water Survey of Canada, 2026. HYDAT Database - Canadian Hydrometric Data. Environment and Climate Change Canada. Retrieved from https://www.canada.ca/en/environment-climate-change/services/water-overview/quantity/monitoring/survey/data-products-services/national-archive-hydat.html

For Water Office portal data:

Water Survey of Canada, 2026. Water Office Portal Operational Metadata (Rating Curves, Corrections, Field Visits). Archived extraction dataset. DOI: <add_DOI>. Extraction date: 2026-02-03.

For HYSETS:

Arsenault, R., Brissette, F., Martel, J. L., Troin, M., Lévesque, G., Davidson-Chaput, J., … & Poulin, A. (2020). A comprehensive, multisource database for hydrometeorological modeling of 14,425 North American watersheds. Scientific Data, 7(1), 243. https://doi.org/10.17605/OSF.IO/RPC3W

For Caravan:

Kratzert, F., Nearing, G., Addor, N., Erickson, T., Gauch, M., Gilon, O., … & Matias, Y. (2023). Caravan - A global community dataset for large-sample hydrology. Scientific Data, 10(1), 61. https://doi.org/10.5281/zenodo.15529786

Last Updated: February 3, 2026 Schema Version: 1.0.0