GRIND-Lab-Core/night_owl_research_agent

Fork 0

mirror of https://github.com/GRIND-Lab-Core/night_owl_research_agent.git synced 2026-11-05 00:17:32 +00:00

ESRI handbooks + fix runnability bugs #6

New issue

Open

opened 2026-10-30 21:24:14 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-30 21:24:14 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/GRIND-Lab-Core/night_owl_research_agent/pull/5
Author: @gladcolor
Created: 4/28/2026
Status: 🔄 Open

Base: main ← Head: fix/data-download-overpass-esri

📝 Commits (1)

e71eb7a feat(data-download): add OSM Overpass + Census + ESRI handbooks; fix runnability bugs

📊 Changes

1 file changed (+258 additions, -35 deletions)

View changed files

📝 skills/data-download/SKILL.md (+258 -35)

📄 Description

Scope

This PR is intentionally narrow: skills/data-download/SKILL.md only (1 file, +258/−35 vs main).

What's added

skills/data-download/SKILL.md gains source-specific download handbooks for the most common geospatial data sources:

OpenStreetMap (OSM) via OSMnx + Overpass API — single boundary, street networks, POIs / features within a boundary, multi-boundary queries by admin_level. Includes a critical-rule callout for the map_to_area pattern.
US Census TIGER/Line — URL template, parameter rules, the "no state-level county file" gotcha, GEOID widths.
US Census ACS — API endpoint, variable groups, county/tract geography hierarchy, FIPS join keys.
ESRI World Imagery — tile endpoint, Web Mercator notes, full download-and-mosaic-to-GeoTIFF pattern.

What's fixed

Runnability bugs that surfaced during end-to-end testing against live services:

OSM Overpass examples switched from requests.get to requests.post with a descriptive User-Agent header. The default python-requests/x.y.z UA is rejected by the public Overpass API with HTTP 406.
Annotated why area(osm_id)->.rel silently returns 0 elements: raw osm_id is not an area_id (for relations the area_id is osm_id + 3_600_000_000); the recommended relation({osm_id}); map_to_area->.searchArea; pattern is what produces results.
Multi-boundary US states query: noted that relation(area.us)['admin_level'='4'] returns ~75 elements (50 states + DC + territories + bordering CA/MX admin units sharing nodes inside the US area), not exactly 50.
ESRI World Imagery snippet: fixed NameError. The previous code referenced west_merc/south_merc/east_merc/north_merc in rasterio.transform.from_bounds without ever computing them. Now uses pyproj.Transformer to reproject the lon/lat bbox to EPSG:3857.
Replaced deprecated gdf.unary_union with gdf.geometry.union_all() for Shapely 2.x / GeoPandas 1.x.
Default place changed from "Tokyo, Japan" (prefecture spans Ogasawara Islands ~1,000 km offshore → ~50,000 tiles at z=12) to "Shibuya, Tokyo, Japan" at z=14 (~9 tiles).
Added User-Agent header, raise_for_status(), .convert("RGB"), and an assert n_tiles < 500 guard for courtesy and to prevent abuse of the public ArcGIS service.

Verified end-to-end against live services

Atlanta hospitals via Overpass relation + map_to_area → 7 hospitals
The warned-against area(osm_id) pattern → silently 0 elements (confirms the documented warning)
US states multi-boundary query → 75 relations
TIGER counties (national) → 3,234 rows, GEOID width 5
TIGER tracts (CA, FIPS 06) → 9,109 rows, GEOID width 11
TIGER state-only-county claim: cb_2021_06_county_500k.zip → HTTP 404; cb_2021_us_county_500k.zip → HTTP 200
ACS Georgia counties → 159 rows with population, median income, 5-char GEOID
Shibuya imagery at z=14 → valid 768×768 EPSG:3857 GeoTIFF

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/GRIND-Lab-Core/night_owl_research_agent/pull/5 **Author:** [@gladcolor](https://github.com/gladcolor) **Created:** 4/28/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `fix/data-download-overpass-esri` --- ### 📝 Commits (1) - [`e71eb7a`](https://github.com/GRIND-Lab-Core/night_owl_research_agent/commit/e71eb7aa624dee156c34d813c357a5ef446aac6a) feat(data-download): add OSM Overpass + Census + ESRI handbooks; fix runnability bugs ### 📊 Changes **1 file changed** (+258 additions, -35 deletions) <details> <summary>View changed files</summary> 📝 `skills/data-download/SKILL.md` (+258 -35) </details> ### 📄 Description ## Scope This PR is intentionally narrow: **`skills/data-download/SKILL.md` only** (1 file, +258/−35 vs `main`). ## What's added `skills/data-download/SKILL.md` gains source-specific download handbooks for the most common geospatial data sources: - **OpenStreetMap (OSM)** via OSMnx + Overpass API — single boundary, street networks, POIs / features within a boundary, multi-boundary queries by `admin_level`. Includes a critical-rule callout for the `map_to_area` pattern. - **US Census TIGER/Line** — URL template, parameter rules, the "no state-level county file" gotcha, GEOID widths. - **US Census ACS** — API endpoint, variable groups, county/tract geography hierarchy, FIPS join keys. - **ESRI World Imagery** — tile endpoint, Web Mercator notes, full download-and-mosaic-to-GeoTIFF pattern. ## What's fixed Runnability bugs that surfaced during end-to-end testing against live services: - **OSM Overpass examples** switched from `requests.get` to `requests.post` with a descriptive `User-Agent` header. The default `python-requests/x.y.z` UA is rejected by the public Overpass API with HTTP 406. - Annotated *why* `area(osm_id)->.rel` silently returns 0 elements: raw `osm_id` is not an area_id (for relations the area_id is `osm_id + 3_600_000_000`); the recommended `relation({osm_id}); map_to_area->.searchArea;` pattern is what produces results. - Multi-boundary US states query: noted that `relation(area.us)['admin_level'='4']` returns ~75 elements (50 states + DC + territories + bordering CA/MX admin units sharing nodes inside the US area), not exactly 50. - **ESRI World Imagery** snippet: fixed `NameError`. The previous code referenced `west_merc/south_merc/east_merc/north_merc` in `rasterio.transform.from_bounds` without ever computing them. Now uses `pyproj.Transformer` to reproject the lon/lat bbox to EPSG:3857. - Replaced deprecated `gdf.unary_union` with `gdf.geometry.union_all()` for Shapely 2.x / GeoPandas 1.x. - Default place changed from `"Tokyo, Japan"` (prefecture spans Ogasawara Islands ~1,000 km offshore → ~50,000 tiles at z=12) to `"Shibuya, Tokyo, Japan"` at z=14 (~9 tiles). - Added `User-Agent` header, `raise_for_status()`, `.convert("RGB")`, and an `assert n_tiles < 500` guard for courtesy and to prevent abuse of the public ArcGIS service. ## Verified end-to-end against live services - [x] Atlanta hospitals via Overpass `relation + map_to_area` → 7 hospitals - [x] The warned-against `area(osm_id)` pattern → silently 0 elements (confirms the documented warning) - [x] US states multi-boundary query → 75 relations - [x] TIGER counties (national) → 3,234 rows, GEOID width 5 - [x] TIGER tracts (CA, FIPS 06) → 9,109 rows, GEOID width 11 - [x] TIGER state-only-county claim: `cb_2021_06_county_500k.zip` → HTTP 404; `cb_2021_us_county_500k.zip` → HTTP 200 - [x] ACS Georgia counties → 159 rows with population, median income, 5-char GEOID - [x] Shibuya imagery at z=14 → valid 768×768 EPSG:3857 GeoTIFF --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-30 21:24:14 +00:00

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

GRIND-Lab-Core/night_owl_research_agent#6

No description provided.

Rows
Columns