Accuracy & Validation
How accurate is Van Permit Audit? We publish every case we validate.
The fastest way to know whether an AI compliance tool is rigorous is to compare its output against a real City of Vancouver decision on the same bylaw question. That is what this page does. Every case below is a public Vancouver Board of Variance decision, Development Permit Board staff report, or Council rezoning staff report, a permit where the city's reasoning is documented. We run the same permit through Van Permit Audit and publish the comparison.
This is an ongoing project. Cases are validated on a rolling basis and new ones are added as Vancouver publishes new adjudicated decisions. Last updated: April 23, 2026.
Running accuracy
Computed from validated cases below. Numbers update automatically as new cases are reviewed. Pre-validation phase: first case reviews under way. Numbers below will populate as each case is validated.
10
Cases in pool
Public Vancouver decisions queued for validation
0
Validated
VPA output compared + scored
-
Issue identification
VPA surfaced the same bylaw question the city decided
-
Verdict match
VPA's pass / warning / fail matched the city's outcome
-
Audits run
Permits analyzed through the engine to date
-
Jurisdictions · bylaws indexed
Municipalities and bylaw documents in the knowledge base
Jun 26, 2026
Permit data refreshed
Last refresh of the permit warehouse the engine cites
Citation grounding · secondary process metric
Building baseline
We hold this number back until it is statistically meaningful, rather than publish a percentage off a handful of findings. Every finding the engine cites is still checked against the retrieved bylaw text: the source PDF must match and the quote must appear word for word. Anything that cannot be grounded has its citation removed and is flagged for professional review.
The verified-grounding rate publishes once we reach 50 cited findings or 25 audits with citations. Until then, the counters above are the honest measure of growth.
Methodology, in plain English
Vancouver publishes three kinds of documented, adjudicated permit decisions that make excellent validation ground truth because the city's reasoning is on the record:
- Board of Variance decisions: where an applicant asks for a zoning relaxation. The Board's written decision cites the bylaw section at issue, the applicant's argument, and the outcome with reasoning.
- Development Permit Board staff reports: published before Board decisions on larger projects, with detailed zoning-analysis findings.
- Council staff reports on rezoning and CD-1 applications: zoning-compliance analysis written by City planners and published at every Public Hearing.
For each case in the pool, we:
- Extract the bylaw question the city formally answered.
- Reconstruct the relevant permit details from the public record.
- Run it through Van Permit Audit and save the output.
- Score VPA on three axes, independently:
- Issue identification: did VPA surface the same bylaw question the city decided?
- Citation accuracy: did VPA cite the correct bylaw section?
- Verdict match: did VPA's pass / warning / fail align with the city's outcome?
- Publish the case below with a direct link to the public source record so anyone can verify.
What these numbers do and do not prove
Sample bias toward edge cases. Board of Variance cases are specifically the permits where compliance was disputed. VPA's accuracy on routine, uncontroversial permits is likely higher than this sample shows. The edge cases are harder.
"Agreement with City staff" is not the same as "absolutely correct." City staff can be wrong too (rarely). This methodology measures whether VPA and a licensed City planner reach the same conclusion, which is the next-best ground truth available without commissioning a first-principles professional review of every case.
Sample size grows over time. We start with ~25 cases and add 2-3 per month as Vancouver publishes new decisions. Our current number is never the final number.
This is not a substitute for professional review. High agreement rates do not remove the need for a licensed P.Eng. or Registered Architect to sign off on your actual permit submission. See our Terms for the full disclaimer.
Validation case log (10 cases)
Every case below is a publicly-documented Vancouver decision. Click the Source record link to verify the city's reasoning directly. Cases marked Validated have VPA's output recorded and scored. Cases marked Scheduled are queued for review.
BOV-2024-4212WIN
Single-family · crawlspace-to-habitable
Board of Variance · 2024-05-21 · Zoning R1-1
Bylaw at issue
R1-1 District Schedule §3.2.1: Floor Space Ratio
City decision
Relaxation granted (4-0). Board allowed appeal 4-0 granting FSR relaxation of approximately 442 sq ft over the district maximum to permit interior alterations converting a crawlspace into habitable area at an existing one-family dwelling.
Common query type: crawlspace conversions routinely push FSR over the R1-1 maximum.
BOV-2023-644WOOD
Multi-family · 4-unit multiplex
Board of Variance · 2023-11-21 · Zoning R1-1
Bylaw at issue
R1-1 District Schedule: FSR
City decision
Relaxation granted (FSR 1.0 → 1.22). Board allowed appeal overturning Director of Planning's refusal. Approved four dwelling units on the corner lot and increased FSR from 1.0 to 1.22. Site hardship found to support multiplex use given the updated R1-1 By-law permitting multiplex units.
Explicit quantitative FSR numbers (1.0 → 1.22). Strong numeric validation target.
BOV-2025-R11-FEB
Secondary suite · R1-1 basement conversion
Board of Variance · 2025-02-11 · Zoning R1-1
Bylaw at issue
R1-1 District Schedule §3.2.1: Density and Floor Area
City decision
Relaxation denied (2-2, no majority). Appellant sought to add a secondary suite in the basement with a new covered rear deck, new front stairs, and a side-yard door. Board found no site hardship warranting relaxation of the density/floor-area limit.
Direct test of VPA's secondary-suite analysis: exactly the permit type homeowners ask about most.
BOV-2024-2773EH
Commercial · Cannabis Retail Store siting
Board of Variance · 2024-06-18 · Zoning C-2C1
Bylaw at issue
Zoning §573(1)(a): 300m school / community-facility buffer
City decision
Relaxation denied. Board disallowed appeal despite 555 letters of community support. Majority found no site hardship because the site was within 300m of Hastings Elementary, Our Lady of Sorrows Elementary, Hastings Community Centre, and a youth centre.
Negative case: the buffer rule is strictly enforced. Strong test of use-regulation analysis.
BOV-2024-42W8
Institutional · parking variance
Board of Variance (Parking) · 2024-01-16 · Zoning C-3A
Bylaw at issue
Parking By-law 6059: Minimum off-street parking space requirement
City decision
Relaxation granted. Parking Variance Board allowed a relaxation to five on-site parking spaces for the Japanese Community Association. Board accepted that the majority of visitors are seniors and noted proximity to transit including the future Main Street & East Broadway SkyTrain station. Five letters of support, zero opposition.
Tests VPA's Parking By-law handling and its ability to reason about transit proximity arguments.
BOV-2024-1125W12
Multi-family · Housekeeping vs Sleeping Unit classification
Board of Variance · 2024-12-03 · Zoning RM-3A
Bylaw at issue
Zoning §11.3 and §11.13.3: Housekeeping Units vs Sleeping Units
City decision
Relaxation granted (5-0). Board allowed appeal 5-0, deleting Conditions 1.1 and 2.1 of DP-2023-00670. Approved total of 216 dwelling units (change of use from previously-approved Sleeping Units to Housekeeping Units). Board found site hardship including the age of the building (built early 1970s). Tenants allowed to stay.
Flagship high-profile case: 216-tenant renoviction decision. Covered by CityHallWatch.
DPB-2024-1525ROB
Mixed-use · Two residential towers (28 + 29 storeys)
Development Permit Board · 2025-03-10 · Zoning C6 (Downtown South)
Bylaw at issue
C6 District Schedule: max FSR 8.75, +10% via Heritage Amenity Share
City decision
Approved with conditions. Board approved DP-2024-00900 for two residential towers on a two-storey podium with four levels of underground parking. Staff found the proposal meets the intent of C6 District Schedule and applicable policies. Urban Design Panel supported 5-0 (Dec 11, 2024).
1525 and 1555 Robson Street. FSR-uplift via heritage density transfer is the interesting test.
COUNCIL-2024-5490ASH
Multi-family · rezoning R1-1 → CD-1
Council rezoning staff report · 2024-04-09 · Zoning R1-1 → CD-1
Bylaw at issue
Rezoning: FSR uplift 0.70 → 2.03, height uplift 11.5m → 13.7m
City decision
Approved at Public Hearing. Council approved rezoning at Public Hearing on April 9, 2024. Rezones 5490 Ash Street from R1-1 to CD-1 to permit a 4-storey residential building with 14 strata-titled units. Exact FSR and height numbers approved.
Explicit quantitative density-uplift test: ideal for validating how VPA handles CD-1 rezoning analyses.
BOV-2024-DEC3-HERITAGE-B
Heritage-B single-detached house
Board of Variance · 2024-12-03 · Zoning RS (Heritage-B)
Bylaw at issue
Zoning: minimum side-yard width (site ≤ 15.0m) and impermeable area
City decision
Relaxation granted (5-0). Board allowed appeal on a Heritage-B Single Detached House for non-compliance with minimum side-yard width where site width does not exceed 15.0m, plus additional setback issues. Board also approved a Site Impermeable Area of 0.75 (75%).
Heritage overlay analysis is the most complex VPA workflow: good rigor test.
BOV-2025-NOV-MULTIPLEX
Multi-family · R1-1 multiplex (post-2023 missing-middle)
Board of Variance · 2025-11-18 · Zoning R1-1
Bylaw at issue
R1-1 District Schedule: Multiplex siting + parking
City decision
Relaxation granted. Board allowed appeal approving a Multiple Dwelling (Multiplex) with two three-storey two-unit front buildings and a two-storey two-unit rear building, with five parking spaces accessed from the lane.
Post-2023 missing-middle R1-1 multiplex: tests VPA's awareness of recent bylaw reforms.
Data sources
Every case in the validation pool is sourced from one of the following public Vancouver records:
Related
Free analysis · $49 one-time for the readiness review PDF · No account required to try