How to collect utility data automatically: methods that scale

Collecting utility data by hand works for ten accounts and breaks at a hundred. Someone logs into each utility portal, downloads a file, renames it, and pastes it into a workbook. Bills that arrive by mail get scanned, or do not. When that person is away, the data stops. Automated utility data collection replaces that routine with feeds that run on a schedule, but setting it up well involves more than signing up for Green Button: you need to know which method fits each account, how customers authorize access, how to keep feeds alive, and how to check what comes back.

This is a step-by-step guide to how to collect utility data automatically for a portfolio of any size. It covers the inventory, the choice of method per account (Green Button Connect My Data, Download My Data, portal exports, PDF bill capture), the authorization step, scheduling and refresh, validation, delivery, and monitoring, with a comparison table of the methods. If you are a consultant, the companion piece automating utility data collection for consultants covers the client-facing side; this guide is about the mechanics.

Step 1: inventory every account and meter

You cannot automate what you have not listed. Start with a table that has one row per utility account, and add the meters under each account. For every row capture the utility, the commodity (electricity, natural gas, water, other), the account number as printed on the bill, the service address, the customer of record (who will need to authorize access), whether the bill is paid by you or by a tenant, and the current source of data (portal, emailed PDF, paper, nothing). Most portfolios discover accounts they did not know about at this stage, usually small meters for signage, parking or irrigation. This inventory becomes the control document for the project: each row ends up with a collection method, an authorization status and a last-successful-retrieval date. The onboarding utility accounts at scale guide describes how to run this across hundreds of accounts without a support backlog.

Step 2: choose a collection method per account

Green Button Connect My Data

Where the utility offers it, Connect My Data is the method to prefer. Ontario's government states that most regulated Ontario electricity and natural gas utilities had to provide Green Button access by November 1, 2023, that customers can download natural gas and hourly electricity usage data, and that they can authorize the secure, automatic transfer of that data to Green Button applications of their choice. The Green Button Alliance's release on the regulation identifies it as Ontario Regulation 633/21: Energy Data, filed 2 September 2021, covering approximately 60 electricity and natural gas utilities province-wide, and its Canadian initiatives page reports that 55 Ontario-based utilities and platforms achieved certification by the November 2023 deadline. The O. Reg. 633/21 explained guide covers the regulation in detail.

With Connect My Data, the customer authorizes once and the utility transfers data to the authorized third party from then on. The Green Button Alliance describes it as a customer-controlled authorization where the customer can define the length of that authorization and revoke authorization at any time, and notes the standard supports electricity, natural gas and water data at 1-minute, 15-minute, hourly, daily or monthly granularity. This is a utility data feed in the proper sense: no logins, no downloads, no re-keying.

Green Button Download My Data

Download My Data is the manual sibling. The Green Button Alliance describes it as letting customers select and download their electricity, gas or water usage data from their energy provider's website in an industry-standard XML format. The format is the same as Connect My Data, so the same parser applies, but it is not automatic. It suits a one-time history pull or a customer who prefers not to grant ongoing access. The Connect My Data vs Download My Data guide compares them.

Portal exports and scheduled reports

Some utilities offer business portals with their own export and scheduling features. Hydro One's Interval Data Portal, for example, lets commercial and industrial customers view hourly energy usage, download a breakdown by meter, facility and account, and schedule reports to be sent automatically by email, with a universal login for third-party representatives managing multiple accounts. Portal exports are a reasonable interim method, but every portal is different and each is a login that expires, changes or locks.

PDF bill capture

For water accounts, propane and fuel deliveries, utilities without Green Button, sites outside Ontario, and any account where the customer has not authorized a feed, the bill is the source. Automated bill retrieval means the PDF is picked up from an email inbox or a portal and digitized without a person reading it. BillScan does this for EZGB customers, extracting consumption, demand, billing period and line items from PDF and scanned bills and validating them before delivery.

Outside Ontario

The Green Button Alliance reports only minor discussions of Green Button in British Columbia, no known Connect My Data deployments in Alberta, and no information about deployments in Quebec, with Nova Scotia Power certified for Download My Data in May 2023. There, portals and bills carry the load. See how to get utility data in BC, Alberta and Quebec.

Comparison of collection methods

Method	Setup effort	Freshness	Coverage	Granularity
Green Button Connect My Data	One authorization per account, then none	Automatic after each utility update	Most regulated Ontario electricity and gas utilities	Interval and billing, per utility
Green Button Download My Data	Login and download each time	As often as someone downloads	Same utilities as CMD	Same format as CMD
Portal export or scheduled report	Login per portal, report configuration	Per schedule, until the login breaks	Utilities with a business portal	Varies by portal
Automated PDF bill capture	Inbox or portal pickup, account mapping	Each billing cycle	Any utility, any commodity, any province	Monthly billing data

Step 3: the authorization step

Connect My Data uses OAuth. RFC 6749 describes the OAuth 2.0 framework as enabling a third-party application to obtain limited access to an HTTP service on behalf of a resource owner, with access tokens representing the authorization issued to the client and refresh tokens used to obtain new access tokens. In practice the customer of record is sent to the utility's site, authenticates, chooses what to share and for how long, and is returned to the application. The OEB states that customers must authenticate as the account holder before permitting their utility to share data and can withdraw authorization at any time by contacting their utility. The OAuth authorization in Green Button Connect My Data guide walks through the flow.

The authorization has to be completed by the account holder, so plan the communication to tenants or site managers as carefully as the technical work. And each utility onboards third parties separately: Toronto Hydro, for example, requires a third party to submit an online application, complete a connectivity test and accept its terms before customers can connect to it, and Enbridge Gas lets customers revoke consent through the Share My Data section of My Account. A utility data automation platform that already holds these registrations saves repeating them; the aggregator vs build your own guide weighs that choice.

Step 4: schedule and refresh

An authorized feed still has to be polled. Decide the refresh cadence per commodity and use case: daily retrieval is typical for electricity interval data, while gas and water usually update on the billing cycle. Pull the full available history when an account is first connected, then retrieve incrementally. Keep the raw response from each retrieval as well as the parsed result, so a parser fix can be replayed against stored data. For bill capture, set an expected cycle per account and treat a missed bill as an exception to investigate rather than a quiet gap.

Step 5: validate what comes back

Automatic collection removes typing errors and introduces different problems: missing intervals, estimated reads, duplicated periods, unit mismatches, and daylight saving shifts. Every retrieval should be checked for interval counts per day, totals against billed consumption where both exist, period continuity, and plausibility against the account's history. The Green Button data quality gaps and validation guide lists the specific gaps seen in Ontario feeds and how to handle them.

Step 6: deliver to the systems that use it

The usual options are a REST API returning JSON for applications, SFTP drops for batch systems such as accounting and BI, and CSV for analysts. Choose based on the consuming system, and insist on one schema regardless of which utility or method the data came from. The how to receive utility data: API, SFTP or CSV guide compares the options.

Step 7: monitor for broken authorizations and gaps

Feeds break. Customers revoke authorizations, authorizations expire, portal passwords change, utilities migrate billing systems, and accounts close when tenants move. A process that scales has monitoring built in: a last-successful-retrieval date per account, alerts when an account exceeds its expected interval without data, a queue of accounts needing re-authorization, and a report of accounts with no method assigned. Review it weekly. The point of automation is to look at exceptions instead of at everything.

Frequently asked questions

What is the fastest way to collect utility data automatically in Ontario?

Green Button Connect My Data, where the account's utility offers it. Ontario requires most regulated electricity and natural gas utilities to provide it, the customer authorizes once, and data transfers automatically from then on. Pair it with automated PDF bill capture for the accounts it does not cover.

Can utility data collection be automated for water?

Partly. The Green Button standard supports water usage data, but Ontario's regulation applies to electricity and natural gas utilities, and most municipal water bills still arrive as PDFs. Automated bill capture is the practical method for water today.

How long does a Green Button authorization last?

The Green Button Alliance states that the customer defines the length of the authorization and can revoke it at any time. Ask customers to choose the longest period they are comfortable with, record the expiry date, and monitor for it.

How EZGB fits

EZGB collects billing and interval data from 55+ Ontario electricity and natural gas utilities through authorized utility connections and Green Button (Connect My Data and Download My Data), digitizes PDF and scanned bills with BillScan for the accounts without a feed, and retrieves on a schedule, typically daily. Every retrieval is validated, normalized to one schema, and delivered by REST API, SFTP, JSON and CSV, with authorization status and missed retrievals visible per account. The automated bill and meter data retrieval page and how EZGB works describe the pipeline, and EZGB pricing lists the plans from the one-time Freedom download to Enterprise.

If you have an account list, we can map each account to a collection method and show you what the first retrieval looks like for your utilities and delivery format. Request a demo on the pricing page.

Sources

Source: greenbuttonalliance.org