Real-Time Energy Price Prediction

The electricity industry in most of Texas is deregulated, which means that consumers can choose their electricity provider. This is different from regulated areas of Texas and other parts of the country, where a single utility company delivers electricity to homes and businesses. Texas deregulation process started with the passage of Senate Bill 373 in 1995 and culminated with granting independence to the Electric Reliability Council of Texas (ERCOT) in 2002.

My work at a green energy trading company/hedge fund (battery energy storage system or "BESS" company) involved building an entire backend real-time, multi-cloud data storage system and ELT/ETL pipeline. Our goal was to make real-time energy predictions as accurately as possible and as far in advance as possible.

I cannot provide specifics on the data vendors or code used to build my machine learning models out of confidentiality, but here is an example of the type of weather data that such models use:

Looks cool, right?

It was a fun challenge to work with both public and private, proprietary data sets to build a data pipeline on Snowflake, AWS and with an app on Slack. The less fun part was discovering the vagaries, nuances and problems within the datasets. For example, what do `DW_PT`, `DW_PT_3` and `PCP_7DAY` mean? Why are there nulls sometimes? How is daylight savings handled in the data? As a data scientist, each of these questions had to be answered as neat, organized data catalogues often aren't available, even when one is paying very large sums of money for "cleaned" data.

The tangible outputs of weather are simple and easy to grasp: sunshine, rain, etc. But this becomes a more complex topic when considering that solar irradiance, wind velocity and direction, precipitation, temperature, and many more variables vary by location and over time. This is an especially interesting problem because these weather patterns have an impact on the physical capacity of the power lines that transmit electricity, and then there is the idea of these power lines have limited capacity much like highways, areas of higher demand at different times and how that varies by location. There is also the interesting consideration that solar power generation and wind generation obviously varies with the day-night cycle, and is also impacted by cloud cover.

Building the model was the fun part.

Here is a brief article that I wrote about my work can be read on this industry publication. Note that the date of publication was not the date of authorship. https://www.energy-storage.news/the-future-of-battery-trading-addressing-the-risks-and-rewards-of-ai-optimisation/