prereise.gather.flexibilitydata.doe package

Subpackages

• prereise.gather.flexibilitydata.doe.examples package
  • Submodules
  • prereise.gather.flexibilitydata.doe.examples.bus_example module
  • prereise.gather.flexibilitydata.doe.examples.doe_example module
  • prereise.gather.flexibilitydata.doe.examples.geo_example module
  • Module contents
• prereise.gather.flexibilitydata.doe.tests package
  • Submodules
  • prereise.gather.flexibilitydata.doe.tests.test_bus_data module
    • test_get_bus_fips()
    • test_get_bus_zip()
  • prereise.gather.flexibilitydata.doe.tests.test_geo_data module
    • test_batch_download()
    • test_cache_production()
  • Module contents

Submodules

prereise.gather.flexibilitydata.doe.batch_process module

prereise.gather.flexibilitydata.doe.batch_process.collect_all_raw_data(download_path)

Download all required raw data needed for producing cached files

Parameters:

download_path (str) – folder to store the downloaded file

prereise.gather.flexibilitydata.doe.batch_process.create_geo_cache_files(raw_path, cache_path)

Process downloaded raw files and create cached intermediate files

Parameters:

• raw_path (str) – folder that contains downloaded raw data

• cache_path (str) – folder to store processed cache files

prereise.gather.flexibilitydata.doe.batch_process.get_cache_from_blob(cache_path)

Download previously uploaded cached files from BLOB storage

Parameters:

cache_path (str) – folder to store downloaded cache files

prereise.gather.flexibilitydata.doe.bus_data module

prereise.gather.flexibilitydata.doe.bus_data.cleanup_zip(zipdict)

Try to cleanup a zip dictionary obtained using online query by converting to 5-digit integers. Several possible mis-format are considered

Parameters:

zipdict (dict) – a dictionary containing raw zip-code of buses

Returns:

(dict) – a dictionary containing 5-digit zip codes

prereise.gather.flexibilitydata.doe.bus_data.get_all_bus_eiaid(bus_csv_path, doe_csv_path, cache_path, bus_pos, out_path)

Compute the EIA ID of each bus in bus.csv from powersimdata using cached files

Parameters:

• bus_csv_path (str) – bus.csv in a powersimdata network model

• doe_csv_path (str) – aggregated .csv DOE flexibility data

• cache_path (str) – folder to store processed cache files

• bus_pos (pandas.DataFrame) – (n x 3) dataframe containing bus coordinates

• out_path (str) – output path to store the bus.csv with EIA ID

Raises:

FileNotFoundError – when any required cache file is not present

prereise.gather.flexibilitydata.doe.bus_data.get_bus_fips(bus_pos, cache_path, start_idx=0)

Try to get FIPS of each bus in a case mat using FCC AREA API Can take hours to run, save to cache file for future use

Parameters:

• bus_pos (pandas.DataFrame) – a dataframe of (bus, lat, lon)

• cache_path (str) – folder to store processed cache files

• start_idx (int) – pointer to the index of a bus to start query from

prereise.gather.flexibilitydata.doe.bus_data.get_bus_pos(grid)

Read raw files of synthetic grid and extract the lat/lon coordinate of all buses

Parameters:

grid (powersimdata.input.grid.Grid) – a Grid instance

Returns:

(pandas.DataFrame) – a data frame of bus position

prereise.gather.flexibilitydata.doe.bus_data.get_bus_zip(bus_pos, cache_path, start_idx=0)

Try to get ZIP of each bus in a case mat using geopy Can take hours to run, save to cache file for future use

Parameters:

• bus_pos (pandas.DataFrame) – a dataframe of (bus, lat, lon)

• cache_path (str) – folder to store processed cache files

• start_idx (int) – pointer to the index of a bus to start query from

prereise.gather.flexibilitydata.doe.doe_data module

prereise.gather.flexibilitydata.doe.doe_data.aggregate_doe(root, out_path)

Aggregate sector flexibilties by summing up the percentage flexibility from all sectors and store to output csv file

Parameters:

• root (str) – the root directory containing raw de-compressed DOE flexibility data

• out_path (str) – the output file where the aggregated data will be stored

prereise.gather.flexibilitydata.doe.doe_data.cleanup_directory(root)

Recursively cleanup a folder by deleting meaningless or empty files

Parameters:

root (str) – the root directory containing raw de-compressed DOE flexibility data

prereise.gather.flexibilitydata.doe.doe_data.download_doe(download_path='data')

Download demand flexibility filters from OEDI, extract and cleanup

Parameters:

download_path (str) – the directory where the original DOE data will be stored

prereise.gather.flexibilitydata.doe.geo_data module

prereise.gather.flexibilitydata.doe.geo_data.eiaid_to_fips(raw_data_path, cache_path)

Find the service region (list of FIPS codes) for every LSE identified by their EIA ID Create a dictionary with EIA ID as keys for list of FIPS codes in the cache folder

Parameters:

• raw_data_path (str) – folder that contains downloaded raw data

• cache_path (str) – folder to store processed cache files

prereise.gather.flexibilitydata.doe.geo_data.eiaid_to_zip(raw_data_path, cache_path)

Find the service region (list of ZIP codes) for every LSE identified by their EIA ID Create a dictionary with EIA ID as keys for list of zip codes in the cache folder

Parameters:

• raw_data_path (str) – folder that contains downloaded raw data

• cache_path (str) – folder to store processed cache files

prereise.gather.flexibilitydata.doe.geo_data.fips_to_eiaid(raw_data_path, cache_path)

Find the corresponding LSE for all counties identified by their FIPS number

Parameters:

• raw_data_path (str) – folder that contains downloaded raw data

• cache_path (str) – folder to store processed cache files

prereise.gather.flexibilitydata.doe.geo_data.fips_zip_conversion(raw_data_path, cache_path)

Create a two-way mapping for all ZIP and FIPS in the crosswalk data save to dictionary files for future use

Parameters:

• raw_data_path (str) – folder that contains downloaded raw data

• cache_path (str) – folder to store processed cache files

prereise.gather.flexibilitydata.doe.geo_data.get_census_data(download_path)

Download county population data from USA Census website

Parameters:

download_path (str) – folder to store the downloaded file

prereise.gather.flexibilitydata.doe.geo_data.get_county_fips_data(download_path)

Download county FIPS data

Parameters:

download_path (str) – folder to store the downloaded file

prereise.gather.flexibilitydata.doe.geo_data.get_crosswalk_data(download_path)

Download FIPS-ZIP crosswalk data from USPS and convert to csv

Parameters:

download_path (str) – folder to store the downloaded file

prereise.gather.flexibilitydata.doe.geo_data.get_fips_population(raw_data_path, cache_path)

Match county population and county FIPS data to produce concise FIPS population save to a dictonary in cache folder with key being 5-digit FIPS codes.

Parameters:

• raw_data_path (str) – folder that contains downloaded raw data

• cache_path (str) – folder to store processed cache files

prereise.gather.flexibilitydata.doe.geo_data.get_lse_region_data(download_path)

Download LSE service region data

Parameters:

download_path (str) – folder to store the downloaded file

prereise.gather.flexibilitydata.doe.geo_data.get_zip_population(raw_data_path, cache_path)

Compute population of each ZIP code using percentage share and FIPS population save to a dictonary in cache folder with key being zip codes.

Parameters:

• raw_data_path (str) – folder that contains downloaded raw data

• cache_path (str) – folder to store processed cache files

prereise.gather.flexibilitydata.doe.geo_data.zip_to_eiaid(raw_data_path, cache_path)

Find the LSE of all zip codes listed in the EIA LSE service regoin data

Parameters:

• raw_data_path (str) – folder that contains downloaded raw data

• cache_path (str) – folder to store processed cache files

Module contents