GeneralProcessing package
Submodules
GeneralProcessing.gen_streamflow_file module
Streamflow File Preparation
gen_streamflow_file.py contains a class GenStreamflowFile that handles fetching and combining streamflow data from USGS and Environment Canada and generating output in the OBSTXT and ENSIM formats.
Parameters:
fetch_hydrometric_data_ca: Fetches flow data from Canadian stations. extract_flow_data_us: Fetches flow data from US stations. write_flow_data_to_file_obstxt: Writes the data in OBSTXT format. write_flow_data_to_file_ensim: Writes the data in ENSIM format.
Example Usage (Please check MESH_streamflowFile_example.ipynb for step by step example)
>>> from GeneralProcessing.gen_streamflow_file import GenStreamflowFile
>>> gen_flow = GenStreamflowFile()
>>>
>>> # 1) Historical daily‐mean (1980–2018)
>>> station_ca = ["05GG001", "05AC012"]
>>> station_us = ["06132200", "05020500"]
>>> start_date = "1980-03-01"
>>> end_date = "2018-01-10"
>>> df_ca, meta_ca = gen_flow.fetch_hydrometric_data_ca(station_ca, start_date, end_date)
>>> df_us, meta_us = gen_flow.extract_flow_data_us(station_us, start_date, end_date)
>>> combined = pd.merge(df_ca, df_us, on="Date", how="outer")
>>>
>>> # 2) Realtime provisional (last 1 month by 1-day windows)
>>> from datetime import datetime, timezone
>>> from dateutil.relativedelta import relativedelta
>>> end_dt = datetime.now(timezone.utc).replace(microsecond=0)
>>> start_dt = end_dt - relativedelta(months=1)
>>> start = start_dt.strftime("%Y-%m-%dT%H:%M:%SZ")
>>> end = end_dt.strftime("%Y-%m-%dT%H:%M:%SZ")
>>> df_rt, meta_rt = gen_flow.fetch_hydrometric_realtime_full_range(
... station_numbers=["05GG001","05AC012"],
... start=start, end=end,
... window_days=1, freq_hours=12
... )
>>>
>>> # 3) Write files
>>> all_meta = meta_ca + meta_us + meta_rt
>>> gen_flow.write_flow_data_to_file_obstxt("streamflow_obs.txt", combined, all_meta)
>>> gen_flow.write_flow_data_to_file_ensim("streamflow.tb0", combined, all_meta)
- class GeneralProcessing.gen_streamflow_file.GenStreamflowFile[source]
Bases:
object- extract_flow_data_us_with_metadata(station_list: list[str], start_date: str, end_date: str, limit: int = 1000) tuple[DataFrame, list[dict]][source]
Fetch daily‐mean discharge and full station metadata for USGS gauges.
- Parameters:
station_list (list[str]) – USGS station numbers (e.g. [‘06132200’]).
start_date (str) – Start date in ‘YYYY-MM-DD’ format.
end_date (str) – End date in ‘YYYY-MM-DD’ format.
limit (int) – Page size for API requests (default=1000).
- Returns:
df (pd.DataFrame) – Indexed by Date, one column per station with daily mean discharge [m³/s].
station_info (list[dict]) –
- Metadata for each station, including:
Station_Number, Station_Name, Latitude, Longitude, Drainage_Area, Contrib_Drainage_Area, Elevation_m, Datum
- fetch_hydrometric_data_ca(station_numbers: list[str], start_date: str, end_date: str, limit: int = 1000) tuple[DataFrame, list[dict]][source]
Fetch daily‐mean discharge and full station metadata for Canadian hydrometric stations.
- Parameters:
station_numbers (list of str) – Environment Canada station IDs (e.g. [‘05HG001’, ‘05AG006’]).
start_date (str) – Start date in ‘YYYY-MM-DD’ format.
end_date (str) – End date in ‘YYYY-MM-DD’ format.
limit (int, optional) – API page size (default=1000).
- Returns:
df (pandas.DataFrame) – DataFrame indexed by Date, with one column per station containing daily mean discharge.
metadata (list of dict) –
- List of station metadata dicts with keys:
Station_Number, Station_Name, CONTRIBUTOR_EN, PROV_TERR_STATE_LOC, DRAINAGE_AREA_GROSS, DRAINAGE_AREA_EFFECT, REAL_TIME, RHBN, STATUS_EN, VERTICAL_DATUM, Latitude, Longitude
- fetch_hydrometric_realtime_full_range(station_numbers: list[str], start: str, end: str, window_days: int = 1, freq_hours: int | None = None, limit: int = 1000) tuple[DataFrame, list[dict]][source]
Fetches provisional real-time discharge by slicing [start,end] into window_days-day windows. If freq_hours is given, the result is resampled to that interval; otherwise you get the native timestamps.
- write_flow_data_to_file_ensim(file_path: str, flow_data: DataFrame, site_details: list, column_width: int = 12, initial_spacing: int = 28)[source]
Write a pandas DataFrame of streamflow time series to an EnSim‐formatted ASCII file.
Parameters:
- file_pathstr
Path to the output file.
- flow_datapd.DataFrame
Time series data. May include a ‘Date’ column (case‐insensitive) or have a DatetimeIndex.
- site_detailslist of dict
One dict per data column, each with ‘Latitude’ and ‘Longitude’ keys.
- column_widthint
Fixed width for each numeric field.
- initial_spacingint
Number of spaces before the first data column on each line.
- write_flow_data_to_file_obstxt(file_path: str, flow_data: DataFrame, site_details: list)[source]
Write a pandas DataFrame of observed streamflow to a space-delimited text file.
Parameters:
- file_pathstr
Path for the output .txt file.
- flow_datapd.DataFrame
Time series data. May include a ‘Date’ column (case-insensitive) or have a DatetimeIndex.
- site_detailslist of dict
One dict per data column, each with keys: ‘Station_Number’, ‘Latitude’, ‘Longitude’, ‘Drainage_Area’, ‘Station_Name’.
GeneralProcessing.generate_mesh_class_ini_from_excel module
- GeneralProcessing.generate_mesh_class_ini_from_excel.generate_mesh_class_ini_from_excel(excel_file, output_file='MESH_class_output.ini', num_cels=None, lat=None, lon=None, sheet_name='class_ini')[source]
Description:
This function processes an Excel file containing land cover parameter values and generates a MESH-compatible .ini file. The function is designed to extract, format, and write the required parameters for active land covers into the output file, ensuring compatibility with the MESH hydrological model. The function includes detailed steps for processing the Excel file, extracting relevant data, and formatting it into the required structure for MESH. It also validates the input data and ensures that all necessary parameters are present. The existing description below provides further details about the function’s parameters, processing steps, and output format.
Parameters:
- excel_filestr
Path to the Excel file containing parameter values.
- output_filestr
Path to the output .ini file where processed values will be written.
- num_celsint
Number of grid cells in the model domain.
- latfloat
Latitude of the location.
- lonfloat
Longitude of the location.
Overview:
This function extracts land cover parameter values from an Excel file and writes them into a MESH-compatible .ini file. Only active land covers are included, as indicated by the ‘status’ row in the Excel sheet.
Processing Steps:
- Load the Excel file and normalize column names.
The function reads the Excel file using pandas and converts all column names to lowercase for consistency.
Identify active land covers (status > 0): - The function identifies the row labeled ‘status’ in the Excel file, which contains numerical values for each land cover column. - These numerical values indicate the activation status of each land cover. A value greater than 0 signifies that the land cover is active. - The function filters out inactive land covers (status <= 0) and retains only the active ones. - The active land covers are then sorted in ascending order based on their status values to ensure a consistent and logical ordering in the output file.
Verify required rows such as ‘colum’.
Extract vegetation and land cover parameters.
Write formatted values into an .ini file with the required MESH structure.
Output Format:
The output file is structured to be compatible with MESH, including: - Header defining basin information. - Land cover-specific vegetation and hydrological parameters. - Footer containing model time initialization values.
File Structure:
The output file consists of:
Header Information: Includes metadata such as location, author, and details.
Land Cover Blocks: Each selected land cover is written separately, including: - Vegetation parameters (written in pairs) - One-to-One parameter assignments (written in pairs) - Multi-value parameter assignments (written in structured format)
Final Footer: Contains three mandatory lines required for MESH processing.
Example Usage:
>>> # The excel database can be found in the GeneralProcessing folder. >>> # Directly dowolnload the excel file from the GeneralProcessing folder. >>> import requests >>> url = "https://raw.githubusercontent.com/MESH-Scripts-PyLib/MESH-Scripts-PyLib/main/GeneralProcessing/meshparametersvalues2.xlsx" >>> # Local path to save the file >>> local_path = "D:/Coding/GitHub/Repos/MESH-Scripts-PyLib/GeneralProcessing/meshparametersvalues2.xlsx" >>> # Send a GET request to download the file >>> response = requests.get(url) >>> # Check if the request was successful (status code 200) >>> if response.status_code == 200: >>> # Write the content to a local file >>> with open(local_path, 'wb') as file: >>> file.write(response.content) >>> print(f"File downloaded successfully and saved to {local_path}") >>> else: >>> print(f"Failed to download file. Status code: {response.status_code}")
>>> # Example usage of the function >>> from GeneralProcessing.generate_mesh_class_ini_from_excel import generate_mesh_class_ini_from_excel as gen_classini >>> gen_classini( ... excel_file="D:/Coding/GitHub/Repos/MESH-Scripts-PyLib/GeneralProcessing/meshparametersvalues2.xlsx", ... output_file="MESH_output2.ini", ... num_cels=7408, ... lat=53.18, ... lon=-99.25 ... )
Example Output:
Basin Author Org 53.18 -99.25 40.00 40.00 50.00 -1.0 1 7408 5 0.000 0.000 0.000 0.000 1.000 0.000 0.000 0.000 0.000 # fcan, lamx 0.000 0.000 0.000 0.000 -5.352 0.000 0.000 0.000 0.000 # lnz, lamn 0.000 0.000 0.000 0.000 0.400 0.000 0.000 0.000 0.000 # alvc, cmas 0.000 0.000 0.000 0.000 0.250 0.000 0.000 0.000 0.000 # alir, root 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 # rsmn, qa50 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 # vpda, vpdp 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 # psga, psgb 1.000 2.000 1.000 50.000 # drn, sdep, fare, dden 0.100 0.100 0.388 0.070 8.000 Barren # xslp, grkf, man, wfci, mid, name 73.000 45.000 66.000 # sand 6.400 29.600 20.440 # clay 7.400 0.000 0.000 # org 4.000 2.000 1.000 4.000 0.000 4.000 # soit, cant, snot, pndt 0.250 0.250 0.250 0.000 0.000 0.000 0.000 # soiwf, soiif, pond 0.000 0.000 0.000 0.500 100.000 1.000 # rcan, scan, sno, albs, rho, gro 0.000 1.000 0.000 0.000 0.000 0.000 4.148 0.000 0.000 # fcan, lamx 0.000 0.475 0.000 0.000 0.000 0.000 0.512 0.000 0.000 # lnz, lamn 0.000 0.049 0.000 0.000 0.000 0.000 23.583 0.000 0.000 # alvc, cmas 0.000 0.257 0.000 0.000 0.000 0.000 1.249 0.000 0.000 # alir, root 0.000 110.930 0.000 0.000 0.000 37.506 0.000 0.000 # rsmn, qa50 0.000 0.427 0.000 0.000 0.000 0.765 0.000 0.000 # vpda, vpdp 0.000 100.000 0.000 0.000 0.000 5.000 0.000 0.000 # psga, psgb 1.000 2.000 1.000 50.000 # drn, sdep, fare, dden 0.040 0.100 0.119 0.100 2.000 MixedForest # xslp, grkf, man, wfci, mid, name 75.000 66.000 66.000 # sand 5.000 30.000 17.000 # clay 5.000 0.000 0.000 # org 4.000 2.000 1.000 4.000 0.000 4.000 # soit, cant, snot, pndt 0.250 0.250 0.250 0.000 0.000 0.000 0.000 # soiwf, soiif, pond 0.000 0.000 0.000 0.500 100.000 1.000 # rcan, scan, sno, albs, rho, gro 1.000 0.000 0.000 0.000 0.000 1.162 0.000 0.000 0.000 # fcan, lamx -0.187 0.000 0.000 0.000 0.000 0.509 0.000 0.000 0.000 # lnz, lamn 0.046 0.000 0.000 0.000 0.000 13.624 0.000 0.000 0.000 # alvc, cmas 0.154 0.000 0.000 0.000 0.000 1.768 0.000 0.000 0.000 # alir, root 236.390 0.000 0.000 0.000 27.913 0.000 0.000 0.000 # rsmn, qa50 0.502 0.000 0.000 0.000 0.812 0.000 0.000 0.000 # vpda, vpdp 100.000 0.000 0.000 0.000 5.000 0.000 0.000 0.000 # psga, psgb 1.000 2.000 1.000 50.000 # drn, sdep, fare, dden 0.040 0.122 0.151 1.380 1.000 Forest # xslp, grkf, man, wfci, mid, name 73.200 43.300 72.903 # sand 5.200 29.100 21.362 # clay 5.000 0.000 0.000 # org 4.000 2.000 1.000 4.000 0.000 4.000 # soit, cant, snot, pndt 0.250 0.250 0.250 0.000 0.000 0.000 0.000 # soiwf, soiif, pond 0.000 0.000 0.000 0.500 100.000 1.000 # rcan, scan, sno, albs, rho, gro 0.000 0.000 1.000 0.000 0.000 0.000 0.000 5.807 0.000 # fcan, lamx 0.000 0.000 -2.541 0.000 0.000 0.000 0.000 0.000 0.000 # lnz, lamn 0.000 0.000 0.061 0.000 0.000 0.000 0.000 1.923 0.000 # alvc, cmas 0.000 0.000 0.298 0.000 0.000 0.000 0.000 1.200 0.000 # alir, root 0.000 0.000 63.107 0.000 0.000 0.000 22.541 0.000 # rsmn, qa50 0.000 0.000 0.568 0.000 0.000 0.000 0.804 0.000 # vpda, vpdp 0.000 0.000 100.000 0.000 0.000 0.000 5.000 0.000 # psga, psgb 1.000 2.000 1.000 40.000 # drn, sdep, fare, dden 0.010 0.503 0.164 1.432 4.000 Crop # xslp, grkf, man, wfci, mid, name 50.000 55.000 60.000 # sand 35.000 35.000 35.000 # clay 15.000 10.000 5.000 # org 4.000 2.000 1.000 4.000 0.000 4.000 # soit, cant, snot, pndt 0.250 0.250 0.250 0.000 0.000 0.000 0.000 # soiwf, soiif, pond 0.000 0.000 0.000 0.500 100.000 1.000 # rcan, scan, sno, albs, rho, gro 0.000 0.000 0.000 1.000 0.000 0.000 0.000 0.000 2.000 # fcan, lamx 0.000 0.000 0.000 -2.806 0.000 0.000 0.000 0.000 0.500 # lnz, lamn 0.000 0.000 0.000 0.050 0.000 0.000 0.000 0.000 1.200 # alvc, cmas 0.000 0.000 0.000 0.340 0.000 0.000 0.000 0.000 1.200 # alir, root 0.000 0.000 0.000 100.000 0.000 0.000 0.000 30.000 # rsmn, qa50 0.000 0.000 0.000 0.500 0.000 0.000 0.000 1.000 # vpda, vpdp 0.000 0.000 0.000 100.000 0.000 0.000 0.000 5.000 # psga, psgb 1.000 2.000 1.000 50.000 # drn, sdep, fare, dden 0.010 0.200 0.233 0.100 5.000 Grass # xslp, grkf, man, wfci, mid, name 72.100 49.700 70.020 # sand 5.000 21.400 23.682 # clay 7.800 0.000 0.000 # org 4.000 2.000 1.000 4.000 0.000 4.000 # soit, cant, snot, pndt 0.250 0.250 0.250 0.000 0.000 0.000 0.000 # soiwf, soiif, pond 0.000 0.000 0.000 0.500 100.000 1.000 # rcan, scan, sno, albs, rho, gro 0 0 0 0 20 0 0 0 0 21 0 0 0 0 22 IHOUR/IMINS/IJDAY/IYEAR
GeneralProcessing.generate_mesh_hydrology_ini_from_excel module
- GeneralProcessing.generate_mesh_hydrology_ini_from_excel.generate_mesh_hydrology_ini_from_excel(excel_file, output_file='MeshHydrology.ini', sheet_name='hydrology_ini')[source]
Description:
This function reads hydrology parameter values from an Excel file and generates a formatted MESH Hydrology .ini file. The generated file is structured into various parameter categories with appropriate headers, formatting, and spacing.
Parameters:
excel_file : str -> Path to the input Excel file containing parameter values.
- sheet_namestr -> The specific sheet in the Excel file that holds the data.
Default is “hydrology_ini”.
output_file : str -> Path to the output .ini file.
Function Overview:
Reads the Excel file and extracts three main parameter categories: - Channel Routing Parameters - GRU-Independent Parameters - GRU-Dependent Parameters
Filters parameters where status == 1 (active parameters).
Sorts and formats values for proper alignment: - Parameter names are left-aligned (12 characters). - Values are right-aligned (10 characters, formatted as .3f if float).
Writes the output .ini file with section headers, formatted values, and comments where necessary.
Example Hydrology Input File (Version 2.0):
2.0: MESH Hydrology parameters input file (Version 2.0) !> Any line leading with '!' is ignored !> Spacing, whether by tabs or spaces doesn't matter !> Parameter/field names are not case sensitive ##### Option Flags ##### ----# 0 # Number of option flags ##### Channel routing parameters ##### -----# 4 # Number of channel routing parameters R2N 0.100 0.045 0.049 0.013 0.050 0.150 0.141 0.150 0.135 0.314 R1N 0.200 0.167 0.156 0.049 0.064 0.596 0.514 0.422 0.171 0.544 FLZ 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 PWR 1.618 2.570 2.721 2.993 2.823 1.361 1.361 1.361 1.361 1.361 ##### GRU-independent parameters ##### -------# 5 # Number of GRU independent hydrologic parameters SOIL_POR_MAX 0.200 SOIL_DEPTH 4 S0 1 T_ICE_LENS -10 t0_ACC 0 0 0 0 0 ##### GRU-dependent parameters ##### -------# !> Active headers in sorted order: Forest, BroadLeafForest, MixedForest, wetland, water, Shrub 24 # Number of GRU-dependent parameters IWF 1 1 1 1 1 1 ZSNL 0.173 0.393 0.393 0.083 0.110 0.268 ZPLS 0.023 0.097 0.097 0.090 0.090 0.149 ZPLG 0.190 0.282 0.282 0.260 0.090 0.045 FREZTH 0 0 0 0 0 0 SWELIM 0 0 0 0 0 0 SNDENLIM 0 0 0 0 0 0 fetch 300 300 300 300 300 600 Ht 3 4 4 1.500 0.030 1.200 N_S 1 1 1 1 1 1 A_S 1 1 1 1 1 1 Distrib 6 6 6 0 0 1 CMAX 1.500 1.500 1.500 1.500 1.500 1.500 CMIN 0 0 0 0 0 0 B 3 3 3 3 3 3 K1 0 0 0 0 0 0 K2 0 0 0 0 0 0 irflg 0 0 0 0 0 0 irthlmin 0.500 0.500 0.500 0.500 0.500 0.500 irignd 0 0 0 0 0 0 irt1 6 6 6 6 6 6 irt2 10 10 10 10 10 10 irijday1 0 0 0 0 0 0 irijday2 0 0 0 0 0 0File Structure:
Header section: Explains format rules.
Option Flags: Indicates no optional flags.
Channel Routing Parameters: Defines routing-related parameters.
GRU-Independent Parameters: Contains parameters that apply to all land units.
GRU-Dependent Parameters: Contains parameters that vary by land category.
GRU-Dependent Parameters Selection and Ordering:
The GRU_class_dependent_active row in the Excel sheet is used to determine which GRU-dependent headers (columns) are active. Each column corresponding to a GRU class (e.g., Forest, Grass, Wetland) has a value indicating its active status:
A value > 0 indicates the column is active.
A value <= 0 indicates the column is inactive.
Active columns are sorted in ascending order based on their values in the GRU_class_dependent_active row. This sorted order determines the sequence in which GRU-dependent parameters are written to the output file.
The sorted active column names are included as a comment in the output file for reference.
Example Usage:
>>> # The excel database can be found in the GeneralProcessing folder. >>> # Directly dowolnload the excel file from the GeneralProcessing folder. >>> import requests >>> url = "https://raw.githubusercontent.com/MESH-Scripts-PyLib/MESH-Scripts-PyLib/main/GeneralProcessing/meshparametersvalues2.xlsx" >>> # Local path to save the file >>> local_path = "D:/Coding/GitHub/Repos/MESH-Scripts-PyLib/GeneralProcessing/meshparametersvalues2.xlsx" >>> # Send a GET request to download the file >>> response = requests.get(url) >>> # Check if the request was successful (status code 200) >>> if response.status_code == 200: >>> # Write the content to a local file >>> with open(local_path, 'wb') as file: >>> file.write(response.content) >>> print(f"File downloaded successfully and saved to {local_path}") >>> else: >>> print(f"Failed to download file. Status code: {response.status_code}")
>>> from GeneralProcessing.generate_mesh_hydrology_ini_from_excel import generate_mesh_hydrology_ini_from_excel >>> gen_hydini = generate_mesh_hydrology_ini_from_excel >>> gen_hydini( sheet_name="hydrology_ini", output_file="MeshHydrology2.ini", excel_file="D:/Coding/GitHub/Repos/MESH-Scripts-PyLib/GeneralProcessing/meshparametersvalues2.xlsx" )
This will generate a properly formatted MeshHydrology.ini file in the specified path.