Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable user to import weather file as .dat #3713

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Enable user to import weather file as .dat #3713

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f894bee
Enable user to import weather file as .dat
Mar-Ges Nov 26, 2024
f45142a
Update weather_helper.py
Mar-Ges Dec 5, 2024
acaa4ba
Update weather_helper.py
Mar-Ges Dec 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion cea/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -561,14 +561,21 @@ def decode(self, value):
weather_path = self.locator.get_weather(value)
elif os.path.exists(value) and value.endswith('.epw'):
weather_path = value
elif os.path.exists(value) and value.endswith('.dat'):
# if dat convert to epw
dat_weather_file = value
epw_weather_file = os.path.join(os.path.dirname(value), os.path.basename(value).replace('.dat', '.epw'))
from cea.datamanagement.weather_helper import parse_dat_weather_file
parse_dat_weather_file(dat_weather_file, epw_weather_file)
weather_path = epw_weather_file
Comment on lines +564 to +570

This comment was marked as off-topic.

Sign in to view

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Mar-Ges we should only return the path of the file when parsing the weather path parameter in the config. The conversion would be done inside the weather-helper instead.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like the humans are having a chat. I'll hop back into my burrow for now. If you need me again, just tag @coderabbitai in a new comment, and I'll come hopping out!

elif any(w.lower().startswith(value.lower()) for w in self.locator.get_weather_names()) and value.strip():
# allow using shortcuts
weather_path = self.locator.get_weather(
[w for w in self.locator.get_weather_names() if w.lower().startswith(value.lower())][0])
elif value in self.THIRD_PARTY_WEATHER_SOURCES:
weather_path = value
else:
raise cea.ConfigError(f"Invalid weather path: {value}")
raise cea.ConfigError(f"Invalid weather path: {value}. Supported formats: .epw, .dat")
return weather_path

@property
Expand Down
1 change: 0 additions & 1 deletion cea/datamanagement/weather_helper/generate_weather_data_sources.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,6 @@

WEATHER_DATA_LOCATION = os.path.join(os.path.realpath(os.path.dirname(__file__)), 'weather.geojson')


def main():
# Combine for global data
excels = [pd.read_excel(url) for url in URL_LIST]
Expand Down
33 changes: 32 additions & 1 deletion cea/datamanagement/weather_helper/weather_helper.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,31 @@ def copy_weather_file(source_weather_file, locator):
source_weather_file=source_weather_file
))

def parse_dat_weather_file(source_dat_file, output_epw_file):
"""
convert a DWD .dat-file to a epw-file

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you be able to add the header definitions of the .dat file in english in the function's docstring as well. I think this would help others understand the provided columns when debugging this in the future.

Reihenfolge der Parameter:
RW Rechtswert                                                    [m]       {3670500;3671500..4389500}
HW Hochwert                                                      [m]       {2242500;2243500..3179500}
MM Monat                                                                   {1..12}
DD Tag                                                                     {1..28,30,31}
HH Stunde (MEZ)                                                            {1..24}
t  Lufttemperatur in 2m Hoehe ueber Grund                        [GradC]
p  Luftdruck in Standorthoehe                                    [hPa]
WR Windrichtung in 10 m Hoehe ueber Grund                        [Grad]    {0..360;999}
WG Windgeschwindigkeit in 10 m Hoehe ueber Grund                 [m/s]
N  Bedeckungsgrad                                                [Achtel]  {0..8;9}
x  Wasserdampfgehalt, Mischungsverhaeltnis                       [g/kg]
RF Relative Feuchte in 2 m Hoehe ueber Grund                     [Prozent] {1..100}
B  Direkte Sonnenbestrahlungsstaerke (horiz. Ebene)              [W/m^2]   abwaerts gerichtet: positiv
D  Diffuse Sonnenbetrahlungsstaerke (horiz. Ebene)               [W/m^2]   abwaerts gerichtet: positiv
A  Bestrahlungsstaerke d. atm. Waermestrahlung (horiz. Ebene)    [W/m^2]   abwaerts gerichtet: positiv
E  Bestrahlungsstaerke d. terr. Waermestrahlung                  [W/m^2]   aufwaerts gerichtet: negativ
IL Qualitaetsbit bezueglich der Auswahlkriterien                           {0;1;2;3;4}

:param string source_dat_file: source .dat
:param string output_epw_file: output EPW
"""
import pandas as pd

print(f"Parsing .dat file: {source_dat_file}")
try:
weather_data = pd.read_csv(source_dat_file, sep="\s+", skiprows=6, header=None)
weather_data.columns = ['Year', 'Month', 'Day', 'Hour', 'Temperature', 'Radiation', 'WindSpeed', '...']
reyery marked this conversation as resolved.
Show resolved Hide resolved

# from dat to epw
with open(output_epw_file, 'w') as epw_file:
# add EPW-Header
epw_file.write("LOCATION,Example,DEU,,,0,0,0\n")
epw_file.write("DESIGN CONDITIONS,0\n")
reyery marked this conversation as resolved.
Show resolved Hide resolved

for _, row in weather_data.iterrows():
epw_line = f"{row['Year']},{row['Month']},{row['Day']},{row['Hour']},...,{row['Temperature']},...\n"
epw_file.write(epw_line)
reyery marked this conversation as resolved.
Show resolved Hide resolved
except Exception as e:
raise ValueError(f"Error parsing .dat-Datei: {e}")
reyery marked this conversation as resolved.
Show resolved Hide resolved

def main(config):
"""
Expand All @@ -85,7 +110,13 @@ def main(config):
locator = cea.inputlocator.InputLocator(config.scenario)
weather = config.weather_helper.weather

if config.weather_helper.weather == 'climate.onebuilding.org':
if weather.endswith('.dat'):
print(f"Detected .dat file: {weather}")
# convert .dat in .epw
dat_weather_file = weather
epw_weather_file = locator.get_weather_file()
parse_dat_weather_file(dat_weather_file, epw_weather_file)
elif config.weather_helper.weather == 'climate.onebuilding.org':
Comment on lines +129 to +135
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Add file validation and error handling

The .dat file handling needs additional validation and cleanup.

     if weather.endswith('.dat'):
+        if not os.path.exists(weather):
+            raise ValueError(f"Weather file not found: {weather}")
         print(f"Detected .dat file: {weather}")
-        # convert .dat in .epw
         dat_weather_file = weather
         epw_weather_file = locator.get_weather_file()
+        try:
             parse_dat_weather_file(dat_weather_file, epw_weather_file)
+        except Exception as e:
+            # Clean up any partially written EPW file
+            if os.path.exists(epw_weather_file):
+                os.remove(epw_weather_file)
+            raise
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
if weather.endswith('.dat'):
print(f"Detected .dat file: {weather}")
# convert .dat in .epw
dat_weather_file = weather
epw_weather_file = locator.get_weather_file()
parse_dat_weather_file(dat_weather_file, epw_weather_file)
elif config.weather_helper.weather == 'climate.onebuilding.org':
if weather.endswith('.dat'):
if not os.path.exists(weather):
raise ValueError(f"Weather file not found: {weather}")
print(f"Detected .dat file: {weather}")
dat_weather_file = weather
epw_weather_file = locator.get_weather_file()
try:
parse_dat_weather_file(dat_weather_file, epw_weather_file)
except Exception as e:
# Clean up any partially written EPW file
if os.path.exists(epw_weather_file):
os.remove(epw_weather_file)
raise
elif config.weather_helper.weather == 'climate.onebuilding.org':

print("No weather provided, fetching from online sources.")
fetch_weather_data(locator.get_weather_file(), locator.get_zone_geometry())
else:
Expand Down
2 changes: 1 addition & 1 deletion cea/default.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,7 @@ trees.help = Path to shapefile containing the geometries of tree canopies in pol
[weather-helper]
weather =
weather.type = WeatherPathParameter
weather.help = Either a full path to a weather file (.epw) or the name of one of the weather files contained within the CEA. Leave blank to fetch from third-party sources.
weather.help = Either a full path to a weather file (.epw or .dat) or the name of one of the weather files contained within the CEA. Leave blank to fetch from third-party sources.

[radiation]
buildings =
Expand Down