Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable user to import weather file as .dat #3713

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Enable user to import weather file as .dat #3713

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f894bee
Enable user to import weather file as .dat
Mar-Ges Nov 26, 2024
f45142a
Update weather_helper.py
Mar-Ges Dec 5, 2024
acaa4ba
Update weather_helper.py
Mar-Ges Dec 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion cea/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -561,14 +561,21 @@ def decode(self, value):
weather_path = self.locator.get_weather(value)
elif os.path.exists(value) and value.endswith('.epw'):
weather_path = value
elif os.path.exists(value) and value.endswith('.dat'):
# if dat convert to epw
dat_weather_file = value
epw_weather_file = os.path.join(os.path.dirname(value), os.path.basename(value).replace('.dat', '.epw'))
from cea.datamanagement.weather_helper import parse_dat_weather_file
parse_dat_weather_file(dat_weather_file, epw_weather_file)
weather_path = epw_weather_file
Comment on lines +564 to +570

This comment was marked as off-topic.

Sign in to view

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Mar-Ges we should only return the path of the file when parsing the weather path parameter in the config. The conversion would be done inside the weather-helper instead.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like the humans are having a chat. I'll hop back into my burrow for now. If you need me again, just tag @coderabbitai in a new comment, and I'll come hopping out!

elif any(w.lower().startswith(value.lower()) for w in self.locator.get_weather_names()) and value.strip():
# allow using shortcuts
weather_path = self.locator.get_weather(
[w for w in self.locator.get_weather_names() if w.lower().startswith(value.lower())][0])
elif value in self.THIRD_PARTY_WEATHER_SOURCES:
weather_path = value
else:
raise cea.ConfigError(f"Invalid weather path: {value}")
raise cea.ConfigError(f"Invalid weather path: {value}. Supported formats: .epw, .dat")
return weather_path

@property
Expand Down
1 change: 0 additions & 1 deletion cea/datamanagement/weather_helper/generate_weather_data_sources.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,6 @@

WEATHER_DATA_LOCATION = os.path.join(os.path.realpath(os.path.dirname(__file__)), 'weather.geojson')


def main():
# Combine for global data
excels = [pd.read_excel(url) for url in URL_LIST]
Expand Down
49 changes: 48 additions & 1 deletion cea/datamanagement/weather_helper/weather_helper.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,47 @@ def copy_weather_file(source_weather_file, locator):
source_weather_file=source_weather_file
))

def parse_dat_weather_file(source_dat_file, output_epw_file):
"""
convert a DWD .dat-file to a epw-file

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you be able to add the header definitions of the .dat file in english in the function's docstring as well. I think this would help others understand the provided columns when debugging this in the future.

Reihenfolge der Parameter:
RW Rechtswert                                                    [m]       {3670500;3671500..4389500}
HW Hochwert                                                      [m]       {2242500;2243500..3179500}
MM Monat                                                                   {1..12}
DD Tag                                                                     {1..28,30,31}
HH Stunde (MEZ)                                                            {1..24}
t  Lufttemperatur in 2m Hoehe ueber Grund                        [GradC]
p  Luftdruck in Standorthoehe                                    [hPa]
WR Windrichtung in 10 m Hoehe ueber Grund                        [Grad]    {0..360;999}
WG Windgeschwindigkeit in 10 m Hoehe ueber Grund                 [m/s]
N  Bedeckungsgrad                                                [Achtel]  {0..8;9}
x  Wasserdampfgehalt, Mischungsverhaeltnis                       [g/kg]
RF Relative Feuchte in 2 m Hoehe ueber Grund                     [Prozent] {1..100}
B  Direkte Sonnenbestrahlungsstaerke (horiz. Ebene)              [W/m^2]   abwaerts gerichtet: positiv
D  Diffuse Sonnenbetrahlungsstaerke (horiz. Ebene)               [W/m^2]   abwaerts gerichtet: positiv
A  Bestrahlungsstaerke d. atm. Waermestrahlung (horiz. Ebene)    [W/m^2]   abwaerts gerichtet: positiv
E  Bestrahlungsstaerke d. terr. Waermestrahlung                  [W/m^2]   aufwaerts gerichtet: negativ
IL Qualitaetsbit bezueglich der Auswahlkriterien                           {0;1;2;3;4}

:param string source_dat_file: source .dat
:param string output_epw_file: output EPW
"""
import pandas as pd

print(f"Parsing .dat file: {source_dat_file}")
try:
weather_data = pd.read_csv(source_dat_file, sep="\s+", skiprows=6, header=None)
weather_data.columns = ['RW', 'HW', 'MM', 'DD', 'HH', 't', 'p', 'WR', 'WG', 'N', 'x', 'RF', 'B', 'D', 'A', 'E',
'IL']
Comment on lines +88 to +90
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on the sample files that you provided, you could do the following instead.

Suggested change
weather_data = pd.read_csv(source_dat_file, sep="\s+", skiprows=6, header=None)
weather_data.columns = ['RW', 'HW', 'MM', 'DD', 'HH', 't', 'p', 'WR', 'WG', 'N', 'x', 'RF', 'B', 'D', 'A', 'E',
'IL']
weather_data = pd.read_csv(source_dat_file, sep="\s+", skiprows=32, header=0).drop(0)

This would skip all the comment header lines and read the provided column names from the file.

We could use the list of column names to compare with the dataframe to make sure that whatever that is read is correct.

e.g.

dat_columns = ['RW', 'HW', 'MM', 'DD', 'HH', 't', 'p', 'WR', 'WG', 'N', 'x', 'RF', 'B', 'D', 'A', 'E', 'IL']
if not set(dat_columns) == set(weather_data.columns):
    raise ValueError("Invalid .dat file")


# from dat to epw
with open(output_epw_file, 'w') as epw_file:
epw_file.write("LOCATION,Weimar,-,DEU,SRC-TMYx,105550,50.983,11.317,1,268\n")
Copy link

@coderabbitai coderabbitai bot Dec 5, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Replace hardcoded location with configurable parameters

The location information is hardcoded for Weimar, Germany. This limits the function's reusability for other locations.

Consider adding location parameters to the function signature:

-def parse_dat_weather_file(source_dat_file, output_epw_file):
+def parse_dat_weather_file(source_dat_file, output_epw_file, location_name="Weimar", country="DEU", 
+                          latitude=50.983, longitude=11.317, elevation=268):

Then update the location line:

-            epw_file.write("LOCATION,Weimar,-,DEU,SRC-TMYx,105550,50.983,11.317,1,268\n")
+            epw_file.write(f"LOCATION,{location_name},-,{country},SRC-TMYx,105550,{latitude},{longitude},1,{elevation}\n")

Committable suggestion skipped: line range outside the PR's diff.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You would need to remove these hardcoded values and use the correct values that are included in the comment header of the .dat file (the first 5 lines of the .dat file if I am not wrong). This is important for DAYSIM to know where to position the scene during raytracing.

This comment was marked as off-topic.

Sign in to view

epw_file.write("DESIGN CONDITIONS,0\n")
epw_file.write("TYPICAL/EXTREME PERIODS,0\n")
epw_file.write("GROUND TEMPERATURES,0\n")
epw_file.write("HOLIDAYS/DAYLIGHT SAVINGS,No,0,0,0\n")
epw_file.write("COMMENTS 1, converted from .dat to .epw by CEA\n")
epw_file.write("COMMENTS 2\n")
epw_file.write("DATA PERIODS,1,1,Data,Tuesday,1/1,12/31\n")

for _, row in weather_data.iterrows():
mm = int(row['MM'])
dd = int(row['DD'])
hh = int(row['HH'])
epw_line = f"1958,{mm},{dd},{hh},60,B8E7B8B8?9?0?0?0?0?0?0B8B8B8B8?0?0F8F8A7E7, " \
f"{row['t']},99,99,{row['RF']}, {row['p']}, 9999, {row['A']}, {row['D']}, {row['B']}, {row['D']}," \
f"-9999, -9999, -9999, -9999,{row['WR']}, {row['WG']}, 99, 9999, 9999, 9999,0, 999999999,0," \
f" -999,-999,-999\n"
epw_file.write(epw_line)
reyery marked this conversation as resolved.
Show resolved Hide resolved
+ except pd.errors.EmptyDataError as e:
+ raise ValueError("The .dat file is empty or has incorrect format") from e
+ except pd.errors.ParserError as e:
+ raise ValueError("Failed to parse the .dat file - invalid format") from e
+ except Exception as e:
+ raise ValueError(f"Unexpected error while parsing .dat file: {str(e)}") from e
Comment on lines +112 to +117
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are additional characters at the start of these lines that have to be removed.


def main(config):
"""
Expand All @@ -85,7 +126,13 @@ def main(config):
locator = cea.inputlocator.InputLocator(config.scenario)
weather = config.weather_helper.weather

if config.weather_helper.weather == 'climate.onebuilding.org':
if weather.endswith('.dat'):
print(f"Detected .dat file: {weather}")
# convert .dat in .epw
dat_weather_file = weather
epw_weather_file = locator.get_weather_file()
parse_dat_weather_file(dat_weather_file, epw_weather_file)
elif config.weather_helper.weather == 'climate.onebuilding.org':
Comment on lines +129 to +135
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Add file validation and error handling

The .dat file handling needs additional validation and cleanup.

     if weather.endswith('.dat'):
+        if not os.path.exists(weather):
+            raise ValueError(f"Weather file not found: {weather}")
         print(f"Detected .dat file: {weather}")
-        # convert .dat in .epw
         dat_weather_file = weather
         epw_weather_file = locator.get_weather_file()
+        try:
             parse_dat_weather_file(dat_weather_file, epw_weather_file)
+        except Exception as e:
+            # Clean up any partially written EPW file
+            if os.path.exists(epw_weather_file):
+                os.remove(epw_weather_file)
+            raise
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
if weather.endswith('.dat'):
print(f"Detected .dat file: {weather}")
# convert .dat in .epw
dat_weather_file = weather
epw_weather_file = locator.get_weather_file()
parse_dat_weather_file(dat_weather_file, epw_weather_file)
elif config.weather_helper.weather == 'climate.onebuilding.org':
if weather.endswith('.dat'):
if not os.path.exists(weather):
raise ValueError(f"Weather file not found: {weather}")
print(f"Detected .dat file: {weather}")
dat_weather_file = weather
epw_weather_file = locator.get_weather_file()
try:
parse_dat_weather_file(dat_weather_file, epw_weather_file)
except Exception as e:
# Clean up any partially written EPW file
if os.path.exists(epw_weather_file):
os.remove(epw_weather_file)
raise
elif config.weather_helper.weather == 'climate.onebuilding.org':

print("No weather provided, fetching from online sources.")
fetch_weather_data(locator.get_weather_file(), locator.get_zone_geometry())
else:
Expand Down
2 changes: 1 addition & 1 deletion cea/default.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,7 @@ trees.help = Path to shapefile containing the geometries of tree canopies in pol
[weather-helper]
weather =
weather.type = WeatherPathParameter
weather.help = Either a full path to a weather file (.epw) or the name of one of the weather files contained within the CEA. Leave blank to fetch from third-party sources.
weather.help = Either a full path to a weather file (.epw or .dat) or the name of one of the weather files contained within the CEA. Leave blank to fetch from third-party sources.

[radiation]
buildings =
Expand Down
Loading