FENDL-3.2b Retrofitting #42

eitan-weinstein · 2024-06-14T16:38:45Z

This commit includes a user-oriented set of Python scripts that allow for a given input of FENDL 3.2B formatted GENDF or ENDF/PENDF pairs to extract out PKZA, DKZA, emitted particles, and cross-sections data from the new files. The scripts allow for direct downloads of GENDF files from the FENDL database (https://www-nds.iaea.org/fendl/data/neutron/index.html), but this does not include data on nuclear isomers. Therefore, if the user inputs an isomer (i.e. Ti-48m), then it will instead download from the TENDL 2017 database (https://tendl.web.psi.ch/tendl_2017/tendl2017.html), on which FENDL 3.2B is based anyways. Given that TENDL 2017 does not have GENDF file formats and that ALARA requires group-wise, as opposed to point-wise data, data taken from TENDL 2017 will be processed using the NJOY GROUPR module (https://github.com/njoy/NJOY2016) and converted into group-wise data.

The output of the main executable Python script (fendl3_gendf.py) is a CSV file containing the aforementioned extracted data and saves it in the current directory.

There are areas that I know will still need modification, either within these codes or externally, including:

Changing the format from a user CLI input to one consistent with ALARA input files such that these programs can be automatically called given an ALARA input file that contains FENDL 3.2B formatted data.
Another processing script or a modification to an existing C++ library in ../DataLib that can take the data from the gendf_data.csv output and make it readable by ALARA.

gonuke

This looks like a fabulous start @eitan-weinstein, and the overall style is more modular than my first impressions once I understood your design strategy - and a lot cleaner than the first code you wrote a year ago :)

I'm going to publish this first comment now and then dig in to the details a little more in a subsequent review.

Building the comment about logger - print statements are reasonable when debugging & developing something, but by the time we get to PR, its best to either remove them or convert them to something that uses logger primarily to make it easy to hide them.

We also try to avoid any interactive code an rely instead on command-line arguments, input files, and/or calling methods from custom user python code.

gonuke · 2024-06-14T20:45:52Z

src/DataLib/fendl32B_retrofit/fendl3_gendf.py

Separation of concerns - make this more modular with each function performing a specific task.

argparse for command-line input/options

read JSON or YAML for more complicated input

separate methods for reading data, processing data, outputting data

logging for user output

gonuke

Here are a variety of suggestions to cleanup and streamline things.

gonuke · 2024-06-14T22:16:47Z

src/DataLib/fendl32B_retrofit/gendf_tools.py

+
+# Download the GENDF file
+def gendf_download(element, A, M=None, save_path=None):
+    Z = str(elements.index(element) + 1).zfill(2)


rather than searching with index you could process this into something like:

elements = dict(zip(elements, range(1,len(elements)+1)))

and then you can just look up elements[element]

Also, since you use this more than once, you could define it once and import it in each place.

gonuke · 2024-06-14T22:18:51Z

src/DataLib/fendl32B_retrofit/gendf_tools.py

+
+    print(f"Final download URL: {download_url}")
+
+    subprocess.run(['wget', download_url, '-O', save_path])


can't you use requests to do the download rather than wget?

gonuke · 2024-06-14T22:21:44Z

src/DataLib/fendl32B_retrofit/gendf_tools.py

+def gendf_download(element, A, M=None, save_path=None):
+    Z = str(elements.index(element) + 1).zfill(2)
+    A = str(A).zfill(3)
+    gendf_gen_url = 'https://www-nds.iaea.org/fendl/data/neutron/group/'


Ah! These GENDF files are NOT activation files. They are transport files and not reliable for our activation purposes.

I'll hold off reviewing the GENDF stuff until we are sure it's useful

gonuke · 2024-06-14T22:23:33Z

src/DataLib/fendl32B_retrofit/gendf_tools.py

+# Suppress unnecessary warnings from ENDFtk
+@contextlib.contextmanager
+def suppress_output():


I'f definitely curious to see which output you are suppressing here...

gonuke · 2024-06-14T22:28:17Z

src/DataLib/fendl32B_retrofit/GROUPR/groupr_tools.py

+    if 'm' in A:
+        m_index = A.find('m')
+        A = A[:m_index].zfill(3) + 'm'
+    else:
+        A = A.zfill(3)


Isn't the m always last? If so, then I think this will do it:

Suggested change

if 'm' in A:

m_index = A.find('m')

A = A[:m_index].zfill(3) + 'm'

else:

A = A.zfill(3)

A = A.zfill(3 + ('m' in A))

gonuke · 2024-06-14T22:48:10Z

src/DataLib/fendl32B_retrofit/GROUPR/groupr_tools.py

+    # Write the input deck to the groupr.inp file
+    with open('groupr.inp', 'w') as f:
+        f.write('groupr\n')
+        for card_name, card_content in zip(deck_names, deck):
+            f.write(format_card(card_name, card_content, MTs))
+        f.write(' 0/\nstop')


separation of concerns: make a separate function to write the output from the one that generates the data

gonuke · 2024-06-14T22:53:05Z

src/DataLib/fendl32B_retrofit/GROUPR/groupr.py

+matb, MTs = grpt.endf_specs(endf_path)
+
+# Write out the GROUPR input file
+mt_table = pd.read_csv('./mt_table.csv')


can you read this into a dictionary instead of a dataframe?

see: https://docs.python.org/3/library/csv.html#csv.DictReader

gonuke · 2024-06-14T22:56:30Z

src/DataLib/fendl32B_retrofit/GROUPR/groupr_tools.py

+    card9 = []
+    for mt in MTs:
+        mtname = mt_table[mt_table['MT'] == mt]['Reaction'].values[0] # description of section to be processed
+        card9_line = f'{mfd} {mt} "{mtname}"'
+        card9.append(card9_line)


if mt_table is a dictionary, this could be as simple as:

card9 = [f'{mfd} {mt} "{mt_table[mt]}"' for mt in MTs]

gonuke · 2024-06-14T22:59:04Z

src/DataLib/fendl32B_retrofit/GROUPR/groupr_tools.py

+        for line in card_content:
+            card_str += f' {line}/\n'


Suggested change

for line in card_content:

card_str += f' {line}/\n'

card_str = ' ' + '/\n '.join(card_content) + '/\n'

gonuke · 2024-06-14T23:03:49Z

src/DataLib/fendl32B_retrofit/GROUPR/run_njoy_template.sh

Definitely want to replace this with a python script.

…ard formatting function

gonuke

More pythonic use of dictionary

src/DataLib/fendl32B_retrofit/GROUPR/groupr_tools.py

Co-authored-by: Paul Wilson <[email protected]>

…toolkits

…d fendl3_gendf.py

…t to requests

…tain argparse messages in terminal

…vidually

…g over multiple isotopes.

eitan-weinstein · 2024-07-09T21:08:31Z

src/DataLib/fendl32B_retrofit/fendl3_retrofit.py

+    # Set conditionals for local file input 
+    if args.method == 'I':
+        gendf_data = handle_local_file_input(args, mt_dict)
+        cumulative_df = pd.concat([cumulative_df, gendf_data], ignore_index=True)


Execute pd.concat outside of the if-else block

eitan-weinstein · 2024-07-09T21:10:12Z

src/DataLib/fendl32B_retrofit/fendl3_retrofit.py

+    # Extract fundamental data from the GENDF file
+    pKZA = tpp.extract_gendf_pkza(gendf_path, M = M)
+    matb, MTs, file_obj = tpp.extract_endf_specs(gendf_path, 'gendf')
+
+    logger.info(f"GENDF file path: {gendf_path}")
+    logger.info(f"Parent KZA (pKZA): {pKZA}")
+    logger.info(f'MTs: {MTs}')


Have this all be its own function that can be called by both

eitan-weinstein · 2024-07-09T21:17:26Z

src/DataLib/fendl32B_retrofit/fendl3_retrofit.py

+        'Cross Sections'    : []
+    })
+
+def handle_local_file_input(args, mt_dict):


Get this function to go only up to the point that it can be treated the same way as a local file.

eitan-weinstein · 2024-07-09T21:23:17Z

src/DataLib/fendl32B_retrofit/activation_analysis.py

+
+    # Extract data for each MT
+    for MT in MTs:
+        try:


Instead of using a try block, check beforehand to make sure that the MT is in the mt_dict and only proceed if True

eitan-weinstein · 2024-07-09T21:46:02Z

src/DataLib/fendl32B_retrofit/tendl_preprocessing.py

+    return pKZA
+
+@contextlib.contextmanager
+def redirect_ENDFtk_output():


See how things work in avoiding using this function and letting the ENDFtk messages come out organically.

gonuke

Here are some changes we discussed today.

gonuke · 2024-07-09T21:05:16Z

src/DataLib/fendl32B_retrofit/logging_config.py

+logger = logging.getLogger(__name__)
+
+# Redirect stdout and stderr to the logger
+class LoggerWriter:


Probably don't need this - can just access the default logger you created above

gonuke · 2024-07-09T21:05:46Z

src/DataLib/fendl32B_retrofit/logging_config.py

+sys.stdout = LoggerWriter(logger.info)
+sys.stderr = LoggerWriter(logger.error)


I don't think you need to reassign these, and if you don't you won't need so many gymnastics in the arg parsing

gonuke · 2024-07-09T21:23:35Z

src/DataLib/fendl32B_retrofit/fendl3_retrofit.py

+            pKZA = tpp.extract_gendf_pkza(gendf_path, M = M)
+
+            # Recalibrate MT list after GENDF conversion
+            matb, MTs, file_obj = tpp.extract_endf_specs(gendf_path, 'gendf')


Check for any MTs that are not in mt_dict

if {MTs} - {mt_dict} not None:

gonuke · 2024-07-09T21:41:53Z

src/DataLib/fendl32B_retrofit/tendl_preprocessing.py

+    A = A.split(' ')[0]
+    if 'm' in A:
+        m_index = A.find('m')
+        A = A[:m_index]
+    M = str(str(M).count('M')) or '0'
+    pKZA = Z.zfill(2) + A.zfill(3) + M


Suggested change

A = A.split(' ')[0]

if 'm' in A:

m_index = A.find('m')

A = A[:m_index]

M = str(str(M).count('M')) or '0'

pKZA = Z.zfill(2) + A.zfill(3) + M

Z = int(Z)

M = 1 if 'm' in A.lower() else 0

A = int(A.strip().lower().split(''m'))

pKZA = (Z * 1000 + A) * 10 + M

gonuke · 2024-07-09T21:42:16Z

src/DataLib/fendl32B_retrofit/tendl_preprocessing.py

+            Defaults to None and will be otherwise defined internally.
+
+    Returns:
+        pKZA (str): Parent KZA identifier.


If changing below

Suggested change

pKZA (str): Parent KZA identifier.

pKZA (int): Parent KZA identifier.

gonuke · 2024-07-09T22:01:30Z

src/DataLib/fendl32B_retrofit/tendl_preprocessing.py

+
+    try:
+        section = file.section(MT).content
+        lines = section.split('\n')[2:-2:2]


Suggested change

lines = section.split('\n')[2:-2:2]

# only every 2nd line starting at the 3rd line has cross-section data.

lines = section.split('\n')[2:-2:2]

gonuke · 2024-07-09T22:07:53Z

src/DataLib/fendl32B_retrofit/activation_analysis.py

+                particle
+            )
+            for particle in particle_types
+            if count_emitted_particles(particle, emitted_particles)


Suggested change

if count_emitted_particles(particle, emitted_particles)

if particle in emitted_particles)

gonuke · 2024-07-09T22:15:19Z

src/DataLib/fendl32B_retrofit/activation_analysis.py

+    """
+
+    # Neutron capture
+    nucleus_neutrons = A - nucleus_protons + 1


#. emitted delta N delta P delta_NP = { 'n' : np.array([-1, 0]), # neutron emission, N = 1, P = 0 'p' : np.array([0,-1]), 't' : np.array([-2, -1]), .... } N_P = np.array([A - Z + 1, Z]) for particle, num_particles in emission_tuples.items(): # if changed to dictionary N_P += num_particle * change_N_P[particle]

gonuke · 2024-07-09T22:19:00Z

src/DataLib/fendl32B_retrofit/activation_analysis.py

+        emission_tuples = [
+            (
+                count_emitted_particles(particle, emitted_particles),
+                particle
+            )
+            for particle in particle_types
+            if count_emitted_particles(particle, emitted_particles)
+        ]


Suggested change

emission_tuples = [

(

count_emitted_particles(particle, emitted_particles),

particle

)

for particle in particle_types

if count_emitted_particles(particle, emitted_particles)

]

emission_tuples = {

particle : count_emitted_particles(particle, emitted_particles)

for particle in particle_types

if particle in emitted particles

}

And build this as soon as you load the MT CSV data

gonuke · 2024-07-09T22:24:10Z

src/DataLib/fendl32B_retrofit/tendl_preprocessing.py

+    with open(csv_path, 'r') as f:
+        csv_reader = csv.DictReader(f)
+        for row in csv_reader:
+            mt_dict[row['MT']] = row['Reaction']


process all the emitted particles here for once and for all, and store in a data structure that is something like:

you could actually store a dictionary:
mt_data[MT] = { 'delKZA' : (change_P * 1000 + change_P + change_N) * 10 + change_M, 'emitted_particles' : emitted_particles }

…view.

gonuke

A few more comments here

gonuke · 2024-07-16T02:13:48Z

src/DataLib/fendl32B_retrofit/logging_config.py

This may be able to just be moved into your main program now....

gonuke · 2024-07-16T02:17:33Z

src/DataLib/fendl32B_retrofit/reaction_data.py

+            read out as {'n': 1, 'p': 1}.
+    """
+
+    particle_types = ['n', 'd', 'α', 'p', 't', '3He', 'gamma']


Probably makes sense to somehow generate this from the keys of NP_dict below?

gonuke · 2024-07-16T02:18:08Z

src/DataLib/fendl32B_retrofit/reaction_data.py

+
+    #             emitted         delta N       delta P
+    NP_change =            array([ 1      ,      0      ])  # neutron activation
+    NP_dict   = {'n'     : array([-1      ,      0      ]), # neutron emission


What if emission_dict has a gamma in it?

gonuke · 2024-07-16T02:20:13Z

src/DataLib/fendl32B_retrofit/reaction_data.py

+        number_str = int(number_str)
+    elif particle in emitted_particle_string:
+        number_str = 1
+    else:
+        number_str = None
+
+    return number_str


This isn't a string any more so why call it number_str?

gonuke · 2024-07-16T02:24:48Z

src/DataLib/fendl32B_retrofit/groupr_tools.py

+        None
+    """
+
+    # In ENDF-6 formatted files, there are 66 lines of whitespace before


Suggested change

# In ENDF-6 formatted files, there are 66 lines of whitespace before

# In ENDF-6 formatted files, there are 66 characters/columns of whitespace before

gonuke · 2024-07-16T02:28:06Z

src/DataLib/fendl32B_retrofit/groupr_tools.py

+    if not os.path.exists(save_directory):
+        os.makedirs(save_directory)


Use pathlib instead of os

…ular.

…tings for the logger.

…d from DataLib directory.

First commit for FENDL3.2B retrofitting

dbef6ca

eitan-weinstein marked this pull request as ready for review June 14, 2024 16:39

eitan-weinstein changed the title ~~First commit for FENDL3.2B retrofitting~~ FENDL 3.2B Retrofitting Jun 14, 2024

gonuke requested changes Jun 14, 2024

View reviewed changes

gonuke requested changes Jun 16, 2024

View reviewed changes

Eitan Weinstein and others added 9 commits June 17, 2024 11:56

Replacing NJOY Bash script with subprocess execution in Python

cdd7bcd

Simplifying mass number formatting in tendl_download() function

f3d010f

Simplifying endf_specs() function

6c5ac24

Remove now-obsolete ENDFtk warning suppression

03e3af3

Simplify tendl_download() function using data structures

fa4f29e

Switching tendl_download() function over to urllib dependence

0190096

Moving card deck formatting from Pandas DataFrame to dictionary

413ae46

Separating out a write function for the GROUPR input from the input c…

2eb9ffd

…ard formatting function

Removing now-obsolete Pandas dependence

1247db3

gonuke requested changes Jun 17, 2024

View reviewed changes

src/DataLib/fendl32B_retrofit/GROUPR/groupr_tools.py Outdated Show resolved Hide resolved

eitan-weinstein and others added 12 commits June 18, 2024 09:53

Simplifying card writing for groupr_input_file_writer()

1d35b79

Co-authored-by: Paul Wilson <[email protected]>

Fixing indexing on groupr_input_file_writer()

de3cbb4

Storing elements in a single dictionary to be referenced across both …

d20eed8

…toolkits

Removing now-obsolete ENDFtk warning supression from gend_tools.py an…

58a4ede

…d fendl3_gendf.py

Updating gendf_download() function -- notably switching away from wge…

b83be55

…t to requests

Switching CSV reading from Pandas DataFrame to dictionary

e135311

Moving away from direct input to argparse input/options

29528dd

Expanding argparse usage

0abb51b

Moving away from print statements towards logging

582424b

Removed unnecessary file from file cleanup list

77e9a65

Expanding logger to capture 'No bottleneck testing available' message

493a35c

Improving readability of NJOY run message for logger

a29bd66

eitan-weinstein changed the title ~~FENDL 3.2B Retrofitting~~ FENDL-3.2b Retrofitting Jun 19, 2024

Eitan Weinstein added 2 commits June 21, 2024 10:15

Updating the logging to redirect ENDFtk messages to the logger and re…

69fe5f0

…tain argparse messages in terminal

Removing stand-alone groupr script -- unnecessary and not called indi…

d0f7d3b

…vidually

Eitan Weinstein and others added 6 commits July 1, 2024 17:54

Improving readability for nuclear_decay()

121e57a

Increasing readability of argparser

fb1b796

Major overhaul of modularity and including functionality for iteratin…

c8e6cea

…g over multiple isotopes.

Removing time package.

4d99f41

Removing specific example file from GENDF files.

e0529dc

Making the file saving more versatile.

cc064b6

eitan-weinstein commented Jul 9, 2024

View reviewed changes

gonuke requested changes Jul 9, 2024

View reviewed changes

Eitan Weinstein added 6 commits July 11, 2024 16:54

Responding to a majority of the high-level comments from Tuesday's re…

14c5730

…view.

Fixing docstring for ensure_gendf_markers() function.

95815b2

Improving isotope identification methods.

a5997b5

Improving isotope identification methods.

f83a646

Simplifying logging method and usage.

98f23c3

One more logging fix.

498c824

gonuke requested changes Jul 16, 2024

View reviewed changes

Eitan Weinstein added 5 commits July 16, 2024 11:34

Completing response to last review and making arg processing more mod…

a807f1e

…ular.

Improving ability to iterate over all elements.

76b9aa1

Fixing minor bug in execution of handle_TENDL_downloads().

eebeea3

Small formatting change to fit in max line length.

912530f

More minor formatting adjustments and simplifying the line length set…

c374494

…tings for the logger.

eitan-weinstein requested a review from gonuke July 16, 2024 19:28

Eitan Weinstein added 3 commits July 16, 2024 16:53

Allowing for fendle_retrofit.py to be executed from DataLib.

f50b617

Removing unnecessary print statement.

4ba725e

Ensuring that NJOY output is properly handled when program is execute…

6257033

…d from DataLib directory.

eitan-weinstein marked this pull request as draft July 18, 2024 14:34

Small formatting changes before moving over to project individual PRs.

4921dba


		print(f"Final download URL: {download_url}")

		subprocess.run(['wget', download_url, '-O', save_path])

	for line in card_content:
	card_str += f' {line}/\n'
	card_str = ' ' + '/\n '.join(card_content) + '/\n'

		sys.stdout = LoggerWriter(logger.info)
		sys.stderr = LoggerWriter(logger.error)

	pKZA (str): Parent KZA identifier.
	pKZA (int): Parent KZA identifier.

	lines = section.split('\n')[2:-2:2]
	# only every 2nd line starting at the 3rd line has cross-section data.
	lines = section.split('\n')[2:-2:2]

	if count_emitted_particles(particle, emitted_particles)
	if particle in emitted_particles)

	# In ENDF-6 formatted files, there are 66 lines of whitespace before
	# In ENDF-6 formatted files, there are 66 characters/columns of whitespace before

		if not os.path.exists(save_directory):
		os.makedirs(save_directory)

FENDL-3.2b Retrofitting #42

Are you sure you want to change the base?

FENDL-3.2b Retrofitting #42

Conversation

eitan-weinstein commented Jun 14, 2024

gonuke left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gonuke left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gonuke left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gonuke left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gonuke left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment