Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test 4.0.0 #54

Merged
merged 201 commits into from
Apr 1, 2021
Merged
Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
201 commits
Select commit Hold shift + click to select a range
c10faa7
Update project table
steven-murray Jan 13, 2020
aafed0f
Fix project table script
steven-murray Jan 13, 2020
09bed8e
git-bot: Update table
actions-user Jan 14, 2020
8fb17fb
github token in home rather than repo
steven-murray Mar 20, 2020
5d563f3
Merge branch 'project-table' of github.com:HERA-Team/hera-validation …
steven-murray Mar 20, 2020
1463609
Add step 4 structure and add_noise function
steven-murray Mar 20, 2020
84c6b2e
Add notebook scaffolding
steven-murray Mar 20, 2020
0446222
Add a plot_array function to the test-4.0.0 notebook. Quite a bit of …
r-pascua Mar 21, 2020
3d66d9f
One big commit (oops). Updated add_noise function to automatically fi…
r-pascua Mar 27, 2020
1fbca11
Cleaned up most of the mess that was left in the notebook from experi…
r-pascua Mar 27, 2020
8e49032
Add error check in _parse_filename_for_cmp to handle when the compone…
r-pascua Mar 27, 2020
28205de
Write code for choosing which antennas to keep for simulation prepara…
r-pascua Apr 1, 2020
9aab123
Remove code from test notebook that was used to try to figure out how…
r-pascua Apr 1, 2020
86e4970
Merge branch 'test-4.0.0' of https://github.com/hera-team/hera-valida…
Apr 1, 2020
ec77052
Some formatting changes and comments added. Placeholder functions for…
Apr 1, 2020
42d7dd2
Put final touches on tools for making simulation data match observed …
r-pascua Apr 3, 2020
ff0962a
Merge remote changes and fix conflicts.
r-pascua Apr 3, 2020
fe74129
Update comments in sim_prep.add_noise and finish making most of the u…
r-pascua Apr 4, 2020
7fd2cee
Write function for adding bandpass gains to a simulation.
r-pascua Apr 4, 2020
6e3deb7
Write function for simulating cable reflections.
r-pascua Apr 4, 2020
33e4637
Edit some docstrings, start work on xtalk.
r-pascua Apr 4, 2020
7cb3c2d
Move apply_gains function and finish writing add_xtalk function. Now …
r-pascua Apr 6, 2020
5629ed1
Remove options for performing steps in-place or not and update docstr…
r-pascua Apr 6, 2020
0012244
Update add_noise function to use the beam model appropriate for the R…
r-pascua Apr 7, 2020
5263296
Make data directory in tools module; just for storing beam polynomial…
r-pascua Apr 7, 2020
c57861a
Update a line to use np.argmax instead of what it originally did (it'…
r-pascua Apr 7, 2020
85e47f8
Update file chunking routine. Update lst rephasing function. Make add…
r-pascua Apr 8, 2020
10bd106
Add logic for checking if sim files already exist before doing any ca…
r-pascua Apr 8, 2020
03a24f9
Add a message to be issued if chunked sim files exist and are not exp…
r-pascua Apr 8, 2020
7b8808d
Update notebook to show: array selection process; chunked simulation …
r-pascua Apr 8, 2020
9650a17
Remove .pyc files and update gitignore to not track these files. Let'…
r-pascua Apr 8, 2020
0fc872f
Add comment clarifying the choice of which set of blt indices to use …
r-pascua Apr 9, 2020
d4eb9c2
Simplify logic for choosing autocorrelation in add_noise function as …
r-pascua Apr 9, 2020
eab9b51
Update downselect_antennas function to also update the flag and nsamp…
r-pascua Apr 9, 2020
56dd087
Update chunk_sim_and_save to use capture groups for extracting the jd…
r-pascua Apr 9, 2020
6d27ee7
Simplify logic for choosing unique translations in _build_translation…
r-pascua Apr 9, 2020
0802bc3
Simplify calculation of number of overlapping antennas in _get_optima…
r-pascua Apr 9, 2020
5d7af90
Simplify arguments of calls to any() and all() wherever the argument …
r-pascua Apr 9, 2020
9cd793e
Update chunk_sim_and_save function to make the filenames end with '.t…
r-pascua Apr 9, 2020
7a04814
Update notebook and re-run.
r-pascua Apr 9, 2020
9e62d0f
Fix function for checking whether simulation files have already been …
r-pascua Apr 11, 2020
f2597ed
Update section detailing how the simulation files are prepared. There…
r-pascua Apr 11, 2020
4715e2d
Add support for seeding the RNG in add_xtalk (in case jitter is added…
r-pascua Apr 14, 2020
be37bef
Rename adjust_sim_to_data -> prepare_sim_files. Write apply_systemati…
r-pascua Apr 14, 2020
62be3af
Add support for making bandpass gains vary in time.
r-pascua Apr 14, 2020
6b0db8d
Add check in gain amplitude time variation step to ensure gains aren'…
r-pascua Apr 14, 2020
565a168
Bit of refactoring; made a function for varying gain amplitudes in ti…
r-pascua Apr 14, 2020
83117ad
Flesh out function for adding time variation to gains. Start writing …
r-pascua Apr 15, 2020
2d1f069
Larger commit than is responsible. Updated docstrings to remove unnec…
r-pascua Apr 16, 2020
2e0cf64
Add functionality for writing systematics simulation configuration to…
r-pascua Apr 16, 2020
3b9cc37
Write argparser for simulation preparation.
r-pascua Apr 16, 2020
9aba920
Import yaml in sim_prep module.
r-pascua Apr 16, 2020
8c51b78
Add option to argparser to skip writing the true visibilities to disk…
r-pascua Apr 16, 2020
b858cfd
Write script for running simulation file preparation from the command…
r-pascua Apr 16, 2020
408840c
Change relative import of sim_prep in prepare_sim_files to absolute i…
r-pascua Apr 18, 2020
cd8dae0
Bugfix
r-pascua Apr 18, 2020
7887bf2
Bugfix? Let's see if the problem is actually fixed.
r-pascua Apr 18, 2020
06da225
Forgot to import argparse before... now it gets imported.
r-pascua Apr 18, 2020
aab1728
Accidentally changed data import from absolute to relative, so I chan…
r-pascua Apr 18, 2020
b0e46bf
Hopefully fixed now?
r-pascua Apr 18, 2020
46c2934
Make prepare_sim_files function not track individual systematics to s…
r-pascua Apr 19, 2020
612ef4e
Add functionality for choosing whether to return the systematic being…
r-pascua Apr 19, 2020
59b3509
Fix a bad indent.
r-pascua Apr 19, 2020
d340c3b
Bugfix.
r-pascua Apr 20, 2020
aa50ffa
Add verbose option.
r-pascua Apr 20, 2020
ea13b30
Add functionality for performing simulation file preparation in chunk…
r-pascua Apr 22, 2020
49ee043
Copy data preparation batch script into git repo.
r-pascua Apr 23, 2020
9600d1e
Fix a line to update a change in script name in the batch script.
r-pascua Apr 23, 2020
a406e4f
Make script use verbose setting.
r-pascua Apr 23, 2020
07611af
Update prepare_sim_files script pre-processing to actually update the…
r-pascua Apr 23, 2020
4266cf8
Make reference UVData object downselect to times that had a match in …
r-pascua Apr 23, 2020
40843b4
Comment out something put in last commit that allowed us to get away …
r-pascua Apr 23, 2020
ca7b8f4
Make chunk_sim_and_save a little more flexible. Make argparser for ab…
r-pascua Apr 23, 2020
657e21d
Write script for preparing abscal model. *Does not contain delay filt…
r-pascua Apr 23, 2020
0cc1aa2
Remove unused imports from abscal model preparation script.
r-pascua Apr 23, 2020
44df06a
Rename prepare_abscal_model -> prepare_abscal_model_files.
r-pascua Apr 23, 2020
97c7271
Write batch script for preparing (pre-filtered) abscal model files.
r-pascua Apr 23, 2020
caf3903
Forgot to import UVData from pyuvdata in the abscal file prep script.…
r-pascua Apr 23, 2020
ec2c29d
Bugfix; there was a typo where I accidentally used the same variable …
r-pascua Apr 24, 2020
9e57421
Add verbose option to abscal file prep argparser/script.
r-pascua Apr 24, 2020
2766ca9
Make abscal model file preparation batch script make model files for …
r-pascua Apr 24, 2020
5233b21
Modify abscal file prep batch script to do a test for a small LST ran…
r-pascua Apr 25, 2020
fa141b3
Update chunk_sim_and_save function to format JD correctly in filename…
r-pascua Apr 25, 2020
a36e6d8
Update abscal model file prep batch script to do all LSTs, processed …
r-pascua Apr 25, 2020
f067e99
Update abscal preparation script to allow specifying number of chunks…
r-pascua Apr 26, 2020
7f5bacc
Missed a parenthesis. It's there now.
r-pascua Apr 26, 2020
23ff683
Another bugfix.
r-pascua Apr 26, 2020
e797ba9
Update some logic for running file preparation in chunks in preparati…
r-pascua Apr 27, 2020
2940507
Remove a stray 'done' in abscal model preparation batch script.
r-pascua Apr 27, 2020
43dba1f
Fix silent bug in chunk_sim_and_save.
r-pascua Apr 27, 2020
142ff1e
Update rephase_to_reference function.
r-pascua Apr 28, 2020
f670a38
Bugfix in prepare_sim_files function.
r-pascua Apr 28, 2020
bcf63b1
Fix add_noise to work with wrapped LSTs.
r-pascua Apr 28, 2020
0709311
Bugfix regarding return value for add_noise function.
r-pascua Apr 29, 2020
125a4dc
Some bugfixes and some code simplification.
r-pascua Apr 29, 2020
27f71ae
Simplify logic for extracting time variation parameters in prepare_si…
r-pascua Apr 29, 2020
7654c51
Make sure the baseline_array is updated in the antenna downselection …
r-pascua Apr 29, 2020
f6dfe6e
Fix a silent bug in add_xtalk (was assigning xtalk to data rather tha…
r-pascua Apr 30, 2020
f0e16d5
Nick added an argparser for the abscal smoothing script.
Apr 30, 2020
0992ead
Nick wrote a script for smoothing the abscal solutions.
Apr 30, 2020
c75d7e6
Merge branch 'test-4.0.0' of https://github.com/hera-team/hera-valida…
Apr 30, 2020
6c8ab54
Update downselect_antennas function to do a better job of updating si…
r-pascua Apr 30, 2020
aa6ac33
Aesthetic changes to prepare_data batch script.
r-pascua Apr 30, 2020
032487a
Add functionality for specifying LST cuts for file preparation.
r-pascua Apr 30, 2020
eabcb6f
Update default setting for lst_min and lst_max (was 1-11 before, now …
r-pascua Apr 30, 2020
37efb03
Import units from astropy in sim_prep module.
r-pascua Apr 30, 2020
ee77e45
Apply the LST cut to the obsfile glob rather than the loaded in UVDat…
r-pascua Apr 30, 2020
206804f
Forgot to remove lst_min and lst_max from call to adjust_sim_to_data …
r-pascua Apr 30, 2020
08b5ff3
Remove application of LST cuts from prepare_sim_files function and mo…
r-pascua May 1, 2020
594025a
Update _apply_lst_cut function to have a fast and slow implementation…
r-pascua May 1, 2020
34402b5
Add argparser option for specifying whether reference files uniformly…
r-pascua May 1, 2020
fe1bd51
Update prepare_sim_files script in accordance with last commit. Add a…
r-pascua May 1, 2020
9ecda90
Update data preparation batch script in accordance with previous comm…
r-pascua May 1, 2020
ce4175c
Fix logic in determining whether to apply an LST cut (and -> or).
r-pascua May 1, 2020
e0843b0
Slight modification to _apply_lst_cut function to ensure that *all* i…
r-pascua May 1, 2020
fce9e5c
Update configuration handling to ensure that noise realizations are u…
r-pascua May 1, 2020
d579c67
Bugfix.
r-pascua May 1, 2020
1675c2f
Import UVData in prepare_sim_files script.
r-pascua May 1, 2020
b656c56
Explicitly ensure that the time variation parameters determined in pr…
r-pascua May 1, 2020
958b276
Make the systematics application step of data preparation more verbose.
r-pascua May 2, 2020
bd46fb8
Change abscal model prep to only do LST 0-12, and process in chunks o…
r-pascua May 2, 2020
1c46c0f
Make tolerance for smoothing abscal model files smaller.
May 2, 2020
f510025
Merge branch 'test-4.0.0' of https://github.com/hera-team/hera-valida…
May 2, 2020
ccc6210
Change number of chunks in abscal file prep from 2 to 4.
r-pascua May 2, 2020
9dafd8f
Merge branch 'test-4.0.0' of https://github.com/hera-team/hera-valida…
r-pascua May 2, 2020
bad7e0c
Updated notebook, not super clean yet.
r-pascua May 4, 2020
b418d02
Make some verbose statements even more verbose.
r-pascua May 4, 2020
de10879
Add documentation to apply_systematics function in sim_prep. Change d…
r-pascua May 5, 2020
a72b502
Remove some redundant code and explicitly set in the call to in in…
r-pascua May 5, 2020
8a299e5
Change job name in abscal model file prep batch script. Regarding pre…
r-pascua May 5, 2020
ecc6af7
Slight modification to notebook; now shows true visibility plot, corr…
r-pascua May 5, 2020
785cf36
Slightly nicer writing
steven-murray May 6, 2020
184b3d6
Slightly nicer add noise function
steven-murray May 6, 2020
4f7e992
Extra verbosity
steven-murray May 6, 2020
39ee5a6
Fix slice reference
steven-murray May 6, 2020
a1c979d
Make add_noise have same outputs as other functions
steven-murray May 7, 2020
2e2cca1
More direct add_noise functio
steven-murray May 7, 2020
4ecbb3f
Fix ref to Simulator
steven-murray May 7, 2020
2a0e3f6
Fix last ref to sim.data.
steven-murray May 7, 2020
04ac14b
Fix typo in pol
steven-murray May 7, 2020
97679ce
Fix bug in indexing
steven-murray May 7, 2020
a372fb9
Fix import
steven-murray May 7, 2020
cc7a385
Fix indexing
steven-murray May 7, 2020
dfafdb5
Remove old verbose statements that were commented out.
r-pascua May 9, 2020
fe95353
Make sure that the abscal files are conjugated properly.
r-pascua May 9, 2020
d49da30
Make smooth_abscal_model pbs script update read permissions for corre…
r-pascua May 10, 2020
dd9e04d
Make sure that every file has the correct number of integrations (wit…
r-pascua May 11, 2020
32fd28d
Get the file chunking parameters right.
r-pascua May 11, 2020
8b7a0be
Bugfix in vary_gains_in_time.
r-pascua May 12, 2020
b9bf856
Make sure that autos remain real-valued after addition of noise. Also…
r-pascua May 12, 2020
8b00c17
Update to make sure abscal model files have noise added.
r-pascua May 12, 2020
2a3d956
Remove .pyc file from repo.
r-pascua May 12, 2020
3f13459
Change upper LST cut to 7 hours (to avoid problem with missing foregr…
r-pascua May 12, 2020
b4a323c
Add function for interpolating to reference (rather than rephasing). …
r-pascua May 14, 2020
a5528d0
Update rephase_to_reference to handle cases where the reference LST r…
r-pascua May 14, 2020
b74c912
Make adjust_sim_to_data use interpolate_to_reference rather than reph…
r-pascua May 14, 2020
e209d49
Fix interpolate_to_reference.
r-pascua May 14, 2020
3553c83
Bugfix in interpolate_to_reference.
r-pascua May 14, 2020
68cb063
Make sure that simulation Ntimes and Nfreqs parameters get updated in…
r-pascua May 14, 2020
c1bccb3
Fix abscal_model_argparser to accept Trx as an optional parameter. Al…
r-pascua May 14, 2020
34ab7d0
Bugfix in preparing abscal model files when adding noise.
r-pascua May 14, 2020
2ba2465
Modify prepare_data batch script to run foregrounds-only file prep fo…
r-pascua May 14, 2020
7c63a88
Update data preparation script to use summed visibility file rather t…
r-pascua May 16, 2020
7438b5f
Update job name.
r-pascua May 16, 2020
df277f6
Adjust data preparation script to just do JD 2458108.
r-pascua May 19, 2020
d0c7525
Removed some cells from the notebook that were no longer needed. Rera…
r-pascua May 20, 2020
a169bbd
Merge branch 'master' into test-4.0.0
steven-murray May 21, 2020
bf9db7e
Add captions to figures.
r-pascua May 21, 2020
1b54181
Merge branch 'test-4.0.0' of https://github.com/hera-team/hera-valida…
r-pascua May 21, 2020
1f02380
Update some of the text in the notebook to reflect changes to the fil…
r-pascua May 21, 2020
f351c49
Update prepare_sim_files script to properly handle passing no systema…
r-pascua Jun 2, 2020
246f031
Update data preparation batch script to chunk EoR-only files for half…
r-pascua Jun 2, 2020
1b2f3df
Update data preparation to run an hour of LST for each day for only t…
r-pascua Jun 4, 2020
77252be
Use 1d interpolation rather than bivariate.
r-pascua Jun 4, 2020
3aea62f
Adjust data preparation script for re-running file preparation.
r-pascua Jun 4, 2020
d7b3767
Actually import interp1d from scipy.interpolate
r-pascua Jun 4, 2020
d724957
Bugfix in interpolate_to_reference
r-pascua Jun 5, 2020
f7a1ffb
Change where to write output for file prep
r-pascua Jun 5, 2020
be8f7f8
Update downselect_antennas to update antenna positions appropriately …
r-pascua Jun 9, 2020
9c152f5
Update data preparation script to make one day of chunked EoR files.
r-pascua Jun 25, 2020
76971c5
Update analytic power spectrum calculation to match EoR rescaling use…
r-pascua Jun 25, 2020
22ba843
Add support for reflection spectrum.
r-pascua Jul 19, 2020
46aba05
Support varying reflections in time.
r-pascua Jul 19, 2020
e07bd97
Add support for multi-reflections.
r-pascua Jul 19, 2020
1d17bcd
Import itertools.
r-pascua Jul 19, 2020
cb6c69f
Update preparation batch script.
r-pascua Jul 19, 2020
4d0567f
Update time vary params only if config given.
r-pascua Jul 19, 2020
b1255e8
Update output log location.
r-pascua Jul 19, 2020
a110558
Make sure to run data prep with a config.
r-pascua Jul 19, 2020
4512422
Make reflection_spectrum discoverable in pipeline.
r-pascua Jul 19, 2020
507f15f
Fix reflection shoulder code.
r-pascua Jul 19, 2020
32a05d6
Allow more flexibility for multi-reflections.
r-pascua Jul 19, 2020
1b7969e
Bugfix in reflection simulation.
r-pascua Jul 19, 2020
cb5ba6f
Improve prepare_data batch script.
r-pascua Jul 19, 2020
ed5d887
Update batch script to remake sum files.
r-pascua Jul 30, 2020
4fbe9ee
Add more plots to end-to-end notebook.
r-pascua Aug 6, 2020
7dfe927
Update prepare_data for foreground run.
r-pascua Aug 14, 2020
9af0e76
Subtle bugfix in prepare_sim_files script.
r-pascua Sep 30, 2020
1f59484
Get the bugfix right.
r-pascua Sep 30, 2020
057f91a
Update systemtaics sim notebook
r-pascua Mar 26, 2021
4938b03
add test criteria for 4.0.0a
r-pascua Apr 1, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .personal-github-token

This file was deleted.

Binary file added __pycache__/template_tools.cpython-37.pyc
Binary file not shown.
18 changes: 12 additions & 6 deletions make_project_table.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,18 +6,24 @@
from github import Github
import re
import os
import sys

table_header = """Status | # | Simulator(s) | Sim. Components | Analysis Components | Assigned |
-----------| -----|--------------|-----------------|---------------------|----------|
"""

if __name__=="__main__":
if not os.path.exists('.personal-github-token'):
raise ValueError("You do not yet have a personal access token for github installed. Create one at https://github.com/settings/tokens "
"and paste it into the file .personal-github-token (notice leading .) in this directory.")

with open(".personal-github-token") as fl:
GH_TOKEN = fl.read()
if len(sys.argv) > 1:
GH_TOKEN = sys.argv[-1]
else:
if not os.path.exists(os.path.expanduser('~/.validation-github-token')):
raise ValueError(
"You do not yet have a personal access token for github installed. Create one at "
"https://github.com/settings/tokens "
"and paste it into the file ~/.validation-github-token.")

with open(os.path.expanduser("~/.validation-github-token")) as fl:
GH_TOKEN = fl.read()

g = Github(GH_TOKEN)
repo = g.get_repo("HERA-Team/hera-validation")
Expand Down
82 changes: 0 additions & 82 deletions project_table.md

This file was deleted.

Loading