List of changes and TODO tasks

Warning

The changelog is dinamically linked to the standalone markdown files. A documentation rebuild will capture changes in any of these files.

CLEDB UPDATES and TODO list

Github update history

Commit Tag	Date	Description
initial	20210802	Self explanatory.
update-slurm	20210828	- Implemented initial slurm-enabled and batch versions for both jupyter and database build scripts. - Updated and extended documentation.
update-numbaproc	20210906	- Implemented a parallel enable/disable keyword for numba enhanced functions. - Functions that don’t benefit from parallel splitting are hard-coded to parallel=False.
update-tidbits	20210908	- Small updates for optional plotting; save/load datacubes, etc. - Implemented slurm tidbits; standard scripts can now be run inside interactive sessions on RC systems. - Revised how calculations are computed in the CLEDB_BLOS module. - Updated constants and ctrlparams classes.
update-CLEDB_BUILD	20210915	- Updated database building scripts. - Fully implemented the batch run version with $scratch partition use. - Updated documentation accordingly.
update-databugfixes	20220816	- Bug fixes update after running comp inversions. - Fixed bugs in obs_qurotate (PREPINV module) and cledb_quderotate (PROC module). - Fixed the cledb_obsderotate (PROC module) function. - Added angle correction (azimuth derotate) to bphi calculation in cledb_phys (PROC module). - cledb_getsubset routine (PROC module) is optimized and re-written. - Introduced the “integrated” control parameter. - Updated the conda distribuition and python 3.10, numpy, numba, etc. - Bugfix update to CLE 2.0.3 executables in BUILD module.
update-readthedocs	20230205	- Transfered the code documentation from a latex based distribution to a Sphinx webpage format. - The level of detail is increased when compared to the previous latex builds. - The source along with a static PDF are included in the current code distribution. - The html build is hosted on cledb.readthedocs.io.
update-iqud	20230223	Major update. - CLE is updated to 2.0.4. A number of issues with the forward-synthesis are resolved as outlined in the CLE README. - CLE incurred a small fix to height and depth calculations. Databases need to be recomputed with CLE >= 2.0.4 - Small bugfix updates to the task scheduling scripts in BUILD. - The database compression routine sdb_dcompress introduced numerical instability at particular field geometries. It is now disabled. - Databases are now uncompressed and of np.float32 type. Consequentially, double precision in related functions is changed to single float. - The PREPINV obs_integrate routine is rewritten to improve execution time. Noise statistics and observation rms are enabled. - The PROC functions “cledb_invproc”, cledb_matchiquv, cledb_partsort, cledb_getsubset, blos_proc, spectro_proc, cdf_statistics are updated for fixing bugs and mitigating numerical warnings/issues. - Partially implemented the IQUD (IQU+Doppler) functionality to invert vector fields in the absence of Stokes V. - Introduced the additional ctrlparams “iqud” keyword that works alongside the “integrated” control parameter. - Updated functions in the PROC module to enable IQUD setup matches. New functions cledb_matchiqud and cledb_getsubsetiqud are added. The original subset funcrtion is renamed to cledb_getsubsetiquv. - Keyword ingestion partially implemented. A new obs_headstructproc function is added to PREPINV. Additional keys have been added to ctrlparams to support keyword ingestion. - PREPINV module can now recognize CoMP/uCoMP keywords at input. - A new CoMP test example is now available. The test script for Stokes two-line matching had been updated. - Numba activate/deactivate and caching enabled/disabled keywords are implemented in the ctrlparams class. - Updated the numba, numpy, and scipy package versions under the currently stable Python 3.10 conda env. - Numba optimization/improvement of several functions (obs_integrate, cdf_statistics, cledb_matchiquv, cledb_matchiqud, cledb_partsort, cledb_getsubset). - Updated the documentation to describe the functions, parameters and variables related to the IQUD and line-integrated observations.

TODO list last update: 20230106

1. Add the ISSUEMASK setup as outlined in the documentation.

2. Needed observation keywords are currently manually implemented, where a synthetic CLE observation is used. Information from observation keywords will need to be ingested after observations will become available. update-iqud: Keywords are now ingested with data.

3. Develop and use a public test case for more convoluted MURAM data.

4. Implement additional numba compiler flags and options. Make numba active/disabled with a ctrlparam. Implement numba caching.update-iqud

5. After more information on input data is obtained, implement the LEV2CALIB_WAVE and LEV2CALIB_ABSINT functions as outlined in the documentation.update-iqud: DKIST lev-1 data will contain the needed corrections.

6. Implement ML_LOSDISENTANGLE to separate multiple LOS contributions in observations, as outlined in the documentation.

7. Implement a noise uncertainty quantification method (RMS computations and relevance against signal).update-iqud

8. Introduce the option to invert using PyCELp built databases.