sdc issueshttps://gitlab.leibniz-kis.de/groups/sdc/-/issues2023-08-03T07:21:26Zhttps://gitlab.leibniz-kis.de/sdc/gris/gris-interactive/-/issues/2issue with appmode2023-08-03T07:21:26ZVigeesh Gangadharanissue with appmodeLaunching the binder throws<br>
`500: internal server error`
there seems to be an issue with the appmode.<br>
See: https://github.com/oschuett/appmode/issues/64Launching the binder throws<br>
`500: internal server error`
there seems to be an issue with the appmode.<br>
See: https://github.com/oschuett/appmode/issues/64Vigeesh GangadharanVigeesh Gangadharanhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/311DeprecationWarning: pkg_resources is deprecated as an API.2023-07-27T10:00:26ZVigeesh GangadharanDeprecationWarning: pkg_resources is deprecated as an API.```
kistools_env/bin/gris_gen_maps:4: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
__import__('pkg_resources').run_script('kis-tools==3.12.2', 'gris_gen_maps')
``````
kistools_env/bin/gris_gen_maps:4: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
__import__('pkg_resources').run_script('kis-tools==3.12.2', 'gris_gen_maps')
```Vigeesh GangadharanVigeesh Gangadharanhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/310Issue with gris location plot with disable-hmi-cache2023-07-25T10:00:38ZVigeesh GangadharanIssue with gris location plot with disable-hmi-cacheRunning `gris_location_plot` with `--disable-hmi-cache` gives me the following error:
```
gris_location_plot -v -o test --disable-hmi-cache -r 22 /dat/sdc/gris/20140922
/home/vigeesh/conda/envs/kistools_env/bin/gris_location_plot:4: De...Running `gris_location_plot` with `--disable-hmi-cache` gives me the following error:
```
gris_location_plot -v -o test --disable-hmi-cache -r 22 /dat/sdc/gris/20140922
/home/vigeesh/conda/envs/kistools_env/bin/gris_location_plot:4: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
__import__('pkg_resources').run_script('kis-tools==3.12.2', 'gris_location_plot')
__ ___ __ _______. .___________. ______ ______ __ _______.
| |/ / | | / | | | / __ \ / __ \ | | / |
| ' / | | | (----` `---| |----`| | | | | | | | | | | (----`
| < | | \ \ | | | | | | | | | | | | \ \
| . \ | | .----) | | | | `--' | | `--' | | `----.----) |
|__|\__\ |__| |_______/ |__| \______/ \______/ |_______|_______/
1/0 files failed to download. Please check `.errors` for details
Files Downloaded: 0%| | 0/1 [01:27<?, ?file/s]
22sep14.022: 0%| | 0/1 [01:37<?, ?it/s]
Traceback (most recent call last):
File "/home/vigeesh/conda/envs/kistools_env/bin/gris_location_plot", line 4, in <module>
__import__('pkg_resources').run_script('kis-tools==3.12.2', 'gris_location_plot')
File "/home/vigeesh/conda/envs/kistools_env/lib/python3.11/site-packages/pkg_resources/__init__.py", line 722, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/home/vigeesh/conda/envs/kistools_env/lib/python3.11/site-packages/pkg_resources/__init__.py", line 1561, in run_script
exec(code, namespace, namespace)
File "/home/vigeesh/conda/envs/kistools_env/lib/python3.11/site-packages/kis_tools-3.12.2-py3.11.egg/EGG-INFO/scripts/gris_location_plot", line 27, in <module>
make_gris_locplots(folder, run=run, output_folder=output_folder, use_hmi_cache=use_hmi_cache,debug=debug)
File "/home/vigeesh/conda/envs/kistools_env/lib/python3.11/site-packages/kis_tools-3.12.2-py3.11.egg/kis_tools/gris/plotting.py", line 232, in make_gris_locplots
fig, ax = r.plot_location(use_hmi_cache=use_hmi_cache,debug=debug)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/vigeesh/conda/envs/kistools_env/lib/python3.11/site-packages/kis_tools-3.12.2-py3.11.egg/kis_tools/generic/observation.py", line 161, in plot_location
fig, ax = make_loc_plot(
^^^^^^^^^^^^^^
File "/home/vigeesh/conda/envs/kistools_env/lib/python3.11/site-packages/kis_tools-3.12.2-py3.11.egg/kis_tools/util/locplot.py", line 138, in make_loc_plot
hmi_map = get_continuum_map(date, use_hmi_cache)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/vigeesh/conda/envs/kistools_env/lib/python3.11/site-packages/kis_tools-3.12.2-py3.11.egg/kis_tools/util/locplot.py", line 262, in get_continuum_map
continuum_file = download_hmi_continuum(date, context_dir)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/vigeesh/conda/envs/kistools_env/lib/python3.11/site-packages/kis_tools-3.12.2-py3.11.egg/kis_tools/util/locplot.py", line 60, in download_hmi_continuum
continuum_file = pull_continuum(date, target)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/vigeesh/conda/envs/kistools_env/lib/python3.11/site-packages/kis_tools-3.12.2-py3.11.egg/kis_tools/util/locplot.py", line 88, in pull_continuum
file = Fido.fetch(row)[0]
~~~~~~~~~~~~~~~^^^
File "/home/vigeesh/conda/envs/kistools_env/lib/python3.11/collections/__init__.py", line 1241, in __getitem__
return self.data[i]
~~~~~~~~~^^^
IndexError: list index out of range
```Vigeesh GangadharanVigeesh Gangadharanhttps://gitlab.leibniz-kis.de/sdc/gris/grisinv/-/issues/51Array size mismatch2022-07-27T09:45:54ZVigeesh GangadharanArray size mismatch```
SDC:GRIS Inversion Pipeline: v0.6.0+4.gf83d1e4
Reading fits files 0/128: | | 0% (0.0s)
Traceback (most recent call last):
File "/home/vigeesh/conda/envs/gris_env_test/bin/vfisv", line 8, in <module>
...```
SDC:GRIS Inversion Pipeline: v0.6.0+4.gf83d1e4
Reading fits files 0/128: | | 0% (0.0s)
Traceback (most recent call last):
File "/home/vigeesh/conda/envs/gris_env_test/bin/vfisv", line 8, in <module>
sys.exit(main())
File "/home/vigeesh/conda/envs/gris_env_test/lib/python3.9/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/home/vigeesh/conda/envs/gris_env_test/lib/python3.9/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/home/vigeesh/conda/envs/gris_env_test/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/vigeesh/conda/envs/gris_env_test/lib/python3.9/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/home/vigeesh/conda/envs/gris_env_test/lib/python3.9/site-packages/grisinv/invert.py", line 2198, in main
data, stokes, header = vfisv(path, id, line, width,numproc,weights=weights)
File "/home/vigeesh/conda/envs/gris_env_test/lib/python3.9/site-packages/grisinv/invert.py", line 214, in vfisv
vfisv_data = VFISVpackage(path, id, line, width)
File "/home/vigeesh/conda/envs/gris_env_test/lib/python3.9/site-packages/grisinv/invert.py", line 1669, in __init__
self.get_data_slit()
File "/home/vigeesh/conda/envs/gris_env_test/lib/python3.9/site-packages/grisinv/invert.py", line 1921, in get_data_slit
self.SI[iff, ...],
ValueError: could not broadcast input array from shape (424,132) into shape (424,201)
```Vigeesh GangadharanVigeesh Gangadharanhttps://gitlab.leibniz-kis.de/sdc/gris/grisred/-/issues/7Incorrect calibration settings, flat and cal files are mixed up2022-05-27T15:25:08ZTaras YakobchukIncorrect calibration settings, flat and cal files are mixed upTelescope calibration files sometimes specified as flat field files, which results in the pipeline error. For example, below only runs 1, 4, 9 could be processed and added to the web archive. When using correct flat and cal files, pipeli...Telescope calibration files sometimes specified as flat field files, which results in the pipeline error. For example, below only runs 1, 4, 9 could be processed and added to the web archive. When using correct flat and cal files, pipeline completes successfully. Should investigate and correct automatically `gris_calibration_settings.csv` file
Example:
gris_calibration_settings.csv:
```
2019-10-10,1,10oct19.001,gris_v7,10oct19.000,,10oct19.008,,10830,,,,,
2019-10-10,2,10oct19.002,gris_v7,10oct19.008,,10oct19.000,,10830,,,,,
2019-10-10,3,10oct19.003,gris_v7,10oct19.008,,10oct19.005,,10830,,,,,
2019-10-10,4,10oct19.004,gris_v7,10oct19.000,,10oct19.008,,10830,,,,,
2019-10-10,6,10oct19.006,gris_v7,10oct19.008,,10oct19.005,,10830,,,,,
2019-10-10,7,10oct19.007,gris_v7,10oct19.008,,10oct19.005,,10830,,,,,
2019-10-10,9,10oct19.009,gris_v7,10oct19.011,,10oct19.008,,10830,,,,,
2019-10-10,12,10oct19.012,gris_v7,10oct19.008,,10oct19.011,,10830,,,,,
2019-10-10,13,10oct19.013,gris_v7,10oct19.008,,10oct19.011,,10830,,,,,
2019-10-10,14,10oct19.014,gris_v7,10oct19.008,,10oct19.011,,10830,,,,,
```
gris_archive_info.csv:
```
2019-10-10,0,/dat/sdc/gris/20191010/level0/10oct19.000,SLIT,1083,flat field,,complete,100.0
2019-10-10,1,/dat/sdc/gris/20191010/level0/10oct19.001-02,SLIT,1083,scanning,,error,100.0
2019-10-10,2,/dat/sdc/gris/20191010/level0/10oct19.002-01,SLIT,1083,scanning,Spicule,aborted,100.0
2019-10-10,3,/dat/sdc/gris/20191010/level0/10oct19.003-01,SLIT,1083,scanning,Spicule,aborted,100.0
2019-10-10,4,/dat/sdc/gris/20191010/level0/10oct19.004-02,SLIT,1083,scanning,,error,100.0
2019-10-10,5,/dat/sdc/gris/20191010/level0/10oct19.005,SLIT,1083,flat field,,complete,100.0
2019-10-10,6,/dat/sdc/gris/20191010/level0/10oct19.006,SLIT,1083,scanning,,error,100.0
2019-10-10,7,/dat/sdc/gris/20191010/level0/10oct19.007-01,SLIT,1083,scanning,,error,100.0
2019-10-10,8,/dat/sdc/gris/20191010/level0/10oct19.008,SLIT,1083,telescope cal.,,complete,100.0
2019-10-10,9,/dat/sdc/gris/20191010/level0/10oct19.009-02,SLIT,1083,scanning,Spicule,complete,100.0
2019-10-10,10,/dat/sdc/gris/20191010/level0/10oct19.010,SLIT,1083,flat field,,aborted,100.0
2019-10-10,11,/dat/sdc/gris/20191010/level0/10oct19.011,SLIT,1083,flat field,,complete,100.0
2019-10-10,12,/dat/sdc/gris/20191010/level0/10oct19.012,SLIT,1083,scanning,Quiet Sun,complete,100.0
2019-10-10,13,/dat/sdc/gris/20191010/level0/10oct19.013,SLIT,1083,scanning,Quiet Sun,complete,100.0
2019-10-10,14,/dat/sdc/gris/20191010/level0/10oct19.014,SLIT,1083,scanning,Quiet Sun,complete,100.0
```Taras YakobchukTaras Yakobchukhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/309Better Error messages for failure due to untranslated data2022-04-27T10:55:09ZCarl SchafferBetter Error messages for failure due to untranslated dataBoth `gris_location_plot` and `gris_gen_maps` fail when pointed at untranslated data. Te error messages do not list this as the cuse, should be improved.Both `gris_location_plot` and `gris_gen_maps` fail when pointed at untranslated data. Te error messages do not list this as the cuse, should be improved.https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/308Stepsize value might be wrong for old data2022-04-26T07:25:07ZCarl SchafferStepsize value might be wrong for old dataThe Cassda-Gui prints the following message on loading old datasets e.g. 20150603_008:
![image](/uploads/8a990cd860cc47c77c487519629b6bed/image.png)
This implies that the step size value in the header might not be correct. InvestigateThe Cassda-Gui prints the following message on loading old datasets e.g. 20150603_008:
![image](/uploads/8a990cd860cc47c77c487519629b6bed/image.png)
This implies that the step size value in the header might not be correct. Investigatehttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/307use updated tag list for header translation2022-03-31T08:28:31ZCarl Schafferuse updated tag list for header translationThe heqader translator has a `to_OBS_TRGT`method that is not being used. It should be modified to be used. Also the `.csv` file in kis_tools/gris/resources containing the target tag information should be extended with Sanis tags from 202...The heqader translator has a `to_OBS_TRGT`method that is not being used. It should be modified to be used. Also the `.csv` file in kis_tools/gris/resources containing the target tag information should be extended with Sanis tags from 2022:
[missing_targets_main_tag_added.csv](/uploads/6ce45a8503f63c1443b1388488a48dcf/missing_targets_main_tag_added.csv)https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/306Graceful exit for gris_lot_cont_corr2022-03-17T13:34:45ZCarl SchafferGraceful exit for gris_lot_cont_corrPlot cont corr should exit with a warning if no files are found:
```
Wed Mar 16 01:43:35 CET 2022
running gris_plot_cont_corr --store /dat/sdc/gris/20191003/context_data/*cont_corr*sav
__ ___ __ _______. .___________. _____...Plot cont corr should exit with a warning if no files are found:
```
Wed Mar 16 01:43:35 CET 2022
running gris_plot_cont_corr --store /dat/sdc/gris/20191003/context_data/*cont_corr*sav
__ ___ __ _______. .___________. ______ ______ __ _______.
| |/ / | | / | | | / __ \ / __ \ | | / |
| ' / | | | (----` `---| |----`| | | | | | | | | | | (----`
| < | | \ \ | | | | | | | | | | | | \ \
| . \ | | .----) | | | | `--' | | `--' | | `----.----) |
|__|\__\ |__| |_______/ |__| \______/ \______/ |_______|_______/
Traceback (most recent call last):
File "/home/schaffer/anaconda3/envs/sdc/bin/gris_plot_cont_corr", line 7, in <module>
exec(compile(f.read(), __file__, 'exec'))
File "/dat/schaffer/projects/kis_tools/bin/gris_plot_cont_corr", line 27, in <module>
fig = plot_cont_corr(i)
File "/dat/schaffer/projects/kis_tools/kis_tools/gris/plotting.py", line 140, in plot_cont_corr
data = readsav(path)
File "/home/schaffer/anaconda3/envs/sdc/lib/python3.10/site-packages/scipy/io/idl.py", line 738, in readsav
f = open(file_name, 'rb')
FileNotFoundError: [Errno 2] No such file or directory: '/dat/sdc/gris/20191003/context_data/*cont_corr*sav'
/var/lib/condor/execute/dir_326475/condor_exec.exe: line 105: [: ==: unary operator expected
```https://gitlab.leibniz-kis.de/sdc/gris/cassda_gui/-/issues/2write setup instructions2022-03-15T08:23:43ZCarl Schafferwrite setup instructionshttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/305WCS errors for unequal step-sizes2022-03-14T17:37:09ZCarl SchafferWCS errors for unequal step-sizes![image](/uploads/3f72dc7616b40682f70569e5c96d3d30/image.png)
Gris Coordinates recovered from the first wcs sometimes will go in the wrong direction. Also ther is an offset between the coordinates fed into the WCS generator and the ones...![image](/uploads/3f72dc7616b40682f70569e5c96d3d30/image.png)
Gris Coordinates recovered from the first wcs sometimes will go in the wrong direction. Also ther is an offset between the coordinates fed into the WCS generator and the ones reconstructed from the header. It's something with the rotation of the different step widths in X and Y direction. Cant't figure it out, spent one week now.
to reproduce run test_coords /test_wcs for commit 50ee44ce819c3229adb91e2d425e26d4fdbe0b9bhttps://gitlab.leibniz-kis.de/sdc/issue_collection/-/issues/45Exptime values might be broken2022-03-14T13:34:43ZCarl SchafferExptime values might be broken
Mail from @lindner:
> Hi Carl,
>
> das sieht ja schon mal gut aus! Mir ist nur eine Sache aufgefallen: Ich habe eine zufaellige Observation ausgewaehlt und sehe eine Exposure time von Null:
>
> https://archive.sdc.leibniz-kis.de/SDC...
Mail from @lindner:
> Hi Carl,
>
> das sieht ja schon mal gut aus! Mir ist nur eine Sache aufgefallen: Ich habe eine zufaellige Observation ausgewaehlt und sehe eine Exposure time von Null:
>
> https://archive.sdc.leibniz-kis.de/SDCDetailServlet?Instrument=gris&ObjectId=607c9d7140cd264c85d965ec
>
> VG,
>
> Philip
Value in database:
![image](/uploads/ade69e4ee2b589f0c727f53e1eec440e/image.png)
Ran `kharon describe -f -d sdc /dat/sdc/gris/20201017`
Fixed the value.
Check whether more descriptions are broken like this.https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/304stepsizes for penumbral masks2022-03-14T07:54:31ZCarl Schafferstepsizes for penumbral masksPenumbral masks do not support variable step sizes as added in https://gitlab.leibniz-kis.de/sdc/kis_tools/-/merge_requests/233
* [ ] Let masks use WCS of first file with mask dataPenumbral masks do not support variable step sizes as added in https://gitlab.leibniz-kis.de/sdc/kis_tools/-/merge_requests/233
* [ ] Let masks use WCS of first file with mask datahttps://gitlab.leibniz-kis.de/sdc/gris/grisinv/-/issues/50Add an option to apply user-supplied weights2022-03-10T11:38:55ZVigeesh GangadharanAdd an option to apply user-supplied weightsRelated to `inv_utils.f90`:
```fortran
42 WEIGHTS(:)=NOISE
43 !-----------------------------------------------------------
44 WEIGHTS(1)=(2.0*(1.0+4.0*RATUMQS-ICVAL/ICMEAN)/(1.0+RATUMQS*ICVAL/ICMEAN))
45...Related to `inv_utils.f90`:
```fortran
42 WEIGHTS(:)=NOISE
43 !-----------------------------------------------------------
44 WEIGHTS(1)=(2.0*(1.0+4.0*RATUMQS-ICVAL/ICMEAN)/(1.0+RATUMQS*ICVAL/ICMEAN))
45 IF (WEIGHTS(1) > 2.5) WEIGHTS(1)=2.5
46 IF (WEIGHTS(1) < 0.5) WEIGHTS(1)=0.5
47 WEIGHTS(1)=WEIGHTS(1)*NOISE
```
Try to get this out of Fortran into PythonVigeesh GangadharanVigeesh Gangadharanhttps://gitlab.leibniz-kis.de/sdc/gris/grisred/-/issues/6targets from different days2022-04-21T08:48:56ZTaras Yakobchuktargets from different daysGRIS_v8 can and should use target runs even if there was none taken for a specific day.
WE need to implement a mechanism to specify a target file from another day.
Idea: modify the file search to include sibling folders of other days....GRIS_v8 can and should use target runs even if there was none taken for a specific day.
WE need to implement a mechanism to specify a target file from another day.
Idea: modify the file search to include sibling folders of other days.
We also need to add the targets to the calibration settings. @Carl has a list of headers we can use to match up runs with target scans.Taras YakobchukTaras Yakobchukhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/301translate_header gracious error code on missing run2022-02-01T14:05:35ZCarl Schaffertranslate_header gracious error code on missing run```
__ ___ __ _______. .___________. ______ ______ __ _______.
| |/ / | | / | | | / __ \ / __ \ | | / |
| ' / | | | (----` `---| |----`| | | | | | ...```
__ ___ __ _______. .___________. ______ ______ __ _______.
| |/ / | | / | | | / __ \ / __ \ | | / |
| ' / | | | (----` `---| |----`| | | | | | | | | | | (----`
| < | | \ \ | | | | | | | | | | | | \ \
| . \ | | .----) | | | | `--' | | `--' | | `----.----) |
|__|\__\ |__| |_______/ |__| \______/ \______/ |_______|_______/
Traceback (most recent call last):
File "/usr/local/bin/gris_translate_header", line 9, in <module>
run_translate_header(sys.argv[1:])
File "/usr/local/lib/python3.8/site-packages/kis_tools/gris/headers/translate_header.py", line 307, in run_translate_header
files = groupby_gris_run(files)[args.run]
KeyError: 2
```
If translate header is called with a non-existent run, the tool should exit graciously. Same goes for the splitter toolhttps://gitlab.leibniz-kis.de/sdc/gris/grisinv/-/issues/46add proper BTYPE label for uncertainties2022-01-11T10:31:49ZVigeesh Gangadharanadd proper BTYPE label for uncertaintiesVigeesh GangadharanVigeesh Gangadharanhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/295add grepfits for full calib pipeline2021-11-19T17:41:46ZCarl Schafferadd grepfits for full calib pipelinecalibration settings retrieval can be largely improved using grepfits and get last l0:
```
echo $(for i in /instruments/gris/202*;do ./gris_l0_info.sh $i;done) > last_l0_file_per_run.txt
grepfits "IMGSYS WAVELENG MEASURE TARGET FILESTAT...calibration settings retrieval can be largely improved using grepfits and get last l0:
```
echo $(for i in /instruments/gris/202*;do ./gris_l0_info.sh $i;done) > last_l0_file_per_run.txt
grepfits "IMGSYS WAVELENG MEASURE TARGET FILESTAT EXPTIME" $(cat last_l0_file_per_run.txt ) |tee $(date '+%Y-%m-%d')_gris_l0_info.csv
```
the output file can then replace the file in `kis_tools.gris.resources` and be used to generate calibration settings.
The shell scripts have been added to `kis_tools.gris.calib_settings`https://gitlab.leibniz-kis.de/sdc/gris/grisinv/-/issues/44[IFU-integration] Header does not change for timeseries data2022-02-02T16:48:34ZVigeesh Gangadharan[IFU-integration] Header does not change for timeseries dataFor timeseries data, the header is temporarily fixed to the first map. The axes of the preview (animated) gif are fixed and does not change with the FOV. This needs to be corrected.For timeseries data, the header is temporarily fixed to the first map. The axes of the preview (animated) gif are fixed and does not change with the FOV. This needs to be corrected.Vigeesh GangadharanVigeesh Gangadharanhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/293gris/IFU instrument segmentation2021-11-14T21:36:34ZCarl Schaffergris/IFU instrument segmentationa lot of the gris and IFU files are problematic due to their names that could fit to either instrument.
e.g. IFU continuum correction files would be added to the GRIS database by default.
This ambiguity needs to be addressed.a lot of the gris and IFU files are problematic due to their names that could fit to either instrument.
e.g. IFU continuum correction files would be added to the GRIS database by default.
This ambiguity needs to be addressed.