kis_tools issueshttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues2020-05-12T09:46:26Zhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/188Gris PhD Embargos2020-05-12T09:46:26ZCarl SchafferGris PhD EmbargosWrite list of observers per day to NBG, she will check which dates should be embargoed.Write list of observers per day to NBG, she will check which dates should be embargoed.Demleitner VisitCarl SchafferCarl Schaffer2020-05-11https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/297GRIS data is flipped depending on step-angle2022-04-21T10:46:49ZCarl SchafferGRIS data is flipped depending on step-angleDepending on how the slit data is scanned, images are flipped w.r.t. HMI.
We should consider serving flipped data to the end-users instead of what we're doing right now.
* [ ] Check whether this is already done somewhere
* [ ] Evaluat...Depending on how the slit data is scanned, images are flipped w.r.t. HMI.
We should consider serving flipped data to the end-users instead of what we're doing right now.
* [ ] Check whether this is already done somewhere
* [ ] Evaluate how much effort a fix would be
* [ ] decide on actionCarl SchafferCarl Schaffer2021-11-30https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/311DeprecationWarning: pkg_resources is deprecated as an API.2023-07-27T10:00:26ZVigeesh GangadharanDeprecationWarning: pkg_resources is deprecated as an API.```
kistools_env/bin/gris_gen_maps:4: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
__import__('pkg_resources').run_script('kis-tools==3.12.2', 'gris_gen_maps')
``````
kistools_env/bin/gris_gen_maps:4: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
__import__('pkg_resources').run_script('kis-tools==3.12.2', 'gris_gen_maps')
```Vigeesh GangadharanVigeesh Gangadharanhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/310Issue with gris location plot with disable-hmi-cache2023-07-25T10:00:38ZVigeesh GangadharanIssue with gris location plot with disable-hmi-cacheRunning `gris_location_plot` with `--disable-hmi-cache` gives me the following error:
```
gris_location_plot -v -o test --disable-hmi-cache -r 22 /dat/sdc/gris/20140922
/home/vigeesh/conda/envs/kistools_env/bin/gris_location_plot:4: De...Running `gris_location_plot` with `--disable-hmi-cache` gives me the following error:
```
gris_location_plot -v -o test --disable-hmi-cache -r 22 /dat/sdc/gris/20140922
/home/vigeesh/conda/envs/kistools_env/bin/gris_location_plot:4: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
__import__('pkg_resources').run_script('kis-tools==3.12.2', 'gris_location_plot')
__ ___ __ _______. .___________. ______ ______ __ _______.
| |/ / | | / | | | / __ \ / __ \ | | / |
| ' / | | | (----` `---| |----`| | | | | | | | | | | (----`
| < | | \ \ | | | | | | | | | | | | \ \
| . \ | | .----) | | | | `--' | | `--' | | `----.----) |
|__|\__\ |__| |_______/ |__| \______/ \______/ |_______|_______/
1/0 files failed to download. Please check `.errors` for details
Files Downloaded: 0%| | 0/1 [01:27<?, ?file/s]
22sep14.022: 0%| | 0/1 [01:37<?, ?it/s]
Traceback (most recent call last):
File "/home/vigeesh/conda/envs/kistools_env/bin/gris_location_plot", line 4, in <module>
__import__('pkg_resources').run_script('kis-tools==3.12.2', 'gris_location_plot')
File "/home/vigeesh/conda/envs/kistools_env/lib/python3.11/site-packages/pkg_resources/__init__.py", line 722, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/home/vigeesh/conda/envs/kistools_env/lib/python3.11/site-packages/pkg_resources/__init__.py", line 1561, in run_script
exec(code, namespace, namespace)
File "/home/vigeesh/conda/envs/kistools_env/lib/python3.11/site-packages/kis_tools-3.12.2-py3.11.egg/EGG-INFO/scripts/gris_location_plot", line 27, in <module>
make_gris_locplots(folder, run=run, output_folder=output_folder, use_hmi_cache=use_hmi_cache,debug=debug)
File "/home/vigeesh/conda/envs/kistools_env/lib/python3.11/site-packages/kis_tools-3.12.2-py3.11.egg/kis_tools/gris/plotting.py", line 232, in make_gris_locplots
fig, ax = r.plot_location(use_hmi_cache=use_hmi_cache,debug=debug)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/vigeesh/conda/envs/kistools_env/lib/python3.11/site-packages/kis_tools-3.12.2-py3.11.egg/kis_tools/generic/observation.py", line 161, in plot_location
fig, ax = make_loc_plot(
^^^^^^^^^^^^^^
File "/home/vigeesh/conda/envs/kistools_env/lib/python3.11/site-packages/kis_tools-3.12.2-py3.11.egg/kis_tools/util/locplot.py", line 138, in make_loc_plot
hmi_map = get_continuum_map(date, use_hmi_cache)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/vigeesh/conda/envs/kistools_env/lib/python3.11/site-packages/kis_tools-3.12.2-py3.11.egg/kis_tools/util/locplot.py", line 262, in get_continuum_map
continuum_file = download_hmi_continuum(date, context_dir)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/vigeesh/conda/envs/kistools_env/lib/python3.11/site-packages/kis_tools-3.12.2-py3.11.egg/kis_tools/util/locplot.py", line 60, in download_hmi_continuum
continuum_file = pull_continuum(date, target)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/vigeesh/conda/envs/kistools_env/lib/python3.11/site-packages/kis_tools-3.12.2-py3.11.egg/kis_tools/util/locplot.py", line 88, in pull_continuum
file = Fido.fetch(row)[0]
~~~~~~~~~~~~~~~^^^
File "/home/vigeesh/conda/envs/kistools_env/lib/python3.11/collections/__init__.py", line 1241, in __getitem__
return self.data[i]
~~~~~~~~~^^^
IndexError: list index out of range
```Vigeesh GangadharanVigeesh Gangadharanhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/309Better Error messages for failure due to untranslated data2022-04-27T10:55:09ZCarl SchafferBetter Error messages for failure due to untranslated dataBoth `gris_location_plot` and `gris_gen_maps` fail when pointed at untranslated data. Te error messages do not list this as the cuse, should be improved.Both `gris_location_plot` and `gris_gen_maps` fail when pointed at untranslated data. Te error messages do not list this as the cuse, should be improved.https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/308Stepsize value might be wrong for old data2022-04-26T07:25:07ZCarl SchafferStepsize value might be wrong for old dataThe Cassda-Gui prints the following message on loading old datasets e.g. 20150603_008:
![image](/uploads/8a990cd860cc47c77c487519629b6bed/image.png)
This implies that the step size value in the header might not be correct. InvestigateThe Cassda-Gui prints the following message on loading old datasets e.g. 20150603_008:
![image](/uploads/8a990cd860cc47c77c487519629b6bed/image.png)
This implies that the step size value in the header might not be correct. Investigatehttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/307use updated tag list for header translation2022-03-31T08:28:31ZCarl Schafferuse updated tag list for header translationThe heqader translator has a `to_OBS_TRGT`method that is not being used. It should be modified to be used. Also the `.csv` file in kis_tools/gris/resources containing the target tag information should be extended with Sanis tags from 202...The heqader translator has a `to_OBS_TRGT`method that is not being used. It should be modified to be used. Also the `.csv` file in kis_tools/gris/resources containing the target tag information should be extended with Sanis tags from 2022:
[missing_targets_main_tag_added.csv](/uploads/6ce45a8503f63c1443b1388488a48dcf/missing_targets_main_tag_added.csv)https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/306Graceful exit for gris_lot_cont_corr2022-03-17T13:34:45ZCarl SchafferGraceful exit for gris_lot_cont_corrPlot cont corr should exit with a warning if no files are found:
```
Wed Mar 16 01:43:35 CET 2022
running gris_plot_cont_corr --store /dat/sdc/gris/20191003/context_data/*cont_corr*sav
__ ___ __ _______. .___________. _____...Plot cont corr should exit with a warning if no files are found:
```
Wed Mar 16 01:43:35 CET 2022
running gris_plot_cont_corr --store /dat/sdc/gris/20191003/context_data/*cont_corr*sav
__ ___ __ _______. .___________. ______ ______ __ _______.
| |/ / | | / | | | / __ \ / __ \ | | / |
| ' / | | | (----` `---| |----`| | | | | | | | | | | (----`
| < | | \ \ | | | | | | | | | | | | \ \
| . \ | | .----) | | | | `--' | | `--' | | `----.----) |
|__|\__\ |__| |_______/ |__| \______/ \______/ |_______|_______/
Traceback (most recent call last):
File "/home/schaffer/anaconda3/envs/sdc/bin/gris_plot_cont_corr", line 7, in <module>
exec(compile(f.read(), __file__, 'exec'))
File "/dat/schaffer/projects/kis_tools/bin/gris_plot_cont_corr", line 27, in <module>
fig = plot_cont_corr(i)
File "/dat/schaffer/projects/kis_tools/kis_tools/gris/plotting.py", line 140, in plot_cont_corr
data = readsav(path)
File "/home/schaffer/anaconda3/envs/sdc/lib/python3.10/site-packages/scipy/io/idl.py", line 738, in readsav
f = open(file_name, 'rb')
FileNotFoundError: [Errno 2] No such file or directory: '/dat/sdc/gris/20191003/context_data/*cont_corr*sav'
/var/lib/condor/execute/dir_326475/condor_exec.exe: line 105: [: ==: unary operator expected
```https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/305WCS errors for unequal step-sizes2022-03-14T17:37:09ZCarl SchafferWCS errors for unequal step-sizes![image](/uploads/3f72dc7616b40682f70569e5c96d3d30/image.png)
Gris Coordinates recovered from the first wcs sometimes will go in the wrong direction. Also ther is an offset between the coordinates fed into the WCS generator and the ones...![image](/uploads/3f72dc7616b40682f70569e5c96d3d30/image.png)
Gris Coordinates recovered from the first wcs sometimes will go in the wrong direction. Also ther is an offset between the coordinates fed into the WCS generator and the ones reconstructed from the header. It's something with the rotation of the different step widths in X and Y direction. Cant't figure it out, spent one week now.
to reproduce run test_coords /test_wcs for commit 50ee44ce819c3229adb91e2d425e26d4fdbe0b9bhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/304stepsizes for penumbral masks2022-03-14T07:54:31ZCarl Schafferstepsizes for penumbral masksPenumbral masks do not support variable step sizes as added in https://gitlab.leibniz-kis.de/sdc/kis_tools/-/merge_requests/233
* [ ] Let masks use WCS of first file with mask dataPenumbral masks do not support variable step sizes as added in https://gitlab.leibniz-kis.de/sdc/kis_tools/-/merge_requests/233
* [ ] Let masks use WCS of first file with mask datahttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/303skewed boxes in location previews for 2021 data2022-03-14T17:44:20ZCarl Schafferskewed boxes in location previews for 2021 data![image](/uploads/6155a8922f12d28ec42010d5fbb93f41/image.png)
![image](/uploads/e1245cbcd549c057d94ccb78cf144c80/image.png)
![image](/uploads/307d603ecb23984b5c8f376750ea45b2/image.png)![image](/uploads/6155a8922f12d28ec42010d5fbb93f41/image.png)
![image](/uploads/e1245cbcd549c057d94ccb78cf144c80/image.png)
![image](/uploads/307d603ecb23984b5c8f376750ea45b2/image.png)Carl SchafferCarl Schafferhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/302previews: respect aspect ratio step size2022-02-21T12:21:22ZCarl Schafferpreviews: respect aspect ratio step sizeSome observations use a step size perpendicular larger than the resolution along the slit. This leads to skewed preview images, as the pixels would essentially need to be non square.
matplotlib supports this, but it also needs to be che...Some observations use a step size perpendicular larger than the resolution along the slit. This leads to skewed preview images, as the pixels would essentially need to be non square.
matplotlib supports this, but it also needs to be checked through the entire chain of calculations and plotting:
The preview plots should use the wcs keywords to determine the values for the aspect ratio and spatial extent of FOVs.
Example:
![image](/uploads/48b580236f2776f341cb7d7f290d906a/image.png)
https://archive.sdc.leibniz-kis.de/SDCDetailServlet?Instrument=gris&ObjectId=607c9dcd18262e0426a2b2a2
Also see the corresponding issue on [Jira](https://leibniz-kis.atlassian.net/jira/servicedesk/projects/SR/queues/custom/10/SR-173)https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/301translate_header gracious error code on missing run2022-02-01T14:05:35ZCarl Schaffertranslate_header gracious error code on missing run```
__ ___ __ _______. .___________. ______ ______ __ _______.
| |/ / | | / | | | / __ \ / __ \ | | / |
| ' / | | | (----` `---| |----`| | | | | | ...```
__ ___ __ _______. .___________. ______ ______ __ _______.
| |/ / | | / | | | / __ \ / __ \ | | / |
| ' / | | | (----` `---| |----`| | | | | | | | | | | (----`
| < | | \ \ | | | | | | | | | | | | \ \
| . \ | | .----) | | | | `--' | | `--' | | `----.----) |
|__|\__\ |__| |_______/ |__| \______/ \______/ |_______|_______/
Traceback (most recent call last):
File "/usr/local/bin/gris_translate_header", line 9, in <module>
run_translate_header(sys.argv[1:])
File "/usr/local/lib/python3.8/site-packages/kis_tools/gris/headers/translate_header.py", line 307, in run_translate_header
files = groupby_gris_run(files)[args.run]
KeyError: 2
```
If translate header is called with a non-existent run, the tool should exit graciously. Same goes for the splitter toolhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/300Map Overplotting2022-04-21T10:45:08ZCarl SchafferMap OverplottingImplement map overplotting as composite maps with hm data.
Implemented in IFUFitsFile as example.
Will only be useful once map rotation has been solved.Implement map overplotting as composite maps with hm data.
Implemented in IFUFitsFile as example.
Will only be useful once map rotation has been solved.Carl SchafferCarl Schafferhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/299'scanedge' in SCANCNTR keyword2022-01-28T15:50:32ZCarl Schaffer'scanedge' in SCANCNTR keywordSometimes gris coordinates do not reference off of the slit center. Add a `NotImplemented` error for these cases and open an issue for futher investigation at a future point in time. There are not sufficiently many examples to warrant a ...Sometimes gris coordinates do not reference off of the slit center. Add a `NotImplemented` error for these cases and open an issue for futher investigation at a future point in time. There are not sufficiently many examples to warrant a lot of time spent.
* [ ] Ask manolo, berke or lucia about this... or reiner?Carl SchafferCarl Schafferhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/298IFU coordinates2022-01-28T15:50:52ZCarl SchafferIFU coordinatesifu coordinates have improved significantly. THe code in `IFUFitsFile`does not account for derotator presence information or STEPANGL information.
Both of these need to be evaluated and a processed before the coordinate calculation can ...ifu coordinates have improved significantly. THe code in `IFUFitsFile`does not account for derotator presence information or STEPANGL information.
Both of these need to be evaluated and a processed before the coordinate calculation can be called done.Carl SchafferCarl Schafferhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/296Misleading Labels for GRIS Previews2022-02-01T16:08:40ZCarl SchafferMisleading Labels for GRIS PreviewsThe labels of the GRIS slit previews imply alignment along the X and Y axes of the Helioprojective coordinate system, while they are actually relative coordinates within the map. Make this clearer.
Also the colorbar of the plots does no...The labels of the GRIS slit previews imply alignment along the X and Y axes of the Helioprojective coordinate system, while they are actually relative coordinates within the map. Make this clearer.
Also the colorbar of the plots does not seem to match up with the data. Investigate and label/omit.
Checklist:
* [ ] Axis Lables
* [ ] Colorbar
* [ ] Re-run Preview generation
* [ ] Upload new Previews
## Excerpt of conversation:
> yakobchuk
> 4:00 PM
> Hi Carl! What's up? Don't remember if I raised this: I think gris image preview labels are confusing, if not wrong. X/Y coords on top indicate center of the frame in helio-projective coord.sys., while deltaX and deltaY labels refer to instrumental relative coords. Here is a good example: https://archive.sdc.leibniz-kis.de/SDCDetailServlet?Instrument=gris&ObjectId=5ee10fdf871689ed8458baad
> The spot is clearly stretched along Y (LAT) (see whole sun image), while on the preview image it goes along deltaX. What do you think?
>
>
> yakobchuk
> 4:21 PM
> Then regarding the colorbar: what are these numbers? I see they are much smaller than intensity data raw counts I get in grisview. Also, I think colorbar label is missing showing what is this (average intensity? polarization?) and in which units.
>
> schaffer
> schaffer
> 9:38 AM
> Hey Taras,
>
> you're completely right concerning the images, this is an issue that hadn't occured to me earlier, but it just came up to me a couple of days ago while checking the IFU coordinates. I agree that the names x and y are misleading as they don't represent actual coordinates. For slit data we should probably move to something like "parallel/horizontal" to slit. Philip also once proposed having two arrows on the image indicating the directions of Solar North and Disk center. I'll open an issue on kis_tools 👍 Thanks for the feedback.
>
> The colorbar we should probably omit or label as arbitrary units: The image data is taken from manolos .cm files, I'm not sure whether any preprocessing is done there beforehand, but there might well be something that causes a discrepancy to the values in the split files. I'll also add it to the issue.https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/295add grepfits for full calib pipeline2021-11-19T17:41:46ZCarl Schafferadd grepfits for full calib pipelinecalibration settings retrieval can be largely improved using grepfits and get last l0:
```
echo $(for i in /instruments/gris/202*;do ./gris_l0_info.sh $i;done) > last_l0_file_per_run.txt
grepfits "IMGSYS WAVELENG MEASURE TARGET FILESTAT...calibration settings retrieval can be largely improved using grepfits and get last l0:
```
echo $(for i in /instruments/gris/202*;do ./gris_l0_info.sh $i;done) > last_l0_file_per_run.txt
grepfits "IMGSYS WAVELENG MEASURE TARGET FILESTAT EXPTIME" $(cat last_l0_file_per_run.txt ) |tee $(date '+%Y-%m-%d')_gris_l0_info.csv
```
the output file can then replace the file in `kis_tools.gris.resources` and be used to generate calibration settings.
The shell scripts have been added to `kis_tools.gris.calib_settings`https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/294Labels Vanish on IFU Map2021-11-24T08:42:50ZCarl SchafferLabels Vanish on IFU MapExcerpt from sunpy matrix:
> cschaffer
>
> Hi all,
>
> I'm preparing data from one of our instruments for release to the public. This includes reformatting file format and FITS headers as well as generating preview images for the webs...Excerpt from sunpy matrix:
> cschaffer
>
> Hi all,
>
> I'm preparing data from one of our instruments for release to the public. This includes reformatting file format and FITS headers as well as generating preview images for the website.
>
> I've encountered a problem with plotting sunpy maps: I read a file using sunpy.map.Map and simply do a .plot() on the resulting object. My issue is that sometimes the map will contain the auto-generated labels for the x and y axis and sometimes not resulting in inconsistent plots (I'm stitching the individual images into a GIF afterwards, so I need consistency). Of course I could just fix or omit the labels manually, but I suspect there's something wrong with the WCS of the files as mentioned in https://github.com/sunpy/sunpy/issues/2654 but I'm at a loss on how to debug/validate my WCS keywords.
>
> I've collected a set of sample files and plots here happy for any help!
> Cadair
> Hey
> cschaffer that looks like fun 😆
> The fact you have ticks and no labels is interesting.
> I can't promise I will be able to look tomorrow, but I will have a poke sometime.
> dstansby
> It's because the rotation matrix of your images is almost 90 degrees, but by default sunpy tries to label longitude on the x-axis and latitude on the y-axis
> Doing something like this gives nice labels, with longitude on the y-axis and latitude on the x-axis:
>
> 12345678910111213141516fig = plt.figure()
> ax = fig.add_subplot(111, projection=m)
> m.plot(axes=ax)
>
> lon = ax.coords[0]
> lat = ax.coords[1]
> \# Put longitude ticks on left side
> lon.set_ticks_position('l')
> lon.set_ticklabel_position('l')
> lon.set_axislabel_position('l')
> \# Put latitude ticks on bottom side
> lat.set_ticks_position('b')
> lat.set_ticklabel_position('b')
> lat.set_axislabel_position('b')
>
> fig.savefig(...)
>
> Perhaps a nice action to take on the sunpy side would be automatically doing this labelling 'flip' when the rotation of the map is > 45 degreeshttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/293gris/IFU instrument segmentation2021-11-14T21:36:34ZCarl Schaffergris/IFU instrument segmentationa lot of the gris and IFU files are problematic due to their names that could fit to either instrument.
e.g. IFU continuum correction files would be added to the GRIS database by default.
This ambiguity needs to be addressed.a lot of the gris and IFU files are problematic due to their names that could fit to either instrument.
e.g. IFU continuum correction files would be added to the GRIS database by default.
This ambiguity needs to be addressed.