kis_tools issueshttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues2023-07-27T10:00:26Zhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/311DeprecationWarning: pkg_resources is deprecated as an API.2023-07-27T10:00:26ZVigeesh GangadharanDeprecationWarning: pkg_resources is deprecated as an API.```
kistools_env/bin/gris_gen_maps:4: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
__import__('pkg_resources').run_script('kis-tools==3.12.2', 'gris_gen_maps')
``````
kistools_env/bin/gris_gen_maps:4: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
__import__('pkg_resources').run_script('kis-tools==3.12.2', 'gris_gen_maps')
```Vigeesh GangadharanVigeesh Gangadharanhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/310Issue with gris location plot with disable-hmi-cache2023-07-25T10:00:38ZVigeesh GangadharanIssue with gris location plot with disable-hmi-cacheRunning `gris_location_plot` with `--disable-hmi-cache` gives me the following error:
```
gris_location_plot -v -o test --disable-hmi-cache -r 22 /dat/sdc/gris/20140922
/home/vigeesh/conda/envs/kistools_env/bin/gris_location_plot:4: De...Running `gris_location_plot` with `--disable-hmi-cache` gives me the following error:
```
gris_location_plot -v -o test --disable-hmi-cache -r 22 /dat/sdc/gris/20140922
/home/vigeesh/conda/envs/kistools_env/bin/gris_location_plot:4: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
__import__('pkg_resources').run_script('kis-tools==3.12.2', 'gris_location_plot')
__ ___ __ _______. .___________. ______ ______ __ _______.
| |/ / | | / | | | / __ \ / __ \ | | / |
| ' / | | | (----` `---| |----`| | | | | | | | | | | (----`
| < | | \ \ | | | | | | | | | | | | \ \
| . \ | | .----) | | | | `--' | | `--' | | `----.----) |
|__|\__\ |__| |_______/ |__| \______/ \______/ |_______|_______/
1/0 files failed to download. Please check `.errors` for details
Files Downloaded: 0%| | 0/1 [01:27<?, ?file/s]
22sep14.022: 0%| | 0/1 [01:37<?, ?it/s]
Traceback (most recent call last):
File "/home/vigeesh/conda/envs/kistools_env/bin/gris_location_plot", line 4, in <module>
__import__('pkg_resources').run_script('kis-tools==3.12.2', 'gris_location_plot')
File "/home/vigeesh/conda/envs/kistools_env/lib/python3.11/site-packages/pkg_resources/__init__.py", line 722, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/home/vigeesh/conda/envs/kistools_env/lib/python3.11/site-packages/pkg_resources/__init__.py", line 1561, in run_script
exec(code, namespace, namespace)
File "/home/vigeesh/conda/envs/kistools_env/lib/python3.11/site-packages/kis_tools-3.12.2-py3.11.egg/EGG-INFO/scripts/gris_location_plot", line 27, in <module>
make_gris_locplots(folder, run=run, output_folder=output_folder, use_hmi_cache=use_hmi_cache,debug=debug)
File "/home/vigeesh/conda/envs/kistools_env/lib/python3.11/site-packages/kis_tools-3.12.2-py3.11.egg/kis_tools/gris/plotting.py", line 232, in make_gris_locplots
fig, ax = r.plot_location(use_hmi_cache=use_hmi_cache,debug=debug)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/vigeesh/conda/envs/kistools_env/lib/python3.11/site-packages/kis_tools-3.12.2-py3.11.egg/kis_tools/generic/observation.py", line 161, in plot_location
fig, ax = make_loc_plot(
^^^^^^^^^^^^^^
File "/home/vigeesh/conda/envs/kistools_env/lib/python3.11/site-packages/kis_tools-3.12.2-py3.11.egg/kis_tools/util/locplot.py", line 138, in make_loc_plot
hmi_map = get_continuum_map(date, use_hmi_cache)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/vigeesh/conda/envs/kistools_env/lib/python3.11/site-packages/kis_tools-3.12.2-py3.11.egg/kis_tools/util/locplot.py", line 262, in get_continuum_map
continuum_file = download_hmi_continuum(date, context_dir)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/vigeesh/conda/envs/kistools_env/lib/python3.11/site-packages/kis_tools-3.12.2-py3.11.egg/kis_tools/util/locplot.py", line 60, in download_hmi_continuum
continuum_file = pull_continuum(date, target)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/vigeesh/conda/envs/kistools_env/lib/python3.11/site-packages/kis_tools-3.12.2-py3.11.egg/kis_tools/util/locplot.py", line 88, in pull_continuum
file = Fido.fetch(row)[0]
~~~~~~~~~~~~~~~^^^
File "/home/vigeesh/conda/envs/kistools_env/lib/python3.11/collections/__init__.py", line 1241, in __getitem__
return self.data[i]
~~~~~~~~~^^^
IndexError: list index out of range
```Vigeesh GangadharanVigeesh Gangadharanhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/309Better Error messages for failure due to untranslated data2022-04-27T10:55:09ZCarl SchafferBetter Error messages for failure due to untranslated dataBoth `gris_location_plot` and `gris_gen_maps` fail when pointed at untranslated data. Te error messages do not list this as the cuse, should be improved.Both `gris_location_plot` and `gris_gen_maps` fail when pointed at untranslated data. Te error messages do not list this as the cuse, should be improved.https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/308Stepsize value might be wrong for old data2022-04-26T07:25:07ZCarl SchafferStepsize value might be wrong for old dataThe Cassda-Gui prints the following message on loading old datasets e.g. 20150603_008:
![image](/uploads/8a990cd860cc47c77c487519629b6bed/image.png)
This implies that the step size value in the header might not be correct. InvestigateThe Cassda-Gui prints the following message on loading old datasets e.g. 20150603_008:
![image](/uploads/8a990cd860cc47c77c487519629b6bed/image.png)
This implies that the step size value in the header might not be correct. Investigatehttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/291IFU header_validation2022-04-21T10:49:01ZCarl SchafferIFU header_validation`kis_tools.gris.ifu_fits_file.IFUFitsFile.get_cleaned_header` overwrites the parent implementation and omits the call to `validate_headers`
Asess whether this is ok`kis_tools.gris.ifu_fits_file.IFUFitsFile.get_cleaned_header` overwrites the parent implementation and omits the call to `validate_headers`
Asess whether this is okhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/297GRIS data is flipped depending on step-angle2022-04-21T10:46:49ZCarl SchafferGRIS data is flipped depending on step-angleDepending on how the slit data is scanned, images are flipped w.r.t. HMI.
We should consider serving flipped data to the end-users instead of what we're doing right now.
* [ ] Check whether this is already done somewhere
* [ ] Evaluat...Depending on how the slit data is scanned, images are flipped w.r.t. HMI.
We should consider serving flipped data to the end-users instead of what we're doing right now.
* [ ] Check whether this is already done somewhere
* [ ] Evaluate how much effort a fix would be
* [ ] decide on actionCarl SchafferCarl Schaffer2021-11-30https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/300Map Overplotting2022-04-21T10:45:08ZCarl SchafferMap OverplottingImplement map overplotting as composite maps with hm data.
Implemented in IFUFitsFile as example.
Will only be useful once map rotation has been solved.Implement map overplotting as composite maps with hm data.
Implemented in IFUFitsFile as example.
Will only be useful once map rotation has been solved.Carl SchafferCarl Schafferhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/307use updated tag list for header translation2022-03-31T08:28:31ZCarl Schafferuse updated tag list for header translationThe heqader translator has a `to_OBS_TRGT`method that is not being used. It should be modified to be used. Also the `.csv` file in kis_tools/gris/resources containing the target tag information should be extended with Sanis tags from 202...The heqader translator has a `to_OBS_TRGT`method that is not being used. It should be modified to be used. Also the `.csv` file in kis_tools/gris/resources containing the target tag information should be extended with Sanis tags from 2022:
[missing_targets_main_tag_added.csv](/uploads/6ce45a8503f63c1443b1388488a48dcf/missing_targets_main_tag_added.csv)https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/306Graceful exit for gris_lot_cont_corr2022-03-17T13:34:45ZCarl SchafferGraceful exit for gris_lot_cont_corrPlot cont corr should exit with a warning if no files are found:
```
Wed Mar 16 01:43:35 CET 2022
running gris_plot_cont_corr --store /dat/sdc/gris/20191003/context_data/*cont_corr*sav
__ ___ __ _______. .___________. _____...Plot cont corr should exit with a warning if no files are found:
```
Wed Mar 16 01:43:35 CET 2022
running gris_plot_cont_corr --store /dat/sdc/gris/20191003/context_data/*cont_corr*sav
__ ___ __ _______. .___________. ______ ______ __ _______.
| |/ / | | / | | | / __ \ / __ \ | | / |
| ' / | | | (----` `---| |----`| | | | | | | | | | | (----`
| < | | \ \ | | | | | | | | | | | | \ \
| . \ | | .----) | | | | `--' | | `--' | | `----.----) |
|__|\__\ |__| |_______/ |__| \______/ \______/ |_______|_______/
Traceback (most recent call last):
File "/home/schaffer/anaconda3/envs/sdc/bin/gris_plot_cont_corr", line 7, in <module>
exec(compile(f.read(), __file__, 'exec'))
File "/dat/schaffer/projects/kis_tools/bin/gris_plot_cont_corr", line 27, in <module>
fig = plot_cont_corr(i)
File "/dat/schaffer/projects/kis_tools/kis_tools/gris/plotting.py", line 140, in plot_cont_corr
data = readsav(path)
File "/home/schaffer/anaconda3/envs/sdc/lib/python3.10/site-packages/scipy/io/idl.py", line 738, in readsav
f = open(file_name, 'rb')
FileNotFoundError: [Errno 2] No such file or directory: '/dat/sdc/gris/20191003/context_data/*cont_corr*sav'
/var/lib/condor/execute/dir_326475/condor_exec.exe: line 105: [: ==: unary operator expected
```https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/245comon entry point for all fits files2022-03-15T07:55:47ZCarl Schaffercomon entry point for all fits filesImplement a common for all fits files, something like:
`open_kis_fits(filename) -> any of {GrisFitsFile, ChroTelFitsFile, BBI...)`
* [ ] add documentation about this to the base README.mdImplement a common for all fits files, something like:
`open_kis_fits(filename) -> any of {GrisFitsFile, ChroTelFitsFile, BBI...)`
* [ ] add documentation about this to the base README.mdhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/270coordinates: slit offset2022-03-15T07:54:50ZCarl Schaffercoordinates: slit offsetVerify whether slitposxy reference the first slit's center or the center of the expected map. Current behavior assumes center of first slit.Verify whether slitposxy reference the first slit's center or the center of the expected map. Current behavior assumes center of first slit.https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/303skewed boxes in location previews for 2021 data2022-03-14T17:44:20ZCarl Schafferskewed boxes in location previews for 2021 data![image](/uploads/6155a8922f12d28ec42010d5fbb93f41/image.png)
![image](/uploads/e1245cbcd549c057d94ccb78cf144c80/image.png)
![image](/uploads/307d603ecb23984b5c8f376750ea45b2/image.png)![image](/uploads/6155a8922f12d28ec42010d5fbb93f41/image.png)
![image](/uploads/e1245cbcd549c057d94ccb78cf144c80/image.png)
![image](/uploads/307d603ecb23984b5c8f376750ea45b2/image.png)Carl SchafferCarl Schafferhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/305WCS errors for unequal step-sizes2022-03-14T17:37:09ZCarl SchafferWCS errors for unequal step-sizes![image](/uploads/3f72dc7616b40682f70569e5c96d3d30/image.png)
Gris Coordinates recovered from the first wcs sometimes will go in the wrong direction. Also ther is an offset between the coordinates fed into the WCS generator and the ones...![image](/uploads/3f72dc7616b40682f70569e5c96d3d30/image.png)
Gris Coordinates recovered from the first wcs sometimes will go in the wrong direction. Also ther is an offset between the coordinates fed into the WCS generator and the ones reconstructed from the header. It's something with the rotation of the different step widths in X and Y direction. Cant't figure it out, spent one week now.
to reproduce run test_coords /test_wcs for commit 50ee44ce819c3229adb91e2d425e26d4fdbe0b9bhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/304stepsizes for penumbral masks2022-03-14T07:54:31ZCarl Schafferstepsizes for penumbral masksPenumbral masks do not support variable step sizes as added in https://gitlab.leibniz-kis.de/sdc/kis_tools/-/merge_requests/233
* [ ] Let masks use WCS of first file with mask dataPenumbral masks do not support variable step sizes as added in https://gitlab.leibniz-kis.de/sdc/kis_tools/-/merge_requests/233
* [ ] Let masks use WCS of first file with mask datahttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/302previews: respect aspect ratio step size2022-02-21T12:21:22ZCarl Schafferpreviews: respect aspect ratio step sizeSome observations use a step size perpendicular larger than the resolution along the slit. This leads to skewed preview images, as the pixels would essentially need to be non square.
matplotlib supports this, but it also needs to be che...Some observations use a step size perpendicular larger than the resolution along the slit. This leads to skewed preview images, as the pixels would essentially need to be non square.
matplotlib supports this, but it also needs to be checked through the entire chain of calculations and plotting:
The preview plots should use the wcs keywords to determine the values for the aspect ratio and spatial extent of FOVs.
Example:
![image](/uploads/48b580236f2776f341cb7d7f290d906a/image.png)
https://archive.sdc.leibniz-kis.de/SDCDetailServlet?Instrument=gris&ObjectId=607c9dcd18262e0426a2b2a2
Also see the corresponding issue on [Jira](https://leibniz-kis.atlassian.net/jira/servicedesk/projects/SR/queues/custom/10/SR-173)https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/296Misleading Labels for GRIS Previews2022-02-01T16:08:40ZCarl SchafferMisleading Labels for GRIS PreviewsThe labels of the GRIS slit previews imply alignment along the X and Y axes of the Helioprojective coordinate system, while they are actually relative coordinates within the map. Make this clearer.
Also the colorbar of the plots does no...The labels of the GRIS slit previews imply alignment along the X and Y axes of the Helioprojective coordinate system, while they are actually relative coordinates within the map. Make this clearer.
Also the colorbar of the plots does not seem to match up with the data. Investigate and label/omit.
Checklist:
* [ ] Axis Lables
* [ ] Colorbar
* [ ] Re-run Preview generation
* [ ] Upload new Previews
## Excerpt of conversation:
> yakobchuk
> 4:00 PM
> Hi Carl! What's up? Don't remember if I raised this: I think gris image preview labels are confusing, if not wrong. X/Y coords on top indicate center of the frame in helio-projective coord.sys., while deltaX and deltaY labels refer to instrumental relative coords. Here is a good example: https://archive.sdc.leibniz-kis.de/SDCDetailServlet?Instrument=gris&ObjectId=5ee10fdf871689ed8458baad
> The spot is clearly stretched along Y (LAT) (see whole sun image), while on the preview image it goes along deltaX. What do you think?
>
>
> yakobchuk
> 4:21 PM
> Then regarding the colorbar: what are these numbers? I see they are much smaller than intensity data raw counts I get in grisview. Also, I think colorbar label is missing showing what is this (average intensity? polarization?) and in which units.
>
> schaffer
> schaffer
> 9:38 AM
> Hey Taras,
>
> you're completely right concerning the images, this is an issue that hadn't occured to me earlier, but it just came up to me a couple of days ago while checking the IFU coordinates. I agree that the names x and y are misleading as they don't represent actual coordinates. For slit data we should probably move to something like "parallel/horizontal" to slit. Philip also once proposed having two arrows on the image indicating the directions of Solar North and Disk center. I'll open an issue on kis_tools 👍 Thanks for the feedback.
>
> The colorbar we should probably omit or label as arbitrary units: The image data is taken from manolos .cm files, I'm not sure whether any preprocessing is done there beforehand, but there might well be something that causes a discrepancy to the values in the split files. I'll also add it to the issue.https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/301translate_header gracious error code on missing run2022-02-01T14:05:35ZCarl Schaffertranslate_header gracious error code on missing run```
__ ___ __ _______. .___________. ______ ______ __ _______.
| |/ / | | / | | | / __ \ / __ \ | | / |
| ' / | | | (----` `---| |----`| | | | | | ...```
__ ___ __ _______. .___________. ______ ______ __ _______.
| |/ / | | / | | | / __ \ / __ \ | | / |
| ' / | | | (----` `---| |----`| | | | | | | | | | | (----`
| < | | \ \ | | | | | | | | | | | | \ \
| . \ | | .----) | | | | `--' | | `--' | | `----.----) |
|__|\__\ |__| |_______/ |__| \______/ \______/ |_______|_______/
Traceback (most recent call last):
File "/usr/local/bin/gris_translate_header", line 9, in <module>
run_translate_header(sys.argv[1:])
File "/usr/local/lib/python3.8/site-packages/kis_tools/gris/headers/translate_header.py", line 307, in run_translate_header
files = groupby_gris_run(files)[args.run]
KeyError: 2
```
If translate header is called with a non-existent run, the tool should exit graciously. Same goes for the splitter toolhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/288IFU header WCS stepsize is a factor of 200 too small2022-01-28T15:50:53ZCarl SchafferIFU header WCS stepsize is a factor of 200 too smallCarl SchafferCarl Schafferhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/298IFU coordinates2022-01-28T15:50:52ZCarl SchafferIFU coordinatesifu coordinates have improved significantly. THe code in `IFUFitsFile`does not account for derotator presence information or STEPANGL information.
Both of these need to be evaluated and a processed before the coordinate calculation can ...ifu coordinates have improved significantly. THe code in `IFUFitsFile`does not account for derotator presence information or STEPANGL information.
Both of these need to be evaluated and a processed before the coordinate calculation can be called done.Carl SchafferCarl Schafferhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/289IFU Coordinates2022-01-28T15:50:52ZCarl SchafferIFU CoordinatesIFU coordinate calculation is *horrible* the stepsize per pixel is off by about an order of 200 and it is not clear where where the center coordinates reference off or what the angles mean.
The calculation in `kis_tools.gris.ifu_fits_f...IFU coordinate calculation is *horrible* the stepsize per pixel is off by about an order of 200 and it is not clear where where the center coordinates reference off or what the angles mean.
The calculation in `kis_tools.gris.ifu_fits_file.IFUFitsFile._coords_from_simple_header` need to be completely re-written or at least validated.
I don't have time to do this now as we need to get IFU data running for the V1 release, so the best I can do is to increase the uncertainties to a full solar disk and issue a warning each time the function is used.Carl SchafferCarl Schaffer