kis_tools issueshttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues2022-04-27T10:55:09Zhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/309Better Error messages for failure due to untranslated data2022-04-27T10:55:09ZCarl SchafferBetter Error messages for failure due to untranslated dataBoth `gris_location_plot` and `gris_gen_maps` fail when pointed at untranslated data. Te error messages do not list this as the cuse, should be improved.Both `gris_location_plot` and `gris_gen_maps` fail when pointed at untranslated data. Te error messages do not list this as the cuse, should be improved.https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/308Stepsize value might be wrong for old data2022-04-26T07:25:07ZCarl SchafferStepsize value might be wrong for old dataThe Cassda-Gui prints the following message on loading old datasets e.g. 20150603_008:
![image](/uploads/8a990cd860cc47c77c487519629b6bed/image.png)
This implies that the step size value in the header might not be correct. InvestigateThe Cassda-Gui prints the following message on loading old datasets e.g. 20150603_008:
![image](/uploads/8a990cd860cc47c77c487519629b6bed/image.png)
This implies that the step size value in the header might not be correct. Investigatehttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/307use updated tag list for header translation2022-03-31T08:28:31ZCarl Schafferuse updated tag list for header translationThe heqader translator has a `to_OBS_TRGT`method that is not being used. It should be modified to be used. Also the `.csv` file in kis_tools/gris/resources containing the target tag information should be extended with Sanis tags from 202...The heqader translator has a `to_OBS_TRGT`method that is not being used. It should be modified to be used. Also the `.csv` file in kis_tools/gris/resources containing the target tag information should be extended with Sanis tags from 2022:
[missing_targets_main_tag_added.csv](/uploads/6ce45a8503f63c1443b1388488a48dcf/missing_targets_main_tag_added.csv)https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/305WCS errors for unequal step-sizes2022-03-14T17:37:09ZCarl SchafferWCS errors for unequal step-sizes![image](/uploads/3f72dc7616b40682f70569e5c96d3d30/image.png)
Gris Coordinates recovered from the first wcs sometimes will go in the wrong direction. Also ther is an offset between the coordinates fed into the WCS generator and the ones...![image](/uploads/3f72dc7616b40682f70569e5c96d3d30/image.png)
Gris Coordinates recovered from the first wcs sometimes will go in the wrong direction. Also ther is an offset between the coordinates fed into the WCS generator and the ones reconstructed from the header. It's something with the rotation of the different step widths in X and Y direction. Cant't figure it out, spent one week now.
to reproduce run test_coords /test_wcs for commit 50ee44ce819c3229adb91e2d425e26d4fdbe0b9bhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/303skewed boxes in location previews for 2021 data2022-03-14T17:44:20ZCarl Schafferskewed boxes in location previews for 2021 data![image](/uploads/6155a8922f12d28ec42010d5fbb93f41/image.png)
![image](/uploads/e1245cbcd549c057d94ccb78cf144c80/image.png)
![image](/uploads/307d603ecb23984b5c8f376750ea45b2/image.png)![image](/uploads/6155a8922f12d28ec42010d5fbb93f41/image.png)
![image](/uploads/e1245cbcd549c057d94ccb78cf144c80/image.png)
![image](/uploads/307d603ecb23984b5c8f376750ea45b2/image.png)Carl SchafferCarl Schafferhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/302previews: respect aspect ratio step size2022-02-21T12:21:22ZCarl Schafferpreviews: respect aspect ratio step sizeSome observations use a step size perpendicular larger than the resolution along the slit. This leads to skewed preview images, as the pixels would essentially need to be non square.
matplotlib supports this, but it also needs to be che...Some observations use a step size perpendicular larger than the resolution along the slit. This leads to skewed preview images, as the pixels would essentially need to be non square.
matplotlib supports this, but it also needs to be checked through the entire chain of calculations and plotting:
The preview plots should use the wcs keywords to determine the values for the aspect ratio and spatial extent of FOVs.
Example:
![image](/uploads/48b580236f2776f341cb7d7f290d906a/image.png)
https://archive.sdc.leibniz-kis.de/SDCDetailServlet?Instrument=gris&ObjectId=607c9dcd18262e0426a2b2a2
Also see the corresponding issue on [Jira](https://leibniz-kis.atlassian.net/jira/servicedesk/projects/SR/queues/custom/10/SR-173)https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/301translate_header gracious error code on missing run2022-02-01T14:05:35ZCarl Schaffertranslate_header gracious error code on missing run```
__ ___ __ _______. .___________. ______ ______ __ _______.
| |/ / | | / | | | / __ \ / __ \ | | / |
| ' / | | | (----` `---| |----`| | | | | | ...```
__ ___ __ _______. .___________. ______ ______ __ _______.
| |/ / | | / | | | / __ \ / __ \ | | / |
| ' / | | | (----` `---| |----`| | | | | | | | | | | (----`
| < | | \ \ | | | | | | | | | | | | \ \
| . \ | | .----) | | | | `--' | | `--' | | `----.----) |
|__|\__\ |__| |_______/ |__| \______/ \______/ |_______|_______/
Traceback (most recent call last):
File "/usr/local/bin/gris_translate_header", line 9, in <module>
run_translate_header(sys.argv[1:])
File "/usr/local/lib/python3.8/site-packages/kis_tools/gris/headers/translate_header.py", line 307, in run_translate_header
files = groupby_gris_run(files)[args.run]
KeyError: 2
```
If translate header is called with a non-existent run, the tool should exit graciously. Same goes for the splitter toolhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/299'scanedge' in SCANCNTR keyword2022-01-28T15:50:32ZCarl Schaffer'scanedge' in SCANCNTR keywordSometimes gris coordinates do not reference off of the slit center. Add a `NotImplemented` error for these cases and open an issue for futher investigation at a future point in time. There are not sufficiently many examples to warrant a ...Sometimes gris coordinates do not reference off of the slit center. Add a `NotImplemented` error for these cases and open an issue for futher investigation at a future point in time. There are not sufficiently many examples to warrant a lot of time spent.
* [ ] Ask manolo, berke or lucia about this... or reiner?Carl SchafferCarl Schafferhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/297GRIS data is flipped depending on step-angle2022-04-21T10:46:49ZCarl SchafferGRIS data is flipped depending on step-angleDepending on how the slit data is scanned, images are flipped w.r.t. HMI.
We should consider serving flipped data to the end-users instead of what we're doing right now.
* [ ] Check whether this is already done somewhere
* [ ] Evaluat...Depending on how the slit data is scanned, images are flipped w.r.t. HMI.
We should consider serving flipped data to the end-users instead of what we're doing right now.
* [ ] Check whether this is already done somewhere
* [ ] Evaluate how much effort a fix would be
* [ ] decide on actionCarl SchafferCarl Schaffer2021-11-30https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/296Misleading Labels for GRIS Previews2022-02-01T16:08:40ZCarl SchafferMisleading Labels for GRIS PreviewsThe labels of the GRIS slit previews imply alignment along the X and Y axes of the Helioprojective coordinate system, while they are actually relative coordinates within the map. Make this clearer.
Also the colorbar of the plots does no...The labels of the GRIS slit previews imply alignment along the X and Y axes of the Helioprojective coordinate system, while they are actually relative coordinates within the map. Make this clearer.
Also the colorbar of the plots does not seem to match up with the data. Investigate and label/omit.
Checklist:
* [ ] Axis Lables
* [ ] Colorbar
* [ ] Re-run Preview generation
* [ ] Upload new Previews
## Excerpt of conversation:
> yakobchuk
> 4:00 PM
> Hi Carl! What's up? Don't remember if I raised this: I think gris image preview labels are confusing, if not wrong. X/Y coords on top indicate center of the frame in helio-projective coord.sys., while deltaX and deltaY labels refer to instrumental relative coords. Here is a good example: https://archive.sdc.leibniz-kis.de/SDCDetailServlet?Instrument=gris&ObjectId=5ee10fdf871689ed8458baad
> The spot is clearly stretched along Y (LAT) (see whole sun image), while on the preview image it goes along deltaX. What do you think?
>
>
> yakobchuk
> 4:21 PM
> Then regarding the colorbar: what are these numbers? I see they are much smaller than intensity data raw counts I get in grisview. Also, I think colorbar label is missing showing what is this (average intensity? polarization?) and in which units.
>
> schaffer
> schaffer
> 9:38 AM
> Hey Taras,
>
> you're completely right concerning the images, this is an issue that hadn't occured to me earlier, but it just came up to me a couple of days ago while checking the IFU coordinates. I agree that the names x and y are misleading as they don't represent actual coordinates. For slit data we should probably move to something like "parallel/horizontal" to slit. Philip also once proposed having two arrows on the image indicating the directions of Solar North and Disk center. I'll open an issue on kis_tools 👍 Thanks for the feedback.
>
> The colorbar we should probably omit or label as arbitrary units: The image data is taken from manolos .cm files, I'm not sure whether any preprocessing is done there beforehand, but there might well be something that causes a discrepancy to the values in the split files. I'll also add it to the issue.https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/295add grepfits for full calib pipeline2021-11-19T17:41:46ZCarl Schafferadd grepfits for full calib pipelinecalibration settings retrieval can be largely improved using grepfits and get last l0:
```
echo $(for i in /instruments/gris/202*;do ./gris_l0_info.sh $i;done) > last_l0_file_per_run.txt
grepfits "IMGSYS WAVELENG MEASURE TARGET FILESTAT...calibration settings retrieval can be largely improved using grepfits and get last l0:
```
echo $(for i in /instruments/gris/202*;do ./gris_l0_info.sh $i;done) > last_l0_file_per_run.txt
grepfits "IMGSYS WAVELENG MEASURE TARGET FILESTAT EXPTIME" $(cat last_l0_file_per_run.txt ) |tee $(date '+%Y-%m-%d')_gris_l0_info.csv
```
the output file can then replace the file in `kis_tools.gris.resources` and be used to generate calibration settings.
The shell scripts have been added to `kis_tools.gris.calib_settings`https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/287Add interface to gristagger2021-09-30T10:30:05ZCarl SchafferAdd interface to gristaggerthe gristagger utility in the bin folder is undocumented even though it's pretty useful. Add a cleaner interface based on the kis_tools interface structure and add documentation.the gristagger utility in the bin folder is undocumented even though it's pretty useful. Add a cleaner interface based on the kis_tools interface structure and add documentation.https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/286Add interface to grisplot2021-09-30T10:30:06ZCarl SchafferAdd interface to grisplotthe grisplot utility in the bin folder is undocumented even though it's pretty useful. Add a cleaner interface based on the kis_tools interface structure and add documentation.the grisplot utility in the bin folder is undocumented even though it's pretty useful. Add a cleaner interface based on the kis_tools interface structure and add documentation.https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/282wcs wavelength retrieval shifted by one pixel2021-05-06T14:48:42ZCarl Schafferwcs wavelength retrieval shifted by one pixelPhilip pointed out to me that the wcs retrieval of wavelengths might be off by one pixel. We narrowed the problem down to the definition of the `CRPIX`keyword. We set it to `0` meaning to indicate the beginning of the array, but using th...Philip pointed out to me that the wcs retrieval of wavelengths might be off by one pixel. We narrowed the problem down to the definition of the `CRPIX`keyword. We set it to `0` meaning to indicate the beginning of the array, but using the `WCS.wcs_pix2world(pixels,0)` method to retrieve the value yields a wavelength array with larger by `CDELT`than we would expect.
We either need to set the CRPIX keyword to 1 or use 1 for the origin value of the retrieval method.
It is unclear which of these solutions should be preferred.
Links:
* [related issue on github](https://github.com/astropy/astropy/issues/9989)
* [wcs paper](https://www.aanda.org/articles/aa/pdf/2002/45/aah3859.pdf)
* [astropy docs for WCS.wcs_pix2world](https://docs.astropy.org/en/stable/api/astropy.wcs.WCS.html#astropy.wcs.WCS.wcs_pix2world)Carl SchafferCarl Schafferhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/280gris headers changes2021-10-11T16:36:09ZCarl Schaffergris headers changes* [x] Origin: KIS
* [x] Timeunit for exposure times/ change time values to seconds* [x] Origin: KIS
* [x] Timeunit for exposure times/ change time values to secondshttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/279gris header: AO_MODES=02021-10-11T16:01:08ZCarl Schaffergris header: AO_MODES=0Corrected modes by ao are set to 0 in the sample file I sent to Oslo, we should check where that Value comes from and whether it's correct. If we don't know, we should probably switch to NAN or something similar.Corrected modes by ao are set to 0 in the sample file I sent to Oslo, we should check where that Value comes from and whether it's correct. If we don't know, we should probably switch to NAN or something similar.https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/271clean GRIS header2021-03-22T08:26:49ZCarl Schafferclean GRIS headerAfter discussing with Manolo and Carlos while working on [the slit orientation issue](https://leibniz-kis.atlassian.net/browse/SR-99?atlOrigin=eyJpIjoiMTE2ZjAzN2EwNjlhNDc5MmFiYTdmMDYzZTNkOGYyOTkiLCJwIjoiaiJ9) I found out that some of the...After discussing with Manolo and Carlos while working on [the slit orientation issue](https://leibniz-kis.atlassian.net/browse/SR-99?atlOrigin=eyJpIjoiMTE2ZjAzN2EwNjlhNDc5MmFiYTdmMDYzZTNkOGYyOTkiLCJwIjoiaiJ9) I found out that some of the values we are propagating in the GRIS header are not correct, they might:
* be historic values such as `SLITORIE` which do not have a well-defined or obvious function
* be values that apply only to observations but not single slit positions (e.g. `AZIMUT`, `ELEVATION`)
* be unreliable due to different ephemerides code used for producing them/not be compliant with the telescope's definitions (again `AZIMUTH`)
For fully consistent headers, we would need to go through each value, assess the definition and validity, document that and throw out all confusing values.https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/269headers gris nmaps2021-04-22T10:02:39ZCarl Schafferheaders gris nmapsCheck if NMAPS is corrected in cases of trailing observations?
@yakobchuk reported that it doesn't always match for the split files.Check if NMAPS is corrected in cases of trailing observations?
@yakobchuk reported that it doesn't always match for the split files.WCS fixesCarl SchafferCarl Schafferhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/262location plot: 20150417 run 2 Uncertainties in locplot should be small2021-03-23T09:13:50ZCarl Schafferlocation plot: 20150417 run 2 Uncertainties in locplot should be smallThe plot is made with large uncertainties, even though there is a cross correlation result with HMI. Check!The plot is made with large uncertainties, even though there is a cross correlation result with HMI. Check!WCS fixesCarl SchafferCarl Schafferhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/259GRIS Coordinates without cross correlation2021-02-11T17:32:42ZCarl SchafferGRIS Coordinates without cross correlation
In general we want to have a full cross correlation of coordinates fro GRIS maps. This is the preferred method of getting coordinates for WCS keywords. Back when we did this, we assumed SLITPOS_X and SLITPOS_Y to give coordinates in a ...
In general we want to have a full cross correlation of coordinates fro GRIS maps. This is the preferred method of getting coordinates for WCS keywords. Back when we did this, we assumed SLITPOS_X and SLITPOS_Y to give coordinates in a system that is aligned with the solar axis and for that matter with the HMI system.
Looking at the location guesses from the Mercury transit of last year, it seems very likely, that there is an additional rotation between the systems that we don't account for. This would also affect our uncertainty estimates for non-correlated files. See https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/199
![transit](/uploads/9e4bb8c70cef34cb3b68c6fa0964b603/transit.gif)WCS fixes