kis_tools issueshttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues2020-06-04T12:23:19Zhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/69Gris Run was aborted not robust2020-06-04T12:23:19ZCarl SchafferGris Run was aborted not robustif first map was aborted, is doesnt workif first map was aborted, is doesnt workStreamlining Codebasehttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/71namespace cleaning2019-12-04T09:42:58ZCarl Schaffernamespace cleaningmove gris tools to sdc_tools. gris namespace etc.move gris tools to sdc_tools. gris namespace etc.Streamlining Codebasehttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/73GrisArchive Dataframe does not create runs if no level1 data is present2020-12-22T13:33:28ZCarl SchafferGrisArchive Dataframe does not create runs if no level1 data is presentbug, problematic when working on Lindner databug, problematic when working on Lindner dataremove importer utilitieshttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/110importer gris: error logging not working2020-04-09T08:03:16ZCarl Schafferimporter gris: error logging not workingLogging does not seem to work correctly... Errors occurr but the error log files are emptyLogging does not seem to work correctly... Errors occurr but the error log files are emptyhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/138GrisFitsFile slit coords full slit instead of center2019-12-03T17:17:04ZCarl SchafferGrisFitsFile slit coords full slit instead of center# Bug Report
## Summary
`slit_coords` method provides center instead of full slit coordinate array
## Steps to Reproduce
get an instance of GrisFitsFile and access the `slit_coords` method
## Expected Results
array of coords
## Actual R...# Bug Report
## Summary
`slit_coords` method provides center instead of full slit coordinate array
## Steps to Reproduce
get an instance of GrisFitsFile and access the `slit_coords` method
## Expected Results
array of coords
## Actual Results
single pointGRIS CoordinatesCarl SchafferCarl Schafferhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/188Gris PhD Embargos2020-05-12T09:46:26ZCarl SchafferGris PhD EmbargosWrite list of observers per day to NBG, she will check which dates should be embargoed.Write list of observers per day to NBG, she will check which dates should be embargoed.Demleitner VisitCarl SchafferCarl Schaffer2020-05-11https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/194GRIS Importer: New Map Previews2020-02-26T14:41:17ZCarl SchafferGRIS Importer: New Map PreviewsNew Map previews have been implemented, but so far, the importer doesn't use them by default.New Map previews have been implemented, but so far, the importer doesn't use them by default.https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/195GRIS Location: Indicate uncertainty2019-12-03T17:16:58ZCarl SchafferGRIS Location: Indicate uncertaintyIndicate the uncertainty of the measurements in the location plots.Indicate the uncertainty of the measurements in the location plots.GRIS CoordinatesCarl SchafferCarl Schafferhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/196GRIS importer: add masks2020-04-09T08:01:52ZCarl SchafferGRIS importer: add maskshttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/198GRISFitsFile: Re-think Coordinate access2019-12-05T10:44:57ZCarl SchafferGRISFitsFile: Re-think Coordinate accessRe think coordinate access:
plotting of slit should show:
* coords from wcs
* coords from slitpos and slitorieRe think coordinate access:
plotting of slit should show:
* coords from wcs
* coords from slitpos and slitorieGRIS Coordinateshttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/201GrisRun bounding Box2019-12-05T10:44:54ZCarl SchafferGrisRun bounding BoxThe bounding box for gris run is calculated from the centers of the slits instead of the edges of the slits.The bounding box for gris run is calculated from the centers of the slits instead of the edges of the slits.GRIS Coordinateshttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/203GRIS: Cross correlation file structure2019-12-12T09:25:18ZCarl SchafferGRIS: Cross correlation file structureThere are different structures for the cross correlation files. sometimes `slit_orientation` is used, sometimes `slit_angle` this should be made uniform.There are different structures for the cross correlation files. sometimes `slit_orientation` is used, sometimes `slit_angle` this should be made uniform.GRIS CoordinatesCarl SchafferCarl Schafferhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/218Gris: Some runs not translated2020-05-27T11:54:13ZCarl SchafferGris: Some runs not translatedSome runs do not seem to be fully translated:
![image](/uploads/847f0203e1fd5ad24816ffa21726e518/image.png)
List of the first split files of these runs:
gris_20140503_140502_l1p_012_001_0001.fits
gris_20140503_145531_l1p_015...Some runs do not seem to be fully translated:
![image](/uploads/847f0203e1fd5ad24816ffa21726e518/image.png)
List of the first split files of these runs:
gris_20140503_140502_l1p_012_001_0001.fits
gris_20140503_145531_l1p_015_001_0001.fits
gris_20140503_154906_l1p_018_001_0001.fits
gris_20140503_161857_l1p_020_001_0001.fits
gris_20140503_162943_l1p_021_001_0001.fits
gris_20140503_164102_l1p_022_001_0001.fits
gris_20140503_165223_l1p_023_001_0001.fits
gris_20140503_171735_l1p_026_001_0001.fits
gris_20140503_174110_l1p_027_001_0001.fits
gris_20150603_082644_l1p_004_001_0001.fits
gris_20150603_084246_l1p_005_001_0001.fits
gris_20150603_091312_l1p_006_001_0001.fits
gris_20150603_095403_l1p_008_001_0001.fits
gris_20150603_102339_l1p_009_001_0001.fits
gris_20160602_075844_l1p_001_001_0001.fits
gris_20160602_081620_l1p_002_001_0001.fits
gris_20160602_095847_l1p_006_001_0001.fits
gris_20160602_101825_l1p_007_001_0001.fits
gris_20170616_092551_l1p_005_001_0001.fits
gris_20170616_103511_l1p_007_001_0001.fits
gris_20180504_155639_l1p_016_001_0001.fits
gris_20180504_155905_l1p_017_001_0001.fits
gris_20180504_163932_l1p_020_001_0001.fits
gris_20180504_170742_l1p_021_001_0001.fitshttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/239mapgen: plot in HPC2020-10-16T09:52:20ZCarl Schaffermapgen: plot in HPCGenerate maps in helioprojective coordinates instead of pixel coords unse Sunpy Maps for thisGenerate maps in helioprojective coordinates instead of pixel coords unse Sunpy Maps for thishttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/243translate header: don't run git info for every file2020-09-21T11:06:29ZCarl Schaffertranslate header: don't run git info for every filegit info is currently being run for every file during translation. this is unnecessary overheadgit info is currently being run for every file during translation. this is unnecessary overheadCarl SchafferCarl Schafferhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/247gris run: Coord save from other runs2020-10-01T15:19:35ZCarl Schaffergris run: Coord save from other runsGrisRuns `coord_saves` property returns saves for other runs, this shouldn't happen.
![Selection_164](/uploads/f46e62d437aea9f49bf6dd63ea358011/Selection_164.png)GrisRuns `coord_saves` property returns saves for other runs, this shouldn't happen.
![Selection_164](/uploads/f46e62d437aea9f49bf6dd63ea358011/Selection_164.png)https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/254Gris WCS: CTYPE should be HP instead of HG2021-02-11T17:34:08ZCarl SchafferGris WCS: CTYPE should be HP instead of HGExcempt from rocketchat:
> Vigeesh:
> Question about GRIS data header
> ```
> WCSNAME = 'Helioprojective Cartesian' / nan
> CTYPE1 = 'HGLN-TAN' / Type of coordinates along axis 1
> CUNIT1 = 'arcsec ' / Units alon...Excempt from rocketchat:
> Vigeesh:
> Question about GRIS data header
> ```
> WCSNAME = 'Helioprojective Cartesian' / nan
> CTYPE1 = 'HGLN-TAN' / Type of coordinates along axis 1
> CUNIT1 = 'arcsec ' / Units along axis 1
> CRPIX1 = 0 / Reference pixel
> CRVAL1 = -393.014 / Value at reference pixel on axis 1
> CDELT1 = 0.027174 / Sampling along axis 1
> ```
> HGLN-TAN according to Thompson et al refers to Stonyhurst heliographic longitude
> so, I'm a bit confused about why it is called "Helioprojective Cartesian"?
> also the CDELT1 is in HGLN-TAN or HPLN-TAN?
>
> schaffer
> 10:56 AM
> You're right, that is inconsistent. I believe CTYPE should be HPLT-TAN, at least that is what is being done under the hood. I'll open an issue and correct this in all files sometime in the near future.
* [x] fix code
* [x] re-format files
* [ ] re-upload files to archiveWCS fixeshttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/256GRIS header: RSUN_REF should be in meters2020-12-08T08:33:32ZVigeesh GangadharanGRIS header: RSUN_REF should be in metersIt seems like the RSUN_REF is in arcsec, but should be in meters ([Thomson et al 2010](https://ui.adsabs.harvard.edu/abs/2010A&A...515A..59T/abstract)).<br>It seems like the RSUN_REF is in arcsec, but should be in meters ([Thomson et al 2010](https://ui.adsabs.harvard.edu/abs/2010A&A...515A..59T/abstract)).<br>https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/257CTYPE inconsistency in header for data in spectroscopic mode2021-02-11T17:33:36ZVigeesh GangadharanCTYPE inconsistency in header for data in spectroscopic modeFor dataset in spectroscopic mode, the first axis seems to be always wavelength.<br>
But this is not consistent with the CTYPE-_i_ keywords later.
e.g.
```
SIMPLE = T / conforms to FITS standard ...For dataset in spectroscopic mode, the first axis seems to be always wavelength.<br>
But this is not consistent with the CTYPE-_i_ keywords later.
e.g.
```
SIMPLE = T / conforms to FITS standard
BITPIX = 32 / array data type
NAXIS = 3 / number of array dimensions
NAXIS1 = 1010 / Length of data axis 1
NAXIS2 = 484 / Length of data axis 2
NAXIS3 = 1 / Length of data axis 3
FILENAME= 'gris_20160613_120408_l1s_008_120_0029.fits' / Name of file
EXTNAME = '20160613_008_120_0029' / Unique HDU name
POINT_ID= '20160613_008' / Unique (re-)pointing ID
OBS_TRGT= 'Sunspot(s)' / Observation Target
WCSNAME = 'Helioprojective Cartesian' / nan
CTYPE1 = 'HGLN-TAN' / Type of coordinates along axis 1
CUNIT1 = 'arcsec ' / Units along axis 1
CRPIX1 = 0 / Reference pixel
CRVAL1 = 297.671 / Value at reference pixel on axis 1
CDELT1 = 0.080792 / Sampling along axis 1
CSYER1 = 1900 / Systematic Error along axis 1
CTYPE2 = 'HGLT-TAN' / Type of coordinates along axis 2
CUNIT2 = 'arcsec ' / Units along axis 2
CRPIX2 = 0 / Reference pixel
CRVAL2 = 408.258 / Value at reference pixel on axis 2
CDELT2 = 0.080792 / Sampling along axis 2
CSYER2 = 1900 / Systematic Error along axis 2
CTYPE3 = 'WAVE ' / Type of coordinates along axis 3
CUNIT3 = 'Angstrom' / Units along axis 3
CRPIX3 = 0 / Reference pixel
CRVAL3 = 10823.076 / Value at reference pixel on axis 3
CDELT3 = 0.017998 / Sampling along axis 3
CSYER3 = 0.017998 / Systematic Error along axis 3
```WCS fixesCarl SchafferCarl Schafferhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/258GRIS Header Observer Metadata2021-02-23T10:01:48ZCarl SchafferGRIS Header Observer Metadata![Clipboard_-_November_26__2020_2_54_PM](/uploads/100ec7c094c94884841d2cfe7fccea67/Clipboard_-_November_26__2020_2_54_PM.png)
We should add the HGLN_obs and dsun_ref keywords to the header to silence the astropy warning![Clipboard_-_November_26__2020_2_54_PM](/uploads/100ec7c094c94884841d2cfe7fccea67/Clipboard_-_November_26__2020_2_54_PM.png)
We should add the HGLN_obs and dsun_ref keywords to the header to silence the astropy warningWCS fixesCarl SchafferCarl Schafferhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/259GRIS Coordinates without cross correlation2021-02-11T17:32:42ZCarl SchafferGRIS Coordinates without cross correlation
In general we want to have a full cross correlation of coordinates fro GRIS maps. This is the preferred method of getting coordinates for WCS keywords. Back when we did this, we assumed SLITPOS_X and SLITPOS_Y to give coordinates in a ...
In general we want to have a full cross correlation of coordinates fro GRIS maps. This is the preferred method of getting coordinates for WCS keywords. Back when we did this, we assumed SLITPOS_X and SLITPOS_Y to give coordinates in a system that is aligned with the solar axis and for that matter with the HMI system.
Looking at the location guesses from the Mercury transit of last year, it seems very likely, that there is an additional rotation between the systems that we don't account for. This would also affect our uncertainty estimates for non-correlated files. See https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/199
![transit](/uploads/9e4bb8c70cef34cb3b68c6fa0964b603/transit.gif)WCS fixeshttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/262location plot: 20150417 run 2 Uncertainties in locplot should be small2021-03-23T09:13:50ZCarl Schafferlocation plot: 20150417 run 2 Uncertainties in locplot should be smallThe plot is made with large uncertainties, even though there is a cross correlation result with HMI. Check!The plot is made with large uncertainties, even though there is a cross correlation result with HMI. Check!WCS fixesCarl SchafferCarl Schafferhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/269headers gris nmaps2021-04-22T10:02:39ZCarl Schafferheaders gris nmapsCheck if NMAPS is corrected in cases of trailing observations?
@yakobchuk reported that it doesn't always match for the split files.Check if NMAPS is corrected in cases of trailing observations?
@yakobchuk reported that it doesn't always match for the split files.WCS fixesCarl SchafferCarl Schafferhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/279gris header: AO_MODES=02021-10-11T16:01:08ZCarl Schaffergris header: AO_MODES=0Corrected modes by ao are set to 0 in the sample file I sent to Oslo, we should check where that Value comes from and whether it's correct. If we don't know, we should probably switch to NAN or something similar.Corrected modes by ao are set to 0 in the sample file I sent to Oslo, we should check where that Value comes from and whether it's correct. If we don't know, we should probably switch to NAN or something similar.https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/280gris headers changes2021-10-11T16:36:09ZCarl Schaffergris headers changes* [x] Origin: KIS
* [x] Timeunit for exposure times/ change time values to seconds* [x] Origin: KIS
* [x] Timeunit for exposure times/ change time values to secondshttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/282wcs wavelength retrieval shifted by one pixel2021-05-06T14:48:42ZCarl Schafferwcs wavelength retrieval shifted by one pixelPhilip pointed out to me that the wcs retrieval of wavelengths might be off by one pixel. We narrowed the problem down to the definition of the `CRPIX`keyword. We set it to `0` meaning to indicate the beginning of the array, but using th...Philip pointed out to me that the wcs retrieval of wavelengths might be off by one pixel. We narrowed the problem down to the definition of the `CRPIX`keyword. We set it to `0` meaning to indicate the beginning of the array, but using the `WCS.wcs_pix2world(pixels,0)` method to retrieve the value yields a wavelength array with larger by `CDELT`than we would expect.
We either need to set the CRPIX keyword to 1 or use 1 for the origin value of the retrieval method.
It is unclear which of these solutions should be preferred.
Links:
* [related issue on github](https://github.com/astropy/astropy/issues/9989)
* [wcs paper](https://www.aanda.org/articles/aa/pdf/2002/45/aah3859.pdf)
* [astropy docs for WCS.wcs_pix2world](https://docs.astropy.org/en/stable/api/astropy.wcs.WCS.html#astropy.wcs.WCS.wcs_pix2world)Carl SchafferCarl Schafferhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/286Add interface to grisplot2021-09-30T10:30:06ZCarl SchafferAdd interface to grisplotthe grisplot utility in the bin folder is undocumented even though it's pretty useful. Add a cleaner interface based on the kis_tools interface structure and add documentation.the grisplot utility in the bin folder is undocumented even though it's pretty useful. Add a cleaner interface based on the kis_tools interface structure and add documentation.https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/287Add interface to gristagger2021-09-30T10:30:05ZCarl SchafferAdd interface to gristaggerthe gristagger utility in the bin folder is undocumented even though it's pretty useful. Add a cleaner interface based on the kis_tools interface structure and add documentation.the gristagger utility in the bin folder is undocumented even though it's pretty useful. Add a cleaner interface based on the kis_tools interface structure and add documentation.https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/296Misleading Labels for GRIS Previews2022-02-01T16:08:40ZCarl SchafferMisleading Labels for GRIS PreviewsThe labels of the GRIS slit previews imply alignment along the X and Y axes of the Helioprojective coordinate system, while they are actually relative coordinates within the map. Make this clearer.
Also the colorbar of the plots does no...The labels of the GRIS slit previews imply alignment along the X and Y axes of the Helioprojective coordinate system, while they are actually relative coordinates within the map. Make this clearer.
Also the colorbar of the plots does not seem to match up with the data. Investigate and label/omit.
Checklist:
* [ ] Axis Lables
* [ ] Colorbar
* [ ] Re-run Preview generation
* [ ] Upload new Previews
## Excerpt of conversation:
> yakobchuk
> 4:00 PM
> Hi Carl! What's up? Don't remember if I raised this: I think gris image preview labels are confusing, if not wrong. X/Y coords on top indicate center of the frame in helio-projective coord.sys., while deltaX and deltaY labels refer to instrumental relative coords. Here is a good example: https://archive.sdc.leibniz-kis.de/SDCDetailServlet?Instrument=gris&ObjectId=5ee10fdf871689ed8458baad
> The spot is clearly stretched along Y (LAT) (see whole sun image), while on the preview image it goes along deltaX. What do you think?
>
>
> yakobchuk
> 4:21 PM
> Then regarding the colorbar: what are these numbers? I see they are much smaller than intensity data raw counts I get in grisview. Also, I think colorbar label is missing showing what is this (average intensity? polarization?) and in which units.
>
> schaffer
> schaffer
> 9:38 AM
> Hey Taras,
>
> you're completely right concerning the images, this is an issue that hadn't occured to me earlier, but it just came up to me a couple of days ago while checking the IFU coordinates. I agree that the names x and y are misleading as they don't represent actual coordinates. For slit data we should probably move to something like "parallel/horizontal" to slit. Philip also once proposed having two arrows on the image indicating the directions of Solar North and Disk center. I'll open an issue on kis_tools 👍 Thanks for the feedback.
>
> The colorbar we should probably omit or label as arbitrary units: The image data is taken from manolos .cm files, I'm not sure whether any preprocessing is done there beforehand, but there might well be something that causes a discrepancy to the values in the split files. I'll also add it to the issue.https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/297GRIS data is flipped depending on step-angle2022-04-21T10:46:49ZCarl SchafferGRIS data is flipped depending on step-angleDepending on how the slit data is scanned, images are flipped w.r.t. HMI.
We should consider serving flipped data to the end-users instead of what we're doing right now.
* [ ] Check whether this is already done somewhere
* [ ] Evaluat...Depending on how the slit data is scanned, images are flipped w.r.t. HMI.
We should consider serving flipped data to the end-users instead of what we're doing right now.
* [ ] Check whether this is already done somewhere
* [ ] Evaluate how much effort a fix would be
* [ ] decide on actionCarl SchafferCarl Schaffer2021-11-30https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/299'scanedge' in SCANCNTR keyword2022-01-28T15:50:32ZCarl Schaffer'scanedge' in SCANCNTR keywordSometimes gris coordinates do not reference off of the slit center. Add a `NotImplemented` error for these cases and open an issue for futher investigation at a future point in time. There are not sufficiently many examples to warrant a ...Sometimes gris coordinates do not reference off of the slit center. Add a `NotImplemented` error for these cases and open an issue for futher investigation at a future point in time. There are not sufficiently many examples to warrant a lot of time spent.
* [ ] Ask manolo, berke or lucia about this... or reiner?Carl SchafferCarl Schafferhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/302previews: respect aspect ratio step size2022-02-21T12:21:22ZCarl Schafferpreviews: respect aspect ratio step sizeSome observations use a step size perpendicular larger than the resolution along the slit. This leads to skewed preview images, as the pixels would essentially need to be non square.
matplotlib supports this, but it also needs to be che...Some observations use a step size perpendicular larger than the resolution along the slit. This leads to skewed preview images, as the pixels would essentially need to be non square.
matplotlib supports this, but it also needs to be checked through the entire chain of calculations and plotting:
The preview plots should use the wcs keywords to determine the values for the aspect ratio and spatial extent of FOVs.
Example:
![image](/uploads/48b580236f2776f341cb7d7f290d906a/image.png)
https://archive.sdc.leibniz-kis.de/SDCDetailServlet?Instrument=gris&ObjectId=607c9dcd18262e0426a2b2a2
Also see the corresponding issue on [Jira](https://leibniz-kis.atlassian.net/jira/servicedesk/projects/SR/queues/custom/10/SR-173)https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/303skewed boxes in location previews for 2021 data2022-03-14T17:44:20ZCarl Schafferskewed boxes in location previews for 2021 data![image](/uploads/6155a8922f12d28ec42010d5fbb93f41/image.png)
![image](/uploads/e1245cbcd549c057d94ccb78cf144c80/image.png)
![image](/uploads/307d603ecb23984b5c8f376750ea45b2/image.png)![image](/uploads/6155a8922f12d28ec42010d5fbb93f41/image.png)
![image](/uploads/e1245cbcd549c057d94ccb78cf144c80/image.png)
![image](/uploads/307d603ecb23984b5c8f376750ea45b2/image.png)Carl SchafferCarl Schaffer