kis_tools issueshttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues2022-04-27T10:55:09Zhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/309Better Error messages for failure due to untranslated data2022-04-27T10:55:09ZCarl SchafferBetter Error messages for failure due to untranslated dataBoth `gris_location_plot` and `gris_gen_maps` fail when pointed at untranslated data. Te error messages do not list this as the cuse, should be improved.Both `gris_location_plot` and `gris_gen_maps` fail when pointed at untranslated data. Te error messages do not list this as the cuse, should be improved.https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/308Stepsize value might be wrong for old data2022-04-26T07:25:07ZCarl SchafferStepsize value might be wrong for old dataThe Cassda-Gui prints the following message on loading old datasets e.g. 20150603_008:
![image](/uploads/8a990cd860cc47c77c487519629b6bed/image.png)
This implies that the step size value in the header might not be correct. InvestigateThe Cassda-Gui prints the following message on loading old datasets e.g. 20150603_008:
![image](/uploads/8a990cd860cc47c77c487519629b6bed/image.png)
This implies that the step size value in the header might not be correct. Investigatehttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/304stepsizes for penumbral masks2022-03-14T07:54:31ZCarl Schafferstepsizes for penumbral masksPenumbral masks do not support variable step sizes as added in https://gitlab.leibniz-kis.de/sdc/kis_tools/-/merge_requests/233
* [ ] Let masks use WCS of first file with mask dataPenumbral masks do not support variable step sizes as added in https://gitlab.leibniz-kis.de/sdc/kis_tools/-/merge_requests/233
* [ ] Let masks use WCS of first file with mask datahttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/301translate_header gracious error code on missing run2022-02-01T14:05:35ZCarl Schaffertranslate_header gracious error code on missing run```
__ ___ __ _______. .___________. ______ ______ __ _______.
| |/ / | | / | | | / __ \ / __ \ | | / |
| ' / | | | (----` `---| |----`| | | | | | ...```
__ ___ __ _______. .___________. ______ ______ __ _______.
| |/ / | | / | | | / __ \ / __ \ | | / |
| ' / | | | (----` `---| |----`| | | | | | | | | | | (----`
| < | | \ \ | | | | | | | | | | | | \ \
| . \ | | .----) | | | | `--' | | `--' | | `----.----) |
|__|\__\ |__| |_______/ |__| \______/ \______/ |_______|_______/
Traceback (most recent call last):
File "/usr/local/bin/gris_translate_header", line 9, in <module>
run_translate_header(sys.argv[1:])
File "/usr/local/lib/python3.8/site-packages/kis_tools/gris/headers/translate_header.py", line 307, in run_translate_header
files = groupby_gris_run(files)[args.run]
KeyError: 2
```
If translate header is called with a non-existent run, the tool should exit graciously. Same goes for the splitter toolhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/299'scanedge' in SCANCNTR keyword2022-01-28T15:50:32ZCarl Schaffer'scanedge' in SCANCNTR keywordSometimes gris coordinates do not reference off of the slit center. Add a `NotImplemented` error for these cases and open an issue for futher investigation at a future point in time. There are not sufficiently many examples to warrant a ...Sometimes gris coordinates do not reference off of the slit center. Add a `NotImplemented` error for these cases and open an issue for futher investigation at a future point in time. There are not sufficiently many examples to warrant a lot of time spent.
* [ ] Ask manolo, berke or lucia about this... or reiner?Carl SchafferCarl Schafferhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/296Misleading Labels for GRIS Previews2022-02-01T16:08:40ZCarl SchafferMisleading Labels for GRIS PreviewsThe labels of the GRIS slit previews imply alignment along the X and Y axes of the Helioprojective coordinate system, while they are actually relative coordinates within the map. Make this clearer.
Also the colorbar of the plots does no...The labels of the GRIS slit previews imply alignment along the X and Y axes of the Helioprojective coordinate system, while they are actually relative coordinates within the map. Make this clearer.
Also the colorbar of the plots does not seem to match up with the data. Investigate and label/omit.
Checklist:
* [ ] Axis Lables
* [ ] Colorbar
* [ ] Re-run Preview generation
* [ ] Upload new Previews
## Excerpt of conversation:
> yakobchuk
> 4:00 PM
> Hi Carl! What's up? Don't remember if I raised this: I think gris image preview labels are confusing, if not wrong. X/Y coords on top indicate center of the frame in helio-projective coord.sys., while deltaX and deltaY labels refer to instrumental relative coords. Here is a good example: https://archive.sdc.leibniz-kis.de/SDCDetailServlet?Instrument=gris&ObjectId=5ee10fdf871689ed8458baad
> The spot is clearly stretched along Y (LAT) (see whole sun image), while on the preview image it goes along deltaX. What do you think?
>
>
> yakobchuk
> 4:21 PM
> Then regarding the colorbar: what are these numbers? I see they are much smaller than intensity data raw counts I get in grisview. Also, I think colorbar label is missing showing what is this (average intensity? polarization?) and in which units.
>
> schaffer
> schaffer
> 9:38 AM
> Hey Taras,
>
> you're completely right concerning the images, this is an issue that hadn't occured to me earlier, but it just came up to me a couple of days ago while checking the IFU coordinates. I agree that the names x and y are misleading as they don't represent actual coordinates. For slit data we should probably move to something like "parallel/horizontal" to slit. Philip also once proposed having two arrows on the image indicating the directions of Solar North and Disk center. I'll open an issue on kis_tools 👍 Thanks for the feedback.
>
> The colorbar we should probably omit or label as arbitrary units: The image data is taken from manolos .cm files, I'm not sure whether any preprocessing is done there beforehand, but there might well be something that causes a discrepancy to the values in the split files. I'll also add it to the issue.https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/295add grepfits for full calib pipeline2021-11-19T17:41:46ZCarl Schafferadd grepfits for full calib pipelinecalibration settings retrieval can be largely improved using grepfits and get last l0:
```
echo $(for i in /instruments/gris/202*;do ./gris_l0_info.sh $i;done) > last_l0_file_per_run.txt
grepfits "IMGSYS WAVELENG MEASURE TARGET FILESTAT...calibration settings retrieval can be largely improved using grepfits and get last l0:
```
echo $(for i in /instruments/gris/202*;do ./gris_l0_info.sh $i;done) > last_l0_file_per_run.txt
grepfits "IMGSYS WAVELENG MEASURE TARGET FILESTAT EXPTIME" $(cat last_l0_file_per_run.txt ) |tee $(date '+%Y-%m-%d')_gris_l0_info.csv
```
the output file can then replace the file in `kis_tools.gris.resources` and be used to generate calibration settings.
The shell scripts have been added to `kis_tools.gris.calib_settings`https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/293gris/IFU instrument segmentation2021-11-14T21:36:34ZCarl Schaffergris/IFU instrument segmentationa lot of the gris and IFU files are problematic due to their names that could fit to either instrument.
e.g. IFU continuum correction files would be added to the GRIS database by default.
This ambiguity needs to be addressed.a lot of the gris and IFU files are problematic due to their names that could fit to either instrument.
e.g. IFU continuum correction files would be added to the GRIS database by default.
This ambiguity needs to be addressed.https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/291IFU header_validation2022-04-21T10:49:01ZCarl SchafferIFU header_validation`kis_tools.gris.ifu_fits_file.IFUFitsFile.get_cleaned_header` overwrites the parent implementation and omits the call to `validate_headers`
Asess whether this is ok`kis_tools.gris.ifu_fits_file.IFUFitsFile.get_cleaned_header` overwrites the parent implementation and omits the call to `validate_headers`
Asess whether this is okhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/287Add interface to gristagger2021-09-30T10:30:05ZCarl SchafferAdd interface to gristaggerthe gristagger utility in the bin folder is undocumented even though it's pretty useful. Add a cleaner interface based on the kis_tools interface structure and add documentation.the gristagger utility in the bin folder is undocumented even though it's pretty useful. Add a cleaner interface based on the kis_tools interface structure and add documentation.https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/286Add interface to grisplot2021-09-30T10:30:06ZCarl SchafferAdd interface to grisplotthe grisplot utility in the bin folder is undocumented even though it's pretty useful. Add a cleaner interface based on the kis_tools interface structure and add documentation.the grisplot utility in the bin folder is undocumented even though it's pretty useful. Add a cleaner interface based on the kis_tools interface structure and add documentation.https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/284integrate test reports into gitlab2021-08-23T13:29:21ZCarl Schafferintegrate test reports into gitlabGitlab can parse error reports from unittest xml reports
See here for a manual:
https://gitlab.leibniz-kis.de/help/ci/unit_test_reports#how-to-set-it-upGitlab can parse error reports from unittest xml reports
See here for a manual:
https://gitlab.leibniz-kis.de/help/ci/unit_test_reports#how-to-set-it-uphttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/276refactor gdbs implementation2021-03-26T10:59:38ZCarl Schafferrefactor gdbs implementationGDBS implementation is not universal, testing is dependent on the availablity of the gdbs host, needs more flexible and clearer implementation and interfaceGDBS implementation is not universal, testing is dependent on the availablity of the gdbs host, needs more flexible and clearer implementation and interfacehttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/266gitinfo breaks pip install2021-02-22T17:05:57ZCarl Schaffergitinfo breaks pip install# Bug Report
## Summary
pip install form repo fails
## Steps to Reproduce
```
pip install git+http://gitlab.leibniz-kis.de/sdc/kis_tools
```
or
```bash
git clone http://gitlab.leibniz-kis.de/sdc/kis_tools
cd kis_tools
pip install .
``...# Bug Report
## Summary
pip install form repo fails
## Steps to Reproduce
```
pip install git+http://gitlab.leibniz-kis.de/sdc/kis_tools
```
or
```bash
git clone http://gitlab.leibniz-kis.de/sdc/kis_tools
cd kis_tools
pip install .
```
## Expected Results
install
## Actual Results
```bash
ERROR: Command errored out with exit status 1:
command: /usr/local/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-bp2qmtu2/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-bp2qmtu2/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-pnyed1_p
cwd: /tmp/pip-req-build-bp2qmtu2/
Complete output (15 lines):
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-req-build-bp2qmtu2/setup.py", line 11, in <module>
info = get_git_info(force=True)
File "/tmp/pip-req-build-bp2qmtu2/kis_tools/util/gitinfo/__init__.py", line 43, in get_git_info
write_git_info(path_in)
File "/tmp/pip-req-build-bp2qmtu2/kis_tools/util/gitinfo/__init__.py", line 25, in write_git_info
commit = git_info(dirpath)
File "/tmp/pip-req-build-bp2qmtu2/kis_tools/util/gitinfo/gitinfo.py", line 99, in get_git_info
return get_git_info_dir(gitdir)
File "/tmp/pip-req-build-bp2qmtu2/kis_tools/util/gitinfo/gitinfo.py", line 80, in get_git_info_dir
return get_pack_info(idx_file, gi)
File "/tmp/pip-req-build-bp2qmtu2/kis_tools/util/gitinfo/pack_reader.py", line 112, in get_pack_info
data = zlib.decompress(fin.read(obj_len))
zlib.error: Error -5 while decompressing data: incomplete or truncated stream
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
```https://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/265dev needs pytest-cov2020-12-11T08:29:51ZCarl Schafferdev needs pytest-covadd pytest cov to a developer extra installadd pytest cov to a developer extra installhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/264location plot image size2020-12-08T12:32:10ZCarl Schafferlocation plot image sizeLocation plot images are unnecessarily large and take very long to load with a poor internet connectionLocation plot images are unnecessarily large and take very long to load with a poor internet connectionhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/253location plot: package to nice function2020-11-18T14:45:47ZCarl Schafferlocation plot: package to nice functionThere is too much logic in the gris_locplot script in the bin/ folder. Move that to a function.There is too much logic in the gris_locplot script in the bin/ folder. Move that to a function.GRIS Pipeline Automationhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/252add documentation to calib file parsing2020-10-26T17:34:52ZCarl Schafferadd documentation to calib file parsingCarl SchafferCarl Schafferhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/245comon entry point for all fits files2022-03-15T07:55:47ZCarl Schaffercomon entry point for all fits filesImplement a common for all fits files, something like:
`open_kis_fits(filename) -> any of {GrisFitsFile, ChroTelFitsFile, BBI...)`
* [ ] add documentation about this to the base README.mdImplement a common for all fits files, something like:
`open_kis_fits(filename) -> any of {GrisFitsFile, ChroTelFitsFile, BBI...)`
* [ ] add documentation about this to the base README.mdhttps://gitlab.leibniz-kis.de/sdc/kis_tools/-/issues/243translate header: don't run git info for every file2020-09-21T11:06:29ZCarl Schaffertranslate header: don't run git info for every filegit info is currently being run for every file during translation. this is unnecessary overheadgit info is currently being run for every file during translation. this is unnecessary overheadCarl SchafferCarl Schaffer