README.md 6.07 KB
Newer Older
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
1
# SDC:GRIS Inversion Pipeline
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
2

Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
3
4
This is the SDC pipeline code for performing inversions of GRIS spectropolarimeteric data.<br>
The code currently uses the [Very Fast Inversion of the Stokes Vector (VFISV)](https://gitlab.leibniz-kis.de/borrero/vfisv_spec) code v5.0 (node for spectrograph data)<br>
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
5
6
7
as the main backend to carry out a Milne-Eddington Stokes inversion for individual spectral lines.<br>
The manual for the original code can be found here: [manual_vfisv](https://gitlab.leibniz-kis.de/borrero/vfisv_spec/-/blob/master/manual_vfisv.pdf)<br>

Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
8
This is an implementation of VFISV code to facilitate a seamless processing of the data provided by the SDC.
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
9

10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

## Implementation

The current implementation of the pipeline is a modified version of VFISV code to work with the GRIS data. The GRIS L1 header and Stokes data is extracted
using a Python core module and sent to a bare VFISV via an MPI intercommunicator. The inversion is performed using VFISV
and the buffer with the inversion results is communicated back to the Python module. The Python module propagates the
keywords from L1 and packages the inversion results and outputs a FITS file (when used as a command-line interface) or
returns an NDarray (when called within a script).

## Install and usage

You can either use Docker or install it natively.

### 1. Using Docker

25
Go to the directory where the data files (`level1split`) are located.
26
27
28
29

```shell
docker run -it --rm -v $PWD:/home/grisuser ghcr.io/vigeesh/sdc-grisinv
```
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
30
This will run/pull the latest image from the container repo.
31

Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
32
Within the new shell,
33
34
35
36
37
38
39
40
41
42
```shell
mpiexec -n 1 vfisv \
  --path='data/' \
  --id=3 \
  --line=15648.514 \
  --numproc=20 \
  --width=1.8 \
  --out='output.fits'
```

Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
43

44
45
46
### 2. Native installation

#### Requirements
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
47

Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
48
It is highly recommended to install the pipeline in a new `conda` environment. <br>
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
49

Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
50
If you don't have conda installed, check [Miniconda](https://docs.conda.io/en/latest/miniconda.html)  for a minimal installer for conda.<br>
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
51
To install Miniconda:
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
52

Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
53
54
55
56
```sh
wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh
```
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
57
when prompted, install it in your home path, e.g. ~/conda
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
58

Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
59
Conda is cross-language and can install non-Python libraries and tools (compilers, OpenMPI) in the user space.<br>
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
60
61
62
63
We recommend, you use the conda-forge channel for installing the required packages.
> Note: In the conda-forge channel, NumPy is built against a dummy “BLAS” package. <br> When a user installs NumPy from conda-forge, that BLAS package then gets  <br> installed together with the actual library - this defaults to OpenBLAS ... <br> (see [Numpy Documentation](https://numpy.org/install/) for more info.)

First, create a conda environment (e.g named `gris_env`) and install the required packages from conda-forge. <br>
64
Next, activate the newly created conda environment.
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
65
```sh
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
66
conda create -n gris_env -c conda-forge mpi4py numpy scipy gfortran_linux-64 lapack matplotlib
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
67
conda activate gris_env
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
68
69
```

70
#### Installation
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
71
72
You can install the pipeline code from the GitLab repo directly using `pip`
```
73
pip install git+https://gitlab.leibniz-kis.de/sdc/gris/grisinv.git
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
74
```
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
75
76
or clone the repo and run setup
```sh
77
git clone https://gitlab.leibniz-kis.de/sdc/gris/grisinv.git
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
78
cd grisinv
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
79
80
python setup.py install
```
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
81
This builds the code and installs the command-line tool (`vfisv`) and in your `conda` environment.
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
82

83
84
85
86
Set the `LD_LIBRARY_PATH` so that the MPI libraries provided by conda takes precedence.
```shell
export LD_LIBRARY_PATH=$CONDA_PREFIX/lib:$LD_LIBRARY_PATH
```
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
87

Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
88

Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
89
90
91
92
93
94
95
96
97
98
## Help

```sh
Usage: vfisv [OPTIONS]

Options:
  -p, --path TEXT        Path to the fits files
  -d, --id INTEGER       Observation ID
  -o, --out TEXT         Output fits file
  -n, --numproc INTEGER  Number of processors to run on
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
99
100
  -l, --line FLOAT       Wavelength of the line (Angstrom)
  -w, --width FLOAT      Wavelength range +/- (Angstrom)
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
101
102
103
104
105
  -h, --help             Show this message and exit.
```


## Example usage
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
106

Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
107
### Command line utility
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
108

Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
109
To run the VFISIV inversion on multi-processor, you can either call `mpiexec` with one processor
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
110
111
and set the `--numproc` to the required number of processors to run the inversion on.

Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
112
113
114
115
116
117
118
```sh
mpiexec -n 1 vfisv \
  --path='/dat/sdc/gris/20150919/level1_split/' \
  --id=3 \
  --line=15648.514 \
  --numproc=20 \
  --width=1.8 \
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
119
  --out='output.fits'
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
120
```
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
121
122
or, call `mpiexec` directly with the required number of processors and the python core will distribute the inversion.

Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
123
```shell
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
124
125
126
127
128
mpiexec -n 20 vfisv \
  --path='/dat/sdc/gris/20150919/level1_split/' \
  --id=3 \
  --line=15648.514 \
  --width=1.8 \
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
129
  --out='output.fits'
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
130
```
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
131

Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
132
```shell
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
133
134
135
136
137
138
139
140
mpiexec -n 20 vfisv \
  --path='/dat/sdc/gris/20150919/level1_split/' \
  --id=3 \
  --line=15648.514 \
  --width=1.8 \
  --out='output.fits'
```

Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
141
142
To output uncertainities, use the `--errors='<filename_erros.fits>'` option.<br>

143
If you want to check the line fits, use `--diagnose='<filename_diagnose.fits>'`.<br>
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
144
145
146
To account for observations with multiple maps, `filename_diagnose` is appended by the map-number, e.g.
`filename_diagnose_001.fits`

147
148
A run with all the options will look like,

Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
149
150
151
152
153
154
155
156
```shell
mpiexec -n 1 vfisv \
  --path='/dat/sdc/gris/20150919/level1_split/' \
  --id=3 --line=15648.514 \
  --numproc=20 \
  --width=1.8 \
  --out='test_inversion.fits' \
  --preview='test_preview.png'  \
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
157
158
  --errors='test_errors.fits' \
  --diagnose='test_diagnose.fits' \
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
159
  --log='test_log.txt'
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
160
```
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
161

Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
162
163
### Python interface

Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
164
The pipeline can be also be run within a python script. To do so, you need to call the python using `mpiexec` with one processor.
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
165

Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
166
With `example.py`:
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
167
168
```python
from grisinv import vfisv
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
169
inversions, fits, header = vfisv('/dat/sdc/gris/20150920/level1_split/',5,15648.514,1.8,20)
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
170
```
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
171
172
173
174
175
you can run,
```sh
mpiexec -n 1 python example.py
```

176

Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
177
### Displaying results
178

Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
179

180
181
```shell
ds9 -multiframe -match frame image -lock slice image -lock frame image -single -zoom to fit output.fits 
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
182
```
Vigeesh Gangadharan's avatar
Vigeesh Gangadharan committed
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198

or, with python (using `sunpy`)
```python
from sunpy.map import Map
m = Map('output.fits')
#Fitted continuum
m[0].peek()
#Line-of-sight velocity
m[1].peek()
#Inclination
m[2].peek()
#Azimuth
m[3].peek()
#Field Strength
m[3].peek()
```