pycrtools.tasks.fitbaseline¶

`ApplyBaseline`([ws, parfile])	Usage:
`CalcBaseline`([ws, parfile])	Usage:
`FitBaseline`([ws, parfile])	Usage:
`fitbaseline_calc_freqs`(self)
`fitbaseline_calc_numax_i`(self)
`fitbaseline_calc_numin_i`(self)

Spectrum documentation¶

Module author: Heino Falcke <h.falcke@astro.ru.nl>

Usage:

>>> sp=cr.hArrayRead('/Users/falcke/data/Pulses/oneshot_level4_CS017_19okt_no-0.h5.spec.pcr')  ... old (to be updated)

class pycrtools.tasks.fitbaseline.ApplyBaseline(ws=None, parfile=None, **kwargs)¶

Usage:
>>> ApplyBaseline(spectrum)
contains spectrum.par.baseline

Also does RFI excision.

Input parameters

FitParameters: Parameters of the baseline fitting routine.
adaptive_peak_threshold [default value: False ]: If True then calculate the threshold above which to cut peaks in nbins separate bins and thus let it vary over the spectrum
apply_baseline [default value: True ]: If True then divide spectrum by baseline before removing peaks.
baseline [default value: None ]: Array containing the calculated baseline with the same dimensions as spectrum - can be provided as spectrum.par.baseline (will be created if not).
blocklen: The block length of one bin used to subdivide spectrum for selecting the best RMS.
dim_spectrum: Dimension of input spectrum (typically n antennas times m spectral points) or just one dimensional for one antenna.
dirty_channels: Array of indices pointing to dirty bins, i.e. with high RFI. (work vector)
doplot [default value: False ]: Plot progress information. If value > 1, plot more information (the number of leves varies from task to task).
filename: Filename to write output to.
frequency: Frequency values in Hz for each spectral channel (dimension: [nchannels])
mean [default value: 1 ]: Median mean value of blocks in downsampled spectrum - used to replace flagged data with (output only)
means: Mean value per block in input spectrum
nbins: The number of bins to be used to subdivide spectrum for selecting the best RMS.
nofAntennas: Number of antennas in input spectrum.
nofChannels: Number of channels in input spectrum.
nofChannelsUsed: Number of channels remaining after downsampling and ignoring edges.
numax: Maximum frequency of useable bandwidth. Negative if to be ignored.
numin: Minimum frequency of useable bandwidth. Negative if to be ignored.
plot_center [default value: 0.5 ]: Center plot at this relative distance from start of vector (0=left end, 1=right end).
plot_end: End plotting before this sample number.
plot_filename [default value: “” ]: Base filename to store plot in.
plot_finish: Function to be called after each plot to determine whether to pause or not (see ::func::plotfinish)
plot_name [default value: “” ]: Extra name to be added to plot filename.
plot_start: Start plotting from this sample number.
plotchannel [default value: 0 ]: Which channel to plot.
plotlen [default value: 131072 ]: How many channels +/- the center value to plot during the calculation (to allow progress checking).
rmsfactor [default value: 5.0 ]: Factor above the RMS in each channel at which a channel is considered poluted by RFI.
save_output [default value: False ]: If True save the results in the header files of the input spectrum with file name given in filename
spectrum [default value: None ]: Array with input spectrum either of dimension [nofAntennas,nchannels] or just [nchannels] for a single spectrum. Note that the frequency values for the array are expected to be provided as spectrum.par.xvalues=cr.hArray(float,[nofChannels],fill=...) otherwise provide the frequencies explicitly in frequency
stddevs: Standard deviation per block in input spectrum
verbose [default value: True ]: Print progress information.
work_spectrum: Wrapper to input spectrum with dimension [nofAntennas,nchannels] even for a single spectrum.

Output parameters

ndirty_channels [default value: 0 ]: Number of dirty channels to be removed as RFI. (output only)
numax_i: Channel number in spectrum of the maxium frequency where to calculate baseline. Apply hanning taper above.
numin_i: Channel number in spectrum of the minium frequency where to calculate baseline. Apply hanning taper below.
rms: Median RMS value of blocks in downsampled spectrum - used to calculate threshold for cutting peaks (output only)

class pycrtools.tasks.fitbaseline.CalcBaseline(ws=None, parfile=None, **kwargs)¶

Usage:
>>> CalcBaseline(spectrum,par1=...,par2=...)
Calculate a smooth baseline from a set of coefficients that determine the baseline (e.g., as calculated by :class:FitBaseline) and an array of frequencies (which need not be equi-spaced).

The frequencies can be provided with the .par.xvalues array of the input spectrum (if available). Otherwise, they have to be explicitly provided.

The result will be returned in an array Task.baseline. This baseline will also be attached to the .par dict of the input spectrum, i.e. spectrum.par.baseline.

The task can return the inverse of the baseline (if invert=True), i.e. what one needs to multiply a measured spectrum with.

Note, sometimes the fit can go wrong if logfit=True the interpolation has wiggles. In this case the exponent of a large number is calculated, yielding a numerical error. In this case switch set logfit=False (to see what is going on).

Gain Calibration (DRAFT!)

Knowing that the antenna is exposed to a noise with a certain power, one can use this to make a rough estimate of the absolute gain of the antenna. E.g., in case for LOFAR antennas the spectrum measured in one antenna should follow a $\nu^{-0.5}$ power law due to emission from the Galaxy (Milky Way). Hence, one can multiply the baseline one has fitted to the measured spectrum with a power law, so that after dividing the gain curve one reproduces the expected behaviour (i.e., Galactic noise). Note: if one wants to use calcbaseline for gain calibration one obviously has to provide a power law that is the inverse of the desired function.

Powerlaws can be specified with the Task.powerlaw parameter, which is either just the power law index, or a tuple with amplitude and index, or a keyword (e.g., ‘Galactic’, of ‘GalacticT’).

The noise temperature of an antenna is:

Tnoise = Power [Watts] /Bandwidth [Hz] / kb (Boltzmann constant) = 7.24296*10^22 K

The sky tempertature is according to Falcke & Gorham (2004) roughly: T_sky = 32*( (freq/408e6)^-2.5 ) (T_sky in K, freq in Hz)

The recevied power of an antenna is Prec = Aeff * S, where S is the power density (in W/m^2).

The effective area is

Aeff = gain* lambda^2/(4 pi)

The power density is S=E^2/R (i.e., Electric field^2 / free space impedance (376.7 Ohm))

The System Equivalent Flux Density (SEFD) is

Flux density for bandwidth limited pulses can be rougly converted to Field strength through Snu= Enu epsilon_0 c Bandwidth (Enu in Volt/Meter/MHz in SI units ..., check this formula)

to be continued ....

In the following a short description from Andreas Horneffer (in German) on how to do a simple gain correction.
Unter der Annahme, dass der ganze Himmel eine konstante, und bekannte
Strahlungstemperatur hat, das gemessene Rauschen davon dominiert ist,
und man das Gain (die Richtwirkung) der Antennen kennt.

Die Strahlungstemperatur des Himmels ist nach Falcke & Gorham (2002) etwa:
T_sky = 32*( (freq/408e6)^-2.5 );
(T_sky in K, freq in Hz)

Die gemessene "Rauschtemperatur" ist:
T_noise = P_noise / dNu / kB
(T_noise in K, P_noise in W, dNu (Bandbreite) in Hz, kB die Boltzmann-Konstante)

Die Rauschleistung ist: P_noise = V^2/R (P_noise in W, V (gemessene
Spannung) in V, R (angenommene ADC-Impedanz) in Ohm) Die ADC-Impedanz
kuertzt sich nachher weg. Wenn Du die Rauschleistung als Funktion der
Frequenz nach der FFT fuers Kalibrieren verwenden willst, aber die
Kalibration fuer die Zeitreihe nutzen willst, musst du noch die
Normierung der FFT korrigieren. (Haengt von der verwendeten
FFT-Routine ab.)

Die Verstaerkung der Elektronik ist dann: K = T_sky / T_noise (K ist
einheitenloses Verhaeltnis der Leistungen)

Die Feldstaerke an der Antenne ist dann:

E = sqrt( 4*pi*freq^2*mu0 / G(d,freq) * K * V^2/R )

(E in V/m, mu0 die magnetische Feldkonstante,
G(d,freq) das Gain der Antennen in die Richtung d, bei Frequenz freq
als einheitenloses Verhaeltnis der Empfindlichkeit der Antennen zu der
Empfindlichkeit eine isotropen Strahlers)

Als Gain der LOFAR Antennen kann man vermutlich einfach das Gain der LOPES
Antennen verwenden. (Natuerlich fuer die Ausrichtung der Dipole rotiert.)

Input parameters

FitParameters: Parameters of the baseline fitting routine.
HanningUp [default value: True ]: Let the Hanning filter at the ends go up rather than down (useful if one wants to divide by baseline and suppress out-of-band-noise)
addHanning [default value: True ]: Add a Hanning filter above nu_max and below nu_min to suppress out-of-band emission smoothly.
baseline [default value: None ]: Array containing the calculated baseline with the same dimensions as spectrum - can be provided as spectrum.par.baseline (will be created if not).
coeffs: Polynomial coefficients of the baseline fit with the dimension [nofAntennas, ncoeffs] or [ncoeffs]
dim_coeffs: Dimension of the coefficients array (which should be [nofAntennas, ncoeff] or [ncoeff] for nofAntennas==1)
dim_spectrum: Dimension of input spectrum (typically n antennas times m spectral points) or just one dimensional for one antenna.
doplot [default value: False ]: Plot progress information. If value > 1, plot more information (the number of leves varies from task to task).
filename: Filename to write output to

fittype

frequency: Frequency values in Hz for each spectral channel (dimension: [nchannels])
height_ends: The heights of the baseline at the left and right endpoints of the usable bandwidth where a hanning function is smoothly added.
invert [default value: True ]: Invert the baseline so that it can later simply be multiplied to take out the gain variations.

logfit

ncoeffs: Number of coefficients for the polynomial.
nofAntennas: Number of antennas in input spectrum.
nofAntennasCoeffs: Number of antennas in coeffcient array.
nofChannels: Number of channels in input spectrum.
normalize [default value: True ]: If True, normalize the baseline to have a total sum of unity.
numax: Maximum frequency of useable bandwidth. Negative if to be ignored.
numax_val_i: Maximum frequency of useable bandwidth corresponding to numax_i (output)
numin: Minimum frequency of useable bandwidth. Negative if to be ignored.
numin_val_i: Minimum frequency of useable bandwidth corresponding to numin_i (output)
plot_antenna [default value: 0 ]: Which antenna to plot?
plot_center [default value: 0.5 ]: Center plot at this relative distance from start of vector (0=left end, 1=right end).
plot_end: End plotting before this sample number.
plot_filename [default value: “” ]: Base filename to store plot in.
plot_finish: Function to be called after each plot to determine whether to pause or not (see ::func::plotfinish)
plot_name [default value: “” ]: Extra name to be added to plot filename.
plot_start: Start plotting from this sample number.
plotlen [default value: 131072 ]: How many channels +/- the center value to plot during the calculation (to allow progress checking).
powerlaw [default value: False ]: Multiply gain curve with a powerlaw depending on value. ‘GalacticT’: return a powerlaw according to Galactic radio emission in Kelvin; alpha (i.e. a number): multiply with powerlaw of form :math`:nu^alpha`.
powers: Array of integers, containing the powers to fit in the polynomial. (work vector)
save_output [default value: False ]: If True save the results in the header files of the input spectrum with file name given in filename
selected_bins: Array of indices pointing to clean bins, i.e. with low RFI. (work vector)
spectrum [default value: None ]: Array with input spectrum either of dimension [nofAntennas,nchannels] or just [nchannels] for a single spectrum. Note that the frequency values for the array are expected to be provided as spectrum.par.xvalues=cr.hArray(float,[nofChannels],fill=...) otherwise provide the frequencies explicitly in frequency
splineorder: Order of the polynomial to fit for the BSpline, splineorder=3 is a bicubic spline.
verbose [default value: True ]: Print progress information.
work_coeffs: Array with coefficients in the form [nofAntennas, ncoeff]
work_spectrum: Wrapper to input spectrum with dimension [nofAntennas,nchannels] even for a single spectrum.

Output parameters

numax_i: Channel number in spectrum of the maxium frequency where to calculate baseline. Apply hanning taper above.
numin_i: Channel number in spectrum of the minium frequency where to calculate baseline. Apply hanning taper below.
polyorder: Order of the polynomial to fit. (output only)

call(spectrum)¶

run()¶

class pycrtools.tasks.fitbaseline.FitBaseline(ws=None, parfile=None, **kwargs)¶

Usage:
FitBaseline(input_spectrum, **keywords)
Description:

Function to fit a baseline using a polynomial function (fittype='POLY') or a basis spline fit (default: fittype='BSPLINE') to a spectrum while trying to ignore positive spikes in the fit (e.g., those coming from RFI = Radio Frequency Interference).

The input_spectrum is a float array with input spectrum either of dimension [nofAntennas,nofChannels] or just [nofChannels] for a single spectrum. Note that the frequency values for the array are expected to be provided as input_spectrum.par.xvalues=cr.hArray(float,[nofChannels],fill=...) the same you would use to provide xvalues for plotting. You can also provide the frequencies explicitly in the keyword frequency, otherwise a simple numbering is assumed frequency=[0,1,2,3...].

RFI Treatment:

The spectra are then downsampled and checked which bins have a large RMS (when divided by the mean in each bin). Those bins are affected by interference and are ignored for the fitting. The final fit is done to the clean bins only.

Quality check:

At the end the fitted baselines are compared to each other (divided by the average spectrum of all antennas) and a list of deviant antennas is returned in Task.badantennas.

Results:

The main results are the fit coefficients Task.coeffs.

Bad antennas are listed in Task.badantennas.

Use baseline.polynomial(Task.frequency,Task.coeffs,Task.powers) to caluclate the baseline from the coefficients for the polynomial fit, or:
>>> task.xpowers[...,[0]:task.nselected_bins].linearfitpolynomialx(task.clean_bins_x[...,[0]:task.nselected_bins],task.powers[...])
>>> task.clean_bins_y[...,[0]:task.nselected_bins].bspline(task.xpowers[...,[0]:(task.nselected_bins)],task.coeffs[...])
Parameters can be provided as additional keywords, e.g.:
baseline.polynomial(frequency,coeffs,powers, parameter1=value1, parameter2=value2)
or in a WorkSpace, e.g.:
baseline.polynomial(frequency,coeffs,powers, ws=WorkSpace),
or in a mix of the two. Otherwise default values are used.

Input parameters

clean_bins_x: Array holding the frequencies of the clean bins. (work vector)
clean_bins_y: Array holding the powers of the clean bins. (work vector)
dim_spectrum: Dimension of input spectrum (typically n antennas times m spectral points) or just one dimensional for one antenna.
doplot [default value: False ]: Plot progress information. If value > 1, plot more information (the number of leves varies from task to task).
extendfit [default value: 0.1 ]: Extend the fit by this factor at both ends beyond numax and numin. The factor is relative to the unused bandwidth. Use this to make sure there is a stable solution at least between numax/numin, i.e. avoid wiggles at the endpoint.
filename: Filename to write output to
fitted_bins_x: Array holding the frequency values for the fitted, binned spectrum - for plotting and control. (work vector)
fitted_bins_y: Array holding the fitted, binned spectrum - for plotting and control. (work vector)
fittype [default value: “BSPLINE” ]: Determine which type of fit to do: fittype='POLY' - do a polynomial fit, else (‘BSPLINE’) do a basis spline fit (default).
flattened_bins_y: Array holding the binned spectrum of each antenna divided by the mean spectrum of all antennas - for plotting and control. (work vector)
freqs: Array of frequency values of the downsampled spectrum. (work vector)
frequency: Frequency values in Hz for each spectral channel (dimension: [nchannels])
iteration [default value: 0 ]: If 0 or False, then this is the first iteration, of the fitting otherwise the n-th iteration (information only at this point).
logfit [default value: True ]: Actually fit the polynomial to the log of the (downsampled) data. (Hence you need to .exp the baseline afterwards).
minrms_blen: Block length within downsampled data to look for the cleanest part of the spectrum.
nbins: The number of bins in the downsampled spectrum used to fit the baseline.
ncoeffs [default value: 18 ]: Number of coefficients for the polynomial.
nofAntennas: Number of antennas in input spectrum.
nofChannels: Number of channels in input spectrum.
nofChannelsUsed: Number of channels remaining after downsampling and ignoring edges.
nsigma [default value: 3 ]: Used to select bad antennas (How many sigma can the rms of the difference spectrum deviate from those of others antennas)
numax [default value: -1 ]: Maximum frequency of useable bandwidth. Negative if to be ignored.
numax_val_i: Maximum frequency of useable bandwidth corresponding to numax_i (output)
numin [default value: -1 ]: Minimum frequency of useable bandwidth. Negative if to be ignored.
numin_val_i: Minimum frequency of useable bandwidth corresponding to numin_i (output)
plot_antenna [default value: 0 ]: Which antenna to plot?
plot_center [default value: 0.5 ]: Center plot at this relative distance from start of vector (0=left end, 1=right end).
plot_end: End plotting before this sample number.
plot_filename [default value: “” ]: Base filename to store plot in.
plot_finish: Function to be called after each plot to determine whether to pause or not (see ::func::plotfinish)
plot_name [default value: “” ]: Extra name to be added to plot filename.
plot_start: Start plotting from this sample number.
plotlen [default value: 131072 ]: How many channels +/- the center value to plot during the calculation (to allow progress checking).
powers: Array of integers, containing the powers to fit in the polynomial. (work vector)
qualitycontrol [default value: False ]: If True then select bad antennas (How many sigma can the rms of the difference spectrum deviate from those of others antennas)
ratio: Array holding the ratio between RMS and power of the downsampled spectrum. (work vector)
rms: Array of RMS values of the downsampled spectrum. (work vector)
rmsfactor [default value: 3.0 ]: Factor above and below the RMS in each bin at which a bin is no longer considered due to too many spikes.
save_output [default value: False ]: If True save the results in the header files of the input spectrum with file name given in filename
selected_bins: Array of indices pointing to clean bins, i.e. with low RFI. (work vector)
small_average_spectrum: Array of power values holding the average (fitted) downsampled spectrum. (work vector)
small_spectrum: Array of power values holding the downsampled spectrum. (work vector)
spectrum [default value: None ]: Array with input spectrum either of dimension [nofAntennas,nchannels] or just [nchannels] for a single spectrum. Note that the frequency values for the array are expected to be provided as spectrum.par.xvalues=cr.hArray(float,[nofChannels],fill=...) otherwise provide the frequencies explicitly in frequency
splineorder [default value: 3 ]: Order of the polynomial to fit for the BSpline, splineorder=3 is a bicubic spline.
verbose [default value: True ]: Print progress information.
weights: Array of weight values for the fit. (work vector)
work_spectrum: Wrapper to input spectrum with dimension [nofAntennas,nchannels] even for a single spectrum.
xpowers: Array holding the x-values and their powers for calculating the baseline fit.

Output parameters

badantennas [default value: [] ]: List of bad antennas (where the rms of the difference spectrum deviated from those of others antennas)
bwipointer [default value: 0 ]: Pointer to the internal BSpline workspace as integer. Don’t change!
chisquare [default value: 0 ]: Returns the $\chi^2$ of the baseline fit. (output only)
coeffs: Polynomial coefficients of the baseline fit with the dimension [nofAntennas,ncoeffs] (output vector)
covariance: Array containing the covariance matrix of the fit. (output only)
goodantennas [default value: [] ]: List of good antennas.
meanrms [default value: 0 ]: Estimate the mean RMS in the spectrum per antenna. (output vector)
minmean: Mean value of data in the part of downsampled spectrum with the smallest RMS (output only)
minrms: RMS value of data in the part of downsampled spectrum with the smallest RMS (output only)
nbadantennas [default value: 0 ]: Number of bad antennas (where the rms of the difference spectrum deviated from those of others antennas)
nselected_bins [default value: 0 ]: Number of clean bins after RFI removal. (output only)
numax_i: Channel number in spectrum of the maximum frequency of the useable bandwidth. Negative if to be ignored.
numin_i: Channel number in spectrum of the minimum frequency of the useable bandwidth. Negative if to be ignored.
polyorder: Order of the polynomial to fit. (output only)

run()¶: Run the program.

pycrtools.tasks.fitbaseline.fitbaseline_calc_freqs(self)¶

pycrtools.tasks.fitbaseline.fitbaseline_calc_numax_i(self)¶

pycrtools.tasks.fitbaseline.fitbaseline_calc_numin_i(self)¶

Table Of Contents

Previous topic

Next topic

This Page

pycrtools.tasks.fitbaseline¶

Spectrum documentation¶

Navigation

Table Of Contents

Previous topic

Next topic

This Page

Quick search

pycrtools.tasks.fitbaseline¶

Spectrum documentation¶

Navigation