1

The OSKAR simulator (version 2!)3GC-II Workshop, 21st September 2011

Benjamin Mort, Fred Dulwich, Stef Salvini

http://www.oerc.ox.ac.uk/research/oskar

2

What is OSKAR?

• Interferometer and beamforming simulator package.– End-to-end simulations of the phase 1 SKA.

• Based on a full sky Measurement equation formalism.• High performance library based on NVIDIA CUDA.

3

Motivation for writing OSKAR?

• Understanding system noise limits on SKA like AA interferometers.– Dynamic range limits

• All sky simulation.• High performance required for large AA interferometer end-to-end

simulation.– e.g. ~1e6 sources, ~25 stations of ~10,000 dual polarisation antennas.

• Interferometer configuration studies with aperture arrays.

4

What I’m going to talk about!

• The OSKAR ME implementation.• OSKAR version 2.0• Some example and results.

5

Measurement Equation

• The ME as implemented by OSKAR– Scalar version currently being tested, polarised version under

development.

• K – interferometer phase.• E – Station beam.• G – Antenna element field pattern.• P – Propagation term.• B – Source brightness.• V – Complex visibility. Baseline p, q for all visible sources, s.

6

Sky Model

• The global sky model – Equatorial coordinate point source model.

• A local sky model is generated for each snapshot– Remove sources below the horizon– Transforming the source Stokes parameters to the

horizontal frame.

• Emission from bright, extended objects can be included as large collections of point sources.

7

G – Antenna field pattern matrix

• The average (fully polarized) embedded element pattern for antennas within a station

– Factored out if the antennas are sufficiently similar.– otherwise, it is absorbed into the calculation of the E-Jones term.

• Requires input from EM simulations (University of Cambridge).– Depends on the station geometry (cross coupling) and element design.

• Simple functional responses also possible.

-80 -60 -40 -20 0 20 40 60 80-50

-45

-40

-35

-30

-25

-20

-15

-10

-5

(º)

dBW

Embedded element patterns

bb

aa

gg

gg

G

8

Station Beams (E-matrix)

• Rotate phase tracking centre (=beam direction) and all sources from equatorial (RA, Dec) to horizontal (azimuth, elevation).

• Evaluate station beam response for every source and station, for the direction of interest.

• Sources far from the phase centre will be suppressed by station “primary” beams.

– (And sources far from thezenith will be suppressedfurther by antenna elementpattern.)

• Obtain complex matrix

s,iE

9

Station Phases (K-matrix)

• K-matrix effectively “phases-up” the array of stations.

• Compute phase of each source s at every station a.– Determine station (u,v,w) coordinates by rotating (x,y,z) onto a

plane perpendicular to direction of phase centre.

112exp 22,

ssi

sisi

is ikηξw

ηvξuK

10

“Correlator”

• Multiplies appropriate Jones matrices with the source brightness to obtain a complex visibility per source and per baseline, and then collapses the source dimension.

• Time-average smearing: each visibility point can be averaged over time.

– K is recomputed to include motion of baseline during integration period.– E is allowed to vary throughout the integration at a slower rate than K.

• Bandwidth smearing: multiply each visibility by fs,i,j before collapsing the source dimension.

cD

cDf

sji

sjijis /

)/sin(

,

,,,

s

s,jss,iji JBJV ,

11

The OSKAR Package

• OSKAR simulation function library based written in C making extensive use of CUDA

• Multiple libraries with simple dependencies

– Liboskar• Core CUDA function library.

– Liboskar_ms• Interface to casacore for writing simple measurement sets.

– Liboskar_apps• Utility library for using OSKAR to write C/C++ applications.

– Liboskar_widgets• Set of utility widgets written in Qt4/Qwt5. Plotting, gui components.

– Liboskar_imaging• FFT imager (CUDA based imager in development) with w-projection.

– Liboskar_fits• Interface to cfitsio for writing UVFITS and image fits files.

12

The OSKAR Package

• Simple C like interface to the main simulation library using intrinsic types where possible.

• Aims to make it possible to quickly construct new C/C++ simulation applications.

• Designed to interface easily with other languages– MATLAB either with loadlibrary() or though a MEX interface.– (Python)

• MATLAB and C/C++ applications– Beamforming simulator– AA Interferometer simulator– DFT imaging– Simple image post processing MATLAB scripts e.g. CLEAN.

13

Why CUDA?

14

Why CUDA?

• What is CUDA?

– CUDA (Compute Unified Device Architecture) is NVIDIA’s

– Program development environment based on C/C++ with some

extensions.

– Compatible with other multi threaded code.

– Multiple GPUs can be used to work together for very large problems.

• Cost and power effective desktop supercomputing.

– SIMT parallelism model.

– Requires tens of thousands of threads to be efficient.

• Multiple GPUs work together for very large problems

15

Summary of current Features / assumptions

Sky

• Equatorial system.• Point source sky model.• Support for large

numbers of sources, ~O(10^6)

• Use of point source catalogues / image pixels.

• Currently only stokes I (polarisation support very soon)

Station

• Support for any configuration of antenna elements.

• Optimised for large numbers of antennas (e.g.10,000+ per station).

• Primary beam evaluated for each station.

• Antennas currently assumed to be all identical within a station.

• EM coupling encapsulated in element pattern.

Interferometer

• Any latitude and longitude

• Any station positions.• Time-averaging

smearing by actual average.

• Analytical bandwidth smearing per source and per baseline.

• Series of visibility snapshots.

16

Some examples of using OSKAR

17

Source distribution, 2-degree “hole” at phase centre (49993 sources)

18

Field of view 2 deg across, nearest source is 1.32 degrees from centre (Telescope at 40 degrees, 480 snapshots, 49993 sources elsewhere)

Peak @ ~3E-3

19

Imperfect source subtraction

• Bright interfering source on the flank of the station beam at position X.

• A number of other sources scattered over the sky.

• Because the source has effectively become highly time-variable, a simple subtraction of its clean-component model leaves large residuals.

• limiting the dynamic range of the image.

20

Source removal

• Solving for differential gains in MeqTrees (Ian Heywood) is far more effective.

21

Simulation Example: Observation setup.

• Telescope at Faro• Pointing around

Cassiopeia.• 24h observation.

22

Simulation Example: AA station setup

• Offset grid geometry.

• ~80m diameter.

• ~2600 antennas.

23

Simulation Example: Beam pattern.

24

Simulation Example: Sky model

(Simulation with E-Jones disabled)

25

Simulation Example: Telescope setup.

26

Simulation Example: results

27

Simulation Example: Dirty image snapshots

28

Simulation Example: Frequency time source brightness profiles

(1)

(3)

(2)

29

Next Steps

• Currently working on– Polarisation.

– Efficient modelling of system noise.

– Antenna gain and phase errors.

– Limited precision numerics (currently floating point).

– Scaling up to very large simulations using multiple GPUs

• Any suggestions?