Parallel Calibration

Parallel processing using Multi-MS (MMS) in CASA is unverified - please use at own discretion.

Please consider parallel imaging using normal MS as alternative.

 

Some of the calibration tasks are internally parallelized and will run in parallel if the input MS is a Multi-MS. Other tasks are not and will work normally in the presence of an input MMS. A typical calibration cascade will work normally in parallel when it sees an input MMS. In order to do that, the first step is to set createmms=True inside importasdm to create a Multi-MS. Once that is done, the calibration steps will distribute the processing in parallel if CASA is started with mpicasa, or in serial otherwise.
 
Contrary to the MS, the calibration tables created by calibration tasks are not partitioned. For instance, when gaincal is run on a Multi-MS, it will create the same output gaincal table as if the input was a normal MS.
 
The following calibration tasks are internally parallelised and will work on each Sub-MS in parallel.
flagdata
setjy
applycal
hanningsmooth
cvel2
uvcontsub
mstransform
split
 

Special considerations when running some tasks in parallel

uvcontsub

When the input is a Multi-MS and CASA is started in parallel using mpicasa, uvcontsub will try to process each Sub-MS in parallel. Depending on the parameters of uvcontsub and the separation axis of the partitioned Multi-MS, processing the input in parallel is not possible. This will happen for example when the input MMS is separated using the default axis 'auto'. The 'auto' axis will partition the MMS  by the scan and spw axes, in a way to balance the content on each Sub-MS.
 
If uvcontsub is called with combine='spw', the task will expect to find all selected spws in each Sub-MS, as each parallel engine will process a Sub-MS independently of the others. In such cases, task uvcontsub will issue some warnings that the process cannot be continued in parallel. The task will internally handle such cases and will continue to process the input in serial, as if the Multi-MS was a normal monolithic MS.
 
The following steps can be informed in order to find out what is the partition axis of the MMS and what is the content of each Sub-MS. First, use task listpartition to obtain information on the MMS.

CASA <2>: listpartition('combspw.mms')
INFO listpartition::::@almahpc05:MPIClient
INFO listpartition::::@almahpc05:MPIClient+ ##########################################
INFO listpartition::::@almahpc05:MPIClient+ ##### Begin Task: listpartition #####
INFO listpartition::::@almahpc05:MPIClient listpartition(vis="combspw.ms",createdict=False,listfile="")
INFO listpartition::::@almahpc05:MPIClient This is a Multi-MS with separation axis = scan,spw
INFO listpartition::::@almahpc05:MPIClient Sub-MS              Scan Spw                 Nchan                     Nrows Size
INFO listpartition::::@almahpc05:MPIClient+combspw.ms.0000.ms  1    [ 1 5 6 9 12 16]    [128 128 128 128 128 128] 252   4.9M
INFO listpartition::::@almahpc05:MPIClient                     2    [ 0 3 13 17 18 21]  [128 128 128 128 128 128] 378
INFO listpartition::::@almahpc05:MPIClient combspw.ms.0001.ms  1    [ 0 4 8 13 17 21]   [128 128 128 128 128 128] 252   4.5M
INFO listpartition::::@almahpc05:MPIClient                     2    [ 2 6 7 10 14 22]   [128 128 128 128 128 128] 378
INFO listpartition::::@almahpc05:MPIClient combspw.ms.0002.ms  1    [ 3 7 10 14 20 22]  [128 128 128 128 128 128] 252   4.5M
INFO listpartition::::@almahpc05:MPIClient                     2    [ 5 11 12 15 19 23] [128 128 128 128 128 128] 378
INFO listpartition::::@almahpc05:MPIClient combspw.ms.0003.ms  1    [ 2 11 15 18 19 23] [128 128 128 128 128 128] 252   4.5M
INFO listpartition::::@almahpc05:MPIClient                     2    [ 1 4 8 9 16 20]    [128 128 128 128 128 128] 378
INFO listpartition::::@almahpc05:MPIClient ##### End Task: listpartition #####
INFO listpartition::::@almahpc05:MPIClient+ ##########################################

In the above example, the MMS was partitioned using the default axis 'auto' (scan,spw). One can see the Sub-MSs do not contain all spws, therefore depending on the selection used in the task, it will not be possible to proceed in parallel. See the following example for the warnings given by the task in this case.

CASA <8>: uvcontsub(vis="combspw.mms",fitspw="1~10:5~122,15~22:5~122",excludechans=False,combine="spw",fitorder=0,spw="6~14",want_cont=False)
2018-02-06 15:45:09 INFO uvcontsub::::@almahpc05:MPIClient
2018-02-06 15:45:09 INFO uvcontsub::::@almahpc05:MPIClient+ ##########################################
2018-02-06 15:45:09 INFO uvcontsub::::@almahpc05:MPIClient+ ##### Begin Task: uvcontsub #####
2018-02-06 15:45:09 INFO uvcontsub::::@almahpc05:MPIClient uvcontsub(vis="combspw.mms",field="",fitspw="1~10:5~122,15~22:5~122",excludechans=False,combine="spw",
2018-02-06 15:45:09 INFO uvcontsub::::@almahpc05:MPIClient+ solint="int",fitorder=0,spw="6~14",want_cont=False)
2018-02-06 15:45:11 WARN uvcontsub::::@almahpc05:MPIClient Cannot run with combine='spw' in parallel because the Sub-MSs do not contain all the selected spws
2018-02-06 15:45:11 WARN uvcontsub::::@almahpc05:MPIClient The Multi-MS will be processed in serial and will create an output MS
2018-02-06 15:45:11 INFO uvcontsub::::@almahpc05:MPIClient split is being run internally, and the selected spws
2018-02-06 15:45:11 INFO uvcontsub::::@almahpc05:MPIClient will be renumbered to start from 0 in the output!
2018-02-06 15:45:11 INFO uvcontsub::::@almahpc05:MPIClient Preparing to add scratch columns.
2018-02-06 15:45:11 INFO uvcontsub::::@almahpc05:MPIClient splitting to /data/users/scastro/work/CAS-10697/combspw.mms.contsubId4wzP with spw="1~5,6~14,15~22"
2018-02-06 15:45:11 INFO SubMS::parseColumnNames() Using DATA column.

A few options are possible at this stage. User can let the process continue in serial, which depending on the size of the MS, can take long, and at the end the continuum subtracted output will be a normal MS. Depending on what the user wants to do next, there is the possibility to recreate the MMS using task partition. If user only wants to run tclean and create an image, having either MS or MMS will work in the same way because tclean can run in parallel regardless whether the input is MS or MMS.
 
If the users opts to recreate the MMS before running uvcontsub, best recommend axis to do combine='spw' is per scan. Partition will have to be called in the following way:
 

partition(vis='myMS.ms', outputvis='myout.ms', createmms=True, separationaxis='scan')