• Data Alignment
  • Declaring the Sleeve Area

Data Alignment

A distributed array consists of array variables for which their alignment with the template is defined according to the align directive. Often the size of the template is defined to be the same size as array to be distributed. However, we can also define a template that is larger than the array. In such a case, the relationship between the template and array locations is indicated by an offset from the index. To map each of the array elements b(1), b(2),… to the templates t(1), t(2),…, we use

  • Fortran
!$xmp align b(i) with t(i+1)
  • C
#pragma xmp align b[i] with t(t+1)

as the expression.

This expresses the offset from the template, and thereby declares the alignment between variables and between the variables and the loop. In the following example, which is different from what we have seen up to now, the alignment of variable b is offset by 1. Therefore, the array elements that align with the same t(i) are a(i) and b(i-1).

  • Fortran
!$xmp nodes p(2)
!$xmp template t(11)
!$xmp distribute t(block) onto p
real a(10),b(10),c(10)
!$xmp align a(i) with t(i)
!$xmp align b(i) with t(i+1)

!$xmp loop on t(i)
do i=2,10
	a(i)=a(i)+1.0
	b(i-1)=a(i)
end do
  • C
#pragma xmp nodes p(2)
#pragma xmp template t(0:10)
#pragma distribute t(block) onto p
double a[10],b[10],c[10];
#pragma align a(i) with t(i)
#pragma xmp align b(i) with t(i+1)

#pragma xmp loop on t(i)
for(i=1;i<9;i++){
	a[i]=a[i]+1.0;
	b[i-1]=a[i];
}

ex1.pngConversely, in XcalableMP, an array is sometimes allowed to be larger than the template. To align an array with a sleeve to a template, the amount the array can extend beyond the template limits is limited to the size of the sleeve.

Declaring the Sleeve Area

In applications that use finite difference methods, we often must reference the value of b(i-1) and b(i+1) to compute array element b(i). This reference involves communication with neighboring nodes for array elements near the boundary of the area handled by the node. However, if the area handled by the node could be only slightly extended, and a copy of the adjacent node’s data could be retained, then the number of communications could be reduced. This extended area is referred to as a sleeve or a shadow.

The sleeve is declared using the shadow directive. In the following example, we specify b to have one sleeve on the left and one on the right, and c to have two on the left and three on the right.

  • Fortran
!$xmp nodes p(3)
!$xmp template t(0:21)
!$xmp distribute t(block) onto p
dimension a(0:21),b(0:21),c(0:21)
!$xmp align a(i) with t(i)
!$xmp align b(i) with t(i)
!$xmp align c(i) with t(i)
!$xmp shadow b(1)
!$xmp shadow c(2:3)
  • C
#pragma xmp nodes p(3)
#pragma xmp template t(0:21)
#pragma xmp distribute t(block) onto p
double a[21], b[21], c[21];
#pragma xmp align a[i] with t(i)
#pragma xmp align b[i] with t(i)
#pragma xmp align c[i] with t(i)
#pragma xmp shadow b[1]
#pragma xmp shadow b[2:3]

ex2.pngThe sleeve area can be freely accessed by the node to which it belongs in the same way as for its other area. Before accessing the area, we must use the reflect directive to synchronize the sleeve area. For details, see this page.