Computers and Tensor Multiplication

After working through a matrix multiplication a student at Mockingbird Academy asked “Would it help if we did this using a computer program?

Our initial discussion led to the idea that we could use “for-loops” to do the matrix multiplication and we could talk about how this could be expanded to tensors with higher ranks.

Assume we multiply

\begin{bmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \\ a_{31} & a_{32} & a_{33} \\ a_{41} & a_{42} & a_{43} \end{bmatrix} \begin{bmatrix} b_{11} & b_{12} & b_{13} & b_{14} & b_{15} \\ b_{21} & b_{22} & b_{23} & b_{24} & b_{25} \\ b_{31} & b_{32} & b_{33} & b_{34} & b_{35} \end{bmatrix}

to get:

\begin{bmatrix} c_{11} & c_{12} & c_{13} & c_{14} & c_{15}\\ c_{21} & c_{22} & c_{23} & c_{24} & c_{25} \\ c_{31} & c_{32} & c_{33} & c_{34} & c_{35} \\ c_{41} & c_{42} & c_{43} & c_{44} & c_{45} \end{bmatrix}

We can designate the matrices as A_{ij}, B_{jk}, C_{ik} where i=4, j=3, k=5.

c_{11} = a_{11}b_{11} + a_{12}b_{21} + a_{13}b_{31}

c_{12} = a_{11}b_{12} + a_{12}b_{22} + a_{13}b_{32}

c_{21} = a_{21}b_{11} + a_{22}b_{21} + a_{23}b_{31}

After scrutinizing several of these calculations, you will notice that they all fit the pattern below:

c_{ik} = a_{i1}b_{1k} + a_{i2}b_{2k} + a_{i3}b_{3k}

The above fact leads us to what we are going to do in the computer program – for each pair of i and k we count from 1 to 3 and do a summation. The variable j will do the summation over 1 to 3:

The rough idea is shown below:

  • For i=1 to 3
  • For k=1 to 3
  • Sum=0
  • For j=1 to 3
  • Term = a(i,j) * b(j,k)
  • Sum += Term
  • Next j
  • c(i,k) = Sum
  • Next k
  • Next i

Hopefully you see the summation is going to be called nine times and each time it is going to cakculate an answer for a unique combination of i and k.

Let’s change a and b to be higher tensors: a(i,j,k,l,m) and b(l,m,n,p).

The for-loops for l and m will both be within the summation. The for-loops for i,j,k,n,p will be outside the summation and the result will be c(i,j,k,n,p).

Regarding notation: we wrote c(i,k) to avoid using subscripts and superscripts, because placement depends on whether the index is over something that is covariant or contravariant.

Appendix A

It would be fair for someone to say, this is not the same thing as matrix multiplication. Correct.

When you learned matrix multiplication there were no indices (actually this isn’t completely true). You were given two grids of numbers and there were two ways we could done the math, across-down or down-across. Someone had to decide which way was the correct way.

If we use our system with the indices, then the indices answer the questions– there is no need for arbitrary rules.

You can do matrix multiplication without ever think about indices, but they actually did have indices involved when you were taught about matrices.

a_{ij}b_{jk} = c_{ik}