Vectorization of Matrices

The term \(\,\) vectorization of a matrix \(\,\) denotes a linear transformation which converts a matrix with \(\,m\,\) rows \(\,\) and \(\,n\,\) columns into a column vector of size \(\,mn.\ \)

Consider a matrix \(\,\boldsymbol{A}=[\,a_{ij}\,]_{m\times n}\in M_{m\times n}(K),\ \) where \(\,M_{m\times n}(K)\ \) is the vector space of \(\,m \times n\ \) matrices over a field \(\,K.\ \) Let \(\,E_{\,m\times n}\,=\,\{\,\boldsymbol{E}_{ij}:\ \ i=1,\ldots,m;\ j=1,\ldots,n\,\}\ \) be the standard basis of \(\,M_{m\times n}(K),\ \) composed of matrices \(\,\boldsymbol{E}_{ij}\ \) with exactly one non-zero entry, \(\,\) equal to unity, \(\,\) in the \(\,i\)-th row and \(\,j\)-th column:

\[\begin{split}\left(\boldsymbol{E}_{ij}\right)_{\,|\,kl}\ =\ \ \delta_{ik}\ \delta_{jl}\,, \qquad \begin{array}{l} i,k=1,2,\ldots,m, \\ j,l=1,2,\ldots,n. \end{array}\end{split}\]

The matrix \(\,\boldsymbol{A}\ \) may be vectorized in two ways:

• by juxtaposing the consecutive rows of the matrix next to each other \(\\\) and taking the transpose of the obtained long “multi-row”:

  \[\boldsymbol{\Lambda}^{mn}(\boldsymbol{A})\ \,:\,=\ \, [\,a_{11}\ \,a_{12}\ \,\ldots\ \,a_{1n}\ \ \, a_{21}\ \,a_{22}\ \,\ldots\ \,a_{2n}\ \ \,\ldots\ \ \, a_{m1}\ \,a_{m2}\ \,\ldots\ \,a_{mn}\,]^{\,T} .\]

  The vector \(\,\boldsymbol{\Lambda}^{mn}(\boldsymbol{A})\ \) is the column of coordinates of matrix \(\,\boldsymbol{A}\ \) in the ordered basis

  \[\mathcal{E}^{\,row}_{m\times n}\ =\ \left(\ \boldsymbol{E}_{11},\ \boldsymbol{E}_{12},\ \ldots,\ \boldsymbol{E}_{1n},\ \ \boldsymbol{E}_{21},\ \boldsymbol{E}_{22},\ \ldots,\ \boldsymbol{E}_{2n},\ \ \ldots,\ \ \boldsymbol{E}_{m1},\ \boldsymbol{E}_{m2},\ \ldots,\ \boldsymbol{E}_{mn}\,\right)\,.\]

• by stacking the columns of the matrix on top of one another:

  \[\boldsymbol{\mathrm{V}}^{mn}(\boldsymbol{A})\ \,:\,=\ \, [\,a_{11}\ \,a_{21}\ \,\ldots\ \,a_{m1}\ \ \, a_{12}\ \,a_{22}\ \,\ldots\ \,a_{m2}\ \ \,\ldots\ \ \, a_{1n}\ \,a_{2n}\ \,\ldots\ \,a_{mn}\,]^{\,T} .\]

  The vector \(\,\boldsymbol{\mathrm{V}}^{mn}(\boldsymbol{A})\ \) is the column of coordinates of matrix \(\,\boldsymbol{A}\ \) in the ordered basis

  \[\mathcal{E}^{\,col}_{m\times n}\ =\ \left(\ \boldsymbol{E}_{11},\ \boldsymbol{E}_{21},\ \ldots,\ \boldsymbol{E}_{m1},\ \ \boldsymbol{E}_{12},\ \boldsymbol{E}_{22},\ \ldots,\ \boldsymbol{E}_{m2},\ \ \ldots,\ \ \boldsymbol{E}_{1n},\ \boldsymbol{E}_{2n},\ \ldots,\ \boldsymbol{E}_{mn}\,\right)\,.\]

  In literature 1 \(\ \) the vector \(\,\boldsymbol{\mathrm{V}}^{mn}(\boldsymbol{A})\ \) is also denoted by \(\,\text{vec}(\boldsymbol{A}).\)

Example. \(\,\) The standard basis of the vector space \(\ M_{2\times 3}(R)\ \) is composed of the matrices

\[\begin{split}\begin{array}{lll} \boldsymbol{E}_{11}\ =\ \left[\begin{array}{ccc} 1 & 0 & 0 \\ 0 & 0 & 0 \end{array}\right], & \boldsymbol{E}_{12}\ =\ \left[\begin{array}{ccc} 0 & 1 & 0 \\ 0 & 0 & 0 \end{array}\right], & \boldsymbol{E}_{13}\ =\ \left[\begin{array}{ccc} 0 & 0 & 1 \\ 0 & 0 & 0 \end{array}\right], \\[12pt] \boldsymbol{E}_{21}\ =\ \left[\begin{array}{ccc} 0 & 0 & 0 \\ 1 & 0 & 0 \end{array}\right], & \boldsymbol{E}_{22}\ =\ \left[\begin{array}{ccc} 0 & 0 & 0 \\ 0 & 1 & 0 \end{array}\right], & \boldsymbol{E}_{23}\ =\ \left[\begin{array}{ccc} 0 & 0 & 0 \\ 0 & 0 & 1 \end{array}\right], \end{array}\end{split}\]

hence the ordered bases are:

\[ \begin{align}\begin{aligned}\mathcal{E}^{\,row}_{2\times 3}\ =\ \left(\ \boldsymbol{E}_{11},\ \boldsymbol{E}_{12},\ \boldsymbol{E}_{13},\ \ \boldsymbol{E}_{21},\ \boldsymbol{E}_{22},\ \boldsymbol{E}_{23}\ \right)\,,\\\mathcal{E}^{\,col}_{2\times 3}\ =\ \left(\ \boldsymbol{E}_{11},\ \boldsymbol{E}_{21},\ \boldsymbol{E}_{12},\ \ \boldsymbol{E}_{22},\ \boldsymbol{E}_{13},\ \boldsymbol{E}_{23}\ \right)\,.\end{aligned}\end{align} \]

For the matrix \(\ \boldsymbol{A}\ =\ \left[\begin{array}{ccc} 1 & 2 & 3 \\ 4 & 5 & 6 \end{array}\right]\ \ \) we get \(\ \ \boldsymbol{\Lambda}^{23}(\boldsymbol{A})\ =\ \left[\begin{array}{c} 1 \\ 2 \\ 3 \\ 4 \\ 5 \\ 6 \end{array}\right],\ \ \boldsymbol{\mathrm{V}}^{23}(\boldsymbol{A})\ =\ \left[\begin{array}{c} 1 \\ 4 \\ 2 \\ 5 \\ 3 \\ 6 \end{array}\right]\,.\)

For \(\ \boldsymbol{A}\in M_{m\times n}(K)\ \) obviously \(\ \boldsymbol{\mathrm{V}}^{mn}(\boldsymbol{A})\ =\ \boldsymbol{\Lambda}^{nm}(\boldsymbol{A}^T)\,,\quad\) \(\ \boldsymbol{\Lambda}^{mn}(\boldsymbol{A})\ =\ \boldsymbol{\mathrm{V}}^{nm}(\boldsymbol{A}^T)\,.\)

Using the Kronecker product, matrix multiplication can be expressed as a linear transformation of vectorized matrices. With this end in view, assume that \(\ \boldsymbol{A}=[a_{ij}]_{m\times p},\ \) \(\ \boldsymbol{B}=[b_{ij}]_{p\times n}\ \) and \(\ \boldsymbol{C}\equiv\boldsymbol{A}\boldsymbol{B}= [c_{ij}]_{m\times n}\ \) are matrices over a field \(\ K.\ \) Then

(1)\[c_{ij}\ =\ \,\sum_{k=1}^p\ a_{ik}\ b_{kj}\,,\qquad i=1,2,\ldots,m;\ j=1,2,\ldots,n.\]

Equation (1) may be rewritten as

\[ \begin{align}\begin{aligned}c_{ij}\ = \ \displaystyle\sum_{v=1}^p\ a_{iv}\ b_{vj} \ = \ \displaystyle\sum_{v=1}^p\sum_{w=1}^n\ a_{iv}\ \delta_{jw}\ b_{vw} \ = \ \displaystyle\sum_{v=1}^p\sum_{w=1}^n\ \left(\boldsymbol{A}\otimes\boldsymbol{I}_n\right)_{ij,\,vw}\ b_{vw}\\i=1,2,\ldots,m;\ j=1,2,\ldots,n.\end{aligned}\end{align} \]

which is equivalent to the matrix equation

\[\blacktriangleright\quad \boldsymbol{\Lambda}^{mn}(\boldsymbol{A}\boldsymbol{B})\ \,=\ \, \left(\,\boldsymbol{A}\otimes\boldsymbol{I}_n\,\right)\ \cdot\ \boldsymbol{\Lambda}^{pn}(\boldsymbol{B})\]

On the other hand, Eq. (1) yields also

\[c_{ij}\ = \ \displaystyle\sum_{w=1}^p\ b_{wj}\ a_{iw} \ = \ \displaystyle\sum_{v=1}^m\sum_{w=1}^p\ \delta_{iv}\ b_{jw}^{\,T}\ a_{vw} \ = \ \displaystyle\sum_{v=1}^m\sum_{w=1}^p\ \left(\boldsymbol{I}_m\otimes\boldsymbol{B}^T\right)_{ij,\,vw}\ a_{vw}\]

wherefrom we obtain an alternative relation for the matrix product \(\ \boldsymbol{A}\boldsymbol{B}\,:\)

\[\blacktriangleright\quad \boldsymbol{\Lambda}^{mn}(\boldsymbol{A}\boldsymbol{B})\ \,=\ \, \left(\,\boldsymbol{I}_m\otimes\boldsymbol{B^{\,T}}\,\right)\ \cdot\ \boldsymbol{\Lambda}^{mp}(\boldsymbol{A})\]

1

see, e.g. https://en.wikipedia.org/wiki/Vectorization_(mathematics)