张量分解 (Tensor Decomposition)

君有张良计, 俺有过墙梯

对于n维张量 T^n \in \mathbb{R}^{d_1 \times d_2 \times \cdots \times d_n}(形式化來看: 张量是維度上是向量的量),

有相关量:

范数(norm):

张量 T^n 的 p范数:

\|T^n\|_p= \sqrt[p]{\sum_{i_{1}=1}^{d_{1}} \sum_{i_{2}=1}^{d_{2}} \cdots \sum_{i_{n}=1}^{d_{n}} (\tau_{i_{1} i_{2} \cdots i_{n}}^{p}) }

内积(inner-product):

對於兩個张量 {T^n_1,T^n_2} \in \mathbb{R}^{d_1 \times d_2 \times \cdots \times d_n},其閒内積為:

T^n_{[1]} \cdot T^n_{[2]} = \sum_{i_{1}=1}^{d_{1}} \sum_{i_{2}=1}^{d_{2}} \cdots \sum_{i_{n}=1}^{d_{n}} (\tau_{[1]i_{1} i_{2} \cdots i_{n}} \cdot \tau_{[2]i_{1} i_{2} \cdots i_{n}})

T^n \cdot T^n =\|T^n\|_2^2

有相關特化:

一階张量 (Rank-One Tensors):

张量 T^n \in \mathbb{R}^{d_1 \times d_2 \times \cdots \times d_n} 是一階张量滿足:

T^n = a_{[1]} \times a_{[2]} \times \cdots \times a_{[n]}, 其中 a_{[i]} \in \mathbb{R}^{d_i}

即有: \tau_{i_{1} i_{2} \cdots i_{n}}=a_{[1]i_{1}} \cdot a_{[2]i_{2}} \cdots a_{[n]i_{n}}

對角张量 (Diaginal Tensors):

只有對角綫上值不爲0

張量乘法

\left(\boldsymbol{X} \times_{n} \mathbf{U}\right)_{i_{1} \cdots i_{n-1} j i_{n+1} \cdots i_{N}}=\sum_{i_{n}=1}^{I_{n}} x_{i_{1} i_{2} \cdots i_{N}} u_{j i_{n}}

張量元乘張量:

\mathbf{A} \otimes \mathbf{B}=\left[\begin{array}{cccc}{a_{11} \mathbf{B}} & {a_{12} \mathbf{B}} & {\cdots} & {a_{1 J} \mathbf{B}} \\ {a_{21} \mathbf{B}} & {a_{22} \mathbf{B}} & {\cdots} & {a_{2 J} \mathbf{B}} \\ {\vdots} & {\vdots} & {\ddots} & {\vdots} \\ {a_{I 1} \mathbf{B}} & {a_{I 2} \mathbf{B}} & {\cdots} & {a_{I J} \mathbf{B}}\end{array}\right]

列乘:

\mathbf{A} \odot \mathbf{B}=\left[\begin{array}{lllll}{\mathbf{a}_{1} \otimes \mathbf{b}_{1}} & {\mathbf{a}_{2} \otimes \mathbf{b}_{2}} & {\cdots} & {\mathbf{a}_{K} \otimes \mathbf{b}_{K}}\end{array}\right]

數乘(Hadamard):

\mathbf{A} * \mathbf{B}=\left[\begin{array}{cccc}{a_{11} b_{11}} & {a_{12} b_{12}} & {\cdots} & {a_{1 J} b_{1 J}} \\ {a_{21} b_{21}} & {a_{22} b_{22}} & {\cdots} & {a_{2 J} b_{2 J}} \\ {\vdots} & {\vdots} & {\ddots} & {\vdots} \\ {a_{I 1} b_{I 1}} & {a_{I 2} b_{I 2}} & {\cdots} & {a_{I J} b_{I J}}\end{array}\right]

张量分解

CP分解 (CANDECOMP/PARAFAC Decomposition, CP)

去皮分解 (Tucker Decomposition)

将一个张量分解成一个核张量与每一维矩阵的乘积

連式分解 (Tensor-train Decomposition)

去皮分解是把張量看成對象; 連式分解是把張量看成運算

還式分解 (Tensor-ring Decomposition)

T\left(i_{1}, i_{2}, \ldots, i_{d}\right)=\operatorname{Tr}\left(\mathbf{Z}_{2}\left(i_{2}\right), \mathbf{Z}_{3}\left(i_{3}\right), \ldots, \mathbf{Z}_{d}\left(i_{d}\right), \mathbf{Z}_{1}\left(i_{1}\right)\right) =\cdots=\operatorname{Tr}\left(\mathbf{Z}_{d}\left(i_{d}\right), \mathbf{Z}_{1}\left(i_{1}\right), \ldots, \mathbf{Z}_{d-1}\left(i_{d-1}\right)\right)

Tensorizing Neural Network

^[1]

^[2]

参考

1. ^Tensor Decompositions and Applications.  https://doi.org/10.1137/07070111X
2. ^Tensor Ring Decomposition. arXiv:1606.05535 https://arxiv.org/pdf/1606.05535.pdf