Theses and Dissertations

Permanent URI for this collection

https://testres.nu.edu.kz/handle/123456789/906

Browse

Browsing Theses and Dissertations by Author "Assylbekov, Zhenisbek"

Now showing 1 - 2 of 2
  • Thumbnail Image
    ItemOpen Access
    Convergence Rate of Fourier Neural Networks
    (Nazarbayev University School of Science and Technology, 2019-04-26) Zhumekenov, Abylay; Assylbekov, Zhenisbek; Tourassis, Vassilios D.
    The paper investigates a convergence rate for 2-layer feedforward Fourier Neural Network (FNN). Such networks are motivated by the approximation properties of wellknown Fourier series. Several implementations of FNNs were proposed since 1990’s: by Gallant and White; A. Silvescu; Tan, Zuo and Cai; Liu. The main focus of this research is Silvescu’s FNN, because such activation function does not fit into the category of networks, where the linearly transformed input is exposed to activation. The latter ones were extensively described by Hornik in 1989. In regard to non-trivial Silvescu’s FNN, its convergence rate is proven to be of order 𝑂(1/𝑛). The paper continues investigating classes of functions approximated by Silvescu FNN, which appeared to be from Schwartz space and space of positive definite functions.
  • Thumbnail Image
    ItemOpen Access
    Explorations on chaotic behaviors of Recurrent Neural Networks
    (Nazarbayev University School of Science and Technology, 2019-04-29) Myrzakhmetov, Bagdat; Assylbekov, Zhenisbek; Takhanov, Rustem; Tourassis, Vassilios D.
    In this thesis work we analyzed the dynamics of the Recurrent Neural Network architectures. We explored the chaotic nature of state-of-the-art Recurrent Neural Networks: Vanilla Recurrent Network, Recurrent Highway Networks and Structurally Constrained Recurrent Network. Our experiments showed that they exhibit chaotic behavior in the absence of input data. We also proposed a way of removing chaos chaos from Recurrent Neural Networks. Our findings show that initialization of the weight matrices during the training plays an important role, as initialization with the matrices whose norm is smaller than one will lead to the non-chaotic behavior of the Recurrent Neural Networks. The advantage of the non-chaotic cells is stable dynamics. At the end, we tested our chaos-free version of the Recurrent Highway Networks (RHN) in a real-world application. In a sequence-to-sequence modeling experiments, particularly in the language modeling task, chaos-free version of RHN perform on par with the original version by using the same hyperparameters.