Bilkent University

Department of Computer Engineering

CS690 SEMINAR

SPECTRAL NORMALIZATION ON ORDINARY DIFFERENTIAL EQUATION NEURAL NETWORKS

Saeid Karimi

PhD Student

(Supervisor: Asst. Prof. Dr. Hamdi Dibeklioğlu)

Computer Engineering Department

Bilkent University

Ordinary differential equation neural networks (ODE nets) are new family of deep neural network models. Instead of specifying a discrete sequence of hidden layers. They parameterize the derivative of the hidden state using a neural network. The output of the network is computed using a blackbox differential equation solver. These continuous-depth models have constant memory cost and adapt their evaluation strategy to each input. They demonstrate these properties in continuous-depth residual networks and continuous-time latent variable models. In this research, we analyze the effect of normalizing methods (spectral normalization and orthogonal normalization) in Ordinary differential equation neural networks (ODEnets) for two metrics: loss and number of evaluation functions (NFEs). We found that by applying these normalization methods NFEs decrease, which is an important factor in training time of ODE-nets, however, we encountered a bit increase in the training loss of the networks. So, by decreasing the NFEs and hence, training time, the ODE nets are more suitable to be used in generative models, since ode nets are reversible networks.

DATE: 03 May 2021, Monday @ 15:30