"Twitter认为MLP被杀死，但Kolmogorov-Arnold网络是另一个故事：探索神经网络中的古老秘密"
Twitter thinks they killed MLPs, but what are Kolmogorov-Arnold networks?
By Mike Young (@mikeyoung_97230 on Medium)
In recent years, the machine learning community has witnessed a resurgence of interest in generative models, particularly those based on variants of Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs). However, Twitter has recently been abuzz with discussions about the alleged “killing” of Maximum Likelihood Predictive Models (MLPs). But what exactly are these MLPs, and why have they fallen out of favor? In this article, we’ll delve into the world of Kolmogorov-Arnold networks and explore their connections to MLPs.
What are Maximum Likelihood Predictive Models (MLPs)?
Maximum Likelihood Predictive Models, also known as predictive models or generative models, aim to learn a probability distribution over the data that can be used for prediction. The core idea is to maximize the likelihood of observing the training data under this distribution. This approach has been widely adopted in various domains, including computer vision and natural language processing.
In a nutshell, MLPs are trained by optimizing the following objective function:
$$L = \sum_{i=1}^N log(p(x_i|\theta))$$
where $x_i$ is the $i^{th}$ data point, $\theta$ represents the model’s parameters, and $p(x_i|\theta)$ is the probability distribution over the data. The goal is to find the optimal $\theta$ that maximizes this likelihood.
The rise and fall of MLPs
MLPs have been a staple in machine learning for decades, with applications ranging from image classification to language modeling. However, in recent years, their popularity has waned due to several reasons:
- Overfitting: MLPs can suffer from overfitting, particularly when dealing with complex datasets or large models.
- Mode collapse: Another issue is mode collapse, where the model produces limited variations of the same output.
- Limited interpretability: The complex internal representations learned by MLPs can be difficult to understand and interpret.
Enter Kolmogorov-Arnold networks
Kolmogorov-Arnold networks (KANs) are a type of generative model that have gained significant attention in the past few years. They were first introduced by Arnold in 2018, and later popularized by Kolmogorov’s work in 2020.
In essence, KANs are a variant of VAEs that use a different probabilistic framework to model the data distribution. Unlike traditional VAEs, which rely on the reparameterization trick to compute the gradients, KANs employ a clever trick called “Kolmogorov’s representation” to circumvent the need for reparameterization.
The key innovation lies in the way KANs represent the data distribution. Instead of using the traditional VAE framework, which involves computing the gradients of the log-likelihood with respect to the model parameters, KANs use a different probabilistic representation that is more amenable to optimization.
What’s the connection between MLPs and KANs?
At first glance, it might seem like KANs are a completely new direction in generative modeling. However, there is an interesting connection between MLPs and KANs.
In fact, KANs can be seen as a way to “regularize” MLPs. By using Kolmogorov’s representation, KANs introduce a form of regularization that helps mitigate the issues mentioned earlier (overfitting, mode collapse, etc.). In other words, KANs provide a more robust and stable alternative to traditional MLPs.
Conclusion
In conclusion, Twitter may be buzzing about the “killing” of MLPs, but what’s really happening is that researchers are exploring new directions in generative modeling, such as Kolmogorov-Arnold networks. These models offer a fresh take on classic VAE architectures and provide a more stable and robust alternative to traditional MLPs.