Maximum Entropy toolbox for MATLAB
If you are familiar with maximum entropy distributions, you might want to skip directly to this code example which demonstrates most of the functionality of the package.
The basic purpose of the maxent toolbox is to learn probability distributions of multi-dimensional patterns of binary data from a limited set of samples, by modeling them as maximum-entropy distributions based on some particular features of the patterns.
As an example, assume there are 20-bit binary patterns: 00000000000000000000, 00000000000000000001, ... , 11111111111111111111 which are distributed in some way - the probability distribution p(x) denotes the probability of encountering each of the patterns. We have obtained 5000 independent samples from p(x) and we want to use these to approximate p(x), i.e. obtain the probabilities of encountering each of the 2^20=1048576 possible patterns. We clump these 5000 patterns of 20 bits in a (20 x 5000) MATLAB matrix called raster:
Name Size Bytes Class Attributes raster 20x5000 400000 uint32
We start by picking which class of maximum-entropy model to use - in this example we will use the pairwise maximum entropy model (sometimes known as Ising model) which is a minimal model based on individual bits and correlations between pairs of bits. We initialize the model:
model = maxent.createModel(20,'ising');
Then we need to train the model. i.e. extract the model parameters from the data:
model = maxent.trainModel(model, raster,'threshold',1);
The optional argument threshold controls how closely we want to fit the model to the experimental data, which is a noisy observation. Setting it to "1" means we allow an error of up to one standard deviation of the expected measurement noise. Note that the measurement noise is smaller when more training samples are available, so this threshold is a relative threshold and not an absolute threshold (refer to the documentation of trainModel function for more details).
Training the model returned a pairwise maximum entropy model trained on the raster dataset. In order to use the model to predict the frequency (probability) of a few input patterns. We can use the getLogProbability function:
x = [0,0,0,0,1,0,0,1,0,0,0,0,0,0,0,0,0,1,0,0;
1,1,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0;
0,1,1,0,0,1,0,0,1,0,0,0,1,0,1,1,0,0,0,0]';
logprobs = maxent.getLogProbability(model, raster);
probabilies = exp(logprobs);
The getLogProbability function returns, for each of the input patterns y, its approximate probability p(y) according to the maximum entropy model.
If the distribution we want to learn is relatively small (less than 30 bits), the default behaviour of the toolbox is to compute the maximum entropy model in an exhaustive fashion. For larger distributions this becomes untractable, and the toolbox uses MCMC (Markov Chain Monte Carlo) methods instead. One drawback of MCMC techniques is that they return an unnormalized model, i.e. its probabilities do not necessarily sum to 1. In order to normalize the model, the package provides an implementation of the Wang Landau algorithm:
model = maxent.wangLandau(model);
Which will return a model which is approximately normalized, and will also approximate the entropy of the model (refer to wangLandau for details).
If you want the obtain of empirical marginals of the data, according to a certain maximum entropy model:
model = maxent.getEmpiricalMarginals(raster,model);
If you want to obtain the marginals of a trained maximum entropy model:
model = maxent.getMarginals(raster);
which returns the exact marginals for small (<30 bits) distributions or approximate marginals for large distributions.
A more complete code example is available, or you can browse through the function reference.