Meta AI published a paper on September 24th, MaD-RL, focusing on a less obvious but very practical issue in large model reinforcement learning: training usually only rewards high scores for a single output, which may cause the model to gradually concentrate its probabilities on one solution or expression. For math problems, converging to an answer with a high success rate might be beneficial; however, in tasks involving synthetic data, strategy exploration, and those that require satisfying population distribution constraints, overly singular outputs can result in the loss of viable options.
The research team proposed the Distribution Matching framework, which does not merely aim to maximize the average reward, but rather to make the distribution of a certain potential category attribute in the model's output approach a pre-set target. The so-called “target distribution” can refer to the proportions of different problem-solving strategies, code structures, or other discernible categories. The paper demonstrates the effectiveness of this method through mathematical reasoning and programming experiments; however, these are still research findings and do not imply that all general models have automatically solved the issue of pattern collapse, nor does it mean that a single set of proportions can be used to control all open-ended texts.
GRPO may push the probability towards a single mode, and adjusting the temperature does not always solve the problem.
Commonly used verifiable rewards in post-training of large models: points are awarded for correct answers, and no points are given for incorrect ones, followed by using reinforcement learning to increase the probability of high-score outputs. Methods like GRPO compare a set of candidate outputs for the same question and then reinforce the samples that perform relatively better. The mechanism is simple and effective, but when multiple answers are correct, training may favor the first dominant pattern, gradually weakening other viable paths.
The paper describes this phenomenon as a concentration of the output distribution. Increasing the sampling temperature can enhance the randomness at the Token level, and entropy regularization can also prevent the probability from becoming too sharp too quickly, but it does not guarantee that the model will achieve the desired proportion in terms of semantic categories. Although there may be many surface changes in the text, the underlying strategies could still be the same; conversely, forcibly pursuing uniformity may not necessarily align with business objectives. Some tasks require a 70% conservative strategy and a 30% exploratory strategy, rather than an equal amount for each category.
MaD-RL transforms the goal into a distribution distance problem. Researchers first define a potential attribute that can classify the output, then compare the current distribution of the model with the target distribution, and use the differences to construct rewards. The paper discusses different degrees of dispersion such as L2, KL, and Jensen-Shannon; each measure has a different sensitivity to rare categories, zero probabilities, and degrees of deviation. The value of the method lies not in a particular formula itself, but in the expansion of the training goal from "whether this answer is good or not" to "whether many answers combined meet the requirements."
This also brings new dependencies: categories must be reliably identifiable. If a classifier misclassifies different strategies as the same, or if the target attribute itself is biased, then distribution matching will only make the errors more stable. Mathematical and programming tasks can easily benefit from validators and structural rules; however, open dialogue, creative writing, and social attributes are more difficult to assign clear, undisputed labels to.
Distributable control does not equate to content fairness; who sets the goals remains the core issue.
The paper mentions applications such as synthetic data and fairness constraints. When the model generates data for downstream training, if a large number of samples repeat the same template, it is difficult to cover real-world variations no matter how large the data scale is; distribution matching can force the retention of multiple types. However, 'diversity' is merely a statistical attribute and does not automatically guarantee that the information is correct, fully representative, or free from harmful content. Both quality and distribution thresholds must be present simultaneously.
In fairness-related scenarios, the risks are higher. Setting different output proportions for different groups may help to correct obvious imbalances, but it may also simplify complex social issues into rigid quotas. The target proportions should come from specific policies, data, and discussions with stakeholders, rather than being arbitrarily determined by training engineers. Models may also learn to cater to classifiers, seemingly meeting the proportions while in reality continuing to replicate biases.
For developers, MaD-RL is more suitable to be viewed as a training tool rather than an out-of-the-box product feature. Before use, it is necessary to clarify the attributes, target distribution, classification accuracy rate, and failure cost; after training, it is also essential to verify on unseen tasks to confirm that the model has not traded off accuracy for other metrics. The mathematical and programming experiments presented in the paper provide evidence of feasibility, but cross-domain generalization still requires more independent reproductions.
This research also reminds the industry to re-examine the definition of “best answer.” Many evaluations only consider the highest score or average accuracy rate, and are unable to determine whether a model relies on a single approach. When AI is used for solution generation, scientific research hypotheses, and code candidates, what users truly need may be a set of different options that all meet quality standards. Therefore, evaluations should also record success rates, distribution coverage, repetition, and calibration errors simultaneously.
After reinforcement learning, training is moving from single-reward approaches to multi-objective constraints. Correctness, safety, cost, and diversity may pull in opposite directions; extreme optimization of any one metric can harm other aspects. The contribution made by MaD-RL is to explicitly incorporate semantic distributions into the optimization objectives and to compare the effects of different distance metrics. While it does not solve the governance issue of 'who defines the ideal distribution', it transforms this problem from a complaint after training into a parameter that must be clearly defined before training begins.
The most cautious judgment at this stage is that the Meta research team has demonstrated that in several verifiable tasks, reinforcement learning can directly match the target output distribution, and they have pointed out that common GRPO training methods may compress diversity. Whether this approach can work stably in larger models, more complex open tasks, and with long-term training still requires further experimentation; however, it has already provided a testable technical path for the idea that "models should not only get the answers right but also retain a reasonable range of choices."
