Académique Documents
Professionnel Documents
Culture Documents
Jack Stillwell1
1 Introduction
In this paper we will be discussing a methodology to obtain recommended config-
urations from performance and configuration data in cases with unreliable labels.
Our implementation utilizes data gathered from SMITE, specifically player per-
formance data and player item data. In SMITE, items alter the statistics of a
player, a player’s allies, or even a player’s opponents. For the purposes of this
2 J. Stillwell
The difficulty faced when presented with the described data is that each data
1 5
instance is only 10 of the data produced by a match in which 10 are assigned
5
a loss label and 10 are assigned a win label. With only a single instance of the
described data, it is uncertain whether the label assigned is the result of the
individual’s contribution or the contribution of their teammates or opponents.
3 Proposed Approach
Third, we use the configuration features of each case associated with the as-
signed positive and negative label to train an explainable classifier which consid-
ers only the individual impact of features. Each potential recommended config-
uration is then assessed by this second classifier and the potential recommended
configurations are ranked by likelihood of belonging to the positive class.
3.1 Clustering
The first step is designed to mitigate the uncertainty in the existing label for
each case – the existing label applies to a larger data instance of which our case
is only a part. Due to having only partial data, we accept the existing label as
informative but not definitive. To obtain more reliable labels we treat the data
as unlabeled and employ an unsupervised learning technique to obtain labels
based upon the performance data associated with each case.
In order to direct the unsupervised learning technique towards the trends
we wish to reveal in the data, we elected to seed the algorithm. Determining
these seeds requires identifying the two instances which best exemplify positive
performance features and negative performance features. Once these instances
have been identified, they are used as the starting point for the unsupervised
learning algorithm.
Once the unsupervised learning algorithm has completed its classification, the
new classes are recorded in each case, replacing the previous unreliable label.
The final step ranks the generated configurations. To accomplish this, we train
a secondary supervised classifier which considers only individual features to de-
termine the label of a case given the case’s rearranged configuration data. We
recommend a Bernoulli Naive Bayes algorithm for this task. Bernoulli Naive
Bayes is recommended due to its completely independent consideration of each
feature. This provides a vital secondary lens through which to view a configu-
ration, capable of recognizing and surfacing the most potent configuration by
individual feature.
After the secondary classifier has been trained, each generated configuration
is evaluated for likelihood of belonging to the desired class. This likelihood is then
used to rank the generated configurations from highest to lowest likelihood.
4 Implementation
We begin by filtering the data obtained to include at least 2000 records for each
possible character, with more records allowed if the age of the records is less
than 2 weeks. We chose 2 weeks as the cutoff because that is the time frame in
which the game is updated by the developers to balance different statistics and
these updates often significantly change how the game is played. Additionally,
we only consider data generated by players currently awarded a rank of Platinum
or above by the in-game ranking system, which indicates that those players are
within the top 50% of players evaluated by number of games won and growth
over the current split which lasts roughly 10 weeks. To account for the extreme
differences in builds for different characters, data is separated by character.
After we have obtained the data and chosen a particular character’s data
to use, the first step is separating the data into 3 matrices with the following
columns (features):
4.2 Clustering
For our implementation we elected to use the KMeans algorithm, but were un-
sure of the impact different seeds might have. To investigate this impact and
sanity check our methodology, we evaluated a variety of techniques, including
Gradient Descent Regression, Relief Regression, and Neural Networks, select-
ing each technique’s most positive and negative output as the seed instance for
KMeans. While each technique provided a different seed instance, the seeded
KMeans algorithm created clusters which were more than 99% similar regard-
less of the seed. These trials confirmed that the trends we were interested in
were present in the data and also gave us several options determining the seed.
In addition to KMeans, we also looked into fuzzy KMeans and DBSCAN as
unsupervised clustering algorithm alternatives. While fuzzy KMeans showed po-
tential, especially with its ability to designate certainty thresholds for acceptance
to a cluster, DBSCAN was unable to find any meaningful patterns in the data.
To reduce the complexity of the experiment (e.g. using different thresholds for
acceptance to each cluster in trial runs) we elected to use the KMeans algorithm,
with possible future work dedicated to discovering the impact of fuzzy KMeans.
The Performance Data Matrix is standardized through zero-centering and
division by standard deviation of each column. After the Performance Data
Matrix has been standardized, it is fed to a Gradient Descent Logistic Regression
algorithm, using the Win Data Matrix as the target column. After the Gradient
Descent Logistic Regression algorithm has been trained, it is used to predict the
case in the Performance Data Matrix most likely to be a win and the case in
the Performance Data Matrix most likely to be a loss. These two cases are then
used to seed a 2-cluster KMeans clustering algorithm.
After the clustering completes, the cluster containing the most wins is as-
signed a positive label and the cluster containing the most losses is assigned a
negative label which supplants the Win Status label. These positive and nega-
tive labels are then correlated with their respective Item Data cases in the Item
Data Matrix.
At this point, we consider the uncertainty in the data to be minimized
through the application of an unsupervised algorithm operating without knowl-
edge of the original label.
Now that we have our uncertainty-mitigated labels, the Item Data Matrix must
take a form in which the relative strengths and importance of each possible
item and their combinations can be evaluated. First, a matrix is created with a
column for each of the 264 possible items and a row for each case, with default
values of 0. If the item was purchased in that match, a 1 replaces the 0 at the
intersection of that column and row. Next, the resulting Item Purchase Matrix
is pruned to contain only columns in which an item was purchased during 3%
or more of cases.
Recommending Configurations based on Cases with Unreliable Labels 7
The next step is to train a decision tree using the Item Purchase Matrix, with
the respective positive and negative cluster labels used as the target column for
each instance.
Our decision tree functions by splitting on the feature resulting in minimally
random children. In more technical terms, it selects the feature for which a
split results in the optimal information gain. This means that items selected
first, towards the top of the tree, are more significant than items selected later,
towards the bottom. To generate potential build recommendations, we trace the
decision tree from the top down, recording the items which are recommended for
purchase until at least 4 and at most 6 items have been recommended, returning
only traces which terminate in a 50% or higher chance of a positive label. This
methodology has the additional effect of capturing item synergies, as items which
synergize with each other will appear consistently in traces together.
Recommended builds are constructed by analyzing each returned trace and
grouping any trace with 5 similar items into a core and their differing items as
optional.
Finally, we train a Bernoulli Naive Bayes algorithm on the same data as our de-
cision tree algorithm and use it to rank the generated configurations. Bernoulli
Naive Bayes (BNB) assumes that each feature (item) affects the results indepen-
dently and therefore does not capture these essential synergies. However, BNB
performs the function of capturing individual item strength, which can then be
used to evaluate the relative strength of different synergistic builds constructed
by the ID3 algorithm.
After the potential build recommendations have been generated from the
decision tree trace, each recommendation core is classified using the BNB algo-
rithm, and the 3 recommendations with the highest likelihood of belonging to
the positive cluster are presented to the user.
Item Value
Warrior’s Blessing -1.61
Ninja Tabi -0.81
Pestilence -0.88
Winged Blade -3.1
Shogun’s Kusari -1.19
Void Shield -0.93
6 Conclusions
References
1. Quinlan, J. Ross.: C4. 5: programs for machine learning. Morgan Kaufmann Pub-
lishers Inc., San Francisco, CA, USA (1993)
2. Basu, S., Banerjee A., Mooney, R.: Semi-supervised clustering by seeding. In: 19th
International Conference on Machine Learning, pp. 19–26. ICML, Sydney, Australia
(2002)