Restricted Boltzmann Machines for Invertible Boolean Logic

Madison Saavan Sayeef Manley ,

1 Patel ,
2 Salahuddin2

Department of Electrical and Computer Engineering, University of Central Florida 1

Department of Electrical Engineering and Computer Science, University of California, Berkeley 2

Abstract Error Metric Time Graphs

• Neuromorphic computing comprises of systems that are based on the human brain or Addition Correct output: 33 2-bit Full Adder Logical Units - Addition
artificial neural networks, with the promise of creating a brain inspired ability to learn 100
100

Decimal Value

Decimal Value
and adapt. 80
80 Stuck at incorrect

Accuracy %
Accuracy %
• These systems can be used to solve optimization and machine learning problems. 60 60 value

40 40
• We are trying to analyze the accuracy and variation of Boltzmann Machines
performing invertible Boolean logic by testing the activation function’s range and how 20 20
different kinds of noise can affect the system. 0 0 Sample # Sample # Sample #
0.5 1 1.5 2 5 0.5 1 1.5 2 5 I0 = 0.5 I0 = 1.5 I0 = 5.0
I0 Figure 1: Accuracy of Full Adders
Introduction I0 Figure 5: Sample vs. time graphs
• The accuracy of the Full Adders decrease when the number of bits increase and when I0 • As I0 increases to 1.5, the system becomes more accurate but at I0 = 5.0, the system
becomes greater than 1.5. becomes stuck in local minima.
• Merged Full Adders are more accurate compared to merged logical units.
AND Gate Directed mode Inverted mode Noise
A B C Dynamic Range of Activation
0 0 0 • Clipped Sigmoid Function • The activation function • Three kinds of gaussian noise was added with μ=0 and 𝛔 = {.01, 0.1, 0.2}.
−1 , was clipped to -1 and 1 1. Noise on activation 𝑡𝑎𝑛ℎ 𝐼𝑖 𝑡 + 𝑁(𝜇, 𝜎)
0 1 0 tanh 𝑥 𝑥 < −𝑎
𝑓 𝑥 = , −𝑎 ≤ 𝑥 ≤ 𝑎 to mimic circuit
− tanh −𝑎 implementation. • Noise added to activation function
1 0 0 𝑥>𝑎 0 −1 5 0 𝑁(μ, σ) 𝑁(μ, σ)
1 ,
1 1 1
2. Fixed noise (circuit mismatch) 𝑊 = −1 0 2 + 𝑁(μ, σ) 0 𝑁(μ, σ)
• Linear Function • The clipped sigmoid
−1 , function had a better • Noise on weight matrix 5 2 0 𝑁(μ, σ) 𝑁(μ, σ) 0
𝑥 < −𝑎
𝐴𝑁𝐷(𝑥, 𝑦) = 0, 1 accuracy than the
𝐴𝑁𝐷(1,1) = 𝐶, 𝐶 = 1 𝑓(𝑥 = ൞ 𝑥 ,
𝑎
−𝑎 ≤ 𝑥 ≤ 𝑎
clipped linear function. 3. Non-deterministic Noise 𝐼𝑖 𝑡 = 𝐼0 {ℎ𝑖 + ෍ 𝐽𝑖𝑗 𝑚𝑗 𝑡 + 𝑁(𝜇, 𝜎)}
𝑥, 𝑦 = 0, 0 , 0, 1 , 1, 0 𝑥>𝑎 𝑗
1 , • Noise added to every taken value of weight matrix
• Increasing the range of
• A Restricted Boltzmann Machine is a 3-bit Full Adder - Addition the functions decreases 2-bit Full Adder - Addition
generative model that can learn a 100 the accuracy. 100
probability distribution.1 [-3,3] 100 100
80 80
80
Accuracy %

80

Accuracy %

Accuracy %

Accuracy %
• Connections are only between the hidden 60
60 60 60
and visible layer. This is what makes a 40 40
Boltzmann Machine “restricted.” 40 40
20 20
20 20
• They can partially reconstruct data. 0 0
0.5 1 1.5 2 5 0.5 1 1.5 2 5 0 0
• Probabilistic spin logic (PSL) is based on unstable, stochastic units known as I0 I0 0.5 1 1.5 2 5 0.5 1 1.5 2 5
Linear Figure 2: Accuracy of 3-bit Full Clipped I0 I0
probabilistic bits (p-bits).2 Adder with a linear and clipped Noise on activation Fixed noise
activation function of range [-3,3]. 100
• They can possibly be used to solve NP-hard and optimization problems, as well as Figure 6: Accuracy of 2-bit Full Adder with
precise “invertible” Boolean logic. 80 noise added
Experimental Activation Functions

Accuracy %
60
• p-bits act as a tunable random number generator and their behavior can be 2 2 • Adding noise to the weights had the
40
mathematically described in Eq. 1 and can be interconnected according to Eq. 2. Width #1 Width #2 Figure 3: Graphs of most significant effect on the accuracy
Activation

1 1 experimental 20 of the system.

Activation

(1) (2) 0 activation functions 0

0
-2 -1 0 1 2 0.5 1 1.5 2 5 • In contrast, adding noise to the
-2 -1 0 1 2 -1
• where hi is the bias vector and Jij is the interconnection matrix that can be found -1 I0 activation function had little effect on the
Voltage (V) Voltage (V) Non-deterministic Noise
mathematically or by machine learning techniques. I0 sets the strength of the -2 • The activation accuracy.
-2
interconnections, acting like the inverse “pseudotemperature” of the system.2 2 functions
Width #3 2
1
Width #4
1
correspond to
different width
Conclusions
Activation

Activation

0 ratios for created • We found the accuracy of merged full adders to be much higher compared to full adders
0
-2 -1 0 1 2 -2 -1 0 1 2 bias circuits. made by connecting logical units.
-1 -1
-2 -2 • We found that increasing I0 up to certain value increases the accuracy of the system, but if
Voltage (V) I0 goes beyond this value, the accuracy decreases due to the system possibly becoming
AdditionVoltage (V) stuck in local minima.
2-bit Full Adder 3-bit Full Adder
100 100
• The system seems to be robust as the range of the activation function increases as well as
• With this mathematical prescription, we can turn any 80
Accuracy %

80
Accuracy %

when noise is added to the activation function, but adding noise to the weights significantly
truth table in terms of Boltzmann machines. 60 60 decreases the accuracy.
40 40
20
0
20
0
References
• We can interconnect 0.5 1 1.5 2 5
0.5 1 1.5 2 5 [1] A. Fischer and C. Igel, “An Introduction to Restricted Boltzmann Machines,” Front. Neurosci., vol. 7, pp. 14–36, 2012.
gates by merging Figure 4: Accuracy of I0
I0
matrixes. activation functions [2] K. Y. Camsari, R. Faria, B. M. Sutton, and S. Datta, “Stochastic p-bits for invertible logic,” Phys. Rev. X, vol. 7, no. 3, pp. 1–
19, 2017.
• The accuracy of the experimental activation functions are high when I0 = 0.5, but Support Information: This work was funded by National Science Foundation Award ECCS-0939514
Contact Information: madison.manley@knights.ucf.edu
decrease as I0 becomes greater.