When you first learn neural networks, you quickly meet the sigmoid activation: a smooth curve that squashes inputs into (0, 1). It feels ideal, continuous, differentiable, and bounded. But there\u2019s also a bipolar sigmoid that outputs values between -1 and 1, and that small change matters a lot during training.<\/p>\n\n\n\n
Which one you pick affects how gradients flow backward through the network. Activations centered around zero (like the bipolar sigmoid) help gradients remain balanced and speed up convergence, while strictly positive outputs (like the standard sigmoid) can bias activations and slow or stall learning. Choosing the right activation can therefore mean the difference between a model that converges slowly or not at all and one that learns quickly.<\/p>\n\n\n\n
In this article, we will walk through everything you need to understand about the bipolar sigmoid function, from its mathematical definition and core properties to how it compares with the standard sigmoid and ReLU, and when it makes sense to use it in your own machine learning projects.<\/p>\n\n\n\n
\n The bipolar sigmoid function is an S-shaped activation function used in neural networks that transforms any input value into an output ranging from -1 to 1. Unlike the standard sigmoid function, which outputs values between 0 and 1, the bipolar sigmoid is zero-centered, helping to produce more balanced gradients during backpropagation and often improving learning efficiency. It is mathematically equivalent to the hyperbolic tangent function ( Before diving into the bipolar sigmoid specifically, it helps to understand what activation functions do and why they matter so much in neural networks.<\/p>\n\n\n\n The bipolar sigmoid is defined as:<\/strong><\/p>\n\n\n\n f(x) = (1 – e^-x) \/ (1 + e^-x)<\/strong><\/p>\n\n\n\n This might seem like a small difference, but as we will see in the next few sections, it has a significant impact on how well neural networks learn.<\/p>\n\n\n\n One thing that surprises many beginners is that the bipolar sigmoid function is not just similar to the hyperbolic tangent; it is mathematically identical to it. The tanh function is commonly written as:<\/p>\n\n\n\n tanh(x) = (e^x – e^-x) \/ (e^x + e^-x)<\/strong><\/p>\n\n\n\n The most important property of the bipolar sigmoid is that its output is centered around zero. This is the defining advantage that separates it from the standard binary sigmoid, and it has real consequences for how quickly and effectively a neural network trains.<\/p>\n\n\n\n The bipolar sigmoid is smooth and differentiable everywhere. That means no abrupt jumps or discontinuities, so gradients are well-defined for all inputs a useful property during optimization because backpropagation relies on those derivatives.<\/p>\n\n\n\n Its S-shaped curve transitions gradually from -1 to +1, and near zero the slope is non\u2011zero and relatively large. Most neuron pre-activations during early training lie near this region, so the bipolar sigmoid provides a healthy learning signal where it matters most.<\/p>\n\n\n\n Outputs are centered around zero, unlike the binary sigmoid. Zero-centered activations lead to more balanced weight updates (less zig-zagging) and help optimizers converge faster because the mean of signals passing through layers is closer to zero.<\/p>\n\n\n\n By producing negative as well as positive values, the bipolar sigmoid lets neurons express inhibitory effects directly. That increases representational flexibility compared with strictly positive activations, helping the network capture symmetric or sign-dependent patterns.<\/p>\n\n\n\n Nonlinear activations like the bipolar sigmoid (and standard sigmoid\/tanh) have derivatives that shrink toward zero for very large or very small inputs. At the extremes of their S-shaped curve, the function becomes nearly flat, so its slope, the derivative, is close to zero.<\/p>\n\n\n\n During backpropagation, gradients are propagated by multiplying derivatives layer by layer. If each layer\u2019s activation derivative is small, the product across many layers becomes exponentially smaller. This is the core mechanism that turns small local derivatives into a globally vanishing gradient.<\/p>\n\n\n\n When gradients vanish, early layers receive almost no learning signal. For example, if an activation\u2019s derivative is about 0.25, after 5 layers the gradient contribution is scaled by 0.2550.25^50.255, producing an extremely small update for the first-layer weights. The network stops improving, not because it\u2019s reached a good solution, but because the training signal has effectively disappeared.<\/p>\n\n\n\n This drawback makes bipolar sigmoid (and sigmoid\/tanh) poor choices for very deep feedforward or convolutional networks. Use non-saturating activations like ReLU or its variants (Leaky ReLU, ELU) for deep architectures, or apply techniques such as careful initialization, batch normalization, or residual connections to mitigate vanishing gradients when you must use saturating activations.<\/p>\n\n\n\n The bipolar sigmoid also carries a computational cost that is worth being aware of, particularly if you are building models that need to run quickly or on limited hardware.<\/p>\n\n\n\n The binary sigmoid outputs values in (0, 1) and is not zero-centered, which causes zig-zagging during optimization. Tanh is zero-centered and has a stronger gradient near zero, so it is preferred over binary sigmoid for internal gating (e.g., LSTM, GRU). <\/p>\n\n\n\n Use the binary sigmoid when you need a probability at the output layer; for hidden layers, the bipolar sigmoid (zero-centered) is a better choice than the binary sigmoid because it reduces optimization zig-zagging.<\/p>\n\n\n\n ReLU and its variants solve vanishing gradients for positive inputs and avoid saturation, giving more stable gradient flow and faster training in deep networks. For most large feedforward and convolutional networks, ReLU or Leaky ReLU are the standards because they handle deep architectures more effectively. <\/p>\n\n\n\n The bipolar sigmoid sits between sigmoid\/tanh and ReLU: better than binary sigmoid for hidden layers due to being zero-centered, but less efficient than ReLU for very deep networks.<\/p>\n\n\n\n Given its limitations, knowing when the bipolar sigmoid is genuinely the right choice matters.<\/p>\n\n\n\ntanh<\/code>) and is commonly used in deep learning applications where centered activations are beneficial.\n <\/p>\n\n <\/div>\n\n<\/div>\n\n\n\nUnderstanding Activation Functions First<\/strong><\/h2>\n\n\n\n
\n
The Standard Sigmoid vs. The Bipolar Sigmoid<\/strong><\/h2>\n\n\n\n
\n
\n
The Relationship Between Bipolar Sigmoid and Tanh<\/strong><\/h2>\n\n\n\n
\n
The Zero-Centered Output: Why It Matters<\/strong><\/h2>\n\n\n\n
\n
Continuous and Non-Linear: The S-Curve Advantage<\/strong><\/h2>\n\n\n\n
\n
\n
\n
\n
The Vanishing Gradient Problem<\/strong><\/h2>\n\n\n\n
\n
\n
\n
\n
Computational Cost<\/strong><\/h2>\n\n\n\n
\n
Bipolar Sigmoid vs. Binary Sigmoid vs. ReLU<\/strong><\/h2>\n\n\n\n
\n
2. Comparison with ReLU<\/strong><\/h3>\n\n\n\n
When to Actually Use the Bipolar Sigmoid<\/strong><\/h2>\n\n\n\n
\n