Maths for programmers: binary, hexadecimal, modulo and why 0.1 + 0.2 is not 0.3

You do not need much mathematics to program, but the little you need comes up constantly, and it is usually taught badly or not at all. This tutorial covers four things: how numbers are written in different bases and why hexadecimal is everywhere; using binary digits as flags; the modulo operator, including the way it misbehaves with negatives; and floating-point numbers, which explain a whole family of "the computer cannot add" bugs. Examples are in JavaScript, but the facts hold in every language.

Number bases

Decimal uses ten digits and each position is worth ten times the one to its right: 253 is 2×100 + 5×10 + 3×1. Binary (base 2) uses two digits and positions worth 1, 2, 4, 8, 16, …; hexadecimal (base 16) uses sixteen digits, 0 to 9 then A to F, with positions worth 1, 16, 256, …. The same number three ways:

decimal       253 = 2*100 + 5*10 + 3*1
binary   11111101 = 128 + 64 + 32 + 16 + 8 + 4 + 0 + 1
hex            FD = 15*16 + 13

Hexadecimal is popular because one hex digit is exactly four binary digits, so a byte (eight bits) is always two hex digits. That is why colours are #RRGGBB: three bytes, red, green and blue, each 00 to FF (0 to 255). #FD0000 is red at 253 out of 255.

Converting in JavaScript:

(253).toString(2)      // '11111101'
(253).toString(16)     // 'fd'
parseInt('fd', 16)     // 253
parseInt('11111101', 2) // 253
0xFD                   // 253  (hex literal)
0b11111101             // 253  (binary literal)
Number('0xFD')         // 253

To convert by hand from decimal to binary, divide by two repeatedly and read the remainders bottom-up: 13 → 6 r1 → 3 r0 → 1 r1 → 0 r1, so 13 is 1101. To go the other way, add up the place values of the 1s.

Bits as flags

Because each binary digit is independent, one integer can store several yes/no values, and the bitwise operators test and set them. This appears in file permissions (chmod 755: each octal digit is three bits, read-write-execute), in keyboard event modifiers, and in many APIs' option fields:

const READ = 0b100, WRITE = 0b010, EXEC = 0b001;
let perms = READ | WRITE;        // 0b110 = 6: set both bits (OR)
perms & WRITE                    // 2, truthy: is the bit set? (AND)
perms & EXEC                     // 0: not set
perms = perms & ~WRITE;          // 0b100: clear a bit (AND with NOT)
perms ^= READ;                   // toggle a bit (XOR)
1 << 3                           // 8: shift left = multiply by 2^n

In JavaScript the bitwise operators work on 32-bit signed integers, so they are only safe up to 2³¹−1 and ~x on a large number gives a surprise. For flags that is never a problem; for arithmetic on big values, do not use them.

Modulo

a % b is the remainder after dividing a by b: 17 % 5 is 2. Its everyday uses:

• Even or odd: n % 2 === 0.
• Wrapping around: the next slide in a carousel of n is (i + 1) % n; a clock hour is (h + 13) % 12.
• Every k-th item: i % k === 0, for striped table rows or batching.
• Extracting digits: n % 10 is the last decimal digit, and Math.floor(n / 10) drops it.
• Seconds to minutes and seconds: Math.floor(s / 60) and s % 60.

The trap is negative numbers. In JavaScript, C, Java and Go, the result takes the sign of the dividend (the left operand): -1 % 5 is -1, not 4. In Python and Ruby it takes the sign of the divisor and gives 4. So "previous slide" written as (i - 1) % n gives -1 when i is 0 in JavaScript, and the array lookup returns undefined. The portable fix:

const mod = (a, b) => ((a % b) + b) % b;
mod(-1, 5)   // 4
mod(7, 5)    // 2

Modulo also works on non-integers in JavaScript (5.5 % 2 is 1.5), unlike in C, which has a separate fmod.

Floating point

JavaScript numbers (and double in most languages) are 64-bit IEEE 754 floating point: one sign bit, 11 bits of exponent, 52 bits of fraction. A number is stored as a binary fraction times a power of two, which has a consequence that surprises everyone once: just as 1/3 cannot be written exactly in decimal (0.3333…), 1/10 cannot be written exactly in binary. The nearest double to 0.1 is 0.1000000000000000055511151231257827…. So:

0.1 + 0.2              // 0.30000000000000004
0.1 + 0.2 === 0.3      // false
(0.1 + 0.2).toFixed(2) // '0.30'
1.005 * 100            // 100.49999999999999  (so Math.round gives 100, not 101)

This is not a JavaScript bug; Python, Java and C give the identical result. The rules for living with it:

1. Never compare floats for equality. Check whether they are close: Math.abs(a - b) < Number.EPSILON * Math.max(1, Math.abs(a), Math.abs(b)), or a tolerance that fits your data (1e-9 is common).
2. Do money in integers. Store cents (or the smallest unit) as whole numbers; 1999 + 1 is exact, 19.99 + 0.01 is not. Divide by 100 only for display.
3. Round at the end, once. Rounding intermediate results compounds the error.
4. Format for display with toFixed or Intl.NumberFormat, and note that toFixed returns a string.

Integers are exact up to 2⁵³ (Number.MAX_SAFE_INTEGER, 9,007,199,254,740,991). Beyond that, adjacent integers cannot be distinguished: 2 ** 53 + 1 === 2 ** 53 is true. For IDs and anything larger, use BigInt (9007199254740993n) or strings.

Three special values: Infinity (from 1 / 0, or overflow), -Infinity, and NaN ("not a number", from 0 / 0, Math.sqrt(-1), or parseInt('x')). NaN is the only value not equal to itself, so test it with Number.isNaN(x). Also -0 exists and equals 0; it matters only in edge cases such as 1 / -0.

Rounding correctly

Math.round rounds half up (towards positive infinity): Math.round(-2.5) is -2, which is not what accountants expect. Math.trunc drops the fraction, Math.floor rounds down and Math.ceil up. To round to two decimals, the common Math.round(x * 100) / 100 suffers from the 1.005 problem above; the robust version nudges the value first:

const round2 = x => Math.round((x + Number.EPSILON) * 100) / 100;
round2(1.005)   // 1.01

Better still, avoid needing it by keeping values as integers in the smallest unit.

Pitfalls

• parseInt('08') is 8 in modern engines, but always pass the radix (parseInt(s, 10)) so the intent is explicit, and prefer Number(s) for whole strings since parseInt('12px') happily returns 12.
• % with a negative left operand goes negative in JavaScript.
• Bitwise operators truncate to 32 bits; |0 as a "to integer" trick breaks above 2³¹.
• toFixed returns a string, and (1.005).toFixed(2) is '1.00' for the same reason as above.
• Summing many floats accumulates error; for a long list of money values, sum integers.

Summary

• One hex digit is four bits; a byte is two hex digits; colours are three bytes.
• Flags: | to set, & to test, & ~ to clear, ^ to toggle; 32-bit only.
• % takes the sign of the left operand; use ((a % b) + b) % b for wrap-around.
• Doubles cannot represent most decimals exactly: compare with a tolerance, keep money in integers, round once at the end.