An ABS brake system modulator does exactly what it sounds like it does: it modulates pressure between individual wheels to keep them right on the verge of lockup. If the computer is your ABS's brain, and the wheel-speed sensors its eyes, then the modulator valve body is its muscle.
When you push on the brake pedal, you're pushing on a rod connected to the back of a piston. This piston sits inside of your car's master cylinder, where it pushes fluid through the lines and to the individual wheels' brake cylinders. When a tire loses traction for whatever reason, pressure supplied by the system causes the brake pads to lock against the rotor or drum, bringing the wheel to a sudden halt called "lockup." Once that happens, the tire slides and is essentially useless.
Without ABS, you'd have to lift your foot off the brake pedal in order to get the wheels spinning and useful again. An ABS system works by doing this for you not only with the system as a whole, but with each wheel individually. But instead of "lifting off the pedal," an ABS modulator reduces pressure to the locked-up wheel by sending some of it back to the brake fluid reservoir through a bypass line. When the computer detects lockup through the wheel-speed sensors, it momentarily opens the bypass valve and, thus, reduces the fluid pressure being applied to that wheel.
Open systems are the simplest type, but they have one serious drawback. An open system works just as described above, sending excess fluid back to the chamber under lockup. The problem with this arrangement is that every time the modulator releases pressure, the brake pedal drops a little bit. So, under sustained braking events, either the brake pedal will drop to the floor, or the driver will have to lift to restore pressure. Some open systems, though, use a pump to re-supply the master cylinder with fluid to keep the pedal from dropping.
With a closed system, vented pressure goes either back to the reservoir or to a spring-loaded cylinder called an accumulator. The accumulator holds pressurized fluid on standby for quick release back into the system after the initial pressure drop. If there isn't enough pressurized fluid stored in the accumulator, a brake fluid pump sends pressurized fluid to it to refill it. A closed system operates in a constant vent-pressurize cycle to keep fluid pressure at the highest it can be without inducing lockup. Because of this, closed systems are typically a bit better at maintaining the type of "threshold braking" needed to extract maximum performance from the brakes.
