The one way check valve allows air to flow from the booster to the intake manifold, but not the opposite. It's really there to maintain a vacuum for the booster incase the motor cuts out, allowing about 1-3 brake pedal "pumps" depending on the design of the system.
If the valve malfunctions and allows air to flow in both directions, it won't cause a vacuum leak or make the motor run any different, but you'll have no "assist" in the brakes if the motor is off since there will be no vacuum present in the booster. The dip in idle that you describe is actually normal. Engaging the brakes causes the brake booster to use up engine vacuum, thus creating a minor vacuum leak. If your motor was running too rich, it would have caused your idle to increase instead of decrease.
The main situation that causes a vacuum leak in the braking system is if the internal rubber diaphragm of the brake booster unit springs a leak. Sometimes it's old age, but in every case I've seen, it was a master cylinder that had gone bad, leaking brake fluid into the booster. The brake fluid eats away at the rubber diaphragm, and whala, a vacuum leak is created. This is exactly what happened on my 1959 Pontiac.