It's probably yer brake booster. I had that issue about 5 years ago and ended up gitting a used one. I should've gotten a new one instead. The OEM version is discontinued (about $1400) and the aftermarket ones range from about $80 to $500. 
www.toyotanation.com
Here's one off of eBay.

Hard brake pedal...
Symptoms of my hard brake pedal. Whooshing sound heard from near the brake pedal when stepping on the brake pedal. I suspect a bad brake booster in particular the atmospheric release valve which is internal to the brake booster and is not serviceable without disassembling the brake booster...
Here's one off of eBay.

Power Brake Booster Fits 95-01 Toyota Tacoma T100 2.4L I4 | eBay
Toyota T100 1994 2.7L 2694CC l4 GAS DOHC Naturally Aspirated. Toyota T100 1995 2.7L 2694CC l4 GAS DOHC Naturally Aspirated. Toyota T100 1995 3.4L 3378CC V6 GAS DOHC Naturally Aspirated. Toyota T100 1996 2.7L 2694CC l4 GAS DOHC Naturally Aspirated.
www.ebay.com