Toyota Repair: Camry V6 extremely rough cold start, valve lifters, valve lifter

Question
My 1996 Camry V6 has 150,000 miles on it and has had a wonderful service and reliability record.  Maintenance has been performed very faithfully and the car has run Mobil One synthetic since day one.  Within the last 6 months, when the car has not been started for several days (4 or more), it always starts quickly but SOMETIMES runs extremely rough with what sounds to me like valve clatter.  It's obnoxious enough that one is scared to leave the engine running.  Restarting it does not seem to help, but if you leave it running it will smooth out within 15-30 seconds and sounds/runs perfectly after that.  Subsequent starts are perfect.

A couple of days ago, it had sat outside for about 15 hours with temperatures below freezing and did the rough/banging at start.  Again, the engine soon smoothed out and performed like nothing was ever wrong.

Winter is virtually here on Oklahoma and I am starting to get worried about damaging the engine.  Where should I start and how do I go about diagnosing this issue.

Answer
There is an internal engine problem and it sounds like it has to do with the valves, possibly a sticking valve guide, or a sticking valve lifter, this engine uses shim type valve lifters and they are adjustable by changing the shims, it is possible that one of them is hanging up until it gets lubrication to it.
I would recommend leaving the car at a toyota dealer overnight and have them start the engine in the AM so they can hear the noise and make some sort of diagnosis, I'm just giving you my best guess because I can't hear the actual noise.