The flapper aka diaphragm (lol) is allowed to move by way of flexible rubber. I don't think I'd be too far off to assume that when it's really cold, the rubber isn't quite ready to move yet. When you're troubleshooting things, it's important to know how/why the way they do so you're not just pulling/replacing parts all willy-nilly.
Anyway, it would be awesome if we could know what kind of boost you're seeing. A few more questions:
Have you checked all theses lines: i.e. wastegate, bypass valve, wastegate ref on the intake? Also, I know you've said you've checked your clamps, but it's cold so I'm guessing you've checked all but two; the ones on the intercooler. They are a pain if someone on the assy line was pissed off and orientated the clamps in a terrible way. Does this sound have a distinct pitch that rises and falls with RPM?