There is a little gem in 2006 that lets you control how large messages are transformed .
Previouslt in 2004 if you had a fairly large transformation , there i s nothing much you could do than scale up the server with more memory. In 2006 you could let BizTalk us the disk after a particular threshold , By default that value us set to 1 MB . That means if your XSLT transformation (which you realize is a tree transformation in memory, Your XML is loaded as the source tree and your declartive Xpath in the xslt is used to match the source tree matches to get the result tree) is larger than 1 MB BizTalk will use the large message transformation engine which will page to disk . This is nice feature for large maps but will slow you down ,Hey it is better than getting an OutofMemoryException and that Administrator's pager going off . You might want to tweak this value (disk is slower than memory ) , Of course you dont want it too high either to beat the whole purpose . You need to figure this value on your available capacity.
To do this you need to change the threshold in the registry
Goto 'HKLM\\Software\\Microsoft\\BizTalk Server\\3.0\\Administration'
and set the DWORD 'TransformThreshold'