When opening up a new page (window.open) from a modal dialog page (window.showmodaldialog), ASP.NET may request authentication again by redirecting the user back to a login page. The reason is IE may have lost its session. Problem and solution to this problem is described in http://support.microsoft.com/kb/831678.
This problem happens to both IE 6 and IE 7.