The problem is in the PerformRedirect method in Sitecore.Pipelines.HttpRequest.ExecuteRequest class. You'll have to create a new class that inherits the ExecuteRequest class and override the PerformRedirect method. Instead of executing HttpContext.Current.Server.Transfer(url) you'll need to add some logic that will make sure the url passed is one to an Item, then you'll have to force the request through the httpRequestBegin pipeline again (actually all you really have to do is run it through the LayoutResolver process, but the only way to do this without ripping out and modifying the LayoutResolver code is to force it through the entire pipeline).

Code example below:

    public class ExecuteRequest : Sitecore.Pipelines.HttpRequest.ExecuteRequest
    {
        protected override void PerformRedirect(string url)
        {
            if (Settings.RequestErrors.UseServerSideRedirect)
            {
                Sitecore.Web.WebUtil.RewriteUrl(url);
                Context.Item = GetItemFromUrl(url);
                if (Context.Item != null)
                {
                    HttpRequestArgs args = new HttpRequestArgs(HttpContext.Current, HttpRequestType.Begin);
                    Sitecore.Pipelines.CorePipeline.Run("httpRequestBegin", (Sitecore.Pipelines.PipelineArgs)args);
                }
            }
            else
            {
                WebUtil.Redirect(url, false);
            }
        }

        ///
        ///     Returns an Item from a URL.
        ///
        ///
The url of the sitecore item.
        /// The Item the url represents.
        public static Item GetItemFromUrl(string url)
        {
            string referring_item_path = null;
            if (Uri.IsWellFormedUriString(url, UriKind.Relative))
                url = Sitecore.Web.WebUtil.GetFullUrl(url);
            if (Uri.IsWellFormedUriString(url, UriKind.Absolute))
            {
                Uri uri = new Uri(url);
                if (uri.AbsolutePath.StartsWith("/sitecore/"))
                    referring_item_path = uri.AbsolutePath;
                else
                    referring_item_path = Sitecore.Context.Site.StartPath + uri.AbsolutePath;
            }
            //Remove the extension and put a '/' instead.
            referring_item_path = Regex.Replace(referring_item_path, @"(?:\.aspx|\.ashx)", "/");
            //Declare a temp return value variable.
            Item retVal = null;
            using (new Sitecore.SecurityModel.SecurityDisabler())
            {
                //Get the item based off the sitecore path.
                retVal = Sitecore.Context.Database.GetItem(referring_item_path);
            }
            //return the item.
            return retVal;
        }
    }

This new class needs to replace the current process in the httpRequestBegin pipeline. That can be accomplished by placing a .config file in the /App_Config/Include directory with the following XML.

  

      
        

          your.namespace.ExecuteRequest, yourAssembly
        
    
  

Or if you're ok with modifying the web.config directly replace the type attribute in the below line in your web.config file.
It should look like this:

<processor type="Sitecore.Pipelines.HttpRequest.ItemResolver, Sitecore.Kernel" />
<processor type="Sitecore.Pipelines.HttpRequest.LayoutResolver, Sitecore.Kernel" />
<processor type="your.namespace.ExecuteRequest, yourAssembly" />

string sURL = Sitecore.Web.WebUtil.GetServerUrl(Request.Url, true) + "/404.aspx";
WebRequest wrGETURL;
wrGETURL = WebRequest.Create(sURL);
Stream objStream;
objStream = wrGETURL.GetResponse().GetResponseStream();
StreamReader objReader = new StreamReader(objStream);
string sLine = "";
int i = 0;
while (sLine != null)
{
    i++;
    sLine = objReader.ReadLine();
    if (sLine != null)
        Response.Output.WriteLine(sLine);
}
Context.Response.StatusCode = 404;
Context.Response.Status = "404 not found";

The other way to do it is by creating a new layout with all the controls and HTML structure in the layout. Don't rely on Sitecore's dynamic rendering binding and set the Sitecore.Context.Item on page load.

That code looks like this:

Server.Transfer("/layouts/404.aspx", true);

The preserveForm bool parameter is optional.
The only problem with this - at least for my instance - is Session state is lost. Hunting for a Solution.

UPDATE*** Before I was even able to publish this post, I've found a solution to the Session State issue above and found a way to force a page through sitecore's httpRequestBegin pipeline (actually a little slower than I thought...).

This solution may not fit the purpose for everyone or every purpose, but here goes:

This - as far as I can tell - needs to be in the httpRequestBegin pipeline before the page starts rendering. For my purpose, I was trying to use the Sitecore setting RequestErrors.UseServerSideRedirect=true so the url of the requested item doesn't change on a 404 error (ItemNotFound). I needed to override the PerformRedirect method in Sitecore.Pipelines.HttpRequest.ExecuteRequest class. To do so I created a new class that would be replacing this one in the httpRequestBegin pipeline. However, you don't need to replace the Sitecore.Pipelines.HttpRequest.ExecuteRequest class, you can create your own process and add it to the end of the httpRequestBegin pipeline.

    public class ExecuteRequest : Sitecore.Pipelines.HttpRequest.ExecuteRequest
    {
        protected override void PerformRedirect(string url)
        {
            if (Settings.RequestErrors.UseServerSideRedirect)
            {
                Sitecore.Web.WebUtil.RewriteUrl(url);
            }
            else
            {
                WebUtil.Redirect(url, false);
            }
        }
    }

Note that this will only work if the URL is to an .aspx page and in the page load method set the Sitecore.Context.Item to whichever item you'd like. This solution will not work for Sitecore Items.

If you'd like to server side redirect to a Sitecore Item, please find out how to do that here.

Now you have to replace the in the web.config, you can do it by modifying the web.config directly OR if you're like me and would like to keep the web.config file as close to the release version as possible, then you can add a .config file to the /App_Config/Include folder with something similar to the below XML.

          your.namespace.ExecuteRequest, yourAssembly

Now the page you previously tried to Server.Transfer to that didn't have state, now has session state!

You need to create a new class with a Process method like so:

using System.Text.RegularExpressions;
using Sitecore.Pipelines.RenderField;

namespace your.namespace.here
{
    public class special_character_replacer
    {
        // Methods
        public void Process(RenderFieldArgs args)
        {
            args.Result.FirstPart = Regex.Replace(args.Result.FirstPart, @"&(?!.{2,5};)(?!\w*=)", "&");
            //More special character replacements can go here.
        }
    }
}

*** notice the first &amp;. This is necessary as it's in the web.config and needs to be encoded, but will search for "&". Also, keep in mind that the Ampersand may already be encoded, don't replace them, and don't replace any ampersands in URLs.

Then you need to add the processor you just created to the renderField pipeline like so:

That's it. Now just sit back and watch the encoding begin.

It was working yesterday and the code didn't change, so what was it?

Digging a little deeper I found this in the immediate window of VS2008.