*** Update (12/5): Had a call w/the MS support rep and it looks like this is going to be an issue that'll be resolved in an upcoming CU. So keep an eye out! - M ***
*** Update (11/29): Got off of a call w/MS support and we were able to duplicate this issue/behavior in their environments. They will be addressing this and getting back to me in the near future. When I receive an update I'll put it out here. Thanks! - M ***
Ok, this issue was busting my you-know-whats for a couple weeks now, but I now can duplicate the issue. So, that's half the battle. What I need to figure out now is the specific cause, but what I wanted to do was get out to the community the actual "how it happens" blurb. So, here we go.
When you have Office Web Apps installed, SharePoint exposes an option to view the relevant piece of content in the browser (see Figures 1 & 2 below).

(Figure 1)

(Figure 2)
One of the additional Site Collection Features installed with OWA is the Open Documents in Client Applications by Default Feature. This feature, per the actual description says "Configures links to documents so they open in client applications instead of Web applications, by default". Basically, click an Office document (.DOCX, .XLSX, .PPTX, etc.) and it'll open up in its respective client app (Word, Excel, PowerPoint). Deactivating this feature means that SharePoint will then open up the item in the browser itself (TechNet Office Web Apps Overview).
The issue under discussion here is exposed when you perform a search, and the result includes Excel documents. When you perform a search, and the results have an Excel (.XLSX) document, and you click the View In Browser link for that Excel item (as in the above Figure 2), the behavior isn't always what is expected. If you perform these steps, and the Open Documents in Client Applications by Default Feature is activated, the Excel spreadsheet opens up in the Excel client. When this feature is not active, clicking the View In Browser link opens it up in the browser. So basically, the View in Browser link in the Search Results page seems to "look at" the status of this feature, and then render it accordingly.
For the other types, Word documents and PowerPoint decks, if you click the View In Browser link in the results of a search, it functions as you'd expect, independent of that feature being activated or not.
To me, this is not expected behavior...when you click View In Browser, all things being equal, that's what you expect. The fact that when you do so, for Excel document in a search results page, and it tries to open up in the client application, seems to me an...inconsistency.
I've tested this on a number of environments, pre and post SP1, and it's consistently showing the above behavior.
I'm still in the process of researching a more detailed reason as to the above, and if there's a fix out there. If anyone's seen the above, and has a fix (the fix cannot be disable the feature) I'd love to hear bout it and put it in this post. Note that commenting has been turned off of late here (spammers are *killing* me!) but you can email me and I'll add it to the post.
Thanks! - M