1. Since I'm not sure what your page does, I went ahead and created an arbitrary test page that just passes back date as "application/octet" (Arbitrary File):
   "test" (Visualforce page) 

   <apex:page contentType="application/octet">
     <!-- Begin Default Content REMOVE THIS -->
     <h1>Congratulations</h1>
     This is your new Page
     <!-- End Default Content REMOVE THIS -->
   </apex:page>

2. Go to your object and create a Custom Detail Page Button. You want behavior: "Execute JavaScript" and Content Source : "OnClick JavaScript". The script is pretty easy:
   Custom Button (on your object):

   function EmbedFormHelper() {
   
       this.formSubmit = i_formSubmit;
   
       function i_formSubmit(ivalue) {
           var saveList = document.getElementsByTagName("td");
           alert(saveList.length);
           saveList[20].innerHTML += "<form name=\"exportAction\" method=\"post\" action=\"<VF_PAGE_ADDRESS>test?id"+ivalue+"\"></form>";
           document.forms.exportAction.submit();
       }
   }
   
   var wrapper = new EmbedFormHelper();
   wrapper.formSubmit("{!<YOUR_OBJECTS.ID>}");

   (BTW - this could be a lot simpler, inlining the functions. Also, you might consider modifying it to look for the form named "exportAction" since once the form is submitted it'll just complain about duplicate tags, etc...)

3. Add the button to your page layout and save it.

How it works

The script gets called when you press the button. Since Salesforce won't allow us use getElementById() for an on-click JavaScript, we'll use a more rudimentary thing to grab any available tag - in our case a "<td>" tag. You could also grab div's or spans, or whatever, but it doesn't really matter... Once you've got a tag, the script just appends the form tag contents. This contains your page and a way to parse the ID that you're passing in. Finally, the script calls the form that if just embedded with the ID as a parameter.

Once the form calls your page, your VF page executes. If the response truly doesn't have any HTML/TEXT/XML data, and contains only a content type of "application/octet", you'll end up remaining on the same page and triggering the download dialogue. If "developer mode" is on, the response object will contain HTML data and that will trigger the browser to display that stuff and redirect you off of your calling page. 

I'm curious how this'll work out for you - could you post your approach and if this worked for you back here?