Hi, 

I guess you want to create a model popup window. on custom button click.

If yes then there is not any predefine function by SF to create model popup.

To create a model popup you need to use jquery.

Please follow below steps to create model popup.

1. Create static resource of jquery files (jquery, jqueryCSS, jqueryUI) and upload these resources on your org. ( Adding jQuery To your salesforce application )
2. Now include those static resources into your button script.
   

   {!REQUIRESCRIPT('/resource/1339509496000/jqueryStaticResource')} 
   {!REQUIRESCRIPT('/resource/1339424935000/jqueryUiStaticResource')} 
   {!REQUIRESCRIPT('/resource/1339424935000/jqueryCSSStaticResource')} 

    

3. Now create new div which will represent your popup window.
4. As we know that we don't have any controller on SF page layout and there CSS. But still I want to open the popup on same page layout. For that I open the pagelayout's source code where I am looking for some common element. I do the same for all different objects page layout then come to know that there is one "div" which is common in most of the page layout. I copy that div id (contentWrapper) and open my popup div into this "contentWrapper" div.

Here I am sharing the sample code which will open your popup window into the objects page layout n show the records name.

{!REQUIRESCRIPT("/soap/ajax/14.0/connection.js")}
{!REQUIRESCRIPT("/soap/ajax/14.0/apex.js")}
{!REQUIRESCRIPT('/resource/1339509496000/jqueryStaticResource')}
{!REQUIRESCRIPT('/resource/1339424935000/jqueryUiStaticResource')}
{!REQUIRESCRIPT('/resource/1339424935000/jqueryCSSStaticResource')}

try{ 
var elementId = 'contentWrapper';
var recordName = '{!Lead.Name}';
var recordMobilePhone = '{!Lead.MobilePhone}';

document.getElementById(elementId).innerHTML += "<div id='InlineEditDialog' role='dialog' aria-live='assetive' aria-describedby='InlineEditDialogTitle' class='overlayDialog inlineEditDialog' style=' width: 450px; height: 450px; max-height: 1366px; max-width: 550px; display: block; left: 506.5px; top: 38.5px; visibility: visible;'><div class='topRight'><a title='Start of Dialog' style='width: 2px; height: 2px; position: absolute; border: 0pt none; margin: 0pt; padding: 0pt; background: none repeat scroll 0% 0% transparent; outline: medium none; z-index: -1; cursor: none;' onclick='return false;' href='javascript&colon;void(0)' id='InlineEditDialogFocusPoint'>Start of Dialog</a><div class='topLeft'><a class='dialogClose' href='javascript&colon;void(0)' tabindex='0' title='Close' id='InlineEditDialogX' onclick='location.reload();'>Close</a><h2 id='InlineEditDialogTitle'>POPUP Dialog</h2></div></div><div class='middle'><div id='InlineEditDialogContent' class='innerContent'><div class='activeField'><div class='inlineEditCompoundDiv' id='lea16_ileinneredit'><table border='0'><tbody><tr><td class='labelCol'><label for='recordName' style='float:left;'>Name</label></td><td id='recordName'>"+recordName+"</td></tr><tr><td class='labelCol'><label for='recordMobilePhone' style='float:left;'>Mobile Number</label></td><td id='recordMobilePhone'>"+recordMobilePhone+"</td></tr></tbody></table></div></div><div></div><div style='display: none;'></div><div class='buttons' id='InlineEditDialog_buttons'><input type='submit' id='close' onclick='' value='Cancel' class='btn' id='close'></div></div></div><div class='bottomRight'><input type='text' style='width: 2px; height: 2px; position: absolute; border: 0pt none; margin: 0pt; padding: 0pt; background: none repeat scroll 0% 0% transparent; outline: medium none; z-index: -1; cursor: none;' id='InlineEditDialogBlurCatch'><div class='bottomLeft'></div></div></div><div class='overlayBackground' id='backgroundOfDiag' style='width: 1366px; height: 1505px; display: block;' ></div>";

$('#close').click(function(){
location.reload();
}); 


}catch(err)
{
alert(err);
}

 I know this is not a standred way to do it. But I found only this workaround. This code may fail if SF made changes into there CSS. I am hopping SF may open some API to create a popup on any layout. But till that we can use this workaround.

I hope you are satisfy with this reply.

Thanks,

Purushottam