Hi Pallavi,

You need to create the Custom button as Detail Page Button with Execute JavaScript behavior, and in the onClick section add the logic to change the status to New. Below is the code segment;
 

{!REQUIRESCRIPT("/soap/ajax/29.0/connection.js")}

// identify the record
var yourObjectName = new sforce.SObject("Name of your Object");
yourObjectName.id = "{!yourObject.Id}";

yourObjectName.Status__c = 'New';

// save the change
sforce.connection.update([yourObjectName]);

//refresh the page
window.location.reload();

As I know, there is no way to inactive or grayed out the custom buttons in page layouts. Hence, you need to create two page layouts here, 

Page layout 1 – for New and Modified statuses à You can insert your custom button in this page layout
Page layout 2 – for Submitted and Synced status à you are not going to add your custom in this page layout 

Then create two Record Types for respective Page Layouts as you need to switch between page layouts.

Record Type 1 à for Page Layout 1
Record Type 2 à for Page Layout 2 
​
Finally, create a workflow rule to change the Record Type. Because to click the button you should be in Page Layout 1, after clicking the button, you should be switched to Page Layout 2.
 

Another Way:

1. Please create a formula field and keep return type a text.
2. Now you can write your logic to show a button image if the status is New otherwise put NULL so that nothing would be visible. Or you can show an image of grayed button. 
3. you can embed an URL on clicking on this formula field so that a new VF page can be hit you can update status and it will get back you on the same page. 
Here is sample code
 

IF((your condition logic) ,HYPERLINK("/apex/yournewVFpage?id="&Id,IMAGE("/resource/Buttonimage", "ButtonName"),"_self"),NULL)

Hope this helps you!
Best Regards,
Jyothsna
​