"with sharing" is not taking care of permissions while deleting attachments and notes

I have to customize edit, view, delete for Attachments in Opportunity Page Layout.

On delete button I have following lines in Visualforce Page:

<apex:commandLink styleClass="ankit" value="del" onclick="return confirm('Are you sure?')" action="{!delAttachAndNote}" > <apex:param name="attachId" value="{!item.Id}" assignTo="{!delId}" /> </apex:commandLink>

And below is code in Apex class which is "with sharing":

public String delId{get;set;}
public void delAttachAndNote(){
Database.delete(delId);
}

I tried changing it to delete new Attachment(Id=delId), etc. But still no luck.

Why Salesforce is not taking care of permissions here and Is there any alternative? Also, we cannot get URL for deleting Attachment or Notes.