I know this post is old, but it seems that a lot of people are interested in it.  I needed this too so I came up with a hacky workaround that works for my needs.  As you probably know, SFDC has a Sort button on the product related list where one can manually change the sorting of each individual product (you do not specify, for instance, sort by the product name). In the background, this interface is posting the results of the form to /oppitm/lineitemsort.jsp which results in the SortOrder field of the OLIs to be updated based on the users selection. Unfortunately, the SortOrder field is not writable via apex (things would be much easier if they were).

So, the first step is to have an apex web service that does the custom sorting logic and returns a comma separate list of OLI IDs in the desired sort order. Here is an example of my web service that sorts the OLIs for me based on my sorting requirements:

webservice static String MRsort(Id oppID)
    {
    	//pull back the OLIs in a specific sort order
    	List<OpportunityLineItem> olis = [Select oli.Id
    			From OpportunityLineItem oli
    			Where oli.OpportunityId = :oppId
    			ORDER BY oli.PSFT_REN__c, oli.PricebookEntry.Product2.Technology_Family__c, oli.PricebookEntry.Product2.Name];
    	
    	//build the comma separated 15 character OLI Id string to send back
    	String sortedIds = '';					  
    	for(OpportunityLineItem oli : olis)
    	{
    		sortedIds += String.valueOf(oli.Id).substring(0,15) + ',';		
    	}
    	
    	//remove the last comma
    	sortedIds = sortedIds.substring(0,sortedIds.length() - 1);
    	return sortedIds;
    }

We are going to use the list of OLI Ids from the above web service in a JavaScript button that sits on the product related list. After reverse engineering how /oppitm/lineitemsort.jsp is called to commit the sort through SFDC's OOTB sort interface, I was able to come up with the correct button JavaScript to recreate a POST request to this JSP with the list of sorted OLI IDs from the web service:

{!REQUIRESCRIPT("/soap/ajax/20.0/connection.js")}
{!REQUIRESCRIPT("/soap/ajax/20.0/apex.js")}

var oppID = '{!Opportunity.Id}';

//call the apex web service to get the OLIs in the desired sort order for this opp
var result = sforce.apex.execute("MyClass","MRsort",{oppID: oppID});   

//need to post a form to /oppitm/lineitemsort.jsp because this is how SFDC
//does it but there is not direct API to do the sorting (thus the awkward workaround)
var form = document.createElement("form");
form.setAttribute("method", "post");
form.setAttribute("action", "/oppitm/lineitemsort.jsp");

//set the id of the request to the opportunity ID
var hiddenField = document.createElement("input");
hiddenField.setAttribute("type", 'hidden');
hiddenField.setAttribute("name", 'id');
hiddenField.setAttribute("value", '{!Opportunity.Id}');   
form.appendChild(hiddenField);

//the name of the sorted OLI list that the JSP is expecting is "duel0"
var hiddenField = document.createElement("input");
hiddenField.setAttribute("type", 'hidden');
hiddenField.setAttribute("name", 'duel0');
hiddenField.setAttribute("value", String(result));
form.appendChild(hiddenField);

var hiddenField = document.createElement("input");
hiddenField.setAttribute("type", 'hidden');
hiddenField.setAttribute("name", 'retURL');
hiddenField.setAttribute("value", '/{!Opportunity.Id}');
form.appendChild(hiddenField);

//set to save
var hiddenField = document.createElement("input");
hiddenField.setAttribute("type", 'hidden');
hiddenField.setAttribute("name", 'save');
hiddenField.setAttribute("value", ' Save ');
form.appendChild(hiddenField);

//need to do this so it works in IE
document.body.appendChild(form);

//submit!!
form.submit();

So, after the user clicks the custom JavaScript sort button, the interaction with the sorting web service and /oppitm/lineitemsort.jsp results in the SortOrder field of the OLIs being set to the desired values and thus the OLIs are displayed in the desired sort order in the related product list.