Not sure about how to do it with a datatable, but you can do it with a repeat, and render your own HTML.

Check out this code snippet:

table border="1" width="100%"> <tbody> <tr> <td style="text-align: center; width: 250px; color: rgb(255, 255, 255); background-color: rgb(51, 102, 255);">Step completed</td> <td style="text-align: center; width: 50px; color: rgb(255, 255, 255); background-color: rgb(51, 102, 255);"># of Opps</td> <td style="text-align: center; width: 188px; color: rgb(255, 255, 255); background-color: rgb(51, 102, 255);">Revenue</td> <td style="text-align: center; width: 39px; color: rgb(255, 255, 255); background-color: rgb(51, 102, 255);">X</td> <td style="text-align: center; width: 172px; color: rgb(255, 255, 255); background-color: rgb(51, 102, 255);">Yield % </td> <td style="text-align: center; color: rgb(255, 255, 255); background-color: rgb(51, 102, 255); width: 54px;">=</td> <td style="text-align: center; color: rgb(255, 255, 255); background-color: rgb(51, 102, 255); width: 215px;">Yield</td> </tr> </tbody> </table> <table border="1" width="100%"> <tbody> <apex:repeat value="{!repstages}" var="rs"> <tr> <td style="width: 250px;text-align: center;">{!rs.stagename}</td> <td style="width: 50px;text-align: center;">{!rs.stagecount }</td> <td style="width: 188px;text-align: center;">{!rs.stageamt }</td> <td style="width: 39px;text-align: center;">X</td> <td style="width: 172px;text-align: center;">{!rs.yieldper}</td> <td style="width: 54px;text-align: center;">=</td> <td style="width: 215px;text-align: center;background-color:{!IF(rs.yieldamt <0,''rgb(255, 0, 0)','rgb(51, 255, 51)')};">{!rs.yieldamt}</td> </tr> </apex:repeat> </tbody> </table>

 Notice the IF merge field in the last table row, that will set your color, in this case one color if negative and the other if 0 or greater.

Hope that helps!