Our production deployments using the (ant) force migration tool have been consistently failing recently, after executing some percentage of our tests, with the error:

System.LimitException: Your runAllTests request is using too many DB resources.

All of our tests are @isTest(seeAllData=false) and we recreate all test data from scratch using factory classes at the beginning of each test method.

We contacted Salesforce support, and after a multiple calls, different people, and a lot of time, they told us that in addition to the per-transaction governor limits, there are undocumented limits that are shared amongst all executed code during the course of a runAllTests request:

• The DML statement limit is 350,000
• The SOQL statement limit is 450,000

Salesforce suggested that we use the @testSetup annotation to setup data in our tests (we have not been using that annotation), and this would help us in avoiding hitting the above limits. I wrote a small unit test to verify that this would actually help with limits, but the unit test suggests that the DML statements accrued in a @testSetup annotated method do indeed count in each testmethod.

@isTest(seeAllData=false)
public class TestSalesforceTestSetupDMLCount {

    @testSetup static void setup(){
        // Assert that no DMLs have occurred yet
        system.assertEquals(0, Limits.getDMLStatements());
        
        // Perform 4 DMLs
        for(Integer i=0; i<4; i++){
            insert new Account(Name='Test' + i);
        }
        
        // Assert that 4 DMLs have been performed
        system.assertEquals(4, Limits.getDMLStatements());
    }
    
    public static testmethod void testDmlCount1(){
		// THIS ASSERTION FAILS
        system.assertNotEquals(4, Limits.getDMLStatements());
    }
    
    public static testmethod void testDmlCount2(){
        // THIS ASSERTION FAILS
        system.assertNotEquals(4, Limits.getDMLStatements());
    }
}

The 2 testmethods fail because Limits.getDMLStatements() returns 4. The question is, regarding the runAllTests DML statement limit of 350,000, does the above test contribute 4 DML statements or does it contribute 12 DML statements (4 for setup(), 4 for testDmlCount1(), 4 for testDmlCount2())?

This is obviously something that only Salesforce can answer, but I at least want this issue documented somewhere, and will provide updates as we get answers from Salesforce.