+ Start a Discussion

Different behavior for unit test execution tests in managed packages

Has anyone else noticed different behavior for unit tests that are in a managed package?  


We have a managed package installed in a few hundred client orgs.  For years we've had clients run a script that would modify page layouts using the metadata API and it worked fine.  In the last couple of weeks, however, clients running our script are seeing unit test failures for tests that are within our managed package.  The test failures vary from client to client with the exception of one class that's pretty consistently failing.  The failures also seem to be more prevalent with asynchronous batch apex code - note we are enclosing the batch processing within the StartTest and StopTest commands.


Our first thought is that there are some sort of customizations or live data in the client org that are interfering with our tests so we tried to replicate the behavior in a trial org with our package and we're seeing similar problems.  


The big challenge is that it's very difficult to debug because the failures only occur in client orgs so the debug info is limited, even when we're logged into the client org using the "Subscriber Access" option.


Did something change on the SF side regarding unit test execution for packaged tests?