Apex Code Development You need to sign in to do that
Don't have an account?
Denis BOURDON Creation of attachment with the API - exception BlobField for a few days
Hello
For a few days (since Thursday 8th, but not on Saturday and Sunday, and never during the night), we encounter exceptions we never had before when creating Attachements using the API. We have 2 different exceptions that occur :
BlobField : U#15f.3fffffff (Attachment) U#15f.aff (Body) got exception Exception by writeBlob: BlobField: Attachment 00PD000000glfYF
It is to be noticed that we have the first one during several hours, then the second one during several hours, and back to the first, and so on.
And during peak hours, these exceptions are totally random. If we regenerate the same query, it works fine.
It seems that there is a problem with the Oracle database of SF. The fact that it occurs on peak hours/days makes us think there is something of a degradation performance ? We are on EU1.
Thanks for the dedicated team of Salesforce for their help with that issue.
Regards,
For a few days (since Thursday 8th, but not on Saturday and Sunday, and never during the night), we encounter exceptions we never had before when creating Attachements using the API. We have 2 different exceptions that occur :
BlobField : U#15f.3fffffff (Attachment) U#15f.aff (Body) got exception Request timed out waiting for connection: [config 200ms, actual 215ms] - SERVER_UNAVAILABLE: Request timed out waiting for connection: [config 200ms, actual 236ms]
BlobField : U#15f.3fffffff (Attachment) U#15f.aff (Body) got exception Exception by writeBlob: BlobField: Attachment 00PD000000glfYF
It is to be noticed that we have the first one during several hours, then the second one during several hours, and back to the first, and so on.
And during peak hours, these exceptions are totally random. If we regenerate the same query, it works fine.
It seems that there is a problem with the Oracle database of SF. The fact that it occurs on peak hours/days makes us think there is something of a degradation performance ? We are on EU1.
Thanks for the dedicated team of Salesforce for their help with that issue.
Regards,
SF support just informed me that the 1st rollout has been deployed, see : https://success.salesforce.com/issues_view?id=a1p300000008XQhAAM (I haven't got the notification mail regarding the known issue yet).
We had lots of occurences yesterday, none today (so far).
Let's wait for several days to double check this, and for the other rollout announced by Shashank and expecred in the forthcoming 2 months.
Regards,
All Answers
You can monitor your instance status @
http://trust.salesforce.com/trust/status/
It says that on 1/8/15 there are few issues with EU1.
The issue on Jan 8th had apparently nothing to deal with our concern.
However, if it has a link, as our problem is still underway but there is no problem ou EU1 on trust.salesforce.com since Jan 9th, the root cause is something else.
The same issue happened again this morning as business recovered. No occurence all over the weekend.
The maintenance on Sunday morning has not changed this issue.
Can the dedicated team @Salesforce give us a feedback?
Thanks in advance.
Regards,
Things are getting worse. Below an exemple of exception.
Anyone @SF to help us?
Thanks
Regards,
[ERROR] [] {SfdcServiceImpl.query} SFDC query error : select Id,IsDeleted,ParentId,Name,IsPrivate,ContentType,BodyLength,Body,OwnerId,CreatedDate,CreatedById,LastModifiedDate,LastModifiedById,SystemModstamp,Description,IsPartnerShared,ConnectionReceivedId,ConnectionSentId from Attachment where parentId = 'XXXXXXXXXXXX'
com.sun.xml.ws.fault.ServerSOAPFaultException: Client received SOAP Fault from server: Unable to serialize property of 'records' on type common.api.soap.wsdl.QueryResult Error Id: 472790309-281489 (-102518988) Please see the server log to find more detail regarding exact cause of the failure.
at com.sun.xml.ws.fault.SOAP11Fault.getProtocolException(SOAP11Fault.java:193)
at com.sun.xml.ws.fault.SOAPFaultBuilder.createException(SOAPFaultBuilder.java:125)
at com.sun.xml.ws.client.sei.StubHandler.readResponse(StubHandler.java:253)
at com.sun.xml.ws.db.DatabindingImpl.deserializeResponse(DatabindingImpl.java:181)
at com.sun.xml.ws.db.DatabindingImpl.deserializeResponse(DatabindingImpl.java:258)
at com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:117)
at com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:91)
at com.sun.xml.ws.client.sei.SEIStub.invoke(SEIStub.java:154)
at com.sun.proxy.$Proxy451.queryMore(Unknown Source)
at com.meilleurtaux.core.service.impl.SfdcServiceImpl.query(SfdcServiceImpl.java:458)
I am Shashank from Salesforce support. Could you please email me at ssrivatsavaya@salesforce.com referencing this post with your org Id and detailed replication steps so that we can investigate further on the issue you are facing?
Thanks,
Shashank
Hi Denis,
The error number you mentioned seems to be related to this known issue when I checked internally: https://success.salesforce.com/issues_view?id=a1p300000008XJlAAM
Summary:
Internal server errors in different areas are occurring while editing Tasks / Events or creating Accounts, Opportunities or creating Reports with Standard Report Types ("Accounts" or "Contacts & Accounts" for example) by Users with the following time zone:
(GMT+07:00) Indochina Time (Asia/Ho_Chi_Minh)
Here, the user account will be the API user account. If the above is the case with your API user, Could you please try the workaround mentioned and check if it resolves the issue?
workaround: Please use (GMT+07:00) Indochina Time (Asia/Bangkok) as timezone.
Thanks,
Shashank
If the above mentioned is not the case with your API user or if you are facing the issue even after implementing the above workaround, you are most probably facing the issue due to this more generic knonw issue: https://success.salesforce.com/issues_view?id=a1p300000008XQhAAM
As mentioned in the known issue page, Salesforce R&D is working on it and the workaround for now would be to implement a robust retry logic so that the transaction that could not be performed is retried at a later time.
Thanks,
Shashank
Following our emails :
- Regarding the 1st known issue : we are at GMT+2 Europe/Paris. Which timezone should use as a workaround ?
- Regarding the 2nd known issue : as I told you by mail, we also thought and added a simple retry logic a few weeks ago, a « one-time » immediate retry. What would you suggest when saying « implement(ing) a robust retry logic so that the transaction that could not be performed is retried at a later time » ?
Thanks.Regards,
Denis
Following our emails, we are not concerned by the 1st known issue as we are not in the affected time zone.
As we don’t get a 503 but exceptions like these :
BlobField : U#15f.3fffffff (Attachment) U#15f.aff (Body) got exception Request timed out waiting for connection: [config 200ms, actual 215ms] - SERVER_UNAVAILABLE: Request timed out waiting for connection: [config 200ms, actual 236ms]
BlobField : U#15f.3fffffff (Attachment) U#15f.aff (Body) got exception Exception by writeBlob: BlobField: Attachment 00PD000000glfYF
Could you please double-check with the R&D Dept that the issue they are working on does include these exceptions to be 100% sure that when the known issue @ https://success.salesforce.com/issues_view?id=a1p300000008XQhAAM is fixed, i twill solve our issue ?
Thank you very much in advance.
Best regards,
We just got a performance degradation on our instance :
10:57 AM UTC: EU1 Performance Degradation - Inital / ResolvedThe Salesforce Technology Team has resolved the performance degradation issue on the EU1 instance. The problem began at 08:57 UTC and was resolved as of 10:45 UTC
During this period of time, the number of "server_unavailable" occurences dramatically rose.
That confirms that this issue is linked with "peak" activity hours and performance.
@@Shashank : have you got a feedback from R&D?
When degradations cumulate like this, it provokes substantial business problems.
Thanks for your reply.
Regards,
The performance issue that occured on 25th was more serious than the known issue and has been resolved now. The known issue is also related to the same performance issue, but it is not as serious as the degradation. I will follow up with R&D and let you know the latest update on this known issue if any.
Thanks,
Shashank
SF support just told me no ETA for this.
About the retry logic, SF support was not able to give us details on what a"robust" retry logic would be. However, I found this page on the SF site
https://help.salesforce.com/apex/HTViewSolution?id=000205947&language=en_US
"The retry logic should use the binary exponential backoff algorithm, e.g. retry after 1 second; if it fails, retry after 2 seconds; if it fails, retry after 4 seconds, etc." http://en.wikipedia.org/wiki/Exponential_backoff
I asked the SF support to update the known issue with this SF material.
Regards,
R&D is in the process of testing out a potential fix for this issue. I will get back to you with more details based on their results.
Thanks,
Shashank
The proposed fix has been tested on the na10 instance and will be rolled out to other instances too over the next 2 weeks. This will predominantly decrease the occurance of this error.
There is another initiative rollout as well which will further decrease the occurance of this error and that will be rolled out over the next 2 months.
Thanks for this update, hope this 1st fix will turn live soon on EU1.
Regards,
The number of occurences have dramatically dropped these last 2 days.
Only 1 two days ago, none yesterday.
Could you confirm that you delivered the fix on EU1 on Wed March 4th?
In the known issue, https://success.salesforce.com/issues_view?id=a1p300000008XQhAAM, "is it fixed?" is not checked for any instance.
Thank you for the fix and in advance for your confirmation.
Regards,
Hello
For a few hours, we can't attach any file on our org.
We have "internal server error" when trying to this on SF.com and "UNKNOWN_EXCEPTION" with the API. It is systematic.
The performance degradation is on trust.salesforce.com, but do not describe the issue :
10:20 AM UTC: EU1 Performance DegradationThe Salesforce Technology Team has become aware of a network connectivity issue which is affecting a subset of customers who may observe slow performance or connection difficulties accessing the salesforce.com application.
Please check the status of trust.salesforce.com frequently for updates regarding this issue.
Hope this has nothing to do with the problem we have on attachments for several weeks now...
Regards,
02:23 PM UTC: EU1 Performance Degradation
The Salesforce Technology Team is working to isolate a performance degradation issue affecting FileForce on the EU1 instance.
Please check the status of trust.salesforce.com frequently for updates regarding this issue.
02:04 PM UTC: EU1 Performance Degradation - Resolved
The Salesforce Technology Team has resolved the performance degradation issue on the EU1 instance. The problem began at 07:30 UTC and was resolved as of 13:10 UTC.
11:25 AM UTC: EU1 Performance Degradation - Update #1
The Salesforce Technology Team is continuing to work on the connectivity performance issue on the EU1 instance. A Subset of customers may observe slow performance or connection difficulties accessing the salesforce.com application.
Please check the status of trust.salesforce.com frequently for updates regarding this issue.
10:20 AM UTC: EU1 Performance Degradation
The Salesforce Technology Team has become aware of a network connectivity issue which is affecting a subset of customers who may observe slow performance or connection difficulties accessing the salesforce.com application.
Never heard of Fileforce, but I found this @ http://highscalability.com/blog/2013/9/23/salesforce-architecture-how-they-handle-13-billion-transacti.html
Fileforce includes a bundler function, developed to reduce the disk seek load on the Fileforce servers. If 100+ objects smaller than 32 KB are stored in the database, a process runs on the app servers to bundle those objects into a single file. A reference to the bundled file remains in the database along with a seek offset into the bundle. This is similar to Facebook's Haystack image storage system but built into an object storage system.
Seems like this is linked.
Regards,
Just to let you know you are not on your own, throughout the majority of yesterday (9th March 2015) our org was unable to attach files to any object and emails containing attachments failed to process via email-to-case. We are also on the EU1 instance.
I also read the same information regarding FileForce and the handling of BLOB fields after testing with several different file types and had found that very small .txt files attached normally as did small .jpegs but anything larger generated an Internal Server Error.
I raised a ticket with SFDC Support yesterday during the problems and got a call at 4pm GMT to say the problem had been resolved - we'd already noticed that we were receiving no further errors.
However, this morning (March 10th 2015) we have received another error generated by an email with an attachment (506k JPG):
The following errors were encountered while processing an incoming email:
UNKNOWN_EXCEPTION : BlobField : U#15f.3fffffff (Attachment) U#15f.aff (Body) got exception Request timed out waiting for connection: [config 200ms, actual 206ms]
Hopefully this will be resolved soon as this causes significant disruption to our operations teams.
Regards,
JDV
Thanks for your post.
We have had these errors since January 8th.
Two this morning :
BlobField : U#15f.3fffffff (Attachment) U#15f.aff (Body) got exception Request timed out waiting for connection: [config 200ms, actual 233ms]
BlobField : U#15f.3fffffff (Attachment) U#15f.aff (Body) got exception Too many requests waiting for connections
Nevertheless, except yesterday when FileForce was down, the number of occurences dropped starting March 4th.
Salesforce should have already fixed something.
However, let us wait for the patch next week on EU1.
Regards,
SF support just informed me that the 1st rollout has been deployed, see : https://success.salesforce.com/issues_view?id=a1p300000008XQhAAM (I haven't got the notification mail regarding the known issue yet).
We had lots of occurences yesterday, none today (so far).
Let's wait for several days to double check this, and for the other rollout announced by Shashank and expecred in the forthcoming 2 months.
Regards,
Regards,
JDV
Last occurence on March 11th. Therefore, I closed the support cas and thank you Shashank for your help on this.
Regards,