function readOnly(count){ }
Starting November 20, the site will be set to read-only. On December 4, 2023,
forum discussions will move to the Trailblazer Community.
+ Start a Discussion
Ted.TsungTed.Tsung 

SSLv3 Poodle vulnerability & outbound messaging

Hi,

Because of SSLv3 Poodle vulnerability, we have turned off SSLv3 support on our web server. This in term is causing Salesforce outbound messaging to fail.

Is there a work around with this from Salesforce end?

The outbound messaging processing issue was resolved once we turn SSLv3 back on our web server.

Ted Tsung
ShashankShashank (Salesforce Developers) 
Hi Ted,

At present, some outbound calls initiated from Salesforce are initiated using SSLv3, so if this is disabled on your server, there'll be a handshake failure. Customers will be notified when Salesforce decides to disable SSLv3 in outbound calls. Until then, it is advised that you support this for incoming calls (received from Salesforce).

Please note, in a recent communication, Salesforce advising to disable SSLv3 when connecting to Salesforce. This is a different scenario as the request is sent by the customer, not by Salesforce, and we do support up to TLSv1.2. 

At present, Salesforce R&D is aggressively working on a strategy around this, but there is no set schedule at present. One this is finalized, there will a technology communication sent out.

Thanks,
Shashank
drewpiston.ax678drewpiston.ax678
We started getting intermittent (~10%) handshake failures with our Salesforce Instance's outgoing WebServiceCallouts starting on Wednesday -- could intermittent handshake failures be explained by a similar situation?  

Thanks,
 Drew
ShashankShashank (Salesforce Developers) 
If the callout is coming from salesforce to an external system, then it is the same situation.
LinvioIncLinvioInc
Hi Shashank,

Could you clarify further please? You say "some outbound calls from Salesforce are initiated using SSLv3"  

Are HTTPRequest callouts done over SSLv3?   ...  or are they made with TLSv.1.0 or higher?

Many SF implementations involve callouts to services managed by third parties - and these parties are responding quickly to the POODLE threat by shutting off SSLv3 support.  If Salesforce is slow to follow suit, there will be a LOT of unhappy SF customers.

- Ron
David Hecht AFDavid Hecht AF
Any update on this? Next week many customer integrations are going to be disabled if Salesforce is using SSLv3

There is nothing on trust.salesforce.com or in the security section there. 

Surely this must be on Salesforce's radar?

Thanks,
David
Bob devBob dev
Shashank - "Salesforce R&D is aggressively working on a strategy around this".  Really?  This has been going on for more than a week now.

...and what do you mean "some outbound calls initiated from Salesforce are initiated using SSLv3" - if that is the case, that implies that *some* are using TLS1.2.  Why can't you have "fewer" try using SSLv3?  This is within your control, no?

This is getting critical - Shopify and several others have already shut off SSLv3 which is having a direct impact on operations of many sf.com customers.  Please provide a schedule for a fix - this is not acceptable.
ShashankShashank (Salesforce Developers) 
Salesforce Outbound Callouts, at present, use both SSLv3 and TLS, though SSLv3 is used much lesser than TLS. I understand the concern around the time being taken, however Salesforce R&D is working on this with high priority to stop the usage of SSLv3 for outbound callouts. There will be a tech communication sent at the earliest possible, but I must agree that there is no ETA yet.
David Hecht AFDavid Hecht AF
This is a significant issue.

Authorize.net is disabling all SSLv3 connections on November 4. Millions of emails from other technology firms have gone out this week, informing customers of the situation with SSLv3.

Meanwhile, publicly it seems that Salesforce is treating this as an insignificant browser issue: https://help.salesforce.com/apex/HTViewSolution?urlname=POODLE-SSL-3-0&language=en_US

I sure hope R&D is working day and night on this. Otherwise this is an integration catastrophe for outbound messaging.
Ted.TsungTed.Tsung
fortunately for us, we have received patch on SSLv3 issue from our hosting provider. Good luck to all...
TehNrdTehNrd
Can confirm doing Apex callouts to a server where they have disabled SSL3.0 are failing. This is a MAJOR issue and it is appaling Apex callouts do not support TLS 1.2 . This should be of the highest priority internally at salesforce.com. Case#11165875
TehNrdTehNrd
Stripe is also ending SSL3.0 support on November 15th. https://stripe.com/blog/poodle This is going to be a cluster if salesforce doesn't resolve this soon.
hemmhemm
Our geocoding service is disabling SSL v3 tomorrow, October 30. Our app will be bricked for new installs as soon as they do. If callouts to external servers are setup to allow a TLS connection, won't TLS be chosen instead of being downgraded to SSL 3.0? Not my area of expertise, but isn't it less about disabling SSL v3 on salesforce than it is making sure that Apex callouts support TLS so the connections made with external services are done over a TLS connection?
drewpiston.ax678drewpiston.ax678
Not my area of expertise either, but currently some percentage of our webservices calls are failing (somewhere betweeen 4-15%, in general).  Best explanation seems to be that sometimes SSL v3 is used and sometimes it isn't.
hemmhemm
Turns out it's the entire AOL network and all web services. So it's more than just a Geocoder that's changing tomorrow.
hemmhemm
No idea why that posted as code. 
Bob devBob dev
We are seeing about 50% of the Apex callouts failing - way more than 15%.  

This is going to become a news item if sf.com doesn't do some damage control.  I think we would all like to hear from the CEO as to why one of the largest business integration services in the world is unable to move more quickly - this is putting customers in serious risk of both security exposure as well as operational exposure.  Can't wait to see this in the press in a few days.....geez
LinvioIncLinvioInc
We've tested calls to the AuthNet sandbox (which, according to AuthNet, has SSLv3 turned off) and they appear to be working from our development orgs.  Some reports (e.g. above), seem to indicate that HTTPRequest callouts are failing on some services that have shut off SSLv3.  Salesforce is giving us nothing to work with - promises of aggressive efforts to solve the problem, but no ETA, and no useful information on when they use SSLv3 and when they don't.  One support rep said he had documentation on a "workaround" but we haven't seen that yet and have little confidence that he even knows what the problem is. 

It's still not clear what's going to happen right now ... just a lot of speculation, and no helpful information from SF.   Very disappointing.
 
KevinPKevinP
In the mean time, there's a simple but (admittedly ugly) work arround.

You can utilize a proxy service that accepts SSLv3 to make your calls for you.

The general idea is to stand up a proxy (I like Cors-anywhere (https://github.com/Rob--W/cors-anywhere)) On a sslv3 enabled server (heroku?) and point your outbound calls at the proxy. The proxy then makes TLSv1.2 callouts to auth.net etc.

I highly suggest utilizing custom certs and reasonable security measures to ensure only you are using the proxy.

 
ryanhca123ryanhca123
+1 this is a huge issue for us. Subscribing for updates...
Always ThinkinAlways Thinkin
If you aliased the URL of your servers and updated your callouts to use the new URL, would that force a new negotiation which hopefully results in a match on TLS instead of SSLv3?
LinvioIncLinvioInc
Salesforce has promised a public response to the incident tomorrow.  There is apparently a plan to update SF instances to TLS 1.1/1.2.
Jim WilliamsJim Williams
This is causing major issues for our college as well.  We integrate with the Touchnet Payment Gateway which like many other providers has disabled SSLv3.   I wanted to mention this in case any other colleges that integrate SFDC with Touchnet  are getting the handshake error - now you know why. We also have a case opened with SFDC.   LinvioInc, please let us know when/where the public response from SFDC is posted.  I checked various locations and have found nothing yet.  This really needs to be addressed ASAP.
Jim WilliamsJim Williams
FYI A recent post on stackexchange: http://salesforce.stackexchange.com/questions/53631/twilio-integration-and-sslv3-poodle-vulnerability is mentioning November 4 as a solution date.  Can anyone here confirm this?
Always ThinkinAlways Thinkin
There has been some ambiguity about the solution date because some sources reference the fix for browser sessions and others for the Java client that drives callouts and outbound messages. Until you see a response from a Salesforce-owned and -authorized channel, I would assume any dates are unofficial.
thisisnotaprilthisisnotapril
Please review the updated knowledge base article on this: https://help.salesforce.com/apex/HTViewSolution?urlname=Salesforce-disabling-SSL-3-0-encryption&language=en_US
David Hecht AFDavid Hecht AF
So Salesforce is not going to disable outbound SSLv3 calls until December 10?

Does anyone know what % of outbound calls are going out as SSLv3?

Is there any way for customers to choose before December 10 that all calls go out as TLS 1.0 or higher?
hemmhemm
As it pertains to Apex Callouts, which seems to be the core issue in this thread, I'm not convinced this is really enough. Today, there are services that have long accepted TLS 1.0 and higher and recently disabled SSL v3. However, Salesforce fails to communicate with them right now (intermittently). Theoretically, the calls to these services should have always been made over TLS 1.0+, but that is obviously not the case. More and more services are disabling SSL v3 every day and many important ones are doing it next week (credit card services)

Should our expectation be that Apex callouts cannot be totally counted on to make a good connection until December 10? That seems an awfully long time for a fundamental platform connectivity issue to exist! It'd be analygous to Salesforce saying that the Save button on page layouts won't work 20% of the time and to just try again if it fails.  Kinda crazy.

The help article states the following...
Salesforce recommends customers disable SSL 3.0 encryption in their own IT environment as soon as possible, unless they use call-out integrations. If a customer uses call-out integrations, and they have not already disabled SSL 3.0 in their own environment, then Salesforce recommends that they wait until after Salesforce has disabled SSL 3.0 for outbound requests.
The problem is that a huge amount of communication is done to services not in our own environments. We cannot control whether Google, Stripe, Authorize.net, etc. turn off SSL v3. 

I've been hearing rumors of the core issue being with cached connections in the Java client and that it should go away over time, but cannot really confirm or deny that officially. The help article does not really address this.

I think the dev community needs more explanation. Especially in the area of Apex callouts to 3rd party services. 
drewpiston.ax678drewpiston.ax678
Well said, hemm.
Bob devBob dev

@David Hecht 7 - We have been tracking this for several days and the % of failures is pretty consistent - right around 50%.  Quite random on success/failure rate related to time of day, etc.

I guess we have a date but having a fix almost 1 month after other sites have responded (e.g. completely disabled a big hole!) is a bit concerning.  Good to see other developers pushing for sf.com to get better - let's hope they do.


 
hemmhemm
User-added image

Here's a chart of the last several days of callouts made to our geocoder. These callouts are occuring from within our customers own Salesforce systems (1000s of orgs). A callout is only being counted here if its successful. These things run on an hourly schedule in customer systems and do fluctuate each day, but notice the dramatic drop in connections starting with the change to disabling SSL v3 by our geocode provider.
drewpiston.ax678drewpiston.ax678
We tried putting our callouts in a try/catch loop, with a short sleep hack in between, but that hasn't seemed to make a significant difference.
Jim WilliamsJim Williams
I do not understand the rationale for saving the outbound/callouts for last.  So basically no SFDC apps are going to be able to reliably send apex calls to payment providers (or many other services for that matter) once they all have SSLv3 disabled? The fact that we cannot reliably communicate with our payment gateway has led to numerous problems with being able to collect application fees - not a good way to begin a student relationship!    I do not envy the rest of you whose payment providers have not made the switch yet; when they do it going to cause a lot of trouble.  This is no way to do business.  There at least should be some way for us to control the protocol we want to use for apex calls.  Leaving it to chance is unacceptable.
David Hecht AFDavid Hecht AF
As of a few hours ago I received the following from Salesforce support. I have asked for clarification on when it will be in place.
 
"I was given the following explanation regarding how we're going to address SSLv3 callouts.

We will use TLS with callouts but if it fails. We drop down to SSL and hard code to send it via SSLv3 for 24 hours or an app restart. Which ever comes first. This should address any changes that occur with the way other companies integrate with Salesforce until we completely disable SSL 3.0 on December 10th."
LinvioIncLinvioInc
Thanks, David for sharing that.  Does anyone know what salesforce means by "app" in that context?  Would the 24hr downgrade apply to a single SF instance?  Or all instances with a specific managed package?  Or all instances on a specific SF server? 

Also, would the downgrade occur for any other reason besides lack of support for TLS .... e.g. network latency?

 
James TrachyJames Trachy
Our application is heavily reliant on outbound requests and has effectively been rendered useless as soon as our IT turned of SSL3.  We do not have the option of turning back on as the same infrastructure serves up data to millions of requests outside the scope of Salesforce.

An additional bit of data we've captured - all (no exceptions) our failures over the last 2 weeks (roughly 18,000) have come from NA4, NA8 or NA14.  All other clients have been performing normally. Also worth noting is a "failure" as we count it actually means 3 consecutive failures as it has a retry mechanism.

Perhaps "app" means "instance" in the case of David's comment and we end up suffering because some other code in the instance fails on a TLS callout and therefore reverts us all (the entire instance) back to SSL3?

Interested in any more data folks can provide.
Ben HubbardBen Hubbard
Since Salesforce is disabling inbound SSL before outbound SSL, does this mean that there will be a period of time where Salesforce cannot make callouts to itself?  Shouldn't they fix outbound before inbound?
Jim WilliamsJim Williams
SFDC is still working on our case but no resolution yet.  Apparently they made some coding changes to the sandboxes that would allow TLS to be used as a first choice, but our testing resulted in the same problem (our sandbox is on CS9).  Our production org is on NA14 - just mentioning this since this was one of the problematic isntances mentioned in this thread.  Has anyone else had any sucesss with a resolution yet?  In case anyone is using authorize.net (we do not), today is the day they shut down SSLv3 support.  
David Hecht AFDavid Hecht AF
Knock on wood, but we haven't had any reported issues yet.

Vantiv / Litle shut off SSLv3 support yesterday, and Authorize.net said they were going to do it today. We have lots of customers on both and sandbox testing last week / over the weekend looked good, and no support issues yesterday or today.

It looks like the Salesforce TLS first, then SSL if that fails approach was implemented over the weekend as best we can tell, support said it was scheduled for yesterday.

Would have been great if we had something to tell customers last week, other than just hoping Salesforce would take care of it, but it does look like they did take care of it.
Jim WilliamsJim Williams
David what instance is your sandbox on?  I am wondering if like others have posted that it falls back to SSLv3 if there is any failed TLS callout on an entire instance, not specific to the org.  I will keep you all posted as to if/when our issue is resolved.
David Hecht AFDavid Hecht AF
We have hundreds of customers making outbound calls, so hard to say what instances sandboxes / production are on.

I asked for clarification from Salesforce support on what an "app" meant in the context of the failover to SSL for 24 hours, and haven't heard back yet. I would assume it would be an org ID and not an entire instance, but maybe even more specific like an appexchange app or apex within Salesforce.
James TrachyJames Trachy
David it looks like all handshake errors ceased for us last night at 9PM CST.  We were getting 1000 per day and today have received zero.

Glad this appears to be taken care of.
KevinPKevinP
It’s my understanding of the vulnerability that this can be “Resolved” by either end (Salesforce or X server).  It’s not so much that SSLV3 is enabled on Salesforce’s side, but rather that it’s in the degradation chain from TLS 1.2 -> TLS 1.x -> SSLV3 etc. I believe Salesforce has (always?) had TLS enabled, but that, for many reasons, the encryption handshake can be forced to downgrade to SSLV3. If the receiving server has resolved any/all issues that would force a downgrade to SSLV3, the TLS handshake would start working.  What would be telling is if we could see what the handshake *resulted* in.  --  Kevin Poorman Sent with Airmail
James TrachyJames Trachy
Kevin while that may be accurate in theory I don't think their servers were attempting TLS correctly in practice.  Granted this is speculation - but based on what I was seeing in our clients only a subset of Salesforce servers were failing even though they were all hitting the same endpoints on our end.  In particular requests from three Salesforce instances *almost* always failed (about 80%) but requests from all other instances always succeeded.

It looks like they fixed the glitch though so we're good.
KevinPKevinP
@james I guess I was just trying to illustrate that the third party server could have fixed it on their end. POODLE demonstrates that the downgrade from TLS1.2 —> SSLv3 can be forced by network configuration etc.  --  Kevin Poorman Sent with Airmail
James TrachyJames Trachy
True Kevin - sometimes I just focus on my own little world :)
hemmhemm
It's pretty evident that, no matter what the degradation chain is, that Apex is not handling things intelligently enough. Apex is here to reliably connect to http servers as reliably as a browser would. No matter the reasons, Apex callouts are not keeping up with the industry. They need to and people like us should not need to be chasing down 3rd parties to find out whether they support TLS version x, y or z.

The level of capability we are talking about (connectivity) is what the Salesforce platform is here for, to "just work". It's the blocking and tackling of a platform.

The mind boggling thing to me is that, after being asked again and again and again, Salesforce seems to ignore this issue. We just need to hear from them that they understand the situation and have a plan. The Help article they published is not enough. 
 
drewpiston.ax678drewpiston.ax678
We stopped getting errors last night as well, on server NA14.  Crossing my fingers they won't crop back up.
hemmhemm
We are noticing successful connections not too. In addition to our geocoding providers, we are building new APIs connecting to Cloudflare and those are now successful consistently. 

I do prioritize getting it fixed over an explanation about it so I am thankful to those at Salesforce working on it. (thanks!) Would just be nice to get some sort of communication during the process.

Sounds like things are on the right path. 
brian_sfdcbrian_sfdc
All I see you guys have various questions and I think I can answer quite a few of them, I manage part of the technical support security team at salesforce.  Email me at bestebez@salesforce.com (you can look at my linkedin profile if you question who i am).

Questions that have been raised around Salesforce’s support of SSL 3.0 and TLS 1.0. While we are in the process of disabling SSL 3.0, Salesforce currently supports TLS 1.0 and TLS 1.2 for inbound requests and TLS 1.0 for outbound call-outs.
Our Technology Team has been actively working to address an  issue that causes outbound call-outs to use SSL 3.0 more frequently than they should, given we have TLS 1.0 enabled. We understand that this may have caused issues for customers who have already disabled SSL 3.0 in their call-out endpoints. We released a fix to Sandboxes last Friday, October 31, and plan to release the fix to production instances during off-peak hours on Wednesday, November 5, 2014.
Many customers and partners who have tested this fix in their Sandboxes have reported successful connections using TLS 1.0. A few customers continued to experience TLS 1.0 issues on their Sandboxes, and our Technology team is working with them to find a solution.

There was an issue specifically to Na14 that was generating more outbound messages that were using SSLv3 but that has since been fixed.  That is probably why a few of you guys saw an issue with

But feel free to send me a note.  Thanks
hemmhemm
Brian, thanks for responding. Your answers are a lot of the information we needed to hear.
@altius_rup@altius_rup
Brian,
Thanks for this communication !
brian_sfdcbrian_sfdc
let me know if you guys have continued questions or feedback, email me at bestebez@salesforce.com
Jim WilliamsJim Williams
So all are aware, as of last night our NA14 instance has been successfully communicating with the TouchNet payment gateway over TLS.  So far, so good.  We are monitoring our payments very closely today and I will report back if there is any recurrence of the issue today.

However, we did also need to increase our Apex timeout settings for the outbound call.  We set it to the maximum of 120 seconds.  We first did this in the sandbox which resolved the issues there (after the coding changes were applied by SFDC), and then deployed to production.  I am not certain why this was necessary (unless the new coding had possibly introduced other latency issues that were not present before?) but it seems to have done the trick - just putting it out there in case others are in a similar situation. 

Thank you Brian from for providing the helpful explanation of the issue.
KevinPKevinP
@brian_sfdc,

First, Thank you for this communication. It's the "official word" needed to pass on to clients who are looking for us to explain whats going on and when it will "just work". 

At the risk of opening a can of worms, and in full knowledge that I'm dropping this deep in a safe harbor:

TLS1.0 and SSLv3 are only sufficiently different to be incompatible with one another. However, a good bit of the Cyphers are shared between TLS1.0 and SSLv3 (source: https://www.openssl.org/docs/apps/ciphers.html#TLS_v1_0_cipher_suites_ scroll up for the SSLv3) 

Since POODLE deomonstrates that the CBC cyphers in SSLv3 are insecure ...

What's the plan to bring TLS1.2 (or even 1.3) to In/Out calls on the Platform?

 
Swetha Kiran NallamothuSwetha Kiran Nallamothu
@Kevin_P

Here is a relevant discussion on the cypher suites shared by these two protocols:

http://security.stackexchange.com/questions/70832/why-doesnt-the-tls-protocol-work-without-the-sslv3-ciphersuites

Hope this helps.
brian_sfdcbrian_sfdc
@KevinP

you can use this as our official word: https://help.salesforce.com/apex/HTViewSolution?urlname=Salesforce-disabling-SSL-3-0-encryption&language=en_US

Specifically for poodle the ability to change the padding characters and the padding length while keeping the packet valid is one of the deficiencies that makes SSLv3 insecure while TLSv1 doesn't have this issue.  Attempting the same attack with TLSv1 wouldn't be successful.

hope this helps.
Operations ManagmentOperations Managment
We see a large percentage of salesforce callouts to our API failing with the error "Received fatal alert: bad_record_mac" but can't recreate when calling our API directly. We have reenabled ssl3 support on the API but the problem persists. 
Always ThinkinAlways Thinkin
I started getting those bad_record_mac errors too in the last 48 hours as well from two different apps that were previously getting the handshake failures. 
brian_sfdcbrian_sfdc
@operations management and @Always Thinkin

Can you guys send me your org id, the endpoint, the time it started, and best way to reach you guys (or any other information you guys have) to bestebez@salesforce.com
CRMScienceKirkCRMScienceKirk
@brian_sfdc - I too have been seeing an increase in bad_record_macs for a client's customers.  I've sent you an email as well.  The client disabled SSL v3 and older on 10/15.  Happy to gather any additional related informaiton.
Swetha Kiran NallamothuSwetha Kiran Nallamothu

@Operations Managment @Always Thinkin @CRMScienceKirk

Are any of you still seeing this error?

This could be because of a buggy implementation of one of the cryptographic algorithms at the client end. You can find more information here:

http://security.stackexchange.com/questions/39844/getting-ssl-alert-write-fatal-bad-record-mac-during-openssl-handshake

Hope it helps.

Best,
Swetha.

Steven LawranceSteven Lawrance
As an update, callout target servers that are contacted only by sandbox instances (CS1, CS2, etc) and not by production instances (NA1, NA2, AP1, EU1, etc) should be able to have SSLv3 disabled on them without seeing callout failures as long as TLSv1 is enabled on those callout target servers. If you have such a callout target server that presently has SSLv3 enabled and can turn off SSLv3 on it in the near future, can you proceed with that and let us know if all is well or if issues arise?

Thanks
Always ThinkinAlways Thinkin
Today, Dec. 10, is the day they are supposedly disabling SSLv3.0 outbound calls for all Production instances. Has anyone gotten confirmation or done any tests that verify this happened? I have opened a Case with Support asking for confirmation FWIW.
Steven LawranceSteven Lawrance
Thanks for the question. As of December 3rd last week, SSLv3 was disabled on all sandbox instances. Today's disablement of SSLv3 is scheduled to occur starting tonight (PST) and will occur during an instance's after-hours / off-peak time window.
Operations ManagmentOperations Managment
@Swetha looks like a problem on the client (i.e. salesforce) side alright. The deployment to disable SSLv3 seems to have had no effect on this specific bad_record_mac issue. It is still occuring for sutomers as of 8am UTC this morning
Always ThinkinAlways Thinkin
Can we get confirmation that they deployment to disabled SSLv3 was succesfully completed please?
Steven LawranceSteven Lawrance
At this time, SSLv3 in outbound callouts is disabled in all instances except for AP0 and AP1.

AP0 and AP1 are in progress and are expected to have SSLv3 disabled momentarily, likely within 30 minutes.
Steven LawranceSteven Lawrance
At this time, SSLv3 in outbound callouts is disabled in all instances, including AP0 and AP1.
apexsutherlandapexsutherland
Posted this idea to promote TLS 1.2 support for outbound callouts/encryption requests:

https://success.salesforce.com/ideaView?id=08730000000DhJaAAK (https://success.salesforce.com/ideaView?id=08730000000DhJaAAK" target="_blank)
KevinPKevinP
@apexsutherland,

I'm sure TLS 1.2 support is coming, but I think there are some significant engineering hurdles before it's available. Specifically, I believe Sf still utilizes JDK 6, which doesn't support 1.1 or 1.2. 

Regardless of Salesforce's support of TLS1.2, there's evidence now that even that is (sometimes, often?) insecure. From the public release: 

The impact of this problem is similar to that of POODLE, with the attack being slightly easier to execute–no need to downgrade modern clients down to SSL 3 first, TLS 1.2 will do just fine. The main target are browsers, because the attacker must inject malicious JavaScript to initiate the attack. A successful attack will use about 256 requests to uncover one cookie character, or only 4096 requests for a 16-character cookie. This makes the attack quite practical.

https://community.qualys.com/blogs/securitylabs/2014/12/08/poodle-bites-tls
Steven LawranceSteven Lawrance
Regarding the TLS version of POODLE, a knowledge article exists for that at https://help.salesforce.com/apex/HTViewSolution?urlname=POODLE-for-TLS&language=en_US
Justin SilverJustin Silver
I am still seeing exceptions when making callouts to Twilio that say "Exception message: Received fatal alert: bad_record_mac". Has this issue been resolved and confirmed or could it still be causing issues?
Steven LawranceSteven Lawrance
Thanks, Justin. Please file a case for that issue if you haven't already. The bad_record_mac issue is under investigation and has some leads, but it's unfortunately not yet resolved. It's possible that a workaround exists, and if one is identified, then it can be posted here.

SSLv3 is fully disabled in the inbound and outbound directions at Salesforce on all instances and endpoints.
Operations ManagmentOperations Managment
The bad_record_mac issue is still occuring for us as of 9:14 UTC this morning 19th Dec 2014. I opened case 11140206 at 10/23/2014 11:35 AM but it was closed as "Request out of Scope" and no luck getting anyone on the salesforce side to investigate. brian_sfdc suggested getting a whireshark capture for our API sever but this problem is intermittant and this is our Production infrastructure so it is not feasible. Salesforce is the only client of our API that has this issue.
Steven LawranceSteven Lawrance
The bad_record_mac issue may have been caused by the change https://bugs.openjdk.java.net/browse/JDK-2222432 in Java 1.7.0_06, though it wasn't fixed until Java 1.7.0_51 with https://bugs.openjdk.java.net/browse/JDK-8026986 . Servers that choose a ciphersuite that make use of a Diffie-Hellman (DH) key exchange can encounter this issue when Salesforce calls out to it, and the probability is a 1 in 256 chance when it's a new connection. Reused persistent https connections shouldn't run into this issue.

Callouts or servers that keep the connection open should run into this issue less often than callouts or servers that close the connection after each request. If your callout or your server configuration uses the 'Connection: close' header in normal circumstances, it's advisable to not do that anymore.

To work around this issue for now, disabling the ciphersuites on the callout's target server that use a DH key exchange should be effective. We believe that this issue is already fixed in the Spring '15 release, though we can't promise that, as per Salesforce's safe harbor statement at http://investor.salesforce.com/about-us/investor/safe-harbor-statement/default.aspx. We may be able to get the fix deployed earlier than the Spring '15 release, but that can't be promised at this time. Disabling the DH ciphersuites on the callout target server should give quick relief for now.
Steven LawranceSteven Lawrance
As an update, a recently deployed fix has addressed the bad_record_mac issue. Diffie-Hellman (DH) ciphersuites can be used with Salesforce callouts at this time without seeing the bad_record_mac issue. Please file a case if the bad_record_mac issue continues to occur for you. Thanks.
篤 武藤篤 武藤
Hello, at instance AP0 it seems the bad_record_mac issue resolved.
But the handshake_failure issue occur again.
On January 29, 2015 10:15 UTC handshake_failure error occured.
 
Steven LawranceSteven Lawrance
Thanks, 篤 武藤. I see that some handshake_failure messages were logged around that date and time on AP0, though it appears to be mostly isolated to that day and hasn't occurred since the UTC end of that day. Please feel free to log a case with Salesforce for more information, to get questions answered directly, and to get root cause information. I see that a related investigation for this incident on this instance exists, and the conditions that led to this issue are being reviewed to help prevent a reoccurrence going forward.
篤 武藤篤 武藤
Thanks, Steven.
I opened a case and got root cause information.
Callout program bug.
Volker_factory42Volker_factory42
@Steven Lawrance Could you solve your problem? We have similar problems and couldn't find a working solution...