You need to sign in to do that
Don't have an account?

Training a model from Apex
Hello,
I'm trying to set up a Lightning app that will allow a user to provide a url to a zip file, have that zip file loaded as a dataset into the predictive vision service, then specify the newly created dataset to be trained into a datamodel. I'm able to load the data model from the menu, and get back a dataset id.
But I'm stuck on how to call the training service. I modified the provided VisionController code as follows:
@AuraEnabled
public static String postDatasetTrain(string datasetId, string datasetName) {
// Get a new token
JWT jwt = new JWT('RS256');
jwt.cert = 'SelfSignedCert_18Jul2016_235259';
jwt.iss = 'developer.force.com';
jwt.sub = 'joseph.daily@careerbuilder.com';
jwt.aud = 'https://api.metamind.io/v1/oauth2/token';
jwt.exp = '3600';
String access_token = JWTBearerFlow.getAccessToken('https://api.metamind.io/v1/oauth2/token', jwt);
String VISION_API = 'https://api.metamind.io/v1/vision';
String DATA_TRAIN = VISION_API + '/train';
string contentType = HttpFormBuilder.GetContentType();
// Compose the form
string form64 = '';
form64 += HttpFormBuilder.WriteBoundary();
form64 += HttpFormBuilder.WriteBodyParameter('name', datasetName);
form64 += HttpFormBuilder.WriteBoundary();
form64 += HttpFormBuilder.WriteBodyParameter('datasetId', datasetId);
form64 += HttpFormBuilder.WriteBoundary(HttpFormBuilder.EndingType.CrLf);
blob formBlob = EncodingUtil.base64Decode(form64);
string contentLength = string.valueOf(formBlob.size());
// Compose the http request
HttpRequest httpRequest = new HttpRequest();
httpRequest.setBodyAsBlob(formBlob);
httpRequest.setHeader('Connection', 'keep-alive');
httpRequest.setHeader('Content-Length', contentLength);
httpRequest.setHeader('Content-Type', contentType);
httpRequest.setMethod('POST');
httpRequest.setTimeout(120000);
httpRequest.setHeader('Authorization','Bearer ' + access_token);
httpRequest.setEndpoint(DATA_TRAIN);
Http http = new Http();
HTTPResponse res = http.send(httpRequest);
system.debug('res: ' + res);
system.debug('res.getBody(): ' + res.getBody());
return res.getBody();
}
But the response that comes back is always a 403 : forbidden message.
Is there anything obvious causing this in my code?
It's not an Apex issue, but how the Vision server handles the connect to the given url.
Having said that it's probably easier to use my wrapper. ;-)
https://github.com/muenzpraeger/salesforce-einstein-vision-apex
Currently the createDatasetFromUrl part is missing, but will be added end of this week.
All Answers
Are you using the .pem file to generate the token?
I'm using the .pem file to generate a token when making calls to the service from my terminal/curl, but otherwise, I just use the jwt issue process coupled with the self signed cert I set up in my environment. That works for uploading a photo and getting a prediction response back. Does that same process not work for uploading a dataset or training a dataset?
I'm not able to upload or train a dataset formatting my request in Apex, but I'm able to do both through the terminal.
I tried replacing the access token in the apex with the token I'm using in my terminal (which I generated from the .pem file) and I continue to get error messages back. When uploading a dataset through an apex request, regardless of which token I use, I get the following error response:
Hello,
This is the URL I've been using:
https://www.dropbox.com/s/vs4wcvp5cvgz88s/PokedexDataset.zip?dl=1
It works fine when I use it in my terminal with cURL.
And when using that url in incognito mode in chrome, it downloads the file automatically.
https://www.dropbox.com/s/vs4wcvp5cvgz88s/PokedexDataset.zip?dl=1
Intiially dropbox had these form of URLs and it worked - >
https://dl.dropboxusercontent.com/u/37532083/cars.zip
I am not able to successfully send the zip file via Apex or train a dataset through Apex.
I am able to do both through my Terminal with cURL.
When I first began this project, I was using File Dropper. The zip url I was sending via Apex was getting a response that included a dataset id, so I thought it was uploading successfully, but then when I checked on the dataset via my Terminal, I found out that the dataset wasn't uploading properly.
Then I switched to using Drop Box. Using the url they provide, I was able to successfully upload the zip file, train it, and compare pictures against it, via my Terminal.
So the problem of training the dataset via Apex, has grown into not being able to upload a dataset via Apex. And I have not found a way to train the dataset via Apex, even if I upload the dataset successfully via the Terminal.
So far, the only thing I've successfully done via Apex, is successfully send picture urls for prediction against the General Image model already available on metamind's end.
I'd be more than happy to share the rest of my code. For now, I'm left to believe this is primarily an issue with how I'm formatting the requests via Apex.
Is there a specific file hosting website anyone would recommend besides Drop Box?
Does anyone have any examples of formatting the zip file upload request or the train dataset request via Apex?
Here's the apex code I was using to set up my dataset upload request:
@AuraEnabled
public static String getDatasetUploadUrl(string myURL) {
// Get a new token
JWT jwt = new JWT('RS256');
jwt.cert = 'SelfSignedCert_18Jul2016_235259';
jwt.iss = 'developer.force.com';
jwt.sub = 'joseph.daily@careerbuilder.com';
jwt.aud = 'https://api.metamind.io/v1/oauth2/token';
jwt.exp = '3600';
String access_token = JWTBearerFlow.getAccessToken('https://api.metamind.io/v1/oauth2/token', jwt);
String VISION_API = 'https://api.metamind.io/v1/vision';
String DATA_UPLOAD = VISION_API + '/datasets/upload/sync';
string contentType = HttpFormBuilder.GetContentType();
// Compose the form
string form64 = '';
form64 += HttpFormBuilder.WriteBoundary();
form64 += HttpFormBuilder.WriteBodyParameter('path', myURL);
form64 += HttpFormBuilder.WriteBoundary(HttpFormBuilder.EndingType.CrLf);
system.debug('form64: ' + form64);
blob formBlob = EncodingUtil.base64Decode(form64);
string contentLength = string.valueOf(formBlob.size());
// Compose the http request
HttpRequest httpRequest = new HttpRequest();
httpRequest.setBodyAsBlob(formBlob);
httpRequest.setHeader('Connection', 'keep-alive');
httpRequest.setHeader('Content-Length', contentLength);
httpRequest.setHeader('Content-Type', contentType);
httpRequest.setMethod('POST');
httpRequest.setTimeout(120000);
httpRequest.setHeader('Authorization','Bearer ' + access_token);
httpRequest.setEndpoint(DATA_UPLOAD);
system.debug('httpRequest: ' + httpRequest);
system.debug('httpRequest header: ' + httpRequest.getHeader('path'));
system.debug('httpRequest body: ' + httpRequest.getBody());
Http http = new Http();
HTTPResponse res = http.send(httpRequest);
system.debug('res: ' + res);
system.debug('res.getBody(): ' + res.getBody());
return res.getBody();
}
First, upload your key file (einstein_platform.pem or predictive_serivces.pem) to Salesforce files by clicking the Files tab > Upload File and navigating to the key file. Then if you update your code to the code below, it should work. In code, you'd always want to generate the token, but if you want to do any testing to see if the token code itself is causing an issue, you can use our token generation page at https://api.metamind.io/token to manually generate a token. On this page you enter your email address and upload your key to get a token and then you could use that for testing.
Let me know if you're still having issues.
Dianne
So, I need those who respond to start actually reading through my other posts on this question before posting. I think you're missing A LOT of updates about where I'm at with this and what I've covered. If that's too much to ask, I'll just resubmit a new question, because after a month I don't see any progress being made.
That very first post where I was getting the 403 error, was a result of the dataset not being uploaded successfully in the first place. NOT AS A RESULT OF AN ISSUE WITH THE ACCESS TOKEN.
403 is a generic Forbidden error. It doesn't mean anything more than that. It certainly is not a definite indicator that the issue is with an acess token. ANYONE RESPONDING NEEDS TO CEASE RELYING ON THAT ARGUMENT.
But your code is different from what I was trying, so I figured I give it a go.
First of all, here's what I was getting before I tried your solution:
If you right click on that image, you can open a larger version and see that I'm getting a "Failed to download the dataset from the public URL path".
When I tried out your code, here's what I got:
If you look at that, you'll see I'm getting an "Invalid access token" response with your code.
My first lightning app is still working just fine with what I set up, and I've had no problems with the other ways I've set up access tokens for my other calls through curl. I have so far not encountered anything to make me think any of this is an issue with the access token. But because I know you all have your doubts, here's more proof.
Here's a screenshot of my first lightning app's access token working just fine today.
Here's the code for creating the request to the metamind api:
@AuraEnabled
public static String getCallVisionUrl(string myURL) {
// Get a new token
JWT jwt = new JWT('RS256');
jwt.cert = 'SelfSignedCert_18Jul2016_235259';
jwt.iss = 'developer.force.com';
jwt.sub = 'joseph.daily@careerbuilder.com';
jwt.aud = 'https://api.metamind.io/v1/oauth2/token';
jwt.exp = '3600';
String access_token = JWTBearerFlow.getAccessToken('https://api.metamind.io/v1/oauth2/token', jwt);
String VISION_API = 'https://api.metamind.io/v1/vision';
String PREDICT = VISION_API + '/predict';
string contentType = HttpFormBuilder.GetContentType();
// Compose the form
string form64 = '';
form64 += HttpFormBuilder.WriteBoundary();
form64 += HttpFormBuilder.WriteBodyParameter('modelId', EncodingUtil.urlEncode('GeneralImageClassifier', 'UTF-8'));
form64 += HttpFormBuilder.WriteBoundary();
form64 += HttpFormBuilder.WriteBodyParameter('sampleLocation', myURL);
form64 += HttpFormBuilder.WriteBoundary(HttpFormBuilder.EndingType.CrLf);
blob formBlob = EncodingUtil.base64Decode(form64);
string contentLength = string.valueOf(formBlob.size());
// Compose the http request
HttpRequest httpRequest = new HttpRequest();
httpRequest.setBodyAsBlob(formBlob);
httpRequest.setHeader('Connection', 'keep-alive');
httpRequest.setHeader('Content-Length', contentLength);
httpRequest.setHeader('Content-Type', contentType);
httpRequest.setMethod('POST');
httpRequest.setTimeout(120000);
httpRequest.setHeader('Authorization','Bearer ' + access_token);
httpRequest.setEndpoint(PREDICT);
system.debug('httpRequest: ' + httpRequest);
system.debug('httpRequest header: ' + httpRequest.getHeader('path'));
system.debug('httpRequest body: ' + httpRequest.getBody());
Http http = new Http();
HTTPResponse res = http.send(httpRequest);
system.debug('res: ' + res);
system.debug('res.getBody(): ' + res.getBody());
return res.getBody();
}
My original authentication is still working for that request, so why would the same access token not work for other requests, especially from the same sandbox?
Ignoring that glaring question no one has answered, let's move on to the other ways of generating and using an access token. Here's me generating a token using the pem file like you suggested.
Here's me using the generated access token to upload a zip file:
Continued in next post due to image restrictions....
Here's me succesfully sending a train command using that same access token:
Here's me successfully checking that the dataset has been trained into a datamodel:
And here's me successfully using that access token to compare an image to the datamodel I just set up:
Here's my lighting app's apex code formatting the request, with the access token I just generated used in place of the original getAccessToken method (which is commmented out for now).
@AuraEnabled
public static String getCallVisionUrl(string myURL) {
// Get a new token
JWT jwt = new JWT('RS256');
jwt.cert = 'SelfSignedCert_18Jul2016_235259';
jwt.iss = 'developer.force.com';
jwt.sub = 'joseph.daily@careerbuilder.com';
jwt.aud = 'https://api.metamind.io/v1/oauth2/token';
jwt.exp = '3600';
string access_token = 'b879e45a69c849696e55c8bd10f69cfc9741bf05';
// String access_token = JWTBearerFlow.getAccessToken('https://api.metamind.io/v1/oauth2/token', jwt);
String VISION_API = 'https://api.metamind.io/v1/vision';
String PREDICT = VISION_API + '/predict';
string contentType = HttpFormBuilder.GetContentType();
// Compose the form
string form64 = '';
form64 += HttpFormBuilder.WriteBoundary();
form64 += HttpFormBuilder.WriteBodyParameter('modelId', EncodingUtil.urlEncode('GeneralImageClassifier', 'UTF-8'));
form64 += HttpFormBuilder.WriteBoundary();
form64 += HttpFormBuilder.WriteBodyParameter('sampleLocation', myURL);
form64 += HttpFormBuilder.WriteBoundary(HttpFormBuilder.EndingType.CrLf);
blob formBlob = EncodingUtil.base64Decode(form64);
string contentLength = string.valueOf(formBlob.size());
// Compose the http request
HttpRequest httpRequest = new HttpRequest();
httpRequest.setBodyAsBlob(formBlob);
httpRequest.setHeader('Connection', 'keep-alive');
httpRequest.setHeader('Content-Length', contentLength);
httpRequest.setHeader('Content-Type', contentType);
httpRequest.setMethod('POST');
httpRequest.setTimeout(120000);
httpRequest.setHeader('Authorization','Bearer ' + access_token);
httpRequest.setEndpoint(PREDICT);
system.debug('httpRequest: ' + httpRequest);
system.debug('httpRequest header: ' + httpRequest.getHeader('path'));
system.debug('httpRequest body: ' + httpRequest.getBody());
Http http = new Http();
HTTPResponse res = http.send(httpRequest);
system.debug('res: ' + res);
system.debug('res.getBody(): ' + res.getBody());
return res.getBody();
}
Here's my apex code making the request to upload the zip file, using the new access token:
@AuraEnabled
public static String getDatasetUploadUrl(string myURL) {
// Get a new token
JWT jwt = new JWT('RS256');
jwt.cert = 'SelfSignedCert_18Jul2016_235259';
jwt.iss = 'developer.force.com';
jwt.sub = 'joseph.daily@careerbuilder.com';
jwt.aud = 'https://api.metamind.io/v1/oauth2/token';
jwt.exp = '3600';
string access_token = 'b879e45a69c849696e55c8bd10f69cfc9741bf05';
//string access_token = JWTBearerFlow.getAccessToken('https://api.metamind.io/v1/oauth2/token', jwt);
String VISION_API = 'https://api.metamind.io/v1/vision';
String DATA_UPLOAD = VISION_API + '/datasets/upload/sync';
string contentType = HttpFormBuilder.GetContentType();
// Compose the form
string form64 = '';
form64 += HttpFormBuilder.WriteBoundary();
form64 += HttpFormBuilder.WriteBodyParameter('path', myURL);
form64 += HttpFormBuilder.WriteBoundary(HttpFormBuilder.EndingType.CrLf);
system.debug('form64: ' + form64);
blob formBlob = EncodingUtil.base64Decode(form64);
string contentLength = string.valueOf(formBlob.size());
// Compose the http request
HttpRequest httpRequest = new HttpRequest();
httpRequest.setBodyAsBlob(formBlob);
httpRequest.setHeader('Connection', 'keep-alive');
httpRequest.setHeader('Content-Length', contentLength);
httpRequest.setHeader('Content-Type', contentType);
httpRequest.setMethod('POST');
httpRequest.setTimeout(120000);
httpRequest.setHeader('Authorization','Bearer ' + access_token);
httpRequest.setEndpoint(DATA_UPLOAD);
system.debug('httpRequest: ' + httpRequest);
system.debug('httpRequest header: ' + httpRequest.getHeader('path'));
system.debug('httpRequest body: ' + httpRequest.getBody());
Http http = new Http();
HTTPResponse res = http.send(httpRequest);
system.debug('res: ' + res);
system.debug('res.getBody(): ' + res.getBody());
return res.getBody();
}
And here's the response I get using my new app (the one to upload the zip file) using the new access token I just created:
If you right click and open that in a new tab, you'll be able to see that I'm back to getting the "Failed to download the dataset from the public URL path."
Fun fact, that's the same zip file (and url) I used with my curl commands, which work just fine.
There is an issue with the way the metamind api accepts apex formatted requests to upload a zip file.
I have yet to see anyone point me to a file hosting service that provides a url I can successfully send via apex to the api.
Dropbox works fine via curl, but the exact same url does not work when I pass it via apex. No one is addressing this.
Hi Joseph, We apologize for the issue you are having and the delayed response. Thank you for clarifying the specific problem that you have troubleshooted to discover. We have recreated your issue and are looking into ways we can accept the APEX formatted URLs. We hope to have a resolution soon and will update the forum once resolved.
Thank you for your patience during this process.
-Michael
Einstein Platform
the issue with your Dropbox example is not Apex, it's the type of Dropbox URL that you're using.
To obtain the direct download link for your file you've to replace "www.dropbox.com" in your URL with "dl.dropboxusercontent.com".
Cheers,
René
Hey Rene,
First off, even if that was true, why does the url I was using work with curl but not with apex? I have multiple examples posted above proving that the url I was using, works when formatting the request via curl, but not from apex. Please account for that in your response.
Secondly, I tried your suggestion using this url:
dl.dropboxusercontent.com/s/vs4wcvp5cvgz88s/PokedexDataset.zip?dl=1
The above url successfully downloads the file in a browser. But what happens when we try to use it in a request to the metamind api?
Below is a screenshot of what happens:
It's a 400 error, Bad request, "Failed to download the dataset from the public URL path."
I then tried it via my terminal and curl. Here's the results of that:
So your url suggestion did not work via apex or curl. Which is even less useful than what I started with.
A url consists per definition out of two elements: a protocol and an address. Without the protocol it's only guesswork for any system to see what you want to do. That's also the reason why curl fails.
Please see also curl example in the official documentation => https://metamind.readme.io/docs/create-a-dataset-zip-async
Thanks for the response!
So per your suggestion I tried the following url:
https://dl.dropboxusercontent.com/s/vs4wcvp5cvgz88s/PokedexDataset.zip?dl=1
In curl it worked!
But in the apex request it didn't:
It's not an Apex issue, but how the Vision server handles the connect to the given url.
Having said that it's probably easier to use my wrapper. ;-)
https://github.com/muenzpraeger/salesforce-einstein-vision-apex
Currently the createDatasetFromUrl part is missing, but will be added end of this week.
The dl url worked with apex and curl. But the url I was using originally:
https://www.dropbox.com/s/vs4wcvp5cvgz88s/PokedexDataset.zip?dl=1
That worked with curl but not apex. Does anyone know or understand why there's a difference in performance there?
Even the zip file you guys are hosting on your end doesn't work!
Joseph, there is AFAIK no API change. The Dropbox URL from your screenshot throws also a 404 error in my browser - which means that the file isn't accessible in general.
The location where a customer hosts files for upload is up to the individual needs. I found Dropbox pretty reliable, but that can also be AWS S3, GDrive, a custom FTP server etc. The content only needs to be publically accessible via https.
Which leads to your comment about the sample data: please change the used protocol to https instead of http. Then it'll work.
Nope! Here I am using the https protocol with the mountain vs beaches zip file.
And here I am using that same access token to pull down info about a previously uploaded zip file, so I know it's not the access token:
And I have no problem downloading the file with the following dropbox url, even in incognito, so I know this url works perfectly:
https://www.dropbox.com/sh/lcic17heze539zk/AABwVUy_KP5rqRl2DO89v7qHa?dl=1
Too bad that doesn't work with the api, because that's the new way drop box formats their direct links now. Notice the use of sh instead of s, as well as the elimination of the use of the name of the zip file in the link.
But the following format you originally recommended:
https://dl.dropboxusercontent.com/sh/lcic17heze539zk/AABwVUy_KP5rqRl2DO89v7qHa?dl=1
Doesn't work at all now, because they changed the formatting of their url's.
I have seen no proof that there is currently any viable way to upload a zip file to the api.
I tried that. Didn't work either. Same response.
Also tried deleting and re-uploading my zip file in dropbox. Got a different url format this time. Still seeing the same issue, no matter how I format the drop box url in the request. Either with the "dl.dropboxusercontent.com" format:
Or the format of the url dropbox gives:
And the following is me using the same access token to retrieve data about previously uploaded datasets, to show that the access token is working, even though the service to upload a zip file does not.
Here is the current zip file url's I have been using.
Dropbox default format:
https://www.dropbox.com/s/idson0w7s3yfh1k/pokemon.zip?dl=1
and your recommended format:
https://dl.dropboxusercontent.com/s/idson0w7s3yfh1k/pokemon.zip
You used the http protocol and didn't use the https protocol when uploading the mountain and beach zip. Which contradicts your advice above:
"please change the used protocol to https instead of http. Then it'll work."
And the metamind documentation states the https protocol is necessary: https://metamind.readme.io/v1/page/troubleshooting
It seems like we have a lot of inconsistency between stated policy protocol and actual functionality here.
And none of this gets me closer to understanding why I have the ability as a user to use my access token to look up data and successfully send requests, except when it involves uploading a dataset.
Does the api have different levels of permissions for different users or something?