Table of contents

Transfer files

Instead of manually copying files such as applications, libraries, and graphics to and from your user home directory ( /user-home/<uid>) in Watson Studio Local, you can automate the process with REST APIs in https://<9.87.654.321>/api/v1/filemgmt/ (where <9.87.654.321> represents the master node IP).</9.87.654.321>

Requirement: A bearer token is needed to authenticate into your user home directory.

File system restriction: Any file or directory in the relative-path following user-home in https://<9.87.654.321>/api/v1/filemgmt/user-home/<file_or_folder> must not exceed 255 characters. Otherwise, the response returns status code 400 and body ENAMETOOLONG: name too long.

Tasks you can do:

Download files from your user home directory

Either of the following endpoints downloads files from your user home directory:

GET https://<9.87.654.321>/api/v2/filemgmt/view/file
GET https://<9.87.654.321>/api/v1/filemgmt/user-home/<file path>

The <file path> denotes a file relative to the user home directory /user-home/<uid> in the Watson Studio Local cluster. The response returns the file contents in the response body. The download API might also be able to resolve subdirectories such as /user-home/<uid>/folder1/folder2/file.ext. The file path can be relative within your user home directory, for example, folder/../folder/file.ext. If you attempt to use a relative path outside of your user home directory, for example, ../<different-uid>/folder/file.ext, then you receive a 404.

Returns:

- [200 OK, {Content of file}]
  - [400 Bad Request, {success: false, error: err.message}]
  - [401 Unauthorized, {success: false, error: 'Authentication failed'}]
  - [404 Not Found, {success: false, error: [message]}]

Examples:

$ curl -i -k -H "authorization: Bearer
$bearerToken" -X GET
"https://9.87.654.321/api/v2/filemgmt/view/file"
curl -i -k --header "authorization: Bearer
$bearerToken"
https://9.87.654.321/api/v1/filemgmt/user-home/test.cpp
HTTP/1.1 200 OK
Server: openresty
Date: Wed, 21 Jun 2017 18:03:49 GMT
Content-Type: application/octet-stream
Transfer-Encoding: chunked
Connection: keep-alive
X-Powered-By: Express

#include <iostream>
#include <cstring>
using namespace std;

int main()
{
  cout << "Testing out";
}

Because the transfer encoding is chunked, the receiver must handle a stream to properly receive the file. Otherwise, only part of the file gets downloaded.

Correct example:

https.get(options, function(res) {
    var ws = fs.createWriteStream(uploadPath);
    //below function is the important part, each data chunk is
handled individually
    //by appending it in a write stream
    res.on('data', function(data) {
      //note in this example we're writing raw data with no
encoding, letting the client treat it with whatever
      //encoding after download is complete
      ws.write(data);
    });
    res.on('error', function(error) {
      logger.error(error);
      callback(error, null);
    });
    //once all the data have been received, it is up to the client
to decide what to do with data
    res.on('end', function() {
      ws.close();
      //do stuff with data here, or elsewhere
      // example, read it with utf 8 encoding
      fs.readFile(uploadPath, 'utf8', (err, data) => {
        if (err)
          return callback(err, null);
        return callback(null, data);
      });

    });
  }).on('error', function(err){
    callback(err, null);
  });

Incorrect example:

https.get(options, function(res) {
    var result = '';
    result = res.body;
  }).on('error', function(err){
    callback(err, null);
  });

The incorrect example returns the body of the first data chunk only, making it appear empty. If the file does not exist, a JSON response returns with a false for the success code and a message that details the error:

HTTP/1.1 400 Bad Request
Server: openresty
Date: Mon, 19 Jun 2017 14:12:00 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 160
Connection: keep-alive
X-Powered-By: Express
ETag: W/"a0-wRwpkehE9MKcyhEBa/CKIl6wiVg"

{"success":false,"error":"ENOENT: no such file or directory, stat
'/user-home/1025/spark-applications/spark-submit-5943f783a9b0a700175f5561/doesnotexist'"}

List files in your user home directory

Either of the following endpoints list the first-level root file hierarchy of your user home directory under /user-home:

GET https://<9.87.654.321>/api/v2/filemgmt/view/
GET https://<9.87.654.321>/api/v1/filemgmt/user-home/

You can append a file path relative within your user home directory, for example, GET https://<9.87.654.321>/api/v2/filemgmt/view/folder/../folder/file.ext. If you attempt to use a relative path outside of your user home directory, for example, GET https://<9.87.654.321>/api/v2/filemgmt/view/../<different-uid>/folder/file.ext, you will receive a 404.

To display the contents of a file, append a filename: GET https://<9.87.654.321>/api/v2/filemgmt/view/file.

Returns:

 - [200 OK, [file1, folder1/, file1.txt]
  - [400 Bad Request, {success: false, error: err.message}]
  - [401 Unauthorized, {success: false, error: 'Authentication failed'}]
  - [404 Not Found, {success: false, error: [message]}]

The folder name ends with / and filters out miscellaneous files like ., .., .DS_Store, and Thumbs.db.

The Content-Type response header is used to indicate the media type of the resource. These examples use JSON.

Examples:

$ curl -i -k -H "authorization: Bearer
$bearerToken" -X GET "https://9.87.654.321/api/v2/filemgmt/view"
$ curl -k -X GET \
  https://9.87.654.321/api/v1/filemgmt/user-home \
  -H "authorization: Bearer $bearerToken" -i
***************************************************
HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: application/json; charset=utf-8
Content-Length: 37
ETag: W/"25-gfF2px8Q7tr78v+ZPi7EBZAjq08"
Date: Thu, 15 Jun 2017 22:45:12 GMT
Connection: keep-alive

["file.txt","folder1/","window x.JPG"]

The following endpoint lists the first-level file hierarchy relative to your user home directory:

GET https://<9.87.654.321>/api/v1/filemgmt/user-home/<file_or_folder>

The <file_or_folder> path denotes the relative path of either file or folder to the user home directory /user-home/<uid> in the Watson Studio Local cluster.

Returns:

- [200 OK, [file1, folder1/, file1.txt]]
  - [400 Bad Request, {success: false, error: err.message}]
  - [401 Unauthorized, {success: false, error: 'Authentication failed'}]
  - [403 Forbidden]

The folder name ends with / and filters out miscellaneous files like ., .., .DS_Store, and Thumbs.db.

Examples:

$ curl -k -X GET \
  https://9.87.654.321/api/v1/filemgmt/user-home/folder1 \
  -H "authorization: Bearer $bearerToken" -i 
***********************************************************
HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: application/json; charset=utf-8
Content-Length: 31

ETag: W/"1f-h2T5Se/SdxAgnrADfQhJBttNTlc"
Date: Thu, 15 Jun 2017 22:54:38 GMT
Connection: keep-alive

["file.txt","kitchen_main.JPG"]
$ curl -k -X GET \
  https://9.87.654.321/api/v1/filemgmt/user-home/invalid/path \
  -H "authorization: Bearer $bearerToken" -i
***************************************************************
HTTP/1.1 400 Bad Request
X-Powered-By: Express
Content-Type: text/html; charset=utf-8
Content-Length: 75
ETag: W/"4b-E9LcYVOA0TX79CugWLOFm905cQg"
Date: Thu, 15 Jun 2017 22:56:07 GMT
Connection: keep-alive

{success: false, error: ENOENT: no such file or directory, stat
'user-home/user-id/invalid/path'}

Upload files to your user home directory

The following endpoint uploads files to your user home directory:

POST https://<9.87.654.321>/api/v1/filemgmt/user-home

The response is either: success with the uploaded file names, or fail with the error messages.

More arguments:

  • Set header: -H "Content-Type: multipart/form-data"
  • Use -F "<field-name>=@<file-path>" to specify a file to upload, where <file-path> is the full path (on Windows) or relative path (on UNIX) to the source file. To upload multiple files, append another -F "<field-name>=@<file-path>". If this argument is not provided, then a status code 400 is returned.

You can upload files to a directory relative to your user home directory. You can also upload directly to subdirectories, for example, /user-home/<uid>/path/subpath. Subdirectories are automatically created if they do not exist, so a file is always assumed. For example, POST -F "file=@file.txt" <base-url>/user-home/folder.txt will assume folder.txt is a directory and the result will be /user-home/<uid>/folder.txt/file.txt.

For a file and a directory with the same name, you can overwrite a file with a directory, but trying to overwrite a directory with a file returns an error to the user, for example, {"success":false,"result":"fname failed to upload\n","error":"/user-home/<uid>/fname is a directory, can't overwrite it with a file"}.

When you upload multiple files at the same time and the upload of one file fails, the process ends immediately with no attempt to upload the remaining files.

Returns:

 - [200 OK, {success: true, result : '1.txt (13 Bytes) uploaded\n'}]
  - [400 Bad Request, {success: false, result: resultString, error: err.message}]
  - [401 Unauthorized, {success: false, error: 'Authentication failed'}]

Returns for specific cases:

  • -F "<field-name>=@<file-path>" is not provided in the call:

    [400 Bad Request, {success: false, error: 'File argument is not provided.'}]

  • Uploading several files but one of them fails:

    [400 Bad Request, {success: false, result: '1.txt (13 Bytes) uploaded\nfail.txt failed to upload\n', error: err.message}]

  • Trying to overwrite a folder with a file that has the same name:

    [400 Bad Request, {"success":false,"result":"fname failed to upload\n","error":"/user-home//fname is a directory, can't overwrite it with a file"}]

Example:

# upload one file to user's home folder
$ curl -i -k -H "authorization: Bearer $bearerToken" -H
"Content-Type: multipart/form-data"
  -F "file=@1.txt" -X POST
"https://9.87.654.321/api/v1/filemgmt/user-home"
HTTP/1.1 100 Continue

HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: application/json; charset=utf-8
Content-Length: 55
ETag: W/"37-xKr0DDE9hC/7TE227q/FzKJ7ZtM"
Date: Mon, 19 Jun 2017 21:23:10 GMT
Connection: keep-alive

{"success":true,"result":"1.txt (13 Bytes) uploaded\n"}

# upload multiple files to a subfolder
$ curl -i -k -H "authorization: Bearer $bearerToken" -H
"Content-Type: multipart/form-data"
  -F "file=@1.txt" -F "file2=@large.zip" -X POST
"https://9.87.654.321/api/v1/filemgmt/user-home/subpath"
HTTP/1.1 100 Continue

HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: application/json; charset=utf-8
Content-Length: 82
ETag: W/"52-+xT515cu7Pmm1TPQoBRRJfW2nIs"
Date: Mon, 19 Jun 2017 21:39:42 GMT
Connection: keep-alive

{"success":true,"result":"Uploaded 1.txt (13 Bytes)\nlarge.zip
(3.94 GB)\n"}

Create folders

The following endpoint creates folders in your user home directory:

POST https://<9.87.654.321>/api/v2/filemgmt/create-folder

The server creates the directory base on the request in /user-home/uid/.

More arguments:

  • Set header: -H "Authorization: {AccessToken}"
  • In the body, create a JSON object that defines the folder that you want to create:
    body {
    "folder" : "folder1/folder2/folder2"
    }

Returns:

 - [200 OK, {success: true, result : 'folder create success'}]
  - [401 Unauthorized, {success: false, error: 'Authentication failed'}]
  - [404 Not Found, {success: false, error: [message]}]

Example:

curl -k -X POST -d '{"folder":"xyz/wxz"}'
https://9.87.654.321/api/v2/filemgmt/create-folder -H
"authorization: Bearer $bearerToken"

Copy a file or folder

The following endpoint copies a file or folder to another folder:

POST https://<9.87.654.321>/api/v2/filemgmt/copy

Additional arguments:

  • In the body, create a JSON object with two fields: source and destination. Source is a path within the user folder that point to a file or folder. Destination must be a folder. For example, if there is a file at /user-home/<uid>/folderA/fileB.txt, the source must be /folderA/fileB.txt. The file path can be relative within your user home directory, for example, folder/../folder/file.ext. If you attempt to use a relative path outside of your user home directory, for example, ../<different-uid>/folder/file.ext, then you receive a 404 since a user cannot access another user's contents.

Returns:

 - [200 OK, {success: true/false}]
  - [400 Bad Request, {success: false, error: err.message}]
  - [401 Unauthorized, {success: false, error: 'Authentication failed'}]
  - [404 Not Found, {success: false, error: [message]}]

Example:

The following example copies folder/user-home/1001/folderA/fileB.txt to /user-home/1001/folderC:

curl -i -k --header "authorization: Bearer
$bearerToken" -H "content-type: application/json"
https://9.87.654.321/api/v2/filemgmt/copy -d
'{"source":"folderA/fileB.txt","destination":"folderC/"}'

{"success":true}

The following table details the API behavior:

source destination result
file existing folder; folder/file does not exist folder/file
file existing folder; folder/file exists error File/Folder exists!
folder1 existing folder2; folder2/folder1/ does not exist folder2/folder1
folder1 existing folder2; folder2/folder1/ exists error File/Folder exists!
file non-existing folder create folder, chmod, chown, and then folder/file
file f1 existing folder f2; f2/f1 exists error File/Folder exists!
non-existing f1 existing folder f2 error ENOENT
non-existing f1 non-existing folder f2 error ENOENT; note: won't create f2

For example, according to the table, if the source is a file /fileA.txt and destination is a folder /folderB/folderC, and there is no /folderB/folderC/fileA.txt, then the folder/file does not exist and a new file is placed at /folderB/folderC/fileA.txt. The returning call is {"success":true} in the body. In the same situation, if there exists a file /folderB/folderC/fileA.txt, then the folder/file exists, and a 400 error would be returned with "Folder/File already exist" in the err.message.

Move a file or folder

The following endpoint moves a file or folder to another folder:

POST https://<9.87.654.321>/api/v2/filemgmt/move

The move API resembles the copy API except that once it completes, the source folder no longer exists at its original location.

More arguments:

  • In the body, create a JSON object with two fields: source and destination. Source is a path within the user folder that points to a file or folder. Destination must be a folder. For example, if there is a file at /user-home/<uid>/folderA/fileB.txt, the source should be /folderA/fileB.txt. The file path can be relative within your user home directory, for example, folder/../folder/file.ext. If you attempt to use a relative path outside of your user home directory, for example, ../<different-uid>/folder/file.ext, then you will receive a 404 because a user cannot access another user's contents.

Returns:

 - [200 OK, {success: true/false}]
  - [400 Bad Request, {success: false, error: err.message}]
  - [401 Unauthorized, {success: false, error: 'Authentication failed'}]
  - [404 Not Found, {success: false, error: [message]}]

Example:

The following examples moves folder/user-home/1001/folderA/fileB.txt to /user-home/1001/folderC:

curl -i -k --header "authorization: Bearer
$bearerToken" -H "content-type: application/json"
https://9.87.654.321/api/v2/filemgmt/move -d
'{"source":"folderA/fileB.txt","destination":"folderC/"}'

{"success":true}

The following table details the behavior of the move API:

source destination result
file existing folder; folder/file does not exist folder/file
file existing folder; folder/file exists error File/Folder exists!
folder1 existing folder2; folder2/folder1/ does not exist folder2/folder1
folder1 existing folder2; folder2/folder1/ exists error File/Folder exists!
file non-existing folder create folder, chmod, chown, and then folder/file
file f1 existing folder f2; f2/f1 exists error File/Folder exists!
non-existing f1 existing folder f2 error ENOENT
non-existing f1 non-existing folder f2 error ENOENT; note: won't create f2

For example, according to the table, if the source is a file /fileA.txt and destination is a folder /folderB/folderC, and there is no /folderB/folderC/fileA.txt, then the folder/file does not exist and a new file will be placed at /folderB/folderC/fileA.txt. The returning call is {"success":true} in the body. In the same situation, if there exists a file /folderB/folderC/fileA.txt, then the folder/file exists, and a 400 error would be returned with "Folder/File already exist" in the err.message.

Delete a file or folder

The following endpoint deletes a file or folder in your user home directory:

DELETE https://<9.87.654.321>/api/v2/filemgmt/delete/

Sub-paths are resolved, such as /user-home/uid/<file> or /user-home/uid/<folder>. Watson Studio Local users cannot delete any pre-created or global files and folders such as DSX_Projects, spark/logs, or spark/apps. Users can still delete contents within those folders. Also, users cannot delete the entire user-home/uid/ (including pre-created files/folders). Any attempt to delete these files or folders results in the following type of error message: {"success":false,"error":"No permission to delete the target file/folder"}.

More arguments:

  • When you delete a file, no option is needed.
  • When you delete a folder, include a JSON object in the HTTP request body: {"option":"-dir"}.

Returns:

- [200 OK, {success: true}]
  - [400 Bad Request, {success: false, error: err.message}]
  - [401 Unauthorized, {success: false, error: 'Authentication failed'}]

Example for deleting a single file:

$ curl -k -X DELETE
https://9.87.654.321/api/v2/filemgmt/delete/file -H "authorization:
Bearer $bearerToken" -H "Content-Type: application/json" -i
***********************************************************
HTTP/1.1 200 OK
Server: openresty
Date: Thu, 24 Aug 2017 22:43:25 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 16
Connection: keep-alive
X-Powered-By: Express
ETag: W/"10-oV4hJxRVSENxc/wX8+mA4/Pe4tA"

{"success":true}

Example for deleting a folder:

$ curl -k -X DELETE
https://9.87.654.321/api/v2/filemgmt/delete/folder -H
"authorization: Bearer $bearerToken" -H "Content-Type:
application/json" -d '{"option":"-dir"}' -i
***********************************************************
HTTP/1.1 200 OK
Server: openresty
Date: Thu, 24 Aug 2017 22:44:38 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 16
Connection: keep-alive
X-Powered-By: Express
ETag: W/"10-oV4hJxRVSENxc/wX8+mA4/Pe4tA"

{"success":true}

Example for deleting all contents inside of a folder:

If the path is a directory, the "option" field must be set to -dir in the HTTP request body. In addition, this operation keeps all pre-created system folders and contents therein, and deletes all non-system pre-created contents.

$ curl -k -X DELETE
https://9.87.654.321/api/v2/filemgmt/delete/target-dir/* -H
"authorization: Bearer $bearerToken" -H "Content-Type:
application/json" -d '{"option":"-dir"}' -i
***********************************************************
HTTP/1.1 200 OK
Server: openresty
Date: Thu, 24 Aug 2017 22:35:27 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 109
Connection: keep-alive
X-Powered-By: Express
ETag: W/"6d-9J+XQTiUKj2kQfQF2uXT+wYhVRo"

{"success":true,"result":"All contents under /target-dir/ other
than system pre-created folders are deleted"}

Example of an error message if the file/folder does not exist:

$ curl -k -X DELETE
https://9.87.654.321/api/v2/filemgmt/delete/notexist -H
"authorization: Bearer $bearerToken" -H "Content-Type:
application/json" -d '{"option":"-dir"}' -i
***********************************************************
HTTP/1.1 400 Bad Request
Server: openresty
Date: Thu, 24 Aug 2017 22:45:58 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 94
Connection: keep-alive
X-Powered-By: Express
ETag: W/"5e-h5nTanurbhxBy/EDJchL/c3NQf0"

{"success":false,"error":"ENOENT: no such file or directory, stat
'/user-home/1001/notexist'"}