Systems
Once you are authorized to make calls to the various services, one of first things you may want to do is view storage and execution resources available to you or create your own. In Tapis a storage or execution resource is referred to as a system.
Overview
A Tapis system represents a server or collection of servers exposed through a single host name or IP address. Each system is associated with a specific tenant. A system can be used for the following purposes:
Running a job, including:
Staging files to a system in preparation for running a job.
Executing a job on a system.
Archiving files and data on a remote system after job execution.
Storing and retrieving files and data.
Each system is of a specific type (such as LINUX or S3) and owned by a specific user who has special
privileges for the system. The system definition also includes the user that is used to access the system,
referred to as effectiveUserId. This access user can be a specific user (such as a service account) or dynamically
specified as ${apiUserId}
. For the case of ${apiUserId}
, the username is extracted from the identity
associated with the request to the service.
Model
At a high level a system represents the following information:
- id
A short descriptive name for the system that is unique within the tenant.
- description
An optional more verbose description for the system.
- systemType - Type of system
LINUX, S3, IRODS or GLOBUS
- owner
A specific user set at system creation. By default this is the resolved value for ${apiUserId}, the user making the request to create the system.
- host - Host name, IP address or Globus ID
FQDN, IP address, Globus endpoint ID or Globus collection ID.
- enabled - Enabled flag
Indicates if system is currently considered active and available for use. By default this is true.
- effectiveUserId - Effective User
The username to use when accessing the system. A specific user (such as a service account) or the dynamic user
${apiUserId}
. By default this is${apiUserId}
.- defaultAuthnMethod - Default authentication method
How access authentication is handled by default. Authentication method can also be specified as part of a request. Supported methods: PASSWORD, PKI_KEYS, ACCESS_KEY, TOKEN.
- bucketName - Bucket name
For an S3 system this is the name of the bucket.
- rootDir - Effective root directory
Directory to be used when listing files or moving files to and from the system. All paths are relative to this directory when using Files to list, copy, move, mkdir, etc. For more information please see Effective Root Directory. May not be updated. Contact support to request a change.
- dtnSystemId - DTN system Id
A system that can be used during job execution as a Data Transfer Node (DTN). Use is optional. The DTN is used if the job submission request or the application defintion specify dtnSystemInputDir or dtnSystemOutputDir. canExec must be true. This execution system and the DTN system must have the same rootDir and the file system must be shared storage. Please see DTN Configuration.
- canExec
Indicates if system can be used to execute jobs.
- canRunBatch
Indicates if system supports running jobs using a batch scheduler. By default this is false.
- enableCmdPrefix
Indicates if system allows a job submission request to specify a cmdPrefix. Since cmdPrefix is a free form command it is a security concern. By default this is false.
- allowChildren
Indicates if system supports creating child systems using this system as the parent. By default this is false.
- Job related attributes
Various attributes related to job execution such as jobRuntimes, jobWorkingDir, batchScheduler, batchSchedulerProfile, batchLogicalQueues
When creating a system the required attributes are: id, systemType, host, defaultAuthnMethod and canExec. Depending on the type of system and specific values for certain attributes there are other requirements.
Note that a system may be created as a storage-only resource (canExec=false) or as a system that can be used for both execution and storage (canExec=true).
Getting Started
Before going into further details about Systems, here we give some examples of how to create and view systems.
In the examples below we assume you are using the TACC tenant with a base URL of tacc.tapis.io
and that you have
authenticated using PySDK or obtained an authorization token and stored it in the environment variable JWT,
or perhaps both.
Creating a System
Create a local file named system_example.json
with json similar to the following:
{
"id":"tacc-sample-<userid>",
"description":"My storage system",
"host":"tapis-vm.tacc.utexas.edu",
"systemType":"LINUX",
"defaultAuthnMethod":"PKI_KEYS",
"effectiveUserId":"${apiUserId}",
"rootDir":"/",
"canExec": false
}
where <userid> is replaced with your username, and your host name is updated appropriately. Note that although credentials may be included in the definition we have not done so here. For security reasons, it is recommended that login credentials be updated using a separate API call as discussed below.
Using PySDK:
import json
from tapipy.tapis import Tapis
t = Tapis(base_url='https://tacc.tapis.io', username='<userid>', password='************')
with open('system_example.json', 'r') as openfile:
my_storage_system = json.load(openfile)
t.systems.createSystem(**my_storage_system)
Using CURL:
$ curl -X POST -H "content-type: application/json" -H "X-Tapis-Token: $JWT" https://tacc.tapis.io/v3/systems -d @system_example.json
Effective Root Directory
Correctly defining the system attribute rootDir is critical because it serves as an effective root directory when referencing file paths through the Tapis Files or Jobs services. All paths are relative to this directory when using Files to list, copy, move, mkdir, etc. When creating a system there are certain restrictions for this attribute that should be kept in mind:
Once a system is created, rootDir may not be updated. Contact support to request a change.
Required for systems of type LINUX or IRODS. Must begin with
/
.Optional for systems of type S3 or GLOBUS.
For S3 type systems, typically will not begin with
/
.S3 keys are usually created and manipulated using URLs and do not have a leading
/
.
Support is provided for resolving the following variables at create time: ${apiUserId}, ${tenant} and ${owner}.
Use of HOST_EVAL at System Creation
There is also a special macro available for rootDir that may be used under certain conditions when a system
is first created. The macro name is HOST_EVAL
.
The syntax for the macro is HOST_EVAL($var)
, where var
is the environment variable to be evaluated
on the system host when the create request is made.
Note that the $
character preceding the environment variable name is optional.
If after resolution the final path does not have the required leading slash (/
) to make it an absolute path,
then one will be prepended.
The following conditions must be met in order to use the macro
System must be of type LINUX
Credentials must be provided when system is created.
Macro
HOST_EVAL()
must only appear once and must be the first element of the path. Including a leading slash is optional.The effectiveUserId for the system must be static. Note that effectiveUserId may be set to
${owner}
.
Here are some examples
HOST_EVAL($SCRATCH)
HOST_EVAL($HOME)
/HOST_EVAL(MY_ROOT_DIR)/scratch
/HOST_EVAL($PROJECT_HOME)/projects/${tenant}/${owner}
Registering Credentials for a System
Now that you have registered a system you will need to register credentials so you can use Tapis to access the host. Various authentication methods can be used to access a system, such as PASSWORD, PKI_KEYS and TOKEN. Note that the TOKEN authentication method is for systems of type GLOBUS. Registering credentials for a GLOBUS type system is a special case that involves steps different from those described in this section. Please see the section below on Registering Credentials for a Globus System for more information.
Please note that there is support for only one set of credentials per user per system. Updating credentials overwrites previously registered data.
Here we will cover registering PKI_KEYS (i.e. ssh keys) as an example.
Create a local file named cred_tmp.json
with json similar to the following:
{
"publicKey": "<ssh_public_key>",
"privateKey": "<ssh_private_key>"
}
where <ssh_public_key> and <ssh_private_key> are replaced with your keys. The keys must be encoded on a single line with embedded newline characters. You may find the following linux command useful in converting a multi-line private key into a single line:
cat $privateKeyFile | awk -v ORS='\\n' '1'
Using PySDK:
t.systems.createUserCredential(systemId='tacc-sample-<userid>', userName='<userid>', publicKey='<ssh_public_key>', privateKey='<ssh_private_key>'))
Using CURL:
$ curl -X POST -H "content-type: application/json" -H "X-Tapis-Token: $JWT" https://tacc.tapis.io/v3/systems/credential/tacc-sample-<userid>/user/<userid> -d @cred_tmp.json
An optional attribute loginUser may be included in the request body in order to map the Tapis user to a username to be used when accessing the system. If the login user is not provided then there is no mapping and the Tapis user is always used when accessing the system. When a loginUser is provided the json would be similar to the following:
{
"publicKey": "<ssh_public_key>",
"privateKey": "<ssh_private_key>",
"loginUser": "<linux_host_username>"
}
Note that credentials are stored in the Security Kernel. Only specific Tapis services are authorized to retrieve credentials.
Use of PKI_KEYS as credentials
When using an ssh keypair as credentials there are several important points to keep in mind. As discussed above, the public key and private key must be encoded on a single line. This can sometimes be challenging. For example, copying and pasting may convert newline characters in a way that is not compatible with processing in Tapis. You may find the following linux command useful in converting a multi-line private key into a single line:
cat $privateKeyFile | awk -v ORS='\\n' '1'
Also, Tapis does not currently support OPENSSH type keys. After generating the keypair, please inspect the first few
lines of the private key file and confirm that it is not of type OPENSSH. Typically, a valid private key file will
start with the line -----BEGIN RSA PRIVATE KEY-----
.
If your private key is of type OPENSSH please use a command similar to the following to generate your keypair:
ssh-keygen -t rsa -b 4096 -m PEM
When generating the keypair, do not use a passphrase. This can interfere with non-interactive use of the keypair.
Finally, please be aware that if the host has multi-factor authentication (MFA) enabled this may prevent Tapis from communicating with the host. Tapis does not currently support MFA.
When encountering problems here are some suggestions on what to check:
Public and private keys are each on one line in the json file. Newline characters in private key are properly encoded.
Keypair is not of type OPENSSH
Keypair does not have a passphrase
Public key has been added to the authorized_keys file for the target user. File ~/.ssh/authorized_keys
File ~/.ssh/authorized_keys has proper permissions.
MFA is not enabled for the target host.
If problems persist you can also attempt to manually validate the keypair using a command similar to this:
ssh -i /tmp/my_private_key testuser@myhost.com
where /tmp/my_private_key contains the original multi-line private key. If everything is set up correctly and the keypair is valid you should be logged into the host without being prompted for a password.
Registering Credentials for a Globus System
Registering credentials for a GLOBUS type system is a special case that involves steps different from those described in
the section above. For a GLOBUS type system, the user will need to use the TOKEN authentication method and generate
an accessToken
and refreshToken
using two special-purpose System service endpoints.
Please note that your Tapis site installation must have been configured by the site administrator to support Globus. Please see Globus_Config.
Viewing Systems
Retrieving details for a system
To retrieve details for a specific system, such as the one above:
Note
See the section below on Selecting to find out how to control the amount of information returned.
Using PySDK:
t.systems.getSystem(systemId='tacc-sample-<userid>')
Using CURL:
$ curl -H "X-Tapis-Token: $JWT" https://tacc.tapis.io/v3/systems/tacc-sample-<userid>
The response should look similar to the following:
{
"result": {
"tenant": "dev",
"id": "tacc-sample-<userid>",
"description": "My storage system",
"systemType": "LINUX",
"owner": "<userid>",
"host": "tapis-vm.tacc.utexas.edu",
"enabled": true,
"effectiveUserId": "<userid>",
"defaultAuthnMethod": "PKI_KEYS",
"authnCredential": null,
"rootDir": "/",
"port": 22,
"useProxy": false,
"proxyHost": "",
"proxyPort": -1,
"dtnSystemId": null,
"canExec": false,
"canRunBatch": false,
"enableCmdPrefix": false,
"allowChildren": false,
"jobRuntimes": [],
"jobWorkingDir": null,
"jobEnvVariables": [],
"jobMaxJobs": 2147483647,
"jobMaxJobsPerUser": 2147483647,
"batchScheduler": null,
"batchSchedulerProfile": null,
"batchLogicalQueues": [],
"batchDefaultLogicalQueue": null,
"jobCapabilities": [],
"tags": [],
"notes": {},
"uuid": "f83606bf-7a1a-4ff0-9953-dd732cc07ac0",
"deleted": false,
"created": "2021-04-26T18:45:40.771Z",
"updated": "2021-04-26T18:45:40.771Z"
},
"status": "success",
"message": "TAPIS_FOUND System found: tacc-sample-<userid>",
"version": "0.0.1",
"metadata": null
}
Note that authnCredential is null. Only specific Tapis services are authorized to retrieve credentials.
Retrieving details for all systems
To see the list of systems that you own:
Using PySDK:
t.systems.getSystems()
Using CURL:
$ curl -H "X-Tapis-Token: $JWT" https://tacc.tapis.io/v3/systems?select=allAttributes
The response should contain a list of items similar to the single listing shown above.
Child Systems
Creating Child Systems
A system that has allowChildren set to true allows for creation of child systems based on it. This ability provides a way to easily clone and manage systems based on existing systems. Child systems allow a user to set only a few fields, and use all other values from an existing parent system. This can reduce the difficulty in managing systems. It allows for all child systems to be updated when the parent is updated.
To create a child system, first ensure that the system intended to serve as the parent as allowChildren set to true. Next, create a local file (for example child_system_example.json) similar to the following:
{
"id": "my-child-<userid>",
"effectiveUserId": "${apiUserId}",
"rootDir": "/home/<userid>"
}
Where <userid> is replaced with your username. Also ensure that the root directory path is correct. Now use the create child system REST endpoint to create the child system. Let’s assume that the new child system will be a child of a parent system called parent-system.
Using PySDK:
import json
from tapipy.tapis import Tapis
t = Tapis(base_url='https://tacc.tapis.io', username='<userid>', password='************')
with open('child_system_example.json', 'r') as openfile:
child_system = json.load(openfile)
t.systems.createChildSystem(parentId="parent-system", **child_system)
Using CURL:
$ curl -X POST -H "content-type: application/json" -H "X-Tapis-Token: $JWT" https://tacc.tapis.io/v3/systems/parent-system/createChildSystem -d @child_system_example.json
These fields are maintained independently for child systems:
Attribute |
Type |
Example |
Notes |
---|---|---|---|
id |
String |
ds1.storage.default |
|
owner |
String |
jdoe |
|
enabled |
boolean |
FALSE |
|
effectiveUserId |
String |
tg869834 |
|
rootDir |
String |
/home/${apiUserId} |
|
deleted |
boolean |
FALSE |
|
created |
Timestamp |
2020-06-19T15:10:43Z |
|
updated |
Timestamp |
2020-07-04T23:21:22Z |
|
During the creation of a child system, any of these fields may be specified except for created, updated and deleted. All other fields are taken from the parent system.
Updating a Child System
Updates are done just like any other system, however, only the following fields may be updated for a child system.
Attribute |
Type |
Example |
Notes |
---|---|---|---|
effectiveUserId |
String |
tg869834 |
|
Some other fields can be updated through special endpoints. For example deleted and enabled are updated through the endpoints for deleting, undeleting, enabling and disabling.
Child System Operations
Most operations other than update are the same for child systems as they are for parent systems. For more information see the appropriate section of the document for the operation.
Delete - see Deletion
Undelete - see Deletion
Enable - see “enabled” in System Attributes Table
Disable - see “enabled” in System Attributes Table
Unlinking a Child System from it’s Parent System
A child system may be unlinked from it’s parent. This is a permanent operation, and cannot be undone. This will make the child a standalone system with all of it’s current settings. When the unlink happens any fields that had previously been linked to the parent will be copied to the child, and it will be as if the child was created as in independent system with those values.
If the owner of the child system wants to unlink the child from it’s parent, the owner may use the unlinkFromParent endpoint.
Using PySDK:
import json
from tapipy.tapis import Tapis
t = Tapis(base_url='https://tacc.tapis.io', username='<userid>', password='************')
t.systems.unlinkFromParent(childSystemId="<child-system-id>")
Using CURL:
$ curl -X POST -H "content-type: application/json" -H "X-Tapis-Token: $JWT" https://tacc.tapis.io/v3/systems/<child-system-id>/unlinkFromParent
Replace <child-system-id> with the id of the child system.
The owner of a parent system can also decide to unlink child systems from the parent. In that case the parent system owner would use the unlinkChildren endpoint. The child systems to unlink may be specified in the request body. First create a json file (for example children_to_unlink.json):
{
"childSystemIds":
[
"<child-system-1-id>",
"<child-system-2-id>"
...
]
}
Using PySDK:
import json
from tapipy.tapis import Tapis
t = Tapis(base_url='https://tacc.tapis.io', username='<userid>', password='************')
with open('children_to_unlink.json', 'r') as openfile:
children_to_unlink = json.load(openfile)
t.systems.unlinkChildren(parentSystemId="<parent-system-id>", **children_to_unlink)
Using CURL:
$curl -X POST -H "content-type: application/json" -H "X-Tapis-Token: $JWT" https://tacc.tapis.io/v3/systems/<parent-system-id>/unlinkChildren -d @./children_to_unlink.json
Or all child systems using all=True (no json file required)
Using PySDK:
import json
from tapipy.tapis import Tapis
t = Tapis(base_url='https://tacc.tapis.io', username='<userid>', password='************')
t.systems.unlinkChildren(parentSystemId="<parent-system-id>", all=True)
Using CURL:
$ curl -X POST -H "content-type: application/json" -H "X-Tapis-Token: $JWT" "https://tacc.tapis.io/v3/systems/<parent-system-id>/unlinkChildren?all=true"
Minimal Definition and Restrictions
When creating a system the required attributes are: id, systemType, host, defaultAuthnMethod and canExec. Depending on the type of system and specific values for certain attributes there are other requirements. The restrictions are:
If systemType is S3 then bucketName is required and canExec must be false.
If systemType is LINUX or IRODS then rootDir is required and must begin with
/
.If effectiveUserId is
${apiUserId}
(i.e. it is not static) then authnCredential may not be specified.If canExec is true then jobWorkingDir is required and jobRuntimes must have at least one entry.
If canRunBatch is true then batchScheduler must be specified.
If canRunBatch is true then batchLogicalQueues must have at least one item.
If batchLogicalQueues has more than one item then batchLogicalDefaultQueue must be specified.
If batchLogicalQueues has exactly one item then batchLogicalDefaultQueue is set to that item.
Permissions
The permissions model allows for fine grained access control of Tapis systems.
At system creation time the owner is given full access to the system. Permissions for other users may be granted and revoked through the systems API. Please note that grants and revokes through this service only impact the default role for the user. A user may still have access through permissions in another role. So even after revoking permissions through this service, when permissions are retrieved the access may still be listed. This indicates access has been granted via another role.
Permissions are specified as either *
for all permissions or some combination of the
following specific permissions: ("READ","MODIFY","EXECUTE")
. Specifying permissions in all
lower case is also allowed. Having MODIFY
implies READ
.
Authentication Credentials
At system creation time the authentication credentials may be specified if the effective
access user effectiveUserId is a specific user (such as a service account) and not
a dynamic user (i.e. not equal to ${apiUserId}
).
If the effective access user is dynamic (i.e. equal to ${apiUserId}
) then authentication credentials for any
user allowed to access the system must be registered in separate API calls. In this case the payload provided may
contain the optional attribute loginUser which will be used to map the Tapis user to a username to be used when
accessing the system. If the login user is not provided then there is no mapping and the Tapis user is always used
when accessing the system.
Note that the Systems service does not store credentials. Credentials are persisted by the Security Kernel service and only specific Tapis services are authorized to retrieve credentials.
Also, note that there is support for only one set of credentials per user per system. Updating credentials overwrites previously registered data.
By default any credentials provided for LINUX and S3 type systems are verified. The query parameter skipCredentialCheck=true may be used to bypass the initial verification of credentials.
Runtime
Runtime environment supported by the system that may be used to run applications, such as docker, singularity or ZIP. Consists of the runtime type and version.
Logical Batch Queue
A queue that maps to a single HPC queue. Logical batch queues provide a uniform front end abstraction for an HPC queue. They also provide more features and flexibility than is typically provided by an HPC scheduler. Multiple logical queues may be defined for each HPC queue. If an HPC queue does not have a corresponding logical queue defined then a user will not be able use the Tapis system to directly submit a job via Tapis to that HPC queue.
Batch Scheduler Profile
The Systems service supports managing Tapis scheduler profiles. An HPC center often has certain conventions and restrictions around the use of batch schedulers. A scheduler profile resource can be defined to provide the Tapis Jobs service with additional site specific information to be used when executing applications using a scheduler. A scheduler profile contains information on options that should be hidden from the scheduler, the module load command to use and which modules should be loaded by default when running a job. Anyone in a tenant may create a scheduler profile for use by all users in the tenant. The owner of a profile or a tenant administrator may modify or delete a profile. A profile may be referenced in a system definition using the attribute batchSchedulerProfile. The profile to be used may also be set in the job submit request using the special scheduler option --tapis-profile. The value in the job submit request takes precedence over a value defined for the execution system.
For example, at TACC there is a certain module that must be loaded when running Slurm jobs via singularity. Also, use of the Slurm option --mem is prohibited. In support of this, most of the tenants at TACC make use of a profile similar to the following:
{
"name": "tacc",
"owner": "testuser1",
"description": "Profile for TACC Slurm",
"moduleLoads": [
{
"moduleLoadCommand": "module load",
"modulesToLoad": ["tacc-singularity"]
}
],
"hiddenOptions": ["MEM"]
}
The moduleLoads array contains one or more entries. Each entry contains a moduleLoadCommand, which specifies the local command used to load each of the modules (in order) in its modulesToLoad list. The hiddenOptions attribute identifies scheduler options that the local implementation prohibits. In this case, “MEM” indicates that the --mem option should never be passed to Slurm.
Deletion
A system may be deleted and undeleted. Deletion means the system is marked as deleted and is no longer available for use. By default deleted systems will not be included in searches and operations on deleted systems will not be allowed. When listing systems the query parameter showDeleted may be used in order to include deleted systems in the results.
System Attributes Table
Attribute |
Type |
Example |
Notes |
---|---|---|---|
tenant |
String |
designsafe |
|
id |
String |
ds1.storage.default |
|
description |
String |
Default storage |
|
systemType |
enum |
LINUX |
|
owner |
String |
jdoe |
|
host |
String |
data.tacc.utexas.edu |
|
enabled |
boolean |
FALSE |
|
effectiveUserId |
String |
tg869834 |
|
defaultAuthnMethod |
enum |
PKI_KEYS |
|
authnCredential |
Credential |
|
|
bucketName |
String |
tapis-ds1-jdoe |
|
rootDir |
String |
/home/${apiUserId} |
|
port |
int |
22 |
|
useProxy |
boolean |
TRUE |
|
proxyHost |
String |
|
|
proxyPort |
int |
|
|
dtnSystemId |
String |
default.corral.dtn |
|
canExec |
boolean |
|
|
canRunBatch |
boolean |
|
|
enableCmdPrefix |
boolean |
|
|
allowChildren |
boolean |
|
|
jobRuntimes |
[Runtime] |
|
|
jobWorkingDir |
String |
HOST_EVAL($SCRATCH) |
|
jobEnvVariables |
[KeyValuePair] |
|
|
jobMaxJobs |
int |
|
|
jobMaxJobsPerUser |
int |
|
|
batchScheduler |
String |
SLURM |
|
batchSchedulerProfile |
String |
|
|
batchLogicalQueues |
[LogicalQueue] |
|
|
batchDefaultLogical Queue |
LogicalQueue |
|
|
tags |
[String] |
|
|
notes |
String |
“{}” |
|
uuid |
UUID |
|
|
deleted |
boolean |
FALSE |
|
created |
Timestamp |
2020-06-19T15:10:43Z |
|
updated |
Timestamp |
2020-07-04T23:21:22Z |
|
Credential Attributes Table
Attribute |
Type |
Example |
Notes |
---|---|---|---|
user |
String |
jsmith |
|
authnMethod |
String |
PKI_KEYS |
|
loginUser |
String |
|
|
password |
String |
|
|
privateKey |
String |
|
|
publicKey |
String |
|
|
accessKey |
String |
|
|
accessSecret |
String |
|
|
accessToken |
String |
|
|
refreshToken |
String |
|
KeyValuePair Attributes Table
Attribute |
Type |
Example |
Notes |
---|---|---|---|
key |
String |
“INPUT_FILE” |
|
value |
String |
“/tmp/file.input” |
|
description |
String |
|
|
inputMode |
enum |
REQUIRED |
|
notes |
String |
“{}” |
|
LogicalQueue Attributes Table
Attribute |
Type |
Example |
Notes |
---|---|---|---|
name |
String |
tapisNormal |
|
hpcQueueName |
String |
normal |
|
maxJobs |
int |
|
|
maxJobsPerUser |
int |
|
|
minNodeCount |
int |
|
|
maxNodeCount |
int |
|
|
minCoresPerNode |
int |
|
|
maxCoresPerNode |
int |
|
|
minMemoryMB |
int |
|
|
maxMemoryMB |
int |
|
|
minMinutes |
int |
|
|
maxMinutes |
int |
|
Scheduler Profile Attributes Table
Attribute |
Type |
Example |
Notes |
---|---|---|---|
name |
String |
tacc |
|
owner |
String |
jdoe |
|
description |
String |
|
|
moduleLoads |
Array |
|
|
hiddenOptions |
String[] |
[“MEM”] |
|
Searching
The service provides a way for users to search for systems based on a list of search conditions provided either as query parameters for a GET call or a list of conditions in a request body for a POST call to a dedicated search endpoint.
Search using GET
To search when using a GET request to the systems
endpoint a list of search conditions may be specified
using a query parameter named search
. Each search condition must be surrounded with parentheses, have three parts
separated by the character .
and be joined using the character ~
.
All conditions are combined using logical AND. The general form for specifying the query parameter is as follows:
?search=(<attribute_1>.<op_1>.<value_1>)~(<attribute_2>.<op_2>.<value_2>)~ ... ~(<attribute_N>.<op_N>.<value_N>)
Attribute names are given in the table above and may be specified using Camel Case or Snake Case.
Supported operators: eq
neq
gt
gte
lt
lte
in
nin
like
nlike
between
nbetween
Example CURL command to search for systems that have Test
in the id, are of type LINUX,
are using a port less than 1024
and have a default authentication method of either PKI_KEYS
or PASSWORD
:
$ curl -H "X-Tapis-Token: $JWT" https://tacc.tapis.io/v3/systems?search="(id.like.*Test*)~(system_type.eq.LINUX)~(port.lt.1024)~(DefaultAuthnMethod.in.PKI_KEYS,PASSWORD)"
Notes:
For the
like
andnlike
operators the wildcard character*
matches zero or more characters and!
matches exactly one character.For the
between
andnbetween
operators the value must be a two item comma separated list of unquoted values.If there is only one condition the surrounding parentheses are optional.
In a shell environment the character
&
separating query parameters must be escaped with a backslash.In a shell environment the query value must be surrounded by double quotes and the following characters must be escaped with a backslash in order to be properly interpreted by the shell:
"
\
`
Attribute names may be specified using Camel Case or Snake Case.
Following complex attributes not supported when searching:
authnCredential
jobRuntimes
jobEnvVariables
batchLogicalQueues
notes
Dedicated Search Endpoint
The service provides the dedicated search endpoint systems/search/systems
for specifying complex queries. Using a GET
request to this endpoint provides functionality similar to above but with a different syntax. For more complex
queries a POST request may be used with a request body specifying the search conditions using an SQL-like syntax.
Search using GET on Dedicated Endpoint
Sending a GET request to the search endpoint provides functionality very similar to that provided for the endpoint
systems
described above. A list of search conditions may be specified using a series of query parameters, one for each attribute.
All conditions are combined using logical AND. The general form for specifying the query parameters is as follows:
?<attribute_1>.<op_1>=<value_1>&<attribute_2>.<op_2>=<value_2>)& ... &<attribute_N>.<op_N>=<value_N>
Attribute names are given in the table above and may be specified using Camel Case or Snake Case.
Supported operators: eq
neq
gt
gte
lt
lte
in
nin
like
nlike
between
nbetween
Example CURL command to search for systems that have Test
in the name, are of type LINUX
,
are using a port less than 1024
and have a default authentication method of either PKI_KEYS
or PASSWORD
:
$ curl -H "X-Tapis-Token: $JWT" https://tacc.tapis.io/v3/systems/search/systems?name.like=*Test*\&enabled.eq=true\&system_type.eq=LINUX\&DefaultAuthnMethod.in=PKI_KEYS,PASSWORD
Notes:
For the
like
andnlike
operators the wildcard character*
matches zero or more characters and!
matches exactly one character.For the
between
andnbetween
operators the value must be a two item comma separated list of unquoted values.In a shell environment the character
&
separating query parameters must be escaped with a backslash.Attribute names may be specified using Camel Case or Snake Case.
Following complex attributes not supported when searching:
authnCredential
jobRuntimes
jobEnvVariables
batchLogicalQueues
tags
notes
Search using POST on Dedicated Endpoint
More complex search queries are supported when sending a POST request to the endpoint systems/search/systems
.
For these requests the request body must contain json with a top level property name of search
. The
search
property must contain an array of strings specifying the search criteria in
an SQL-like syntax. The array of strings are concatenated to form the full search query.
The full query must be in the form of an SQL-like WHERE
clause. Note that not all SQL features are supported.
For example, to search for systems that are owned by jdoe
and of type LINUX
or owned by
jsmith
and using a port less than 1024
create a local file named system_search.json
with following json:
{
"search":
[
"(owner = 'jdoe' AND system_type = 'LINUX') OR",
"(owner = 'jsmith' AND port < 1024)"
]
}
To execute the search use a CURL command similar to the following:
$ curl -X POST -H "content-type: application/json" -H "X-Tapis-Token: $JWT" https://tacc.tapis.io/v3/systems/search/systems -d @system_search.json
Notes:
String values must be surrounded by single quotes.
Values for BETWEEN must be surrounded by single quotes.
Search query parameters as described above may not be used in conjunction with a POST request.
SQL features not supported include:
IS NULL
andIS NOT NULL
Arithmetic operations
Unary operators
Specifying escape character for
LIKE
operator
Map of SQL operators to Tapis operators
Sql Operator |
Tapis Operator |
---|---|
= |
eq |
<> |
neq |
< |
lt |
<= |
lte |
> |
gt |
>= |
gte |
LIKE |
like |
NOT LIKE |
nlike |
BETWEEN |
between |
NOT BETWEEN |
nbetween |
IN |
in |
NOT IN |
nin |
Sort, Limit, Select and ListType
When a list of Systems is retrieved the service provides for sorting, filtering and limiting the results. By default, only resources owned by you will be included. The service provides a way for you to request that all resources accessible to you be included. This is determined by the query parameter listType.
When retrieving either a list of resources or a single resource the service also provides a way to select which fields (i.e. attributes) are included in the results. Sorting, limiting and attribute selection are supported using query parameters.
Selecting
When retrieving systems the fields (i.e. attributes) to be returned may be specified as a comma separated list using
a query parameter named select
. Attribute names may be given using Camel Case or Snake Case.
Notes:
Special select keywords are supported:
allAttributes
andsummaryAttributes
Summary attributes include:
id
,systemType
,owner
,host
,effectiveUserId
,defaultAuthnMethod
,canExec
By default all attributes are returned when retrieving a single resource via the endpoint systems/<system_id>.
By default summary attributes are returned when retrieving a list of systems.
Specifying nested attributes is not supported.
The attribute
id
is always returned.
For example, to return only the attributes host
and effectiveUserId
the
CURL command would look like this:
$ curl -H "X-Tapis-Token: $JWT" https://tacc.tapis.io/v3/systems?select=host,effectiveUserId
The response should look similar to the following:
{
"result": [
{
"id": "CSys_CltSrchGet_011",
"host": "hostCltSrchGet_011",
"effectiveUserId": "effUserCltSrchGet_011"
},
{
"id": "CSys_CltSrchGet_012",
"host": "hostCltSrchGet_012",
"effectiveUserId": "effUserCltSrchGet_012"
},
{
"id": "CSys_CltSrchGet_013",
"host": "hostCltSrchGet_013",
"effectiveUserId": "effUserCltSrchGet_013"
}
],
"status": "success",
"message": "TAPIS_FOUND Systems found: 12 systems",
"version": "1.0.0",
"metadata": {
"recordCount": 3,
"recordLimit": 100,
"recordsSkipped": 0,
"orderBy": null,
"startAfter": null,
"totalCount": -1
}
}
Sorting
The query parameter for sorting is named orderBy
and the value is the attribute name to sort on with an optional
sort direction. The general format is <attribute_name>(<dir>)
. The direction may be asc
for ascending or
desc
for descending. The default direction is ascending.
Examples:
orderBy=id
orderBy=id(asc)
orderBy=name(desc),created
orderBy=id(asc),created(desc)
Limiting
Additional query parameters may be used in order to limit the number and starting point for results. This is useful for implementing paging. The query parameters are:
limit
- Limit number of items returned. For example limit=10.
Use 0 or less for unlimited.
Default is 100.
skip
- Number of items to skip. For example skip=10.
May not be used with startAfter.
Default is 0.
startAfter
- Where to start when sorting. For example limit=10&orderBy=id(asc),created(desc)&startAfter=101
May not be used with
skip
.Must also specify
orderBy
.The value of
startAfter
applies to the majororderBy
field.Condition is context dependent. For ascending the condition is value >
startAfter
and for descending the condition is value <startAfter
.
When implementing paging it is recommend to always use orderBy
and when possible use limit+startAfter
rather
than limit+skip
. Sorting should always be included since returned results are not guaranteed to be in the same order
for each call. The combination of limit+startAfter
is preferred because limit+skip
is more likely to result in
inconsistent results as records are added and removed. Using limit+startAfter
works best when the attribute has a
natural sequential ordering such as when an attribute represents a timestamp or a sequential ID.
ListType
By default, you will only see the resources that you own. The query parameter listType allows you to see additional resources that are available to you.
Options:
- OWNED
Include only items owned by you (Default)
- SHARED_PUBLIC
Include only items shared publicly
- ALL
Include all items you are authorized to view.
Tapis Responses
For requests that return a list of resources the response result object will contain the list of resource records that match the user’s query and the response metadata object will contain information related to sorting and limiting.
The metadata object will contain the following information:
recordCount
- Actual number of records returned.
recordLimit
- The limit query parameter specified in the request. -1 if query parameter was not specified.
recordsSkipped
- The skip query parameter specified in the request. -1 if query parameter was not specified.
orderBy
- The orderBy query parameter specified in the request. Empty string if query parameter was not specified.
startAfter
- The startAfter query parameter specified in the request. Empty string if query parameter was not specified.
totalCount
- Total number of records that would have been returned without a limit query parameter being imposed. -1 if total count was not computed.
For performance reasons computation of totalCount
is only determined on demand. This is controlled by the boolean
query parameter computeTotal
. By default computeTotal
is false.
Example query and response:
Query:
$ curl -H "X-Tapis-Token: $JWT" https://tacc.tapis.io/v3/systems?limit=2&orderBy=id(desc)
Response:
{
"result": [
{
"id": "testMin0",
"systemType": "S3",
"owner": "testuser",
"host": "my.example.host",
"defaultAccessMethod": "ACCESS_KEY",
"canExec": false
},
{
"id": "MinSystem1c",
"systemType": "LINUX",
"owner": "testuser",
"defaultAccessMethod": "PASSWORD",
"host": "data.tacc.utexas.edu",
"canExec": true
}
],
"status": "success",
"message": "TAPIS_FOUND Systems found: 2 systems",
"version": "1.0.0",
"metadata": {
"recordCount": 2,
"recordLimit": 2,
"recordsSkipped": 0,
"orderBy": "id(desc)",
"startAfter": null,
"totalCount": -1
}