File Storage Connection Options

If you are querying a FileSource and additional connection information needs to be provided to access the file storage system, include the options property in the query and define the necessary storage-specific connection parameters. The parameters that the GDI supports for each type of storage system are pulled directly from the Java API for that system. The supported properties for each storage type are listed below.

Amazon S3
FTP & FTPS
Google Cloud Storage
HDFS
SFTP
WebDAV

Amazon S3

Loading parquet files from S3 is not supported. Amazon does not provide a VFS driver that supports random access reads, which are necessary for reading parquet files.

s:options [ 
   s:accessKey "string" ;
   s:region "string" ;
   s:secretKey "string" ;
   s:serviceName "string" ;
   s:sessionToken "string" ;
   s:createBucket boolean ;
   s:disableChunkedEncoding boolean ;
   s:serverSideEncryption boolean ;
   s:useHttps boolean ;
] ;

Option	Type	Description
accessKey	string	The `accessKey` property can be included to specify the access key.
region	string	The `region` property can be included to specify the region.
secretKey	string	The `secretKey` property can be included to specify the secret key.
serviceName	string	For connections to AWS service endpoints, the `serviceName` property can be included to specify the service name.
sessionToken	string	The `sessionToken` property can be included to specify the session token.
createBucket	boolean	Refer to the S3 API documentation.
disableChunkedEncoding	boolean	For increased performance, Amazon S3 requests use chunked encoding by default. To disable chunked encoding, you can include `s:disableChunkedEncoding true` in the query.
serverSideEncryption	boolean	Refer to the S3 API documentation.
useHttps	boolean	Refer to the S3 API documentation.

FTP & FTPS

s:options [ 
   s:autodetectUtf8 boolean ;
   s:connectTimeout int ;
   s:controlEncoding "string" ;
   s:dataTimeout int ;
   s:defaultDateFormat "string" ;
   s:entryParser "string" ;
   s:fileType "string" ;
   s:passiveMode boolean ;
   s:proxy "string" ;
   s:recentDateFormat "string" ;
   s:remoteVerification boolean ;
   s:serverLanguageCode "string" ;
   s:serverTimeZoneId "string" ;
   s:shortMonthNames "string" ;
   s:socketTimeout int ;
   s:userDirIsRoot boolean ;
   s:dataChannelProtectionLevel "string" ;
   s:ftpsMode "string" ;
   s:keyManager "string" ;
   s:trustManager "string" ;
] ;

Option	Type	Description
autodetectUtf8	boolean	For FTP connections, the `autodetectUtf8` property can be included to indicate whether the FTP server is set to UTF-8 mode or Auto-detect encoding.
connectTimeout	int	For FTP connections, you can include the `connectTimeout` property to specify the maximum number of seconds to hold a connection before timing out.
controlEncoding	string	Refer to the FTP API documentation.
dataTimeout	int	For FTP connections, you can include the `dataTimeout` property to specify the maximum number of seconds to transfer data before timing out.
defaultDateFormat	string	Refer to the FTP API documentation.
entryParser	string	Refer to the FTP API documentation.
fileType	string	Refer to the FTP API documentation.
passiveMode	boolean	For FTP connections, the `passiveMode` property can be included to indicate whether the data transfer mode is passive or active. If you use passive mode, set passiveMode to true (`s:passiveMode true`).
proxy	string	If you are using an FTP proxy, include the `proxy` property to specify the proxy connection details.
recentDateFormat	string	Refer to the FTP API documentation.
remoteVerification	boolean	For FTP connections, the `remoteVerification` property can be included to indicate whether remote authentication is enabled. If you use remote authentication, set remoteVerification to true (`s:remoteVerification true`).
serverLanguageCode	string	If the FTP server language is not set to English, include the `serverLanguageCode` property to specify the language code for the server. For example, `s:serverLanguageCode "ES"`.
serverTimeZoneId	string	For FTP connections, the `serverTimeZoneId` property can be included to specify the timezone ID for the server.
shortMonthNames	string	Refer to the FTP API documentation.
socketTimeout	int	For FTP connections, you can include the `socketTimeout` property to specify the maximum number of seconds to transfer data before timing out.
userDirIsRoot	boolean	Refer to the FTP API documentation.
dataChannelProtectionLevel	string	For FTPS connections, the `dataChannelProtectionLevel` property specifies the Data Channel Protection Level for the server.
ftpsMode	string	For FTPS connections, the `ftpsMode` property specifies whether the FTPS is in implicit or explicit mode.
keyManager	string	For FTPS connections, the `keyManager` property specifies the KeyManager value for making an SSL connection to the server.
trustManager	string	For FTPS connections, the `trustManager` property specifies the TrustManager value for the SSL connection to the server.

Google Cloud Storage

s:options [ 
   s:serviceAccountKey "string" ;
] ;

Option	Type	Description
serviceAccountKey	string	For connections to GCS, the `serviceAccountKey` property can be included to specify the key for the service account.

HDFS

s:options [ 
   s:configName "string" ;
   s:configPath "string" ;
   s:configURL "string" ;
] ;

Option	Type	Description
configName	string	For connections to HDFS, the `configName` property can be included to specify the name of the configuration file to read.
configPath	string	For connections to HDFS, the `configPath` property can be included to list the path to the specified configuration file.
configURL	string	Refer to the HDFS API documentation.

SFTP

s:options [ 
   s:compression "string" ;
   s:configRepository "string" ;
   s:fileNameEncoding "string" ;
   s:identityProvider "string" ;
   s:identityRepositoryFactory "string" ;
   s:keyExchangeAlgorithm "string" ;
   s:knownHosts "string" ;
   s:loadOpenSSHConfig boolean ;
   s:preferredAuthentications "string" ;
   s:sessionTimeout int ;
   s:strictHostKeyChecking "string" ;
   s:userInfo "string" ;
] ;

Option	Type	Description
compression	string	Refer to the SFTP API documentation.
configRepository	string	Refer to the SFTP API documentation.
fileNameEncoding	string	Refer to the SFTP API documentation.
identityProvider	string	Refer to the SFTP API documentation.
identityRepositoryFactory	string	Refer to the SFTP API documentation.
keyExchangeAlgorithm	string	For SFTP connections, you can include the `keyExchangeAlgorithm` property to specify the key exchange algorithm to use.
knownHosts	string	Refer to the SFTP API documentation.
loadOpenSSHConfig	boolean	For SFTP connections, the `loadOpenSSHConfig` property indicates whether to read the `~/.ssh/config` file.
preferredAuthentications	string	For SFTP connections, the `preferredAuthentications` property can be included to specify the authentication order to use.
sessionTimeout	int	For SFTP connections, you can include the `sessionTimeout` property to specify the maximum number of seconds to leave the session open before timing out.
strictHostKeyChecking	string	For SFTP connections, you can include the `strictHostKeyChecking` property to specify how host keys are checked.
userInfo	string	Refer to the SFTP API documentation.

WebDAV

s:options [ 
   s:creatorName "string" ;
   s:versioning boolean ;
] ;

Option	Type	Description
creatorName	string	For WebDAV connections, the `creatorName` property can be included to add a description of the creator of the resource.
versioning	boolean	Refer to the WebDAV API documentation.