Class AzureFileSystemProvider
AzureFileSystemProvider
is Azure Storage's implementation of the nio interface on top of Azure Blob
Storage.
Particular care should be taken when working with a remote storage service. This implementation makes no guarantees
on behavior or state should other processes operate on the same data concurrently; file systems from this provider
will assume they have exclusive access to their data and will behave without regard for potential of interfering
applications. Moreover, remote file stores introduce higher latencies. Therefore, additional consideration should be
given to managing concurrency: race conditions are more likely to manifest and network failures occur more frequently
than disk failures. These and other such distributed application scenarios must be considered when working with this
file system. While the AzureFileSystem
will ensure it takes appropriate steps towards robustness and
reliability, the application developer must design around these failure scenarios and have fallback and retry options
available.
The Azure Blob Storage service backing these APIs is not a true FileSystem, nor is it the goal of this implementation
to force Azure Blob Storage to act like a full-fledged file system. Some APIs and scenarios will remain unsupported
indefinitely until they may be sensibly implemented. Other APIs may experience lower performance than is expected
because of the number of network requests needed to ensure correctness. The javadocs for each type and method should
also be read carefully to understand what guarantees are made and how they may differ from the contract defined by
FileSystemProvider
.
The scheme for this provider is "azb"
, and the format of the URI to identify an AzureFileSystem
is
"azb://?endpoint=<endpoint>"
. The endpoint of the Storage account is used to uniquely identify the
filesystem.
An AzureFileSystem
is backed by an account. An AzureFileStore
is backed by a container. Any number of
containers may be specified as file stores upon creation of the file system. When a file system is created,
it will try to retrieve the properties of each container to ensure connection to the account. If any of the
containers does not exist, it will be created. Failure to access or create containers as necessary will result in
an exception and failure to create the file system. Any data existing in the containers will be preserved and
accessible via the file system, though customers should be aware that it must be in a format understandable by
the types in this package or behavior will be undefined.
newFileSystem(URI, Map)
will check for the following keys in the configuration map and expect the named
types. Any entries not listed here will be ignored. Note that AzureFileSystem
has public constants defined
for each of the keys for convenience. Most values are documented in the blob package. Any values which are unique to
nio will be documented here.
AzureStorageSharedKeyCredential:
StorageSharedKeyCredential
AzureStorageSasTokenCredential:
AzureSasCredential
AzureStorageHttpLogDetailLevel:
HttpLogDetailLevel
AzureStorageMaxTries:
Integer
AzureStorageTryTimeout:
Integer
AzureStorageRetryDelayInMs:
Long
AzureStorageMaxRetryDelayInMs:
Long
AzureStorageRetryPolicyType:
RetryPolicyType
AzureStorageSecondaryHost:
String
AzureStorageSecondaryHost:
Integer
AzureStorageBlockSize:
Long
AzureStoragePutBlobThreshold:
Long
AzureStorageMaxConcurrencyPerRequest:
Integer
AzureStorageDownloadResumeRetries:
Integer
AzureStorageFileStores:
String
AzureStorageSkipInitialContainerCheck:
Boolean
. Indicates that the initial check which confirms the existence of the containers meant to act as file stores should be skipped. This can be useful in cases where a sas token that is scoped to only one file is used to authenticate.
Either an account key or a sas token must be specified. If both are provided, the account key will be preferred. If a sas token is specified, the customer must take care that it has appropriate permissions to perform the actions demanded of the file system in a given workflow, including the initial connection check specified above. The same token will be applied to all operations.
An iterable of file stores must also be provided; each entry should simply be the name of a container. The first
container listed will be considered the default file store and the root directory of which will be the file system's
default directory. All other values listed are used to configure the underlying
BlobServiceClient
. Please refer to that type for more information on these values.
- See Also:
-
Field Summary
Modifier and TypeFieldDescriptionstatic final String
A helper for setting the HTTP properties when creating a directory.static final String
A helper for setting the HTTP properties when creating a directory.static final String
A helper for setting the HTTP properties when creating a directory.static final String
A helper for setting the HTTP properties when creating a directory.static final String
A helper for setting the HTTP properties when creating a directory.static final String
A helper for setting the HTTP properties when creating a directory. -
Constructor Summary
-
Method Summary
Modifier and TypeMethodDescriptionvoid
checkAccess
(Path path, AccessMode... accessModes) Checks the existence, and optionally the accessibility, of a file.void
copy
(Path source, Path destination, CopyOption... copyOptions) Copies the resource at the source location to the destination.void
createDirectory
(Path path, FileAttribute<?>... fileAttributes) Creates a new directory at the specified path.void
Deletes the specified resource.<V extends FileAttributeView>
VgetFileAttributeView
(Path path, Class<V> type, LinkOption... linkOptions) Returns a file attribute view of a given type.getFileStore
(Path path) Unsupported.getFileSystem
(URI uri) Returns an existing FileSystem created by this provider.Return a Path object by converting the given URI.Returns the URI scheme that identifies this provider:"azb".
boolean
Always returns false as hidden files are not supported.boolean
isSameFile
(Path path, Path path1) Unsupported.void
move
(Path path, Path path1, CopyOption... copyOptions) Unsupported.newByteChannel
(Path path, Set<? extends OpenOption> set, FileAttribute<?>... fileAttributes) Opens or creates a file, returning a seekable byte channel to access the file.newDirectoryStream
(Path path, DirectoryStream.Filter<? super Path> filter) Returns anAzureDirectoryStream
for iterating over the contents of a directory.newFileSystem
(URI uri, Map<String, ?> config) Constructs a new FileSystem object identified by a URI.newInputStream
(Path path, OpenOption... options) Opens anInputStream
to the given path.newOutputStream
(Path path, OpenOption... options) Opens anOutputStream
to the given path.<A extends BasicFileAttributes>
AreadAttributes
(Path path, Class<A> type, LinkOption... linkOptions) Reads a file's attributes as a bulk operation.readAttributes
(Path path, String attributes, LinkOption... linkOptions) Reads a set of file attributes as a bulk operation.void
setAttribute
(Path path, String attributes, Object value, LinkOption... linkOptions) Sets the value of a file attribute.Methods inherited from class java.nio.file.spi.FileSystemProvider
createLink, createSymbolicLink, deleteIfExists, installedProviders, newAsynchronousFileChannel, newFileChannel, newFileSystem, readSymbolicLink
-
Field Details
-
CONTENT_TYPE
A helper for setting the HTTP properties when creating a directory.- See Also:
-
CONTENT_DISPOSITION
A helper for setting the HTTP properties when creating a directory.- See Also:
-
CONTENT_LANGUAGE
A helper for setting the HTTP properties when creating a directory.- See Also:
-
CONTENT_ENCODING
A helper for setting the HTTP properties when creating a directory.- See Also:
-
CONTENT_MD5
A helper for setting the HTTP properties when creating a directory.- See Also:
-
CACHE_CONTROL
A helper for setting the HTTP properties when creating a directory.- See Also:
-
-
Constructor Details
-
AzureFileSystemProvider
public AzureFileSystemProvider()Creates an AzureFileSystemProvider.
-
-
Method Details
-
getScheme
Returns the URI scheme that identifies this provider:"azb".
- Specified by:
getScheme
in classFileSystemProvider
- Returns:
"azb"
-
newFileSystem
Constructs a new FileSystem object identified by a URI.The format of a
URI
identifying a file system is"azb://?endpoint=<endpoint>"
.Once closed, a file system with the same identifier may be reopened.
- Specified by:
newFileSystem
in classFileSystemProvider
- Parameters:
uri
- URI referenceconfig
- A map of provider specific properties to configure the file system- Returns:
- a new file system.
- Throws:
IllegalArgumentException
- If the pre-conditions for the uri parameter aren't met, or the env parameter does not contain properties required by the provider, or a property value is invalid.IOException
- If an I/O error occurs.SecurityException
- neverFileSystemAlreadyExistsException
- If the file system has already been created.
-
getFileSystem
Returns an existing FileSystem created by this provider.The format of a
URI
identifying a file system is"azb://?endpoint=<endpoint>"
.Trying to retrieve a closed file system will throw a
FileSystemNotFoundException
. Once closed, a file system with the same identifier may be reopened.- Specified by:
getFileSystem
in classFileSystemProvider
- Parameters:
uri
- URI reference- Returns:
- the file system
- Throws:
IllegalArgumentException
- If the pre-conditions for the uri parameter aren't metFileSystemNotFoundException
- If the file system already existsSecurityException
- never
-
getPath
Return a Path object by converting the given URI. The resulting Path is associated with a FileSystem that already exists.- Specified by:
getPath
in classFileSystemProvider
- Parameters:
uri
- The URI to convert- Returns:
- The path identified by the URI.
- Throws:
IllegalArgumentException
- If the URI scheme does not identify this provider or other preconditions on the uri parameter do not holdFileSystemNotFoundException
- if the file system identified by the query does not existSecurityException
- never- See Also:
-
newByteChannel
public SeekableByteChannel newByteChannel(Path path, Set<? extends OpenOption> set, FileAttribute<?>... fileAttributes) throws IOException Opens or creates a file, returning a seekable byte channel to access the file.This method is primarily offered to support some jdk convenience methods such as
Files.createFile(Path, FileAttribute[])
which requires opening a channel and closing it. A channel may only be opened in read mode OR write mode. It may not be opened in read/write mode. Seeking is supported for reads, but not for writes. Modifications to existing files is not permitted--only creating new files or overwriting existing files.This type is not threadsafe to prevent having to hold locks across network calls.
- Specified by:
newByteChannel
in classFileSystemProvider
- Parameters:
path
- the path of the file to openset
- options specifying how the file should be openedfileAttributes
- an optional list of file attributes to set atomically when creating the directory- Returns:
- a new seekable byte channel
- Throws:
UnsupportedOperationException
- Operation is not supported.IllegalArgumentException
- if the set contains an invalid combination of optionsFileAlreadyExistsException
- if a file of that name already exists and the CREATE_NEW option is specified (optional specific exception)IOException
- If an I/O error occurs.SecurityException
- never
-
newInputStream
Opens anInputStream
to the given path.The stream will not attempt to read or buffer the entire file. However, when fetching data, it will always request the same size chunk of several MB to prevent network thrashing on small reads. Mark and reset are supported.
Only
StandardOpenOption.READ
is supported. Any other option will throw.- Overrides:
newInputStream
in classFileSystemProvider
- Parameters:
path
- the path to the file to openoptions
- options specifying how the file is opened- Returns:
- a new input stream
- Throws:
IllegalArgumentException
- if an invalid combination of options is specifiedUnsupportedOperationException
- if an unsupported option is specifiedIOException
- If an I/O error occurs.SecurityException
- never
-
newOutputStream
Opens anOutputStream
to the given path. The resulting file will be stored as a block blob.The only supported options are
StandardOpenOption.CREATE
,StandardOpenOption.CREATE_NEW
,StandardOpenOption.WRITE
,StandardOpenOption.TRUNCATE_EXISTING
. Any other options will throw anUnsupportedOperationException
.WRITE
andTRUNCATE_EXISTING
must be specified or anIllegalArgumentException
will be thrown. Hence, files cannot be updated, only overwritten completely.This stream will not attempt to buffer the entire file, however some buffering will be done for potential optimizations and to avoid network thrashing. Specifically, up to
AzureFileSystem.AZURE_STORAGE_PUT_BLOB_THRESHOLD
bytes will be buffered initially. If that threshold is exceeded, the data will be broken into chunks and sent in blocks, and writes will be buffered into sizes ofAzureFileSystem.AZURE_STORAGE_UPLOAD_BLOCK_SIZE
. The maximum number of buffers of this size to be allocated is defined byAzureFileSystem.AZURE_STORAGE_MAX_CONCURRENCY_PER_REQUEST
, which also configures the level of parallelism with which we may write and thus may affect write speeds as well.The data is only committed when the steam is closed. Hence, data cannot be read from the destination until the stream is closed. When the close method returns, it is guaranteed that, barring any errors, the data is finalized and available for reading.
Writing happens asynchronously. Bytes passed for writing are stored until either the threshold or block size are met at which time they are sent to the service. When the write method returns, there is no guarantee about which phase of this process the data is in other than it has been accepted and will be written. Again, closing will guarantee that the data is written and available.
Flush is a no-op as regards data transfers, but it can be used to check the state of the stream for errors. This can be a useful tool because writing happens asynchronously, and therefore an error from a previous write may not otherwise be thrown unless the stream is flushed, closed, or written to again.
- Overrides:
newOutputStream
in classFileSystemProvider
- Parameters:
path
- the path to the file to open or createoptions
- options specifying how the file is opened- Returns:
- a new output stream
- Throws:
IllegalArgumentException
- if an invalid combination of options is specifiedUnsupportedOperationException
- if an unsupported option is specifiedIOException
- If an I/O error occurs.SecurityException
- never
-
newDirectoryStream
public DirectoryStream<Path> newDirectoryStream(Path path, DirectoryStream.Filter<? super Path> filter) throws IOException Returns anAzureDirectoryStream
for iterating over the contents of a directory. The elements returned by the directory stream's iterator are of type Path, each one representing an entry in the directory. The Path objects are obtained as if by resolving the name of the directory entry against dir. The entries returned by the iterator are filtered by the given filter.When not using the try-with-resources construct, then directory stream's close method should be invoked after iteration is completed to free any resources held for the open directory.
Where the filter terminates due to an uncaught error or runtime exception then it is propagated to the hasNext or next method. Where an IOException is thrown, it results in the hasNext or next method throwing a DirectoryIteratorException with the IOException as the cause.
- Specified by:
newDirectoryStream
in classFileSystemProvider
- Parameters:
path
- the path to the directoryfilter
- the directory stream filter- Returns:
- a new and open
DirectoryStream
object - Throws:
IllegalArgumentException
- If the path type is not an instance ofAzurePath
.NotDirectoryException
- if the file could not otherwise be opened because it is not a directoryIOException
- If an I/O error occurs.SecurityException
- never
-
createDirectory
Creates a new directory at the specified path.The existence of a directory in the
AzureFileSystem
is defined on two levels. Weak existence is defined by the presence of a non-zero number of blobs prefixed with the directory's path. This concept is also known as a virtual directory and enables the file system to work with containers that were pre-loaded with data by another source but need to be accessed by this file system. Strong existence is defined as the presence of an actual storage resource at the given path, which in the case of directories, is a zero-length blob whose name is the directory path with a particular metadata field indicating the blob's status as a directory. This is also known as a concrete directory. Directories created by this file system will strongly exist. Operations targeting directories themselves as the object (e.g. setting properties) will target marker blobs underlying concrete directories. Other operations (e.g. listing) will operate on the blob-name prefix.This method fulfills the nio contract of: "The check for the existence of the file and the creation of the directory if it does not exist are a single operation that is atomic with respect to all other filesystem activities that might affect the directory." More specifically, this method will atomically check for strong existence of another file or directory at the given path and fail if one is present. On the other hand, we only check for weak existence of the parent to determine if the given path is valid. Additionally, the action of checking whether the parent exists, is not atomic with the creation of the directory. Note that while it is possible that the parent may be deleted between when the parent is determined to exist and the creation of the child, the creation of the child will always ensure the existence of a virtual parent, so the child will never be left floating and unreachable. The different checks on parent and child is due to limitations in the Storage service API.
There may be some unintuitive behavior when working with directories in this file system, particularly virtual directories (usually those not created by this file system). A virtual directory will disappear as soon as all its children have been deleted. Furthermore, if a directory with the given path weakly exists at the time of calling this method, this method will still return success and create a concrete directory at the target location. In other words, it is possible to "double create" a directory if it first weakly exists and then is strongly created. This is both because it is impossible to atomically check if a virtual directory exists while creating a concrete directory and because such behavior will have minimal side effects--no files will be overwritten and the directory will still be available for writing as intended, though it may not be empty. This is not a complete list of such unintuitive behavior.
This method will attempt to extract standard HTTP content headers from the list of file attributes to set them as blob headers. All other attributes will be set as blob metadata. The value of every attribute will be converted to a
String
except the Content-MD5 attribute which expects abyte[]
. When extracting the content headers, the following strings will be used for comparison (constants for these values can be found on this type):Content-Type
Content-Disposition
Content-Language
Content-Encoding
Content-MD5
Cache-Control
- Specified by:
createDirectory
in classFileSystemProvider
- Parameters:
path
- the directory to createfileAttributes
- an optional list of file attributes to set atomically when creating the directory- Throws:
IllegalArgumentException
- If the path type is not an instance ofAzurePath
.UnsupportedOperationException
- if the array contains an attribute that cannot be set atomically when creating the directoryFileAlreadyExistsException
- if a directory could not otherwise be created because a file of that name already existsIOException
- If an I/O error occurs.SecurityException
- never
-
delete
Deletes the specified resource.This method is not atomic with respect to other file system operations. It is possible to delete a file in use by another process, and doing so will not immediately invalidate any channels open to that file--they will simply start to fail. Root directories cannot be deleted even when empty.
- Specified by:
delete
in classFileSystemProvider
- Parameters:
path
- the path to the file to delete- Throws:
IllegalArgumentException
- If the path type is not an instance ofAzurePath
.NoSuchFileException
- if the file does not existDirectoryNotEmptyException
- if the file is a directory and could not otherwise be deleted because the directory is not emptyIOException
- If an I/O error occurs.SecurityException
- never
-
copy
Copies the resource at the source location to the destination.This method is not atomic with respect to other file system operations. More specifically, the checks necessary to validate the inputs and state of the file system are not atomic with the actual copying of data. If the copy is triggered, the copy itself is atomic and only a complete copy will ever be left at the destination.
In addition to those in the docs for
FileSystemProvider.copy(Path, Path, CopyOption...)
, this method has the following requirements for successful completion.StandardCopyOption.COPY_ATTRIBUTES
must be passed as it is impossible not to copy blob properties; if this option is not passed, anUnsupportedOperationException
will be thrown. Neither the source nor the destination can be a root directory; if either is a root directory, anIllegalArgumentException
will be thrown. The parent directory of the destination must at least weakly exist; if it does not, anIOException
will be thrown. The only supported option other thanStandardCopyOption.COPY_ATTRIBUTES
isStandardCopyOption.REPLACE_EXISTING
; the presence of any other option will result in anUnsupportedOperationException
.This method supports both virtual and concrete directories as both the source and destination. Unlike when creating a directory, the existence of a virtual directory at the destination will cause this operation to fail. This is in order to prevent the possibility of overwriting a non-empty virtual directory with a file. Still, as mentioned above, this check is not atomic with the creation of the resultant directory.
- Specified by:
copy
in classFileSystemProvider
- Parameters:
source
- the path to the file to copydestination
- the path to the target filecopyOptions
- specifying how the copy should be done- Throws:
UnsupportedOperationException
- if the array contains a copy option that is not supportedFileAlreadyExistsException
- if the target file exists but cannot be replaced because the REPLACE_EXISTING option is not specifiedDirectoryNotEmptyException
- the REPLACE_EXISTING option is specified but the file cannot be replaced because it is a non-empty directoryIOException
- If an I/O error occurs.IllegalArgumentException
- If the path type is not an instance ofAzurePath
.SecurityException
- never- See Also:
-
move
Unsupported.- Specified by:
move
in classFileSystemProvider
- Parameters:
path
- pathpath1
- pathcopyOptions
- options- Throws:
UnsupportedOperationException
- Operation is not supported.IOException
-
isSameFile
Unsupported.- Specified by:
isSameFile
in classFileSystemProvider
- Parameters:
path
- pathpath1
- path- Throws:
UnsupportedOperationException
- Operation is not supported.IOException
-
isHidden
Always returns false as hidden files are not supported.- Specified by:
isHidden
in classFileSystemProvider
- Parameters:
path
- the path- Returns:
- false
- Throws:
IOException
- If an I/O error occurs.SecurityException
- never
-
getFileStore
Unsupported.- Specified by:
getFileStore
in classFileSystemProvider
- Parameters:
path
- path- Returns:
- the file store where the file is stored.
- Throws:
UnsupportedOperationException
- Operation is not supported.IOException
- If an I/O error occurs.SecurityException
- never
-
checkAccess
Checks the existence, and optionally the accessibility, of a file.This method may only be used to check the existence of a file. It is not possible to determine the permissions granted to a given client, so if any mode argument is specified, an
UnsupportedOperationException
will be thrown.- Specified by:
checkAccess
in classFileSystemProvider
- Parameters:
path
- the path to the file to checkaccessModes
- The access modes to check; may have zero elements- Throws:
NoSuchFileException
- if a file does not existAccessDeniedException
- the requested access would be denied or the access cannot be determined because the Java virtual machine has insufficient privileges or other reasonsIOException
- If an I/O error occurs.SecurityException
- never
-
getFileAttributeView
public <V extends FileAttributeView> V getFileAttributeView(Path path, Class<V> type, LinkOption... linkOptions) Returns a file attribute view of a given type.See
AzureBasicFileAttributeView
andAzureBlobFileAttributeView
for more information.Reading attributes on a virtual directory will return
null
for most properties other thanAzureBlobFileAttributes.isVirtualDirectory()
, which will return true. SeecreateDirectory(Path, FileAttribute[])
for more information on virtual directories.- Specified by:
getFileAttributeView
in classFileSystemProvider
- Parameters:
path
- the path to the filetype
- the Class object corresponding to the file attribute viewlinkOptions
- ignored- Returns:
- a file attribute view of the specified type, or null if the attribute view type is not available
-
readAttributes
public <A extends BasicFileAttributes> A readAttributes(Path path, Class<A> type, LinkOption... linkOptions) throws IOException Reads a file's attributes as a bulk operation.See
AzureBasicFileAttributes
andAzureBlobFileAttributes
for more information.Reading attributes on a virtual directory will return
null
for most properties other thanAzureBlobFileAttributes.isVirtualDirectory()
, which will return true. SeecreateDirectory(Path, FileAttribute[])
for more information on virtual directories.- Specified by:
readAttributes
in classFileSystemProvider
- Parameters:
path
- the path to the filetype
- the Class of the file attributes required to readlinkOptions
- ignored- Returns:
- the file attributes
- Throws:
UnsupportedOperationException
- if an attributes of the given type are not supportedIOException
- If an I/O error occurs.SecurityException
- never
-
readAttributes
public Map<String,Object> readAttributes(Path path, String attributes, LinkOption... linkOptions) throws IOException Reads a set of file attributes as a bulk operation.See
AzureBasicFileAttributes
andAzureBlobFileAttributes
for more information.Reading attributes on a virtual directory will return
null
for all properties other thanAzureBlobFileAttributes.isVirtualDirectory()
, which will return true. SeecreateDirectory(Path, FileAttribute[])
for more information on virtual directories.- Specified by:
readAttributes
in classFileSystemProvider
- Parameters:
path
- the path to the fileattributes
- the attributes to readlinkOptions
- ignored- Returns:
- a map of the attributes returned; may be empty. The map's keys are the attribute names, its values are the attribute values
- Throws:
UnsupportedOperationException
- if an attributes of the given type are not supportedIllegalArgumentException
- if no attributes are specified or an unrecognized attributes is specifiedIOException
- If an I/O error occurs.SecurityException
- never
-
setAttribute
public void setAttribute(Path path, String attributes, Object value, LinkOption... linkOptions) throws IOException Sets the value of a file attribute.See
AzureBlobFileAttributeView
for more information.Setting attributes on a virtual directory is not supported and will throw an
IOException
. SeecreateDirectory(Path, FileAttribute[])
for more information on virtual directories.- Specified by:
setAttribute
in classFileSystemProvider
- Parameters:
path
- the path to the fileattributes
- the attribute to setvalue
- the attribute valuelinkOptions
- ignored- Throws:
UnsupportedOperationException
- if an attribute view is not availableIllegalArgumentException
- if the attribute name is not specified, or is not recognized, or the attribute value is of the correct type but has an inappropriate valueClassCastException
- If the attribute value is not of the expected type or is a collection containing elements that are not of the expected typeIOException
- If an I/O error occurs.SecurityException
- never
-