Techniques are provided for on-demand serverless disaster recovery. A primary node may host a primary volume. Snapshots of the primary volume may be backed up to an object store. In response to failure, a secondary node and/or an on-demand volume may be created on-demand. The secondary node may provide clients with failover access to the on-demand volume while a restore process restores a snapshot of the primary volume to the on-demand volume. In some embodiments, there was no secondary node and/or on-demand volume while the primary node was operational. This conserves computing resources that would be wasted by otherwise hosting the secondary node and/or on-demand volume while clients were able to access the primary volume through the primary node. Modifications directed to the on-demand volume are incrementally backed up to the object store for subsequently restoring the primary volume after recovery.
G06F 11/20 - Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
2.
METHODS AND STORAGE NODES TO DECREASE DELAY IN RESUMING INPUT OUTPUT (I/O) OPERATIONS AFTER A NON-DISRUPTIVE EVENT FOR A STORAGE OBECT OF A DISTRIBUTED STORAGE SYSTEM BY UTILIZING ASYNCHRONOUS INFLIGHT REPLAY OF THE I/O OPERATIONS
In one embodiment, a method comprises maintaining state information regarding a data replication status for a storage object of the storage node of a primary storage cluster with the storage object being replicated to a replicated storage object of a secondary storage cluster, temporarily disallowing input/output (I/O) operations when the storage object has a connection loss or failure. The method further includes initiating a resynchronization between the storage object and the replicated storage object including initiating asynchronous persistent inflight tracking and replay of any missing I/O operations that are missing from one of a first Op log of the primary storage cluster and a second Op log of the secondary storage cluster, and allowing new I/O operations to be handled with the storage object of the primary storage cluster without waiting for completion of the asynchronous persistent inflight tracking and replay at the secondary storage cluster.
Techniques are provided for a cell based test automation framework. Conventional plain-text based test automation frameworks require extensive knowledge of scripting/programming languages to create and debug test scripts with numerous lines of code written in plain-text. Thus, conventional plain-text based test automation frameworks are disadvantageous in terms of time, effort, and training costs. The disclosed cell based test automation framework overcomes these disadvantages by utilizing spreadsheets defining a test script to execute. Each sheet within a spreadsheet defines a test case that the cell based test automation framework is to execute. The testing spreadsheet include cells storing test data and control values. The control values are used to control the flow of executing the test script to perform a test using the test data. The spreadsheet is very easy for a developer to create, and the cell based test automation framework outputs logs that are easy to read, understand, and debug.
Approaches to data flow bottleneck management using caching mechanisms in a distributed storage environment are disclosed. A read request is received by a first data storage node having a first set of interface module(s), a first set of data management module(s), a first redirection layer, and a first set of data storage devices. The read request has a corresponding file to be read. The first redirection layer is checked for an entry corresponding to the file. The read request is routed based on a file characteristic corresponding to the read request if there is no corresponding entry in the first redirection layer or to a second data storage node based on the entry in the first redirection layer. Potential bottleneck conditions are monitored on the first node. A redirection layer entry in the first redirection layer is generated in response to determining that a bottleneck condition exists.
The technology disclosed herein enables a higher-level process to perform storage volume management with knowledge of a physical storage backend underlying a storage volume. In a particular example, a method includes mounting a storage volume to a computing node of the computing nodes. The storage volume is stored in a storage pool of a plurality of underlying storage pools. The method further includes determining an identifier for the storage pool, receiving a request to duplicate the storage volume, and determining a second identifier for a second storage pool of the plurality of underlying storage pools to which the storage volume will be duplicated. When the second identifier matches the identifier, creating a clone of the storage volume rather than copying the storage volume to the second storage pool.
G06F 3/06 - Digital input from, or digital output to, record carriers
6.
Facilitating analytics involving cold data tiered to an object store by an on-premises storage solution by creating a virtual storage appliance on the object store
Systems and methods for creating a virtual storage appliance on an object store to which an on-premises storage solution has tiered cold data are provided. According to one embodiment, a physical storage system deployed in an on-premises environment replicates hot data of a file system of the physical storage system stored on a set of disks associated with the physical storage system to an object storage service of a hyperscaler to which cold data of the file system has previously been tiered out by the file system. An instance of a virtual storage system may then be created within the hyperscaler that utilizes the object storage service as a primary storage tier. Finally, analytics involving a dataset, including both the data and the cold data, may be performed by accessing the dataset via the virtual storage system.
The disclosed technology relates to managing input-output operation in a zoned storage system includes identifying a first physical zone and a second physical zone within a zoned namespace solid-state drive associated with a logical zone to perform a received write operation. Data to be written in the received write operation is temporarily staged in a zone random write area associated with the identified second physical zone. Based a storage threshold of the zone random write area, a determination is made regarding when to transfer temporarily staged data to be written area to the identified second physical zone. When the storage threshold of the zone random write area determined to have exceeded, temporarily staged data to be written is transferred to the identified second physical zone.
The disclosed technology relates to determining physical zone data within a zoned namespace solid state drive (SSD), associated with logical zone data included in a first received input-output operation based on a mapping data structure within a namespace of the zoned namespace SSD. A second input-output operation specific to the determined physical zone data is generated wherein the second input-output operation and the received input-output operation is of a same type. The generated second input-output operation is completed using the determined physical zone data within the zoned namespace SSD.
Disclosed are methods and systems for processing storage system monitoring data. The time component may be removed from the storage system monitoring data and may be stored as a mathematical formula, such as a line. The data from the storage system monitoring data may be separately analyzed to determine a compression algorithm to be applied to the data, such that the compressed data may be stored as a mathematical formula. Storing the storage system monitoring data as a mathematical formula lowers the amount of storage needed to store the storage system monitoring data, lowers the amount of network bandwidth needed to transfer the storage system monitoring data, and lowers the processing resources needed to process the storage system monitoring data. The removed time component may be used as an index to access the compressed data.
The technology disclosed herein enables worker nodes to reestablish connections when the connection protocols fail to successfully reconnect on their own. In a particular example, a method includes establishing a plurality of TCP-based connections (e.g., iSCSI and/or NVMe) between a plurality of worker nodes and a storage system and determining status of the plurality of TCP-based connections. When the status indicates a first connection of the plurality of TCP-based connections has failed, the method includes determining credentials used to establish the first connection are invalid. In response to determining the credentials are invalid, the method includes requesting new credentials from a controller node and reestablishing the first connection using the new credentials.
A monitoring system can generate compressed storage system monitoring data segments using monitoring data obtained from a storage system. The monitoring system can obtain compression information for a compressed segment generated from a segment of storage system monitoring data and generate a predicted portion of storage system monitoring data using the compression information. The monitoring system can obtain an additional portion of storage system monitoring data, the additional portion contiguous to the segment of storage system monitoring data, and determine that the predicted portion matches the additional portion and combining the additional portion and the compressed segment. In response to a user query, the monitoring system can perform at least one of: reconstructing and providing the additional portion using the compressed segment; or providing the compressed segment for reconstruction of the additional portion.
A monitoring system can generate compressed storage system monitoring data segments using monitoring data obtained from a storage system. The monitoring system can obtain a segment of storage system monitoring data and generate a compressed segment from the segment and a reconstructed segment from the compressed segment. The monitoring system can identify locations in the segment based on a comparison of the segment and the reconstructed segment and determine values for the identified locations. In response to a user query received from a user system, the user query indicating a portion of the segment, the monitoring system can perform at least one of: reconstructing and providing the portion using the identified locations, the determined values for the identified locations, and the compressed segment; or providing the identified locations, the determined values for the identified locations, and the compressed segment for reconstruction of the portion.
In one embodiment, a computer implemented method includes storing objects in a first bucket and storing files in a second bucket of a first storage cluster of the distributed storage system, synchronously replicating data of the objects into a third mirrored bucket of a second storage cluster of the distributed storage system, synchronously replicating OSP configuration data from the first storage cluster to the second storage cluster during the synchronous replication, and providing non-disruptive operations with zero recovery time objective (RTO), and ensuring consistency between the objects in the first bucket and the objects in the third bucket for a software application that is accessing one or more objects and files using the OSP. The objects and files are accessible through an object storage protocol (OSP).
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database systemDistributed database system architectures therefor
G06F 16/28 - Databases characterised by their database models, e.g. relational or object models
Techniques are provided for migrating a volume utilizing backend object copy operations. Data of the volume is stored within objects stored across a storage tier and capacity tier of a source object store. As part of migrating the volume to a destination object store, the objects are migrated to the destination cluster. Directly copying the objects involves multiple read operations to the source object store and a write operation at the destination object store. The techniques provided herein improve the efficiency of the migration by initially sending metadata from the source object store to the destination object store for performing backend object copy operations to migrate the volume. This results in fewer operations and less network usage, thus improving the efficiency and cost of migrating the volume.
Techniques are provided for migrating a volume utilizing an object copy work queue and an object copy driver module. Data of the volume is stored within objects stored across a storage tier and capacity tier of a source object store. As part of migrating the volume to a destination object store, the objects are migrated to the destination cluster. Directly copying the objects involves multiple read operations to the source object store and a write operation at the destination object store. The techniques provided herein improve the efficiency of the migration by initially sending metadata from the source object store to the destination object store for performing backend object copy operations to migrate the volume. This results in fewer operations and less network usage, thus improving the efficiency and cost of migrating the volume.
The technology disclosed herein enables enforcement of Information Lifecycle Management (ILM) policies across tenants in an object storage system using tags associated with the ILM policies. In a particular example, a method includes identifying ILM policies for an object storage system having multiple tenants and associating respective tags with the ILM policies. The method further includes enabling a subset of the tags to be assigned to data objects of a tenant of the multiple tenants and enforcing a portion of the ILM policies on the data objects, wherein the portion of the ILM policies corresponds to tags of the subset assigned to the data objects.
Disclosed herein are methods, systems, and apparatus for the detection of data integrity anomalies indicative of malware for a datastore of an organization. To identify an anomaly in a file, a portion of a file is identified to be used in a vector comparison. The portion can comprise sentences or paragraphs for text files, entries, rows, or columns for spreadsheet files, or some other divisible portion of a file. A vector having multiple dimensions is generated for the portion based on the content in the portion. Each dimension of the multiple dimensions corresponds to a feature of the portion. A variation is determined between the vector and one other vector associated with one other portion of the file. One or more actions to take with respect to the file is determined based on the variation, such as malware mitigation, and the action is performed with respect to the file
G06F 21/56 - Computer malware detection or handling, e.g. anti-virus arrangements
G06F 16/583 - Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
18.
SYSTEMS AND METHODS FOR MANAGING STORAGE SYSTEM MONITORING DATA
A monitoring system can generate compressed storage system monitoring data segments using monitoring data obtained from a storage system. The monitoring system can segment the obtained storage system monitoring data into storage system monitoring data segments. The monitoring system can then compress the storage system monitoring data segments into the compressed storage system monitoring data segments using at least two compression techniques. In response to a user query, the monitoring system can identify at least one compressed storage system monitoring data segment. The monitoring system can then perform at least one of reconstructing and providing a portion of the storage system monitoring data using the at least one compressed storage system monitoring data segment or providing the at least one compressed storage system monitoring data segment for reconstruction of the portion of the storage system monitoring data.
Disclosed herein are methods, systems, and apparatus for the detection of data integrity anomalies indicative of malware for a datastore of an organization. To identify an anomaly in a file, a portion of a file is identified to be used in a vector comparison. The portion can comprise sentences or paragraphs for text files, entries, rows, or columns for spreadsheet files, or some other divisible portion of a file. A vector having multiple dimensions is generated for the portion based on the content in the portion. Each dimension of the multiple dimensions corresponds to a feature of the portion. A variation is determined between the vector and one other vector associated with one other portion of the file. One or more actions to take with respect to the file is determined based on the variation, such as malware mitigation, and the action is performed with respect to the file.
A monitoring system can generate compressed storage system monitoring data segments using monitoring data obtained from a storage system. The monitoring system can obtain storage system monitoring data and generate a segment of the storage system monitoring data. The monitoring system can determine time information, a compression technique, and a compression parameter set for the segment. In response to the user query, the monitoring system can identify the compression technique and the compression parameter set based on a time interval of the query and the time information and perform at least one of: reconstructing and providing at least a portion of the segment using the time information, the compression technique, and the compression parameter set; or providing the compression parameter set and the indication of the compression technique for reconstruction of the portion.
A monitoring system can generate compressed storage system monitoring data segments using monitoring data obtained from a storage system. The monitoring system can obtain storage system monitoring data and generate a segment by applying the storage system monitoring data to a machine learning model trained to segment the storage system monitoring data. The monitoring system can generate a compressed segment by applying a specified compression technique to the segment. In response to user query, the user query specifying a portion of the storage system monitoring data; the monitoring system can perform at least one of: reconstructing and providing the portion using the compressed segment; or providing the compressed segment for reconstruction of the portion.
Techniques are provided for modifying a read path to process read operations during a storage operation, such as an operation to migrate objects storing data of a volume. The objects are stored across a storage tier and capacity tier of a source object store. As part of migrating the volume to a destination object store, the objects are migrated to the destination cluster. Directly copying the objects involves multiple read operations to the source object store and a write operation at the destination object store. The techniques provided herein improve the efficiency of the migration by initially sending metadata from the source object store to the destination object store for performing backend block copy operations to migrate the volume.
In one embodiment, a computer implemented method includes comprises storing objects in a first bucket and files in a second bucket of a first storage cluster of the distributed storage system, initiating an audit job on the first storage cluster, synchronously replicating audit configuration data and mirroring audit data (e.g., audit files, logs) from the first storage cluster to the second storage cluster, performing a switchover process from the first storage cluster to the second storage cluster, and initiating an audit job on the second storage cluster based on the audit configuration during the switchover process. The first storage cluster initially handles input/output operations for a software application before the switchover process.
G06F 7/00 - Methods or arrangements for processing data by operating upon the order or content of the data handled
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database systemDistributed database system architectures therefor
G06F 16/28 - Databases characterised by their database models, e.g. relational or object models
24.
Temporary Local Protection Against Data Loss When A Node Fails In A Distributed Storage System
The technology disclosed herein enhances data protection in a distributed storage system. In a particular example, a method includes determining a drive in a subject node of the distributed storage system has failed while storing first data of a data set distributed across nodes of the distributed storage system by a data protection mechanism. The method further includes broadcasting failure information indicating the data set from the subject node to other nodes of the distributed storage system. At the other nodes, in response to receiving the failure information, the method includes identifying a subset of the other nodes that also store a portion of the data set. In each identified node of the subset, the method includes identifying second data of the data set stored on a local drive and copying the second data to a different local drive to protect the data set from further drive failure.
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
25.
AGGREGATE INLINE DEDUPLICATION WITH VOLUME GRANULAR ENCRYPTION
Techniques are provided for aggregate inline deduplication and volume granularity encryption. For example, data that is exclusive to a volume of a tenant is encrypted using an exclusive encryption key accessible to the tenant. The exclusive encryption key of that tenant is inaccessible to other tenants. Shared data that has been deduplicated and shared between the volume and another volume of a different tenant is encrypted using a shared encryption key of the volume. The shared encryption key is made available to other tenants. In this way, data can be deduplicated across multiple volumes of different tenants of a storage environment, while maintaining security and data privacy at a volume level.
Systems, methods, and machine-readable media for monitoring a storage system and assigning performance service levels to workloads running on nodes within a cluster are disclosed. A performance manager may estimate the performance demands of each workload within the cluster and assign a performance service level to each workload according to the performance requirements of the workload, and further taking into account an overall budgeting framework. The estimates are performed using historical performance data for each workload. A performance service level may include a service level object, a service level agreement, and latency parameters. These parameters may provide a ceiling to the number of operations per second that a workload may use without guaranteeing the use of the operations per second, a guaranteed number of operations per second that a workload may use before being throttled, and define the permitted delay in completing a request to the workload.
Disclosed are systems, computer-readable mediums, and methods for managing client performance in a storage system. According to one embodiment, a total Input/Output Operations per Second (IOPS) pool and a read/write IOPS pool are managed for clients to ensure their write requests can be accommodated by both pools. In one example, a write request is received from a client by the storage system. A requested number of write IOPS is determined for a time period to accommodate the request. Based on the requested number of write IOPS exceeding a number of allocated write IOPS to the client for the time period, a target total IOPS for the client during the time period is determined by subtracting the number of allocated write IOPS from a number of allocated total IOPS to the client. At least a portion of the request is performed by executing the target total IOPS during the time period.
Systems and methods for multiple device consumption of shared namespaces of ephemeral storage devices by a consumer of a virtual storage system are provided. In an example, multiple namespaces of respective ephemeral storage devices are shared among multiple consumers of a virtual storage system by creating multiple partitions within each of the namespaces for use by respective consumers of the multiple consumers. Corresponding partitions of respective shared namespace may then be treated as a stripe set to facilitate multiple device consumption for a subsystem (e.g., operation log journaling) of the virtual storage system by striping data associated with input/output (I/O) requests of a consumer (e.g., a journaling driver) across one or more stripe units of one or more stripes within the stripe set.
A system is described. The system includes a processing resource and a non-transitory computer-readable medium, coupled to the processing resource, having stored therein instructions that when executed by the processing resource cause the processing resource to receive a plurality of quality of service (QoS) parameters and client preferences from a client device and manage a QoS policy based on a plurality of QoS objectives included in the received QoS parameters, wherein the plurality of QoS objectives comprise input output operations per second (IOPS), throughput and latency.
Systems and methods are described for a cross-site high availability distributed storage system. According to one embodiment, a computer implemented method includes providing a remote direct memory access (RDMA) request for a RDMA stream, and generating, with an interconnect (IC) layer of the first storage node, multiple IC channels and associated IC requests for the RDMA request. The method further includes mapping an IC channel to a group of multiple transport layer sessions to split data traffic of the IC channel into multiple packets for the group of multiple transport layer sessions using an IC transport layer of the first storage node and assigning, with the IC transport layer, a unique transaction identification (ID) to each IC request and assigning a different data offset to each packet of a transport layer session.
One or more techniques and/or computing devices are provided for snapshot creation. For example, a request may be received to create a snapshot of a first storage object, hosted by a first storage controller, having a synchronous replication relationship with a second storage object hosted by a second storage controller. A coordinator workflow is initialized to issue a drain and hold request to a splitter. Responsive to an indication that the splitter has stopped processing and started queuing incoming write operations and has drained current write operations based upon the drain and hold request, snapshot creation requests are sent to the first storage controller and the second storage controller. Responsive to the first storage controller creating a first snapshot of the first storage object and the second storage controller creating a second snapshot of the second storage object, the splitter may be resumed to process write operations.
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
G06F 3/06 - Digital input from, or digital output to, record carriers
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database systemDistributed database system architectures therefor
32.
IN-MEMORY HASH ENTRIES AND HASHES USED TO IMPROVE KEY SEARCH OPERATIONS FOR KEYS OF A KEY VALUE STORE
Techniques are provided for implementing a hash building process and an append hash building process. The hash building process builds in-memory hash entries for bins of keys stored within sorted logs of a log structured merge tree used to store keys of a key-value store. The in-memory hash entries can be used to identify the starting locations of bins of keys within the log structured merge tree so that a key within a bin can be searched for from the starting location of the bin as opposed to having to search the entire log structured merge tree. The append hash building process builds two hashes that can be used to more efficiently locate keys and/or ranges of keys within an unsorted append log that would otherwise require a time consuming binary search of the entire append log.
Techniques are provided for storage tier verification checks. A determination is made that a mount operation of an aggregate of a set of volumes stored within a multi-tier storage environment has completed. A first metafile and a second metafile are maintained to track information related to the storage of objects of a volume of the aggregate within a remote object store that is a tier of the multi-tier storage environment. A distributed verification is performed between the first metafile and the second metafile to identify an inconsistency. Accordingly, the first metafile and the second metafile are reconciled to address the inconsistency so that storage information within the first metafile and the second metafile are consistent.
Techniques are provided for caching data during an on-demand restore using a cloud block map. A client may be provided with access to an on-demand volume during a restore process that copies backup data from a snapshot within a remote object store to the on-demand volume stored within local storage. In response to receiving a request from the client for a block of the backup data not yet restored from the snapshot to the on-demand volume, the block may be retrieved from the snapshot in the remote object store. The block may be cached within a cloud block map stored within the local storage as a cached block. The client may be provided with access to the cached block.
G06F 3/06 - Digital input from, or digital output to, record carriers
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
09 - Scientific and electric apparatus and instruments
42 - Scientific, technological and industrial services, research and design
Goods & Services
Computer operating systems software and computer software for use in connection with data management, data storage, data protection, security, and compliance; computer software for managing, accessing, modifying, updating, organizing, delivering, backing up, synchronizing, processing, transmitting, storing, and restoring data Providing temporary use of online non-downloadable software for use in connection with data management, data storage, data protection, security, and compliance; providing temporary use of online non-downloadable software for managing, accessing, modifying, updating, organizing, delivering, synchronizing, processing, transmitting, storing, and restoring data
36.
FASTER REPLAY OF METADATA AND DATA OPERATIONS USING INODE NUMBER BASED DEPENDENCY GRAPH
Techniques are provided for replay of metadata and data operations. During initial execution of operations, identifiers of objects modified by the execution of each operation are identified and stored in association with the operations. When the operations are to be replayed (e.g., executed again, such as part of a replication operation or as part of flushing content from a cache to persistent storage), the identifiers are evaluated to determine which operations are independent with respect to one another and which operations are dependent with respect to one another. In this way, independent operations are executed in parallel and dependent operations are executed serially with respect to the operations from the dependent operations depend.
G06F 11/20 - Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
G06F 16/14 - Details of searching files based on file metadata
G06F 16/16 - File or folder operations, e.g. details of user interfaces specifically adapted to file systems
G06F 16/178 - Techniques for file synchronisation in file systems
Techniques are provided for maintaining and recomputing reference counts in a persistent memory file system of a node. Primary reference counts are maintained for pages within persistent memory of the node. In response to receiving a first operation to link a page into a persistent memory file system of the persistent memory, a primary reference count of the page is incremented before linking the page into the persistent memory file system. In response to receiving a second operation to unlink the page from the persistent memory file system, the page is unlinked from the persistent memory file system before the primary reference count is decremented. Upon the node recovering from a crash, the persistent memory file system is traversed in order to update shadow reference counts for the pages with correct reference count values, which are used to overwrite the primary reference counts with the correct reference count values.
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
G06F 11/07 - Responding to the occurrence of a fault, e.g. fault tolerance
Techniques are provided for compressing images using compression image containers. Many user devices, such as smart phones, incorporate cameras that be utilized by users to capture photos. The photos can be either locally stored on a user device or uploaded to cloud storage. Managing the storage of a vast amount of photos is cumbersome and expensive where trillions of photos could be stored within the cloud storage. The techniques provided herein are capable of compressing images using compression image containers in order to significantly reduce storage consumed when storing images with similar or duplicate image information. Similar images are compressed together within the same compression image container for improved compression and reduced storage.
H04N 19/127 - Prioritisation of hardware or computational resources
H04N 19/136 - Incoming video signal characteristics or properties
H04N 19/154 - Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion
H04N 19/157 - Assigned coding mode, i.e. the coding mode being predefined or preselected to be further used for selection of another element or parameter
H04N 19/184 - Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being bits, e.g. of the compressed video stream
H04N 19/186 - Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a colour or a chrominance component
H04N 19/187 - Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a scalable video layer
H04N 19/30 - Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability
H04N 19/423 - Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation characterised by memory arrangements
H04N 19/46 - Embedding additional information in the video signal during the compression process
H04N 19/80 - Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation
39.
RESYNC TRANSFER FOR RECOVERING FROM STORAGE SITE FAILURE UTILIZING BACKGROUND SCANNER
Techniques are provided for performing a resync transfer to recover from a storage site failure. During normal operation of a first site hosting a first volume, data is replicated to a second volume hosted by a second site. If the first site fails, when clients are redirected to the second volume at the second site. When the first site recovers, data modifications made to the second volume are resynced back to the first volume. As part of synchronizing the first volume, a data warehouse is rebuilt at the first site in order to track the location of blocks present on the replication destination. Typically, the data modifications are transferred after the data warehouse is rebuilt, which results in significantly long resync times. The techniques provided herein decrease the resync time by either rebuilding the data warehouse in parallel with resyncing the data modifications or circumvent the need for rebuild.
Techniques are provided for performing a resync transfer to recover from a storage site failure. During normal operation of a first site hosting a first volume, data is replicated to a second volume hosted by a second site. If the first site fails, when clients are redirected to the second volume at the second site. When the first site recovers, data modifications made to the second volume are resynced back to the first volume. As part of synchronizing the first volume, a data warehouse is rebuilt at the first site in order to track the location of blocks present on the replication destination. Typically, the data modifications are transferred after the data warehouse is rebuilt, which results in significantly long resync times. The techniques provided herein decrease the resync time by either rebuilding the data warehouse in parallel with resyncing the data modifications or circumvent the need for rebuild.
Techniques are provided for performing a resync transfer to recover from a storage site failure. During normal operation of a first site hosting a first volume, data is replicated to a second volume hosted by a second site. If the first site fails, when clients are redirected to the second volume at the second site. When the first site recovers, data modifications made to the second volume are resynced back to the first volume. As part of synchronizing the first volume, a data warehouse is rebuilt at the first site in order to track the location of blocks present on the replication destination. Typically, the data modifications are transferred after the data warehouse is rebuilt, which results in significantly long resync times. The techniques provided herein decrease the resync time by either rebuilding the data warehouse in parallel with resyncing the data modifications or circumvent the need for rebuild.
Techniques are provided for utilizing a log to free pages from persistent memory. A log is maintained to comprise a list of page block numbers of pages within persistent memory of a node to free. A page block number, of a page, within the log is identified for processing. A reference count, corresponding to a number of references to the page block number, is identified. In response to the reference count being greater than 1, the reference count is decremented and the page block number is removed from the log. In response to the reference count being 1, the page is freed from the persistent memory and the page block number is removed from the log.
Techniques are provided for selectively allowing or blocking services from utilizing snapshots. A splitter intercepts and assigns timestamps to operations for execution upon files of a file system. If a snapshot is created while there are no pending operation, then the snapshot is tagged to indicate that the snapshot was created while there were no pending operations, and the snapshot is considered a tagged snapshot. Otherwise, the snapshot is not tagged, and is considered an untagged snapshot. Storage services (e.g., backup applications, lifecycle management applications, applications that consider timestamps of files, etc.) are allowed to utilize tagged snapshots and are blocked from utilizes untagged snapshots in order to maintain consistency.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database systemDistributed database system architectures therefor
G06F 3/06 - Digital input from, or digital output to, record carriers
Techniques are provided for performing a resync transfer to recover from a storage site failure. During normal operation of a first site hosting a first volume, data is replicated to a second volume hosted by a second site. If the first site fails, when clients are redirected to the second volume at the second site. When the first site recovers, data modifications made to the second volume are resynced back to the first volume. As part of synchronizing the first volume, a data warehouse is rebuilt at the first site in order to track the location of blocks present on the replication destination. Typically, the data modifications are transferred after the data warehouse is rebuilt, which results in significantly long resync times. The techniques provided herein decrease the resync time by either rebuilding the data warehouse in parallel with resyncing the data modifications or circumvent the need for rebuild.
G06F 11/20 - Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
45.
REBALANCING ENGINE FOR USE IN REBALANCING FILES IN A DISTRIBUTED STORAGE SYSTEMS
Redistribution of files in a containerized distributed file system is disclosed. Containers each have an engine and a scanner and each of the containers stores files and parameters for characteristics of files stored on the container. A first engine in a first container monitors characteristics of files stored on the first container and, responsive to determining that the parameters for files on the first container exceed one or more predetermined thresholds, communicates with a second engine in a second container to determine a destination container for one or more files from the first container. The second engine in the second container indicates to the first engine in the first container whether the second container is available to receive one or more files from the first container. The first engine triggers file system scanning by the scanner of the first container to identify files to be moved to the second container.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database systemDistributed database system architectures therefor
H04L 67/1029 - Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers using data related to the state of servers by a load balancer
46.
PROACTIVELY TAKING ACTION RESPONSIVE TO EVENTS WITHIN A CLUSTER BASED ON A RANGE OF NORMAL BEHAVIOR LEARNED FOR VARIOUS USER ROLES
Systems and methods are provided for learning normal behavior for user roles of an application running within a cluster of container orchestration platform and based thereon proactively taking action responsive to suspicious events. According to one embodiment, an event data stream is created by an API server of the cluster. The data for each event includes information regarding a request made to an API exposed by the API server with which the event is associated and a user of the application by which the event was initiated. The data is augmented with a role associated with the user and an anomaly threshold for the role. Normal behavior is learned by an ML algorithm of respective user roles by processing the augmented data. When an anomaly score associated with a particular event is output by the ML algorithm that exceeds the anomaly threshold, a predefined or configurable action may be triggered.
H04L 41/16 - Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks using machine learning or artificial intelligence
47.
COST-EFFECTIVE, FAILURE-AWARE RESOURCE ALLOCATION AND RESERVATION IN THE CLOUD
Systems and methods for an improved HA resource reservation approach are provided. According to one embodiment, for a given HA cluster of greater than two nodes in which a number (f) of concurrent node failures are to be tolerated, more efficient utilization of resources may be achieved by distributing HA reserved capacity across more than f nodes of the cluster rather than naïvely concentrating the HA reserved capacity in f nodes. As node failures are not a common occurrence, those of the nodes of the HA cluster having HA reserved capacity may allow for some bursting of one or more units of compute executing thereon unless or until f concurrent node failures occur, thereby promoting more efficient utilization of node resources.
G06F 11/20 - Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
G06F 9/50 - Allocation of resources, e.g. of the central processing unit [CPU]
42 - Scientific, technological and industrial services, research and design
Goods & Services
consumer strategy business consulting in the field of technology featuring the use of analytic models for the understanding and predicting of consumer, business, and retail market trends and actions; Ccnsulting services in the field of technology featuring models and visual communications tools analyzing market trends Consulting and advisory services in the field of information and telecommunications technology; research services in the field of information and telecommunications technology
Techniques are provided for implementing a persistent memory storage tier to manage persistent memory of a node. The persistent memory is managed by the persistent memory storage tier at a higher level within a storage operating system storage stack than a level at which a storage file system of the node is managed. The persistent memory storage tier intercepts an operation targeting the storage file system. The persistent memory storage tier retargets the operation from targeting the storage file system to targeting the persistent memory. The operation is transmitted to the persistent memory.
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
G06F 3/06 - Digital input from, or digital output to, record carriers
G06F 11/20 - Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
50.
ENHANCING HIGH-AVAILABILITY IN MEDIATOR-LESS DEPLOYMENTS IN A DISTRIBUTED STORAGE SYSTEM
Systems and methods include negotiating a primary bias state for primary and secondary storage sites for a mediator-less deployment. In one example, a computer-implemented method comprises detecting whether an active synchronous replication relationship is created for bi-directional synchronous replication between a first consistency group (CG1) of one or more storage members of the primary storage site and a second consistency group (CG2) of one or more storage members of the secondary storage site when a mediator is not configured for a mediator-less deployment or whether an existing mediator is being unconfigured with the mediator located remotely from the primary storage site and remotely from the secondary storage site, and negotiating a primary bias state and setting the primary bias state on a secondary storage cluster of the secondary storage site upon the active synchronous replication relationship being created with no configured mediator or when the existing mediator is unconfigured.
Systems and methods for performing immediate acknowledgement (ACK) of resent network data packets are provided. In one example, an immediate ACK of a resent packet may be sent by a receiver node before processing of the resent packet, thereby allowing the sender's Tx queue to be cleared more quickly and allow clients to continue sending requests to the sender without interruption. Additionally, in an ideal situation, the immediate ACK of a given resent data packet may avoid the sender retransmitting one or more of the data packets subsequent to the missing packet. In one example, in order to mitigate the more frequent ACKs transmitted by the receiver, ACKs may be piggybacked on other packets (e.g., heartbeat packets and/or data packets) that would otherwise be sent from the receiver to the sender.
Techniques are provided for implementing a garbage collection process and a prediction read ahead mechanism to prefetch keys into memory to improve the efficiency and speed of the garbage collection process. A log structured merge tree is used to store keys of key-value pairs within a key-value store. If a key is no longer referenced by any worker nodes of a distributed storage architecture, then the key can be freed to store other data. Accordingly, garbage collection is performed to identify and free unused keys. The speed and efficiency of garbage collection is improved by dynamically adjusting the amount and rate at which keys are prefetched from disk and cached into faster memory for processing by the garbage collection process.
Techniques are provided for tiering snapshots to archival storage in remote object stores. A restore time metric, indicating that objects comprising snapshot data of snapshots created within a threshold timespan are to be available within a storage tier of a remote object store for performing restore operations, may be identified. A scanner may be executed to evaluate snapshots using the restore time metric to identify a set of candidate snapshots for archival from the storage tier to an archival storage tier of the remote object store. For each candidate snapshot within the set of candidate snapshots, the scanner may evaluate metadata associated with the candidate snapshot to identity one or more objects eligible for archival from the storage tier to the archival storage tier, and may archive the one or more objects from the storage tier to the archival storage tier.
G06F 16/11 - File system administration, e.g. details of archiving or snapshots
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
54.
MULTI-CLOUD PLATFORM FOR MIGRATION AND DEPLOYMENT OF CLOUD SERVICES
Techniques are provided for deploying and migrating services across cloud computing platforms. An organization incurs significant costs when hosting services of the organization on-premises using computing resources maintained by the organization. Instead, the services can be migrated to a cloud computing platform that provides scalable and lower cost computing resources for hosting the services. However, the cloud computing platform may not provide adequate performance or may be more costly than other cloud computing platforms. Unfortunately, migrating the services from the current cloud computing platform to another cloud computing platform is difficult, cumbersome, and generally not feasible (e.g., a service may need to be re-coded from scratch for a particular cloud computing platform). Accordingly, the techniques provided herein are capable of deploying services across multiple cloud computing platforms, and seamlessly migrating services and operational metadata of the services amongst different cloud computing platforms.
CROSS-SITE HIGH-AVAILABILITY DISTRIBUTED CLOUD STORAGE SYSTEM TO MONITOR VIRTUAL CHANNELS BETWEEN STORAGE NODES AND PREVENT CHANNEL SHARING OF A RECEIVE QUEUE
According to one embodiment, a computer implemented method comprises providing multiple channels between a first storage node and a second storage node with each channel having a separate network connection for packets of a transport layer session, assigning packets from each channel to a group of receive queues of the second storage node, continuously monitoring whether two or more channels of the multiple channels share a same receive queue of the second storage node, and sending a communication via a channel to the first storage node to indicate a dynamic change in a hash input field (e.g., a source port, a destination port, a source internet protocol (IP) address, and a destination IP address) when two or more channels of the multiple channels share a same receive queue of the second storage node.
H04L 67/1097 - Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
H04L 67/1023 - Server selection for load balancing based on a hash applied to IP addresses or costs
One or more techniques and/or computing devices are provided for implementing synchronous replication. For example, a synchronous replication relationship may be established between a first storage controller hosting local storage and a second storage controller hosting remote storage (e.g., replication may be specified at a file, logical unit number (LUN), or any other level of granularity). Data operations and offloaded operations may be implemented in parallel upon the local storage and the remote storage. Error handling operations may be implemented upon the local storage and implement in parallel as a best effort on the remote storage, and a reconciliation may be performed to identify any data divergence from the best effort parallel implementation. Storage area network (SAN) operations may be implemented upon the local storage, and upon local completion may be remotely implemented upon the remote storage.
H04L 67/1095 - Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
G06F 3/06 - Digital input from, or digital output to, record carriers
G06F 11/20 - Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
G06F 16/178 - Techniques for file synchronisation in file systems
H04L 67/1097 - Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
57.
METHODS AND SYSTEMS TO IMPROVE INPUT/OUTPUT (I/O) RESUMPTION TIME DURING A NON-DISRUPTIVE AUTOMATIC UNPLANNED FAILOVER FROM A PRIMARY COPY OF DATA AT A PRIMARY STORAGE SYSTEM TO A MIRROR COPY OF THE DATA AT A CROSS-SITE SECONDARY STORAGE SYSTEM
Multi-site distributed storage systems and computer-implemented methods are described for improving a resumption time of input/output (I/O) operations during an automatic unplanned failover (AUFO). A computer-implemented method includes monitoring, with a second cluster, heartbeat information received at ultra-short time intervals from a first connection of one or more storage objects of the first cluster, determining, with the second cluster, whether the heartbeat information from the first connection is received during an ultra-short time interval, and intelligently routing heartbeat information from the one or more storage objects of the first cluster from the first connection to a second connection when the heartbeat information from the first connection is not received during the ultra-short time interval.
G06F 11/20 - Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
G06F 11/07 - Responding to the occurrence of a fault, e.g. fault tolerance
58.
Methods and Systems for Archiving File System Data Stored by a Networked Storage System
Methods and systems for a networked storage environment are provided. One method includes generating a first and a second batch of entries in response to scanning a source file system, each batch of entries associated with one or more directories of the source file system and indicating a path to a file associated with each entry; determining, by a first worker process, a first checksum for data associated with the first batch of entries loaded in a first buffer; appending, the first buffer contents processed by the first worker process to a first archive file; and generating an archive data structure having a manifest file storing metadata for the first batch and a second batch entries with a first checksum determined by the first worker process and a second checksum determined by a second worker process, and data from the first archive file and a second archive file.
A computer-implemented method comprises transitioning from a unidirectional asynchronous replication to initiating bi-directional synchronous replication between one or more storage objects of a first consistency group (CG1) of a primary storage site and one or more storage objects of a second consistency group (CG2) of a secondary storage site, converting the one or more storage objects of the CG2 from data protection read only access to read write access, and performing a reverse synchronization process between the one or more storage objects of the CG2 and the one or more storage objects of the CG1 including instantiating a reverse splitter on each volume of CG2, establishing reverse sync replication sessions for each storage object of the CG2, and allowing input output (IO) access to the one or more storage objects of the CG2.
In one embodiment, a computer-implemented method comprises establishing bi-directional synchronous replication between one or more storage objects of a primary storage site and one or more storage objects of a secondary storage site with each storage site having read/write access while maintaining zero recovery point objective (RPO) and Zero recovery time objective (RTO), initiating a resynchronization process due to a loss of the bi-directional synchronous replication between the one or more storage objects and the one or more storage objects, and performing the resynchronization process based on using inflight tracking replay and reconciliation between a first Op log metafile of the primary storage site and a second Op log metafile of the secondary storage site.
Techniques are provided for caching data during an on-demand restore using a cloud block map. A client may be provided with access to an on-demand volume during a restore process that copies backup data from a snapshot within a remote object store to the on-demand volume stored within local storage. In response to receiving a request from the client for a block of the backup data not yet restored from the snapshot to the on-demand volume, the block may be retrieved from the snapshot in the remote object store. The block may be cached within a cloud block map stored within the local storage as a cached block. The client may be provided with access to the cached block.
G06F 3/06 - Digital input from, or digital output to, record carriers
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
Techniques are provided for implementing write ordering for persistent memory. A set of actions are identified for commitment to persistent memory of a node for executing an operation upon the persistent memory. An episode is created to comprise a first subset of actions of the set of actions that can be committed to the persistent memory in any order with respect to one another such that a consistent state of the persistent memory can be reconstructed in the event of a crash of the node during execution of the operation. The first subset of actions within the episode are committed to the persistent memory and further execution of the operation is blocked until the episode completes.
G06F 3/06 - Digital input from, or digital output to, record carriers
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
63.
METHODS AND SYSTEMS TO REDUCE LATENCY OF INPUT/OUTPUT (I/O) OPERATIONS DURING CREATION OF COMMON SNAPSHOTS BASED ON BATCHING OF SYNCHRONOUS REPLICATED DATASETS AT A PRIMARY STORAGE SYSTEM TO A CROSS-SITE SECONDARY STORAGE SYSTEM
Multi-site distributed storage systems and computer-implemented methods are described for improving a resumption time of input/output (I/O) operations during a common snapshot procedure for storage objects. A computer-implemented method includes initiating a snapshot multi create operation to selectively form a batch of first and second synchronous replicated datasets that belong to a first group of storage disks at the primary storage site and corresponding second group of storage disks at the secondary storage site, performing a batch snapshot create operation on the primary storage site by executing snapshots of storage objects on the primary storage site of the batch of first and second synchronous replicated datasets in parallel multiple threads to effectively utilize processing resources on the primary storage site, and initiating an independent workflow and state machine for each storage object of the batch of first and second synchronous replicated datasets.
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
G06F 3/06 - Digital input from, or digital output to, record carriers
G06F 16/11 - File system administration, e.g. details of archiving or snapshots
G06F 16/178 - Techniques for file synchronisation in file systems
A method, computing device, and non-transitory machine-readable medium for generating and maintaining snapshots. An image (snapshot) of a volume is generated for a point in time that aligns with a base schedule time in a base schedule. The image includes metadata pointing to data blocks stored on the volume at the base schedule time and is designated for use as a base snapshot of the volume at the base schedule time. In response to determining that the base schedule time aligns with a supplementary schedule time in a supplementary schedule, the image can be further designated for use as a supplementary snapshot of the volume at the supplementary schedule time. Restoring data corresponding to the supplementary schedule time includes computing all the deltas between the base snapshots created between the supplementary schedule time and an immediately preceding supplementary schedule time in the supplementary schedule.
Systems and methods are described for performing an instant recovery of data associated with a locked snapshot. In various examples, the amount of time for performing a recovery of data associated with a locked snapshot is significantly reduced by making use of enhanced volume cloning functionality instead of making an actual copy of the data to be recovered. In one embodiment, the resulting volume clone representing the recovery volume is cleared of all data protection information (e.g., WORM flags and/or lock metafiles) that was previously used to protect the content from being changed when stored on the data protection volume so as allow the recovery volume to be used in read-write mode.
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
G06F 16/176 - Support for shared access to filesFile sharing support
Described herein are systems, methods, and software to implement multi-level ransomware detection via file processing. In one example, a computing device conducts a first level of ransomware detection on a file, wherein the first level of ransomware detection comprises identifying features of the file that include a measure of randomness in the file. The computing device further inputs the features to a machine learning model that outputs a determination of whether the file has been attacked. The computing device further determines whether to conduct a second level of ransomware detection based on the determination.
Approaches to data flow bottleneck management using caching mechanisms in a distributed storage environment are disclosed. A read request is received by a first data storage node having a first set of interface module(s), a first set of data management module(s), a first redirection layer, and a first set of data storage devices. The read request has a corresponding file to be read. The first redirection layer is checked for an entry corresponding to the file. The read request is routed based on a file characteristic corresponding to the read request if there is no corresponding entry in the first redirection layer or to a second data storage node based on the entry in the first redirection layer. Potential bottleneck conditions are monitored on the first node. A redirection layer entry in the first redirection layer is generated in response to determining that a bottleneck condition exists.
Described herein are systems, methods, and software to implement ransomware detection by varying chunk size in files. In one example, a computing device extracts a first set of chunks from a file, the first set of chunks each representing a first sized portion of the file. The computing device further first features in association with the first set of chunks, the first features comprising a measure of randomness associated with the first set of chunks. The computing device also inputs the first features to a machine learning model that outputs a determination of whether the file has been attacked and determines whether to reduce the first sized portion based on the determination.
Described herein are systems, methods, and software to implement cloud ransomware detection. In one example, a computing device receives features of a file from a second computing device remote from the cloud environment, the features comprising at least a measure of randomness for the file and an identifier for a user associated with a modification to the file. The computing device further user information associated with a user of the modified the file and applies a machine learning model to determine whether the file was attacked based on the features and the user information. The computing device also communicates a notification to the second computing device indicating whether the file was attacked.
Described herein are systems, methods, and software to provide ransomware detection using variable levels of encryption. In one implementation, a computing device identifies a set of files, wherein the set of files each comprise a label indicative of whether the file is representative of a safe file or a file attacked by ransomware, and wherein the set of files comprises unencrypted files, partially encrypted files, and fully encrypted file. The computing device further identifies features associated with the set of files and generates a machine learning model that outputs a determination of whether a new file has been attacked based at least on the features in relation to whether a file in the set of files was labeled as attacked.
A computer implemented method includes maintaining information indicative of whether a data replication relationship between a dataset associated with the local CG (CG1) and a mirror copy of the dataset stored on a remote CG (CG2) of a remote distributed storage system is in an in-synchronization (InSync) state or an out-of-synchronization (OOS) state, determining whether a CG storage expansion process is in progress, and in response to determining the OOS state and whether the primary storage site has a failure, initiating an automatic unplanned failover (AUFO) workflow without manual intervention on the original volumes in the CG1 of a first storage cluster of the primary storage site and the original volumes in the CG2 of a second storage cluster when the CG storage expansion process is in progress with a new source volume to be a member of CG1 and a new destination volume to be a member of CG2.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database systemDistributed database system architectures therefor
72.
NON-DISRUPTIVE STORAGE EXPANSION OF A PAIRED SET OF CONSISTENCY GROUPS (CG) WITHIN A CROSS-SITE STORAGE SYSTEM
A computer implemented method includes maintaining information indicative of whether a data replication relationship between a dataset associated with the local CG and a mirror copy of the dataset stored on a remote CG of a remote distributed storage system is in an in-synchronization (InSync) state or an out-of-synchronization (OOS) state, initiating a storage expansion process including a source expansion to create a new source volume as a member of the local CG and a destination expansion to create a new destination volume as a member of the remote CG, and initiating a data replication relationship expand to establish a data replication relationship for the new source volume and the new destination volume.
A storage application is disclosed herein that employs a method for managing read and write requests associated with a database distributed across multiple replicas in multiple data centers. The storage application may be implemented on one or more computing devices in the context of a data center, cloud environment, or the like. The method includes the storage application receiving a client request to perform an operation with respect to an object at an any-local-quorum consistency level. In response to client request, the storage application iteratively generates and submits coordinator requests to the multiple data centers at a local-quorum consistency level, until one of the multiple data centers successfully replies to one of the coordinator requests with a result. The storage application then replies to the client request based on the result provided by the one of the multiple data centers.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database systemDistributed database system architectures therefor
74.
Any-Local-Quorum Consistency For Distributed Databases Replicated Across Multiple Sites
A storage application is disclosed herein that employs a method for managing read and write requests associated with a database distributed across multiple replicas in multiple data centers. The storage application may be implemented on one or more computing devices in the context of a data center, cloud environment, or the like. The method includes the storage application receiving a client request to perform an operation with respect to an object at an any-local-quorum consistency level. In response to client request, the storage application iteratively generates and submits coordinator requests to the multiple data centers at a local-quorum consistency level, until one of the multiple data centers successfully replies to one of the coordinator requests with a result. The storage application then replies to the client request based on the result provided by the one of the multiple data centers.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database systemDistributed database system architectures therefor
75.
Bottleneck management techniques using non-disruptive file movement mechanisms in distributed storage environments
Approaches to data flow bottleneck management using caching mechanisms in a distributed storage environment are disclosed. A request is received by a first data storage node having a first set of interface components, a first set of data management components, a first advisory cache, and a first set of data storage devices. The request has a corresponding file. The first advisory cache is checked for an entry corresponding to the file. The request is routed based on a file characteristic corresponding to the request if there is no corresponding entry in the first advisory cache or to a second data storage node based on the entry in the first advisory cache. Potential bottleneck conditions are monitored on the first node. An advisory cache entry in the first advisory cache is generated in response to determining that a bottleneck condition exists.
G06F 12/123 - Replacement control using replacement algorithms with age lists, e.g. queue, most recently used [MRU] list or least recently used [LRU] list
G06F 3/06 - Digital input from, or digital output to, record carriers
G06F 12/0817 - Cache consistency protocols using directory methods
Approaches to data flow bottleneck management using caching mechanisms in a distributed storage environment are disclosed. A read request is received by a first data storage node having a first set of interface module(s), a first set of data management module(s), a first redirection layer, and a first set of data storage devices. The read request has a corresponding file to be read. The first redirection layer is checked for an entry corresponding to the file. The read request is routed based on a file characteristic corresponding to the read request if there is no corresponding entry in the first redirection layer or to a second data storage node based on the entry in the first redirection layer. Potential bottleneck conditions are monitored on the first node. A redirection layer entry in the first redirection layer is generated in response to determining that a bottleneck condition exists.
Techniques are provided for microservices management and orchestration. A chart package is selectively retrieved from a chart repository based upon the chart package corresponding to a set of services to host within a cluster and dependencies amongst the set of services. A set of container images may be retrieved from a container repository based upon the set of container images corresponding to the set of services. A cluster may be created within a computing environment. The set of services may be deployed as resources of the computing environment within the cluster and the dependencies may be configured using the chart package and the set of container images.
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
Systems and methods for performing single I/O writes are provided. According to one embodiment, responsive to receipt of a write operation from a client by a file system layer of a node of a distributed storage system and a data payload of the operation having been determined to meet a compressibility threshold, an intermediate storage layer of the node logically interposed between the file system layer and a block storage media is caused to perform a single input/output (I/O) write operation that persists the compressed data payload and corresponding metadata to support asynchronous journaling of the write operation. The single I/O write operation coupled with the use of a new pool file that maintains a list of available blocks for single I/O write operations and a modified node crash recovery approach allows the write operation to be acknowledged to the client while the journaling is performed asynchronously.
Techniques are provided for coordinating snapshot operations across multiple file systems. A notification may be received that a snapshot of data stored across a persistent memory file system and a storage file system is to be generated. Forwarding, of modify operations from a persistent memory tier to a file system tier for execution through the storage file system, may be enabled. Framing may be initiated to notify the storage file system of blocks within the persistent memory file system that comprise more up-to-date data than corresponding blocks within the storage file system. In response to the framing completing, a consistency point operation is performed to create the snapshot and to create a snapshot image as part of the snapshot.
09 - Scientific and electric apparatus and instruments
42 - Scientific, technological and industrial services, research and design
Goods & Services
Computer operating systems software and computer software for use within a file storage server for local area networks and storage area networks for use by computer network storage professionals; computer software within a file storage server for local area networks and storage area networks for managing, accessing, updating, delivering, storing and restoring data in a data storage systems, computer networks and electronic databases. Providing temporary use of online non-downloadable software for storage hardware users to manage, troubleshoot and monitor network systems.
81.
Aggregate inline deduplication with volume granular encryption
Techniques are provided for aggregate inline deduplication and volume granularity encryption. For example, data that is exclusive to a volume of a tenant is encrypted using an exclusive encryption key accessible to the tenant. The exclusive encryption key of that tenant is inaccessible to other tenants. Shared data that has been deduplicated and shared between the volume and another volume of a different tenant is encrypted using a shared encryption key of the volume. The shared encryption key is made available to other tenants. In this way, data can be deduplicated across multiple volumes of different tenants of a storage environment, while maintaining security and data privacy at a volume level.
A system can comprise a memory that stores computer executable components, and a processor that executes the computer executable components stored in the memory. The computer executable components can comprise a correlation component that, based on performance data, such as current performance data and/or historical performance data, for an application stored at a storage system, correlates a performance category with the application, and an execution component that, based on the performance category correlated to the application, executes a modification at the storage system, wherein the modification at the storage system comprises changing a functioning of the storage system relative to the application. In an embodiment, the data comprised by the application can be maintained in a non-accessed state to execute the modification at the storage system.
G06F 3/06 - Digital input from, or digital output to, record carriers
G06F 11/34 - Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation
83.
METHODS AND MULTI-SITE SYSTEMS TO PROVIDE RECOVERY POINT OBJECTIVE (RPO) PROTECTION, SNAPSHOT RETENTION BETWEEN SECONDARY STORAGE SITE AND TERTIARY STORAGE SITE, AND AUTOMATICALLY INITIATING REALIGNMENT AND RECONFIGURATION OF A PROTECTION CONFIGURATION FROM THE SECONDARY STORAGE SITE TO THE TERTIARY STORAGE SITE UPON PRIMARY STORAGE SITE FAILURE
Multi-site distributed storage systems and computer-implemented methods are described for providing common snapshot retention and automatic fanout reconfiguration for an asynchronous leg after a failure event that causes a failover from a primary storage site to a secondary storage site. A computer-implemented method comprises providing an asynchronous replication relationship with an asynchronous update schedule from one or more storage objects of the first storage node to one or more replicated storage objects of the third storage node, creating a snapshot copy of the one or more storage objects of the first storage node, transferring the snapshot copy to the third storage node based on an asynchronous mirror policy, and intercepting the snapshot create operation on the primary storage site and transferring the snapshot copy to the second storage node to provide a common snapshot between the second storage node and the third storage node.
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
G06F 3/06 - Digital input from, or digital output to, record carriers
G06F 11/20 - Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
Techniques are provided for implementing asynchronous region-aware caching for snapshot prefetching and restore. A first application within a first container of a container orchestration platform may be hosted at a first region of a multi-cloud environment. A second application within a second container of the container orchestration platform may be hosted at a second region of the multi-cloud environment. Data of the first application may be stored within storage located within the first region of the multi-cloud environment. In response to determining that the second application has a threshold probability of accessing the data of the first application (e.g., a snapshot that the second application can restore), the data may be cached as cached data within a network edge of the second region for access by the second application
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
85.
USE OF CLUSTER-LEVEL REDUNDANCY WITHIN A CLUSTER OF A DISTRIBUTED STORAGE MANAGEMENT SYSTEM TO ADDRESS NODE-LEVEL ERRORS
Systems and methods that make use of cluster-level redundancy within a distributed storage management system to address various node-level error scenarios are provided. According to one embodiment, an instance of a key-value (KV) store of a first node of a plurality of nodes of a cluster of a distributed storage system manages storage of data blocks as values and corresponding block identifiers (IDs) as keys. A list of missing block IDs that are in use for one or more volumes associated with the first node but that are missing from the instance of the KV store are identified by performing a data integrity check on the instance of the KV store. After identifying the list of missing block IDs, instead of treating the first node as failed, restoring the missing block IDs by writing redundant data blocks retrieved from other nodes within the cluster to the first node.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database systemDistributed database system architectures therefor
Techniques are provided for lock reservations for shared storage. A reserve command to reserve a storage structure is received by a driver from a node. The reserve command is formatted according to a storage protocol. The driver translates the reserve command into a lease acquire command formatted according to an object store protocol and targeting an object stored within an object store and corresponding to the storage structure. A lease identifier derived from a node identifier of the node is inserted into the lease acquire command. The lease acquire command is routed to the object store for obtaining a lease on the object for granting the node exclusive write access to the object.
G06F 3/06 - Digital input from, or digital output to, record carriers
87.
METHODS AND MULTI-SITE SYSTEMS TO PROVIDE RECOVERY POINT OBJECTIVE (RPO) PROTECTION AND AUTOMATICALLY INITIATE REALIGNMENT AND RECONFIGURATION OF A PROTECTION CONFIGURATION FROM THE SECONDARY STORAGE SITE TO THE TERTIARY STORAGE SITE UPON PRIMARY STORAGE SITE FAILURE
A computer-implemented method comprises providing a synchronous replication relationship from one or more storage objects of a first storage node to one or more replicated storage objects of a second storage node, providing an asynchronous replication relationship with an asynchronous update schedule from the one or more storage objects of the first storage node to one or more replicated storage objects of the third storage node to provide a protection configuration, tracking, with the third storage node of the tertiary site, a state of the secondary storage site, automatically performing a failover from the primary storage site to the secondary storage site and activating a synchronous mirror copy for the one or more replicated storage objects of the second storage node, and automatically initiating realignment and reconfiguration of the protection configuration to the tertiary storage site based upon the state of the secondary storage site.
G06F 11/16 - Error detection or correction of the data by redundancy in hardware
G06F 11/20 - Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database systemDistributed database system architectures therefor
88.
MAINTAINING THE BENEFIT OF PARALLEL SPLITTING OF OPS BETWEEN PRIMARY AND SECONDARY STORAGE CLUSTERS IN SYNCHRONOUS REPLICATION WHILE ADDING SUPPORT FOR OP LOGGING AND EARLY ENGAGEMENT OF OP LOGGING
Systems and methods are described for performing persistent inflight tracking of operations (Ops) within a cross-site storage solution. According to one embodiment, a method comprises maintaining state information regarding a data synchronous replication status for a first storage object of a primary storage cluster and a second storage object of a secondary storage cluster. The method includes performing persistent inflight tracking of I/O operations with a first Op log of the primary storage cluster and a second Op log of the secondary storage cluster, establishing and comparing Op ranges for the first and second Op logs, and determining a relation between the Op range of the first Op log and the Op range of the second Op log to prevent divergence of Ops in the first and second Op logs and to support parallel split of the Ops.
Techniques are provided for failing over an aggregate from one file system instance to a different file system instance of a distributed scale-out storage system. The aggregate may be stored within distributed storage that is accessible to a plurality of file system instances of the distributed scale-out storage system. When the aggregate is failed over from a first file system instance to a second file system instance, the first file system instance may still have a valid read lease that allows the first file system instance to serve client I/O, directed to the aggregate, using a cache. In order to prevent the first file system instance from serving stale data from the cache before the read lease expires, state machines and a set of control data are used to ensure that the second file system instance attaches to the aggregate only after the read lease has expired.
G06F 11/20 - Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
G06F 11/16 - Error detection or correction of the data by redundancy in hardware
90.
METHODS FOR MANAGING STORAGE OPERATIONS FOR MULTIPLE HOSTS COUPLED TO DUAL-PORT SOLID-STATE DISKS AND DEVICES THEREOF
Methods, non-transitory machine readable media, and computing devices that manage storage operations directed to dual-port solid state disks (SSDs) coupled to multiple hosts are disclosed. With this technology, context metadata comprising a checksum is retrieved based on a first physical address mapped, in a cached zoned namespace (ZNS) mapping table, to a logical address. The logical address is extracted from a request to read a portion of a file. A determination is made when the checksum is valid based on a comparison to identification information extracted from the request and associated with the file portion. At least the first physical address is replaced in the cached ZNS mapping table with a second physical address retrieved from an on-disk ZNS mapping table, when the determination indicates the checksum is invalid. The file portion retrieved from a dual-port SSD using the second physical address is returned to service the request.
A method for reducing write latency in a distributed file system. A write request that includes a volume identifier is received at a data management subsystem deployed on a node within a distributed storage system. The data management subsystem maps the volume identifier to a file system volume and maps the file system volume to a set of logical block addresses in a logical block device in a storage management subsystem deployed on the node. The storage management subsystem maps the logical block device to a metadata object for the logical block device on the node that is used to process the write request. The mapping of the file system volume to the set of logical block addresses in the logical block device enables co-locating the metadata object with the file system volume on the node, which reduces the write latency associated with processing the write request.
Methods and systems for providing Quality of Service (QOS) in a cloud-based system are provided. One method includes assigning, by a micro-service, a workload identifier to a cloud volume created by a storage operating system in a cloud-based system; mapping, by the micro-service, the workload identifier to a volume identifier, the volume identifier generated by the storage operating system to identify the cloud volume; associating, by the micro-service, a policy with the cloud volume for providing QoS for the cloud volume; determining, by the micro-service, the workload identifier for the cloud volume from the volume identifier included in a request to store or retrieve data using the cloud volume; and assigning, by the micro-service, the workload identifier to a processing thread deployed by the storage operating system to process the request.
Techniques are provided for dynamically implementing quality of service policies for a distributed storage system based upon resources saturation. A quality of service policy is defined for throttling I/O operations received by a node of the distributed storage system based upon whether resources of the node have become saturated. The quality of service policy is dynamically implemented based upon ever changing resource utilization and saturation. Dynamically implementing the quality of service policy improves the ability to efficiently utilize resources of the node compared to conventional static polices that cannot adequately react to such changing considerations and resource utilization/saturation. With conventional static policies, an administrator manually defines a minimum amount of guaranteed resources and/or a maximum resource usage cap that could be set to values that result in inefficient operation and resource starvation. Dynamically implementing the quality of service policy results in more efficient operation and mitigates resource starvation.
Methods and systems for solid state drives are provided, including assigning a first namespace to a first instance of a storage operating system and a second instance of the storage operating system for enabling read access to a first portion of a flash storage system by the first instance, and read and write access to the second instance; allocating a second namespace to the first instance for exclusive read and write access within a second portion of the flash storage system; generating, by the first instance, a request for the second instance to transfer a data object from the second portion owned by the first instance to the first portion; storing, by the second instance, the data object at the first portion; and updating metadata of the data object at the second portion, the metadata indicating a storage location at the second portion where the data object is stored.
G06F 3/06 - Digital input from, or digital output to, record carriers
95.
METHODS AND SYSTEMS TO IMPROVE INPUT/OUTPUT (I/O) RESUMPTION TIME BY BATCHING MULTIPLE NON-CONFLICTING OPERATIONS DURING A NON-DISRUPTIVE AUTOMATIC UNPLANNED FAILOVER FROM A PRIMARY COPY OF DATA AT A PRIMARY STORAGE SYSTEM TO A MIRROR COPY OF THE DATA AT A CROSS-SITE SECONDARY STORAGE SYSTEM
Multi-site distributed storage systems and computer-implemented methods are described for improving a resumption time of input/output (I/O) operations during an automatic unplanned failover (AUFO). A computer-implemented method includes determining, with a second storage cluster, whether heartbeat information from one or more storage objects of a CG of a first set of CGs is received during a time period, determining an out of sync state for a data replication relationship between the CG of the first set of CGs and a mirrored CG of a second set of CGs when the heartbeat information is not received during the time period and sending a single bulk role change call with a cluster identifier from the second cluster to an external mediator to provide a role change from follower to leader in the second set of CGs.
G06F 11/20 - Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
96.
AUTOMATED TUNING OF A QUALITY OF SERVICE SETTING FOR A DISTRIBUTED STORAGE SYSTEM BASED ON INTERNAL MONITORING
Systems and methods for automated tuning of Quality of Service (QoS) settings of volumes in a distributed storage system are provided. According to one embodiment, one or more characteristics of a workload of a client to which a storage node of multiple storage nodes of the distributed storage system is exposed are monitored. After a determination has been made that a characteristic meets or exceeds a threshold, (i) information regarding multiple QoS settings assigned to a volume of the storage node utilized by the client is obtained, (ii) a new value of a burst IOPS setting of the multiple QoS settings is calculated by increasing a current value of the burst IOPS setting by a factor dependent upon a first and a second QoS setting of the multiple QoS settings, and (iii) the new value of the burst IOPS setting is assigned to the volume for the client.
Techniques are provided for splitting a clone volume from a parent volume whose data is stored within objects of an object store. A transfer map is used to track mappings of selectively created child object identifiers used to subsequently copy the one or more parent objects to create child objects corresponding to the child object identifiers. A consistency point phase is performed. For each child object identifier processed during the consistency point phase, an object state for a corresponding child object is set to a copy pending state. A reverse map is populated with a reverse map entry. The transfer map is traversed to copy the one or more parent objects as the child objects for splitting the clone volume from the parent volume. The reverse map is used to verify that the child objects are successfully created with valid data.
One or more techniques and/or computing devices are provided for resynchronization. For example, a request may be received to create pseudo snapshots of a first consistency group, hosted by a first storage controller, and a second consistency group, hosted by a second storage controller, having a synchronous replication relationship with the first consistency group. Incoming client write requests are logged within an intercept tracking log at the first storage controller. After a first drain without hold of incoming write requests is performed, a first pseudo common snapshot of the second consistency group is created. After a second drain without hold of incoming write operations is performed, a second pseudo common snapshot of the first consistency group and the intercept tracking log is created. The pseudo snapshots and the intercept tracking log (e.g., indicating a delta between the pseudo snapshots) are used to resynchronize the first and second consistency groups.
G06F 3/06 - Digital input from, or digital output to, record carriers
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
G06F 11/20 - Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
99.
PRESERVING STORAGE EFFICIENCY DURING RESTORATION OF DATA FROM THE CLOUD TO A DATA STORAGE SYSTEM
Systems and methods for preserving storage efficiency during restoration of data from the cloud are provided. In one embodiment, a CBMAP is maintained that maps cloud block numbers (CBNs) to respective corresponding block numbers of a volume of a data storage system in which previously restored data has been stored by a previously restored file. By making use of the CBMAP during the restoration process, storage of duplicate file data blocks on the volume may be avoided by sharing with a current file being restored a reference to the corresponding file data block previously stored on the volume and associated with the previously restored file. In addition to preserving storage efficiency, use of the CBMAP facilitates avoidance of repeated GET operations for data associated with CBNs previously retrieved from the cloud and stored to the volume, thereby reducing data access costs as well as latency of the restore operation.
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
100.
ADDITIONAL COMPRESSION FOR EXISTING COMPRESSED DATA
Techniques are provided for implementing additional compression for existing compressed data. Format information stored within a data block is evaluated to determine whether the data block is compressed or uncompressed. In response to the data block being compressed according to a first compression format, the data block is decompressed using the format information. The data block is compressed with one or more other data blocks to create compressed data having a second compression format different than the first compression format.
