The technology disclosed herein enables access to a file system by a portable executable program. In a particular example, a method includes, in a host executing the portable executable program, recognizing the portable executable program is executing on the one or more processing systems and determining the portable executable program is configured to access the file system. The method also includes directing the portable executable program to create a module therein for file system access and creating an abstraction layer with which the module exchanges file system commands. In the abstraction layer, the method includes translating the file system commands to translated commands for the file system and exchanging translated commands between the abstraction layer and the file system.
G06F 21/53 - Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems during program execution, e.g. stack integrity, buffer overflow or preventing unwanted data erasure by executing in a restricted environment, e.g. sandbox or secure virtual machine
Methods and systems for applications and/or virtual machines (“VM”) are provided. As an example, one method includes registering a storage system configured to store data for a VM; creating a policy for the VM for generating a VM backup by a storage management system interfacing with the storage system and a cloud-based microservice; storing, by the cloud-based micro-service, a copy of the VM backup at a cloud-based storage; receiving, by the cloud-based micro-service, an indication from the storage management system of a successful execution of a pre-restore operation upon unregistering the VM from a VM management system; and copying, by the cloud-based micro-service, VM data from the VM backup stored at the cloud-based storage to the storage system for executing a restore operation to restore the VM backup at the storage system.
Systems and methods are described for a cross-site high availability distributed storage system. According to one embodiment, a computer implemented method includes providing a remote direct memory access (RDMA) request for a RDMA stream, and generating, with an interconnect (IC) layer of the first storage node, multiple IC channels and associated IC requests for the RDMA request. The method further includes mapping an IC channel to a group of multiple transport layer sessions to split data traffic of the IC channel into multiple packets for the group of multiple transport layer sessions using an IC transport layer of the first storage node and assigning, with the IC transport layer, a unique transaction identification (ID) to each IC request and assigning a different data offset to each packet of a transport layer session.
A computer implemented method includes maintaining information indicative of whether a data replication relationship between a dataset associated with the local CG (CG1) and a mirror copy of the dataset stored on a remote CG (CG2) of a remote distributed storage system is in an in-synchronization (InSync) state or an out-of-synchronization (OOS) state, determining whether a CG storage expansion process is in progress, and in response to determining the OOS state and whether the primary storage site has a failure, initiating an automatic unplanned failover (AUFO) workflow without manual intervention on the original volumes in the CG1 of a first storage cluster of the primary storage site and the original volumes in the CG2 of a second storage cluster when the CG storage expansion process is in progress with a new source volume to be a member of CG1 and a new destination volume to be a member of CG2.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database systemDistributed database system architectures therefor
The technology disclosed herein enables two-way synchronization between a remote data lifecycle manager and agents (e.g., custom resources and controllers) in the computing environments being managed. In a particular example, a method includes executing a data lifecycle manager external to the cluster and executing an agent of the data lifecycle manager on a node within the cluster. A watcher of the control plane monitors events for resources in the cluster. Upon identifying an event, the watcher establishes a connection with the data lifecycle manager and transmits a notification about the event. This transmission allows the data lifecycle manager to manage data lifecycles in the cluster based on the notification.
The disclosure describes systems, devices, and methods for managing multi-part files and attributes thereof in data storage environments. In an example implementation, a method of operating multiple storage controllers in a data storage environment is provided. In performing the method, a data element of a storage controller of the multiple storage controllers receives an input/output request from a network element of a one of the multiple storage controllers corresponding to data to a multi-part file stored across multiple storage devices in the data storage environment. The input/output request includes current attributes of the multi-part file. The data element executes the input/output request with respect to a part of the multi-part file corresponding to the storage controller and determines whether to delegate attribute authority to the network element based on a set of criteria.
Systems and methods are provided for data migration including cloning a multiple-logical unit number (LUN) volume into a plurality of cloned single-LUN volumes on a first storage node of a computing system; creating a plurality of new volumes on a second storage node of the computing system based at least in part on the plurality of cloned single-LUN volumes; selectively copying snapshot data from the plurality of cloned single-LUN volumes to the plurality of new volumes; establishing snapshot mirror relationships between the plurality of cloned single-LUN volumes and the plurality of new volumes; synchronizing the plurality of cloned single-LUN volumes and the plurality of new volumes; and performing a migration of data logical interface failovers (LIFs) from the plurality of cloned single-LUN volumes to the plurality of new volumes.
G06F 11/20 - Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
37 - Construction and mining; installation and repair services
41 - Education, entertainment, sporting and cultural services
42 - Scientific, technological and industrial services, research and design
Goods & Services
Installation, maintenance, and repair of computer hardware for storing, managing, replicating, transferring, securing, retrieving, and restoring data and files; installation, maintenance, and repair of computer hardware for sharing, updating, partitioning, and accessing files over a computer network; installation, maintenance, and repair of computer hardware for optimizing the accessibility, delivery, backup, security, restoration, and replication of data; installation, maintenance, and repair of computer hardware for managing, monitoring, and securing networks, computer disc drives, electronic data storage systems, databases and other storage devices across computer networks; installation, maintenance, and repair of computer hardware using artificial intelligence for the purpose of managing business data; consulting services related to installation, repair, and maintenance of computer hardware and artificial intelligence (AI) in the fields of artificial intelligence, computers, computer software and hardware, computer storage networks, computer storage devices, data and computer file storage, management, replication, security, retrieval, restoration and distribution, and management, monitoring and security of computer networks, computer disc drives, electronic data storage systems, databases and other storage devices across computer networks. Educational services, namely, arranging and conducting training courses, lectures, classes, workshops, seminars and conferences in the fields of artificial intelligence, computers, computer software and hardware, computer storage networks, computer storage devices, data and computer file storage, management, replication, security, retrieval, restoration and distribution, and management, monitoring and security of computer networks, computer disc drives, electronic data storage systems, databases and other storage devices across computer networks; providing educational testing to determine professional skills in the fields of computers, computer software and hardware, computer storage networks, computer storage devices, data and computer file storage, management, replication, security, retrieval, restoration and distribution, and management, monitoring and security of computer networks, computer disc drives, electronic data storage systems, databases and other storage devices across computer network. Providing online and cloud-based non-downloadable software for monitoring, securing, optimizing, analyzing, and managing electronic and cloud data storage, computer servers, databases, and other storage devices and systems across computer networks; technical support services, namely, monitoring and troubleshooting for problems with computers, computer software and hardware and computer storage networks, and providing back-up computer programs and facilities; software as a Service (SAAS) services featuring software for management, monitoring, modeling, troubleshooting, optimization, reporting and analysis of network storage infrastructure and data; cloud computing featuring software for use in controlling, monitoring, and managing cloud infrastructure, virtual services, and networking configuration; design, deployment and management of software and hardware using artificial intelligence for the purpose of managing business data; technical consulting in the field of artificial intelligence (AI) software customization; technical consulting in the fields of artificial intelligence, computers, computer software and hardware, computer storage networks, computer storage devices, data and computer file storage, management, replication, security, retrieval, restoration and distribution, and management, monitoring and security of computer networks, computer disc drives, electronic data storage systems, databases and other storage devices across computer networks; artificial Intelligence as a Service (AIAAS) services featuring software using artificial intelligence (AI) for use in database management; providing online and cloud-based non-downloadable software for deploying, implementing, monitoring, securing, optimizing, analyzing, storing, managing, and troubleshooting artificial intelligence (AI) platforms and internal processes; providing temporary use of online non-downloadable software for use in connection with data management, data storage, data protection, security, and compliance; providing temporary use of online non-downloadable software for managing, accessing, modifying, updating, organizing, delivering, synchronizing, processing, transmitting, storing, and restoring data.
9.
RESYNC TRANSFER FOR RECOVERING FROM STORAGE SITE FAILURE UTILIZING BACKGROUND SCANNER
Techniques are provided for performing a resync transfer to recover from a storage site failure. During normal operation of a first site hosting a first volume, data is replicated to a second volume hosted by a second site. If the first site fails, when clients are redirected to the second volume at the second site. When the first site recovers, data modifications made to the second volume are resynced back to the first volume. As part of synchronizing the first volume, a data warehouse is rebuilt at the first site in order to track the location of blocks present on the replication destination. Typically, the data modifications are transferred after the data warehouse is rebuilt, which results in significantly long resync times. The techniques provided herein decrease the resync time by either rebuilding the data warehouse in parallel with resyncing the data modifications or circumvent the need for rebuild.
Techniques are provided for a data format for efficient management of checkpoint support. The data format corresponds to a base metafile and a set of instance metafiles used to track storage operations such as a directory restore operation. The base metafile and the set of instance metafiles can be used to resume the storage operation from where the storage operation left off in the event of a failure. The base metafile and the set of instance metafiles can be used to track progress of the storage operation processing objects stored within an object store of a cloud storage environment.
Techniques are provided for restoring a directory from a snapshot of a volume backed up to an object store. The snapshot may be backed up from a node to the object store, such as a cloud computing environment. A user may want to restore the directory within the volume without having to restore the entire volume, which otherwise would waste computing resources, storage, network bandwidth, and time. Accordingly, the techniques provided herein are capable of restoring just the directory from the snapshot that is stored within the object store. Because snapshot data of the snapshot may be stored across multiple objects within the object store, certain objects are identified as comprising snapshot data (backup data) of the directory and content items within the directory. In this way, the snapshot data of the directory is restored from these objects to a restore directory at a restore target.
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
Systems and methods for use of information relating to units of storage space movement and ownership to facilitate performance of disaggregated storage system workflows are provided. In various examples, the unit of disaggregated storage space for purposes of movement and ownership is an allocation area (AA). AA ownership information relating to AAs owned by dynamically extensible file systems (DEFSs) of a cluster may be maintained within two different persistent data sources, including an AA label region and DEFS AA owner metafiles. A node-scoped cache may be implemented to cache the AA ownership information in-memory for DEFSs hosted by a given node to provide a multiprocessing (MP)-safe cache for AA ownership information, a mechanism to coordinate between AA movement and other subsystems needing to know whether an AA is “local” or “remote,” and a mechanism to serialize access and Input/Output (IO) operations to the persistent AA ownership information data sources.
G06F 16/16 - File or folder operations, e.g. details of user interfaces specifically adapted to file systems
G06F 12/0802 - Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
G06F 12/14 - Protection against unauthorised use of memory
13.
FACILITATING PERFORMANCE OF DONOR-RECIPIENT DISAGGREGATED STORAGE SYSTEM WORKFLOWS BASED ON INFORMATION RELATING TO UNITS OF STORAGE FOR STORAGE SPACE MOVEMENT AND OWNERSHIP
Systems and methods for use of information relating to units of storage space movement and ownership to facilitate performance of disaggregated storage system workflows are provided. In various examples, the unit of disaggregated storage space for purposes of movement and ownership is an allocation area (AA). AA ownership information relating to AAs owned by dynamically extensible file systems (DEFSs) of a cluster may be maintained within two different persistent data sources, including an AA label region and DEFS AA owner metafiles. A node-scoped cache may be implemented to cache the AA ownership information in-memory for DEFSs hosted by a given node to provide a multiprocessing (MP)-safe cache for AA ownership information, a mechanism to coordinate between AA movement and other subsystems needing to know whether an AA is “local” or “remote,” and a mechanism to serialize access and Input/Output (IO) operations to the persistent AA ownership information data sources.
FACILITATING PERFORMANCE OF NODE-LEVEL DISAGGREGATED STORAGE SYSTEM WORKFLOWS BASED ON INFORMATION RELATING TO UNITS OF STORAGE SPACE FOR MOVEMENT AND OWNERSHIP
Systems and methods for use of information relating to units of storage space movement and ownership to facilitate performance of disaggregated storage system workflows are provided. In various examples, the unit of disaggregated storage space for purposes of movement and ownership is an allocation area (AA). AA ownership information relating to AAs owned by dynamically extensible file systems (DEFSs) of a cluster may be maintained within two different persistent data sources, including an AA label region and DEFS AA owner metafiles. A node-scoped cache may be implemented to cache the AA ownership information in-memory for DEFSs hosted by a given node to provide a multiprocessing (MP)-safe cache for AA ownership information, a mechanism to coordinate between AA movement and other subsystems needing to know whether an AA is “local” or “remote,” and a mechanism to serialize access and Input/Output (IO) operations to the persistent AA ownership information data sources.
Techniques are provided for backing up and restoring a file system or storage virtual machine located within a remote object store. A specification is parsed to identify resources associated with and including a primary resource hosted within a remote object store and to identify REST API endpoints of the resources. GET operations targeting the REST API endpoints of the resources are performed to retrieve the resources and properties of the resources. A link relationship specification is parsed to identify links corresponding to dependencies amongst the resources. A backup of the primary resource is generated to include the resources, the properties of the resources, and dependency information derived from the links. The backup can be used to restore the primary resource to the remote object store in manner that preserves the dependencies amongst the resources.
Systems and methods for performing space balancing heuristics within a distributed storage system that makes use of disaggregated storage are provided. In various examples described herein, the unit of storage space assignment used to associate blocks of storage space within a storage pod with a given dynamically extensible file system (DEFS) is an AA, representing a large chunk of one or more gigabytes (GB). The use of AAs described herein allows disk space to be used more elastically across individual nodes of a storage cluster. Each DEFS periodically reports its AA usage information by persisting it to a respective heuristic block within the storage pod. An AA director operable within the cluster periodically monitors the reported AA usage information, performs a heuristic algorithm to determine donor and recipient DEFSs at a given instance, and triggers AA movement accordingly by matching one or more donor DEFSs to each recipient DEFS.
Methods and systems involve extracting metadata and contents of source files to produce ingestion record files for the source files, which are transmitted to an external processing service to be processed. The extraction of metadata and contents of the source files includes dividing the metadata and contents of a single source file into multiple ingestion record files when the metadata and contents of the single source file exceeds a predetermined size.
Techniques are provided for parsing and transforming one or more queries using semantic model automation. A platform may store data according to a first format and structure. For example, a reporting platform may store operational data as a project that can be used for generating reports through the reporting platform. The project may not be compatible with other platforms that support other formats and structures. Accordingly, the disclosed technology is used for parsing a query targeting the project to identify components used by the reporting platform to organize the data. The components are used to generate one or more custom queries to retrieve data and metadata of the project. The data and metadata is used to create a model having a format and structure supported by a different reporting platform such that the project can be migrated between different reporting platforms supporting different formats and structures of data.
The disclosure describes a workflow management service that dynamically provisions workflow documents across multiple accounts to avoid reaching document limits imposed by a compute provider. The workflow management service initially allocates workflow documents to a central account, which manages workflow execution and distribution. The workflow management service also provisions shard accounts dedicated to hosting and executing workflow documents. When the number of workflow documents in the central account or an existing shard account approaches the compute provider's imposed limit, the workflow management service provisions a new shard account and redirects future workflow documents to the new shard account.
Techniques are provided for compressing images using compression image containers. Many user devices, such as smart phones, incorporate cameras that be utilized by users to capture photos. The photos can be either locally stored on a user device or uploaded to cloud storage. Managing the storage of a vast amount of photos is cumbersome and expensive where trillions of photos could be stored within the cloud storage. The techniques provided herein are capable of compressing images using compression image containers in order to significantly reduce storage consumed when storing images with similar or duplicate image information. Similar images are compressed together within the same compression image container for improved compression and reduced storage.
H04N 19/127 - Prioritisation of hardware or computational resources
H04N 19/136 - Incoming video signal characteristics or properties
H04N 19/154 - Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion
H04N 19/157 - Assigned coding mode, i.e. the coding mode being predefined or preselected to be further used for selection of another element or parameter
H04N 19/184 - Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being bits, e.g. of the compressed video stream
H04N 19/186 - Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a colour or a chrominance component
H04N 19/187 - Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a scalable video layer
H04N 19/30 - Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability
H04N 19/423 - Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation characterised by memory arrangements
H04N 19/46 - Embedding additional information in the video signal during the compression process
H04N 19/80 - Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation
Techniques are provided for microservices management and orchestration. A chart package is selectively retrieved from a chart repository based upon the chart package corresponding to a set of services to host within a cluster and dependencies amongst the set of services. A set of container images may be retrieved from a container repository based upon the set of container images corresponding to the set of services. A cluster may be created within a computing environment. The set of services may be deployed as resources of the computing environment within the cluster and the dependencies may be configured using the chart package and the set of container images.
Systems and methods for restoring multi-part files and directories in cloud storage environments are described. A catalog inode associated with a multi-part file is restored, containing metadata comprising inode parts corresponding to file parts. The inode parts are iteratively restored based on the catalog inode metadata. Corresponding file parts are then retrieved from cloud storage resources and assembled to reconstruct the multi-part file. The process may involve creating and updating checkpoints, restoring subsets of inode parts, and parallel retrieval of file parts. For directory restoration, the method is applied to multiple catalog inodes within the directory structure. The restoration process may handle flexible volumes and additional directories. Apparatuses implementing these methods include processors and memory storing instructions to perform the restoration steps. The systems and methods enable efficient, granular restoration of complex file structures in distributed cloud environments while maintaining data integrity and providing robust recovery capabilities.
Systems and methods for restoring multi-part files and directories in cloud storage environments are described. A catalog inode associated with a multi-part file is restored, containing metadata comprising inode parts corresponding to file parts. The inode parts are iteratively restored based on the catalog inode metadata. Corresponding file parts are then retrieved from cloud storage resources and assembled to reconstruct the multi-part file. The process may involve creating and updating checkpoints, restoring subsets of inode parts, and parallel retrieval of file parts. For directory restoration, the method is applied to multiple catalog inodes within the directory structure. The restoration process may handle flexible volumes and additional directories. Apparatuses implementing these methods include processors and memory storing instructions to perform the restoration steps. The systems and methods enable efficient, granular restoration of complex file structures in distributed cloud environments while maintaining data integrity and providing robust recovery capabilities.
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
24.
DYNAMIC STORAGE JOURNALING PARTITIONS FOR EFFICIENT RESOURCE USE AND IMPROVED SYSTEM THROUGHPUT
Systems and methods for improving latency and throughput for metadata-heavy workloads and/or workloads including metadata bursts by decoupling data journal records and metadata-only journal records are provided. According to one embodiment, expedited and independent space reclamation is facilitated by differentiating between various types of journal records chains of which should be retained until different conditions are met. For example, data journal records may be added to data journal record chains within a persistent KV store and metadata-only journal records may be added to metadata-only journal record chains within the persistent KV store. Reclamation of spaced utilized by a data journal record chain may be reclaimed after both remote node data flush has been completed and the completion of a local CP for all records in the chain, whereas records of a metadata-only journal chain may be freed independently upon completion of a local CP for all records.
Techniques are provided for rebuilding a destination file system as part of a resync operation. A file system of a first node may be replicated to a second node as a destination file system. The file system may include clone files that were created as clones of parent files. The clone files may have been created in a storage efficient manner where a clone file refers to data of a parent file, instead physically storing duplicate instances of the parent data of the parent file. The techniques provided herein are capable of replicating the clone files to the second node while preserve the storage efficiency of the clone files during a resync operation that rebuilds the destination file system after recovering from a failure of the second node.
G06F 11/20 - Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
Improved techniques for using multicore processors to process storage operations for multithreaded applications. Operation type concurrency is considered for storage operations in a storage system in addition to application concurrency. A storage controller of the storage system identifies a storage operation associated with an application and an available worker thread for processing the storage operation. The storage controller determines whether to use the available worker thread to execute the storage operation based on an application concurrency limit for the application and an operation type concurrency limit for the operation type of the storage operation. The storage controller allocates a worker thread to a storage operation only where the storage operation does not satisfy either limit. By evaluating and limiting the concurrency of specific operation types, the storage system mitigates response latency resulting from a large volume of storage operations of a particular type consuming a disproportionate amount of computing resources.
Methods and systems for mitigating stranded data objects in storage resources used by tenants of an application service. Techniques are disclosed for determining maximum allowable retain-until-dates for objects that have object lock features enabled such that any retain-until-date selection will not result in a stranded object. A request to configure an object lock retention period for an object or for a structure containing data objects is received. The request is associated with a tenant account, based on which a maximum value for the retain-until-date can be determined. The maximum retain-until-date is the greatest retention period that the object or structure can have while not exceeding the expiration of a service relationship associated with the tenant. Service relationships for the tenant are identified and the maximum retain-until-date is determined. Based on the maximum retain-until-date, the retain-until-date for the object can be set without the risk of the objecting becoming stranded.
The disclosure describes systems, devices, and methods for automating the reintegration of a failed virtual storage block into a file system service upon recovery of the virtual storage block. In an example embodiment, a method performed by a storage controller in a file system service is provided. In performing the method, the storage controller determines that a virtual storage volume of an array of virtual storage volumes has become unavailable, and in response to determining that the virtual storage volume has become unavailable, pauses use of the virtual storage volume and monitors for the virtual storage volume to return to an available state. Upon the virtual storage volume returning to the available state, the storage controller tests multiple portions of the virtual storage volume. After successfully testing the multiple portions of the virtual storage volume, the storage controller commences with the use of the virtual storage volume.
Systems and methods for scaling application and/or storage system functions of a distributed storage system based on a heterogeneous resource pool are provided. According to one embodiment, the distributed storage system has a composable, service-based architecture that provides scalability, resiliency, and load balancing. The distributed storage system includes a cluster of nodes each potentially having differing capabilities in terms of processing, memory, and/or storage. The distributed storage system takes advantage of different types of nodes by selectively instating appropriate services (e.g., file and volume services and/or block and storage management services) on the nodes based on their respective capabilities. Furthermore, disaggregation of these services, facilitated by interposing a frictionless layer (e.g., in the form of one or more globally accessible logical disks) therebetween, enables independent and on-demand scaling of either or both of application and storage system functions within the cluster while making use of the heterogeneous resource pool.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database systemDistributed database system architectures therefor
Techniques and systems for enhanced storage system management and failure detection/mitigation are presented. In one example, a method includes, responsive to detection of a trigger event for a first host node monitored by at least a second host node, initiating fencing of the first host node by at least transferring a fencing notification to a controller module for a set of storage drives assigned to the first host node. The method also includes preventing the first host node from communicating with the set of storage drives, and alerting a client node communicating with the first host node that the second host node is handling communication with the set of storage drives.
G06F 11/20 - Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
G06F 11/16 - Error detection or correction of the data by redundancy in hardware
31.
Persisting Mutex Lock Data To Enhance Failover Handling By Storage Controllers
Improved failover handling by persisting lease lock data to storage. Storage controllers of a data storage system handle client I/O requests for files or portions of files in a storage aggregate of the system. The controllers obtain lease locks on behalf of clients for the I/O requests and record lock data in the storage aggregate. If a first controller of the system fails over to a second controller, a grace period is triggered. The second controller gets the lock data from storage and processes an I/O request received from a client during the grace period. From the data, the second controller determines if the first storage controller held the lock corresponding to the I/O request. If the first storage controller held the lock, the second controller immediately executes the request. Where the first storage controller did not hold the lock, the second controller obtains the lock to execute the request.
G06F 9/52 - Program synchronisationMutual exclusion, e.g. by means of semaphores
H04L 67/1097 - Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
32.
Dynamic Balancing Between Stability And Cost Optimization In Compute Clusters
The disclosure describes a node management service that dynamically adjusts resource allocations for workloads in a compute cluster by balancing cost and stability considerations. The node management service implements a disruption budget that allows for a predetermined level of disruptions of applications within the workload. As the disruption level of the workload approaches the disruption budget, the node management service initiates disruption reduction actions that lower the likelihood of further disruptions.
OP logs are utilized for fast recovery of synchronous replication in a storage system. According to one embodiment, a computer-implemented method comprises establishing synchronous replication between one or more storage objects of a first consistency group (CG1) of a primary storage site and one or more storage objects of a second consistency group (CG2) of a secondary storage site, initiating a peer health check to determine a health condition of the primary storage site and concurrently initiating a create snapshot process to create a snapshot of one or more storage objects of the CG2 of the secondary storage site in response to detection of missing heartbeat communications from the primary storage site, and performing inflight tracking replay and reconciliation between a first Op log of the primary storage site and a second Op log of the secondary storage site when an out of sync state or failover occurs.
G06F 11/20 - Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
34.
Managing Parity Operations Using Controller-Specific Parity Bitmaps
The disclosure describes systems, devices, and methods for tracking operations of controllers in a data storage environment on a per-controller basis. In an implementation, a method for re-assigning responsibility of individual controller-owned parity bitmap sections and re-performing an incomplete operation is provided. In the method, a controller detects a failure of a first controller and responsively assign responsibility for a second of a parity bitmap previously assigned to the first controller to the second controller. The controller directs the second controller to read the assigned section of the parity bitmap. The controller further directs the second controller to, for a status indicator in the section indicating incomplete, compute new parity data based on source data associated with the status indicator, store the new parity data in the section, and update the status indicator from incomplete to complete.
In one embodiment, a method comprises providing a first dynamically extensible file system (DEFS) of a first node and a second DEFS of a second node of a storage cluster having a disaggregated storage space within a storage pod, detecting, with the first DEFS of the first node, blocks and associated physical volume block numbers (PVBNs) to be freed, determining whether the first DEFS owns or does not own an allocation area (AA) having the PVBNs to be freed, and transferring non-owned PVBNs to be freed to a remote free log of the first DEFS when the AA having the PVBNs to be freed is not owned or assigned to the first DEFS.
Systems, methods, and software are disclosed herein relating to processing telemetry data of a storage environment for anomaly detection in various implementations. In an implementation, a computing apparatus detects a sequence of missing data in telemetry data associated with a user of a data storage environment and verifies the user was active during a time period of the sequence of missing data. Upon verification, the computing apparatus generates synthetic data to replace the sequence of missing data; the synthetic data is based on activity data of users similar to the given user. The computing apparatus generates augmented data with the telemetry data and the synthetic data and incorporates the augmented data into a historical user activity profile. The computing apparatus detects anomalous behavior in the storage environment based on a comparison of new telemetry data with the historical user activity profile and takes corrective action associated with the anomalous behavior.
The technology disclosed herein enhances data protection in a distributed storage system. In a particular example, a method includes determining a drive in a subject node of the distributed storage system has failed while storing first data of a data set distributed across nodes of the distributed storage system by a data protection mechanism. The method further includes broadcasting failure information indicating the data set from the subject node to other nodes of the distributed storage system. At the other nodes, in response to receiving the failure information, the method includes identifying a subset of the other nodes that also store a portion of the data set. In each identified node of the subset, the method includes identifying second data of the data set stored on a local drive and copying the second data to a different local drive to protect the data set from further drive failure.
With a forever incremental snapshot configuration and a typical caching policy (e.g., least recently used), a storage appliance may evict stable data blocks of an older snapshot, perhaps unchanged data blocks of the snapshot baseline. If stable data blocks have been evicted, restore of a recent snapshot will suffer the time penalty of downloading the stable blocks for restoring the recent snapshot. Creating synthetic baseline snapshots and refreshing eviction data of stable data blocks can avoid eviction of stable data blocks and reduce the risk of violating a recovery time objective.
Systems and methods for sharing a namespace of an ephemeral storage device by multiple consumers are provided. In an example, a storage driver (e.g., a non-volatile memory express (NVMe) driver) of a virtual storage system deployed within a compute instance of a cloud environment facilitates sharing of the namespace by exposing an application programming interface (AP)I through which the multiple consumers access an ephemeral storage device associated with the compute instance. During initialization processing performed by each consumer, for example, during boot processing of the virtual storage system, the consumers may share the namespace by reserving for their own use respective partitions within the namespace via the API and thereafter restrict their usage of the namespace to their respective partitions.
Systems and methods for identifying anomalous activities in a cloud computing environment are provided. According to one embodiment, a customer's infrastructure may be fortified by leveraging deep learning technology (e.g., an encoder-decoder machine-learning (ML) model) to predict events in the cloud environment. During a training phase, the ML model may be trained to make a prediction regarding a next event based on a predetermined or configurable length of a sequence of contextual events. For example, historical events (e.g., cloud application programming interface (API) events logged to a cloud activity trace) observed within the customer's cloud infrastructure over the course of a particular date range may be split into appropriate event/context pairs and fed to the ML model. Subsequently, during a run-time anomaly detection phase, the ML model may be used to predict a next event based on a sequence of immediately preceding events to facilitate identification of anomalous activity.
H04L 41/16 - Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks using machine learning or artificial intelligence
Systems and methods for enhancing API security by identifying anomalous activities in a cloud environment are provided. In one example, the lack of awareness of an external API with respect to how calls to the external API may affect a cluster of a container orchestration platform is addressed. For instance, the views of the external and internal APIs may be combined to achieve better API security by correlating external API calls with undesirable behavior or other anomalies arising in the internal API. Responsive to identifying such undesirable behavior, information (e.g., a host, a source IP, a user, a specific payload) associated with the offending external API call may be added to a network security feature (e.g., a deny list, an IPS, or a WAF) utilized by the external API to facilitate performance of enhanced filtering of subsequent external API calls by the external API on behalf of the internal API.
Disclosed herein are methods and systems for the operation of a resource management service. The resource management service deploys reclaimable compute instances from a resource pool and continuously generates predicted remaining lifespans for the deployed reclaimable compute instances. The predicted remaining lifespan is monitored to determine if the predicted remaining lifespan is below a threshold value. In response to the predicted remaining lifespan for a reclaimable compute instance falling below a threshold value, the resource management service instructs the reclaimable compute instance to create an application state snapshot of an application running thereon. A subsequent compute instance is deployed from the resource pool, on which the application can be restored to a previous state using the application state snapshot.
A method, a computing device, and a non-transitory machine-readable medium for classifying documents. A document collection is sorted into a plurality of categories. A classifier corresponding to a category of the plurality of categories is trained to output a probability that a document associated with the category is of a selected type (e.g., confidential). The training includes determining, by the processor, that a cardinality of a set of negative samples in a train set is not above a pipeline threshold but is at least one and training the classifier via a first pipeline and a second pipeline using a training group that includes a first portion of a group of positive samples in the train set, a second portion of a set of negative samples in the train set, and a third portion of a group of unlabeled samples in the train set
G06F 18/214 - Generating training patternsBootstrap methods, e.g. bagging or boosting
G06F 18/23213 - Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions with fixed number of clusters, e.g. K-means clustering
G06F 18/2415 - Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
Systems and methods are disclosed for implementing a process for graph database storage optimization, applicable to delta-based cloud asset tracking. In certain embodiments, a method may comprise implementing a delta-based graph storage optimization system for asset tracking in a cloud environment, including storing a graph database representing a configuration of a cloud environment, obtaining configuration settings representing a current state of the cloud environment from a cloud platform, and identifying a delta based on changes between the configuration from the graph database and the configuration settings from the cloud platform. The method may further comprise creating an asset property node based on the delta, and adding the asset property node to the graph database without creating a new graph based on the configuration settings.
Systems and methods for performing journal swapping are provided. In one example, the backing storage used for virtual non-volatile random access memory (vNVRAM) is dynamically switched based on the high-availability (HA) state of an HA pair of nodes of a virtual storage system. By default, in-memory logging may be used for write journaling when the HA pair is operating in a normal HA state. When the HA pair is in an HA degraded state, the write journaling may be performed to local ephemeral storage. This allows the file system of the virtual storage system to run in a more performant configuration most of the time (e.g., during which HA is enabled and healthy). At the same time, the file system has the ability to swap to a less performant but more resilient configuration during planned events (e.g., scheduled maintenance and throughput scaling) in which HA is in a degraded state.
Systems and methods for performing journal swapping are provided. In one example, the backing storage used for virtual non-volatile random access memory (vNVRAM) is dynamically switched based on the high-availability (HA) state of an HA pair of nodes of a virtual storage system. By default, in-memory logging may be used for write journaling when the HA pair is operating in a normal HA state. When the HA pair is in an HA degraded state, the write journaling may be performed to local ephemeral storage. This allows the file system of the virtual storage system to run in a more performant configuration most of the time (e.g., during which HA is enabled and healthy). At the same time, the file system has the ability to swap to a less performant but more resilient configuration during planned events (e.g., scheduled maintenance and throughput scaling) in which HA is in a degraded state.
G06F 11/20 - Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
47.
Dynamic Management Of Node Headroom To Accommodate Startup Spikes
The disclosure describes a node management service that dynamically adjusts node capacity in orchestration platforms. The service leverages an extended resource definition to reflect the increased resource needs during the startup phase of workloads, then reallocates these resources once the workload transitions to steady-state operation. Workloads are provisioned with both a native resource request and an extended resource request to cover initialization spikes. Once initialization completes, the node management service updates the node's metadata to increase available extended resource capacity, thereby freeing up headroom for future deployments.
Systems and methods for selectively bypassing an external cache (EC) of a storage system are provided. In one example, when the EC backing storage device is saturated, reads bypass the EC and are completed via a redundant array of independent disks (RAID) subsystem of the storage system. One or more performance metrics for the EC backing storage device may be monitored to predict one or more saturation thresholds (e.g., in terms of latency and/or throughput). Based on this monitoring, tuning may be performed to drive utilization of the EC into a “knee region” of a performance (or response) curve of the EC backing storage device. For example, the depth of one or more request queues at the front-end of an EC lookup may be manipulated based on a current measure of saturation relating to the EC backing storage device to limit the number of in-flight reads pending for the EC.
Systems and methods for selectively bypassing an external cache (EC) of a storage system are provided. In one example, when the EC backing storage device is saturated, reads bypass the EC and are completed via a redundant array of independent disks (RAID) subsystem of the storage system. One or more performance metrics for the EC backing storage device may be monitored to predict one or more saturation thresholds (e.g., in terms of latency and/or throughput). Based on this monitoring, tuning may be performed to drive utilization of the EC into a “knee region” of a performance (or response) curve of the EC backing storage device. For example, the depth of one or more request queues at the front-end of an EC lookup may be manipulated based on a current measure of saturation relating to the EC backing storage device to limit the number of in-flight reads pending for the EC.
A method, a computing device, and a non-transitory machine-readable medium for detecting personal information. Terms that are of interest are extracted from a corpus of raw text that has been extracted from a collection of documents. For each of the terms, a surrounding sentence is extracted to form a target sentence to thereby form a plurality of target sentences. The surrounding sentence includes at least one reference to a data subject. A matrix of feature information is generated for each of the target sentences to form a plurality of matrices. A neural network model is trained, using the matrices as input, to compute an output that indicates a likelihood of a given sentence containing personal information.
According to an example, a computer-implemented method comprises initiating a first process for atomically setting the primary bias state with a first node of a primary storage cluster of a multi-site distributed storage system due to a temporary loss of connectivity to a mediator or a temporary mediator failure, releasing an atomic lock for the first process on the first node of the primary storage cluster, sending the first process and an associated first generation indicator to a first node of a secondary storage cluster of the multi-site distributed storage system to handle the first process for setting the primary bias state, and initiating a second process for atomically clearing a primary bias state with the first node or any node of the primary storage cluster based on detecting a connection to the mediator or detecting that the mediator is available.
Various embodiments of the present technology generally relate to systems and methods for providing cross-grid replication within distributed storage systems. In an example, a method includes identifying an object for ingest into a first storage grid containing a first distributed storage system and replicating the object to one or more nodes within the first storage grid. The method may also include determining a cross-grid replication status of the object to a second storage grid containing a second distributed storage system and performing a cross-grid replication of the object to the second storage grid based on the cross-grid replication status of the object.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database systemDistributed database system architectures therefor
Systems and methods for enhancing container security are provided. In one example, exposure of a containerize application to potential security vulnerabilities is reduced by identifying dynamically loaded symbols by the application via performance of static and/or dynamic symbol analysis to identify dynamically loaded symbols that are potentially and/or actually used, respectively, and that correspond to functions contained within shared libraries. Based on a shared library's usage of functions within a standard library and a known mapping between functions of the standard library and system calls, those system calls potentially and actually accessed by the application may be identified and a security policy may be generated and configured for enforcement by a kernel security module to limit system call usage accordingly. Additionally, removal of files or functions of libraries that are deemed unnecessary for proper execution of the applications may be performed to reduce the footprint of the application.
G06F 21/57 - Certifying or maintaining trusted computer platforms, e.g. secure boots or power-downs, version controls, system software checks, secure updates or assessing vulnerabilities
G06F 8/75 - Structural analysis for program understanding
G06F 21/51 - Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems at application loading time, e.g. accepting, rejecting, starting or inhibiting executable software based on integrity or source reliability
G06F 21/56 - Computer malware detection or handling, e.g. anti-virus arrangements
The disclosure describes systems, devices, and methods for replicating metadata in data storage environments. In an example embodiment, a method for operating a controller in a data storage environment to provide cross-shelf data replication is provided. In performing the method, the controller generates metadata for an input/output (I/O) request upon receiving the I/O request. The controller identifies a primary location at which to store the metadata and identifies a secondary location at which to store a replicated version of the metadata, then stores the metadata at the primary location and the replicated version at the secondary location. The primary and secondary locations correspond to storage devices in the data storage environment located on different physical shelves or enclosures, such that the metadata is stored across multiple drive shelves for redundancy and recovery purposes.
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
G06F 11/20 - Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
55.
MAINTAINING TIMESTAMP PARITY OF OBJECTS WITH ALTERNATE DATA STREAMS DURING TRANSITION PHASE TO SYNCHRONOUS STATE
Techniques are provided for maintaining timestamp parity during a transition replay phase to a synchronous state. During a transition logging phase where metadata operations executed by a primary node are logged into a metadata log and regions modified by data operations executed by the primary node are tracked within a dirty region log, a close stream operation to close a stream associated with a basefile of the primary node is identified. A determination is made as to whether the dirty region log comprises an entry for the stream indicating that a write data operation previously modified the stream. In an example, in response to the dirty region log comprising the entry, an indicator is set to specify that the stream was deleted by the close stream operation. In another example, a modify timestamp of the basefile is logged into the metadata log for subsequent replication to the secondary node.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database systemDistributed database system architectures therefor
56.
DEFRAGMENTATION FOR LOG STRUCTURED MERGE TREE TO IMPROVE READ AND WRITE AMPLIFICATION
Techniques are provided for implementing a defragmentation process during a merge operation performed by a re-compaction process upon a log structured merge tree. The log structured merge tree is used to store keys of key-value pairs within a key-value store. As the log structured merge tree fills with keys over time, the re-compaction process is performed to merge keys down to lower levels of the log structured merge tree to re-compact the keys. Re-compaction can result in fragmentation because there is a lack of spatial locality of where the re-compaction operations re-writes the keys within storage. Fragmentation increases read and write amplification when accessing the keys stored in different locations within the storage. Accordingly, the defragmentation process is performed during a last merge operation of the re-compaction process in order to store keys together within the storage, thus reducing read and write amplification when accessing the keys.
METHODS AND SYSTEMS FOR A NON-DISRUPTIVE AUTOMATIC UNPLANNED FAILOVER FROM A PRIMARY COPY OF DATA AT A PRIMARY STORAGE SYSTEM TO A MIRROR COPY OF THE DATA AT A CROSS-SITE SECONDARY STORAGE SYSTEM
Multi-site distributed storage systems and computer-implemented methods are described for providing an automatic unplanned failover (AUFO) feature to guarantee non-disruptive operations (e.g., operations of business enterprise applications, operations of software application) even in the presence of failures including, but not limited to, network disconnection between multiple data centers and failures of a data center or cluster.
G06F 11/20 - Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
G06F 11/07 - Responding to the occurrence of a fault, e.g. fault tolerance
Various embodiments of the present disclosure relate to compression techniques that can be used for data storage, indexing, and retrieval. In an example embodiment, a controller of or in communication with a data storage system can obtain a request to store a file in the data storage system. The controller identifies time-series data associated with input/output of the file and metrics of the file, generates a feature vector for the file, and compresses the feature vector using a compression technique, resulting in a compressed feature vector having a compression ratio. The controller performs a hash algorithm using the compression ratio to determine a storage device at which to store the compressed feature vector, the compression ratio associated with the file, and an indication of the compression technique. Upon receiving a query for a file, the controller compares compression ratios to identify a storage device storing the requested file.
The disclosure describes a system for enforcing role-based access control in a multi-tenant compute cluster with cross-namespace references. A control plane of the compute cluster receives a custom-resource request from a tenant to create or modify a first custom resource in a tenant namespace. The first custom resource references if a second custom resource in an administrative namespace. In response to the request, the control plane transmits a validating admission request to a data-protection controller registered as a webhook endpoint for admission validation. The data-protection controller retrieves access metadata from the referenced second custom resource and generates an admission determination indicating whether the tenant's request satisfies cross-namespace access conditions defined in the metadata. The controller returns the admission determination to the control plane, which admits or denies the custom-resource request accordingly.
G06F 21/62 - Protecting access to data via a platform, e.g. using keys or access control rules
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
Techniques are provided for forwarding operations to bypass persistent memory. A modify operation, targeting an object, may be received at a persistent memory tier of a node. If a forwarding policy indicates that forwarding is not enabled for the modify operation and the target object, then the modify operation is executed through a persistent memory file system. If the forwarding policy indicates that forwarding is enabled for the modify operation and the target object, then the modify operation is forwarded to a file system tier as a forwarded operation for execution through a storage file system.
Techniques are provided for combining data block and checksum block I/O into a single I/O operation. Many storage systems utilize checksums to verify the integrity of data blocks stored within storage devices managed by a storage stack. However, when a storage system reads a data block from a storage device, a corresponding checksum must also be read to verify integrity of the data in the data block. This results in increased latency because two read operations are being processed through the storage stack and are being executed upon the storage device. To reduce this latency and improve I/O operations per second, a single combined I/O operation corresponding to a contiguous range of blocks including the data block and the checksum block is processed through the storage stack instead of two separate I/O operations. Additionally, I/O operation may be combined into a single request that is executed upon the storage device.
Techniques are provided for parsing files using node objects within a pool. Conventional techniques for parsing files, such as extensible markup language log files, may not efficiently parse the files, and thus cannot keep up with a rate at which the files are generated and/or populated (e.g., a storage system may generate a significant amount of log data stored in log files over time). The disclosed parsing technique is capable of more efficiently processing the files utilizing less memory and time. In particular, the files are parsed by threads that use node objects within a pool of memory (e.g., a sync pool) to store data being parsed and processed by the threads in parallel. When a thread finishes using a node object, the node object is cleared and returned to the pool for subsequent use by the thread or a different thread, which is memory efficient.
An object storage system with enhanced processes for managing replicas of data objects. Storage nodes of the object storage system receive replicas of objects to store. In response, a storage node identifies one of multiple virtual object space and its corresponding placement group to associate with the replica of the object. The storage node identifies a storage device associated with the virtual object space, stores the replica on the storage device, and updates system metadata to associate the placement group with the replicated object. In response to a restoration event for a storage device of the system, a storage node identifies associated virtual object spaces and placement groups for the effected storage device. Using the identified virtual object spaces and placement groups, the storage node identifies the objects needing restoration and restores replicas of the objects to a different storage device of the system.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database systemDistributed database system architectures therefor
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
64.
Migrating data using volume clones in a distibuted storage system
Systems and methods are provided for data migration including cloning a multiple-logical unit number (LUN) volume into a plurality of cloned single-LUN volumes on a first storage node of a computing system; creating a plurality of new volumes on a second storage node of the computing system based at least in part on the plurality of cloned single-LUN volumes; selectively copying snapshot data from the plurality of cloned single-LUN volumes to the plurality of new volumes; establishing snapshot mirror relationships between the plurality of cloned single-LUN volumes and the plurality of new volumes; synchronizing the plurality of cloned single-LUN volumes and the plurality of new volumes; and performing a migration of data logical interface failovers (LIFs) from the plurality of cloned single-LUN volumes to the plurality of new volumes.
G06F 11/20 - Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
Dynamic distribution of compression and decompression job requests to hardware accelerators is disclosed. A set of requests is evaluated to determine a number of compression jobs and a number of decompression jobs in the set of requests. A first set of hardware accelerator engines is allocated to perform compression jobs, and a second set of hardware accelerator engines is allocated to perform decompression jobs. Compression jobs are assigned to the first set of hardware accelerator engines based, at least in part, on a compressibility score of the corresponding job and a workload of the selected hardware accelerator engine. Decompression jobs are assigned to the second set of hardware accelerator engines based, at least in part, on a decompression weight of the corresponding job and a workload of the selected hardware accelerator engine.
A global policy-driven framework for managing, revising, and implementing data coherency policies in a distributed data storage system. An edge node of the distributed data storage system receives a request from an application to store a data object in the system. The edge node, in response to receiving the request to store the data object, generates a coherency policy request, which is then submitted to the primary node of the system. The primary node of the system generates a coherency policy response, which is returned to the edge node. The edge node then stores the data object in the system in accordance with the coherency policy response.
A global policy-driven framework for managing, revising, and implementing data coherency policies in a distributed data storage system. An edge node of the distributed data storage system receives a request from an application to store a data object in the system. The edge node, in response to receiving the request to store the data object, generates a coherency policy request, which is then submitted to the primary node of the system. The primary node of the system generates a coherency policy response, which is returned to the edge node. The edge node then stores the data object in the system in accordance with the coherency policy response.
H04L 67/1097 - Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
68.
Adaptive Request Grouping Based On Node Pool Impact
The disclosure describes a node management service that groups pods based on an impact of the available instance pool. The node management service identifies a request group associated with a scale-up request to scale up a cluster of compute nodes to host pods in the request group. The node management service iteratively determines to add pods to the request group until an impact of a next pod on a pool of available nodes exceeds a threshold. The node management service sends a request to a distributor to distribute the pods in the request group to one or more nodes obtained from the pool of available nodes.
G06F 9/50 - Allocation of resources, e.g. of the central processing unit [CPU]
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
69.
MANAGING SHUTDOWN AND RESET OF A NETWORK INTERFACE CARD (NIC)
Managing shutdown and reset of a network interface card (NIC) in response to an error condition is disclosed. An indication to initiate a network interface card (NIC) reset and reconnection sequence is received. A notification of a link down condition is transmitted. Pending connections are disconnected. Queue pairs corresponding to the interconnect channels are destroyed. Links corresponding to the NIC are disconnected. Packets are cleared from queues corresponding to the NIC. Send and receive queues are reset. Queue pairs corresponding to the NIC are recreated. Queue pairs are connected to corresponding links. Data transfer resumes over the links.
The disclosure describes a system for managing a namespace move between nodes of a data storage environment. During a namespace move associated with a data storage system, the system receives requests at a first node (e.g., a source node) in the data storage system to perform input/output (I/O) operations associated with a namespace subject to the namespace move and stores the requests in a queue at the first node until completion of the namespace move. Upon completion of the namespace move, the system forwards the requests from the first node to a second node (e.g., a destination node) in the data storage system and performs the I/O operations at the second node.
Techniques for efficiently and durably implementing erasure coding. Data objects are processed to extract metadata, which is used to identify fragments of the data object and the stripe to which the fragments belong. The techniques described herein evaluate failure domains at the drive-level rather than at the node-level, thereby greatly expanding the number of failure domains for fragment storage. To support object availability and durability, each drive-level failure may only hold one fragment from any given stripe. Further, the storage drives of each storage node are restricted such that the total number of fragments from any given stripe stored in the storage node does not exceed a limit. With this arrangement, not only can erasure coding can be implemented while using fewer computing resources when compared with node-level failure domains, but data objects can also be reconstructed without compromising data integrity under configurable levels of tolerance to unavailable fragments.
The disclosure describes a node management service that determines to deploy a primary workload to a new instance in a compute cluster. The node management service projects an allocatable capacity for the new instance based on historical capacity data. The allocatable capacity is the amount of compute resources available for running the primary workload in the new instance after deployment of a supporting workload to the new instance. The node management service associates the primary workload with the new instance upon determining that the projected allocatable capacity is sufficient for running the primary workload.
Techniques are provided for a get key bot that securely provides access to passphrases. A set key workflow is executed to generate a get key executable binary that is implemented as the get key bot. The get key executable binary is encapsulated with encrypted information that includes a verified user identifier, a passphrase, and/or a bot expiry time. Upon receiving a request for the passphrase from a requestor, the get key executable binary is invoked. The encrypted information is decrypted and compared to a logged-in user identifier and current time for verification. In response to successful verification, the passphrase is provided to the requestor. Otherwise, the requestor is denied access to the passphrase.
Backup of application data associated with an application executing in a virtual machine managed by a hypervisor is performed. Backup of the application data includes retrieving a Logical Unit Number (LUN) identification (ID) used by the application to store the application data in a storage volume. Backup of the application data also includes performing a virtual storage resolution for the LUN ID to determine whether the application data is stored in the storage volume identified by the LUN ID based on a first virtual mapping or a physical mapping.
Backup of application data associated with an application executing in a virtual machine managed by a hypervisor is performed. Backup of the application data includes retrieving a Logical Unit Number (LUN) identification (ID) used by the application to store the application data in a storage volume. Backup of the application data also includes performing a virtual storage resolution for the LUN ID to determine whether the application data is stored in the storage volume identified by the LUN ID based on a first virtual mapping or a physical mapping.
Backup also includes storing in metadata for the backup the LUN ID and whether the LUN ID is based on the first virtual mapping or the physical mapping. Backup includes creating a backup of the application data stored in the storage volume. Application data can subsequently be restored based on the application data that is backed up.
The disclosure describes a node management service that integrates right-sizing and node scaling operations. The node management service modifies a request parameter for a workload deployed in a compute cluster. The node management service determines an updated set of compute nodes for nodes affected by the updated request parameter. The node management service obtains, from a compute provider, the updated set of compute nodes for the compute cluster. The node management service provides the modified request parameter to a control plane after obtaining the updated set of compute nodes.
In one example, a computer-implemented method includes establishing bi-directional synchronous replication between one or more members of a first consistency group (CG1) of a primary storage site and one or more members of a second consistency group (CG2) of a secondary storage site with each storage site having read/write access while maintaining zero recovery point objective (RPO) and Zero recovery time objective (RTO), initiating a non-disruptive planned failover (PFO) to change a role for the secondary storage site and change a role for the primary storage site, initiating, with the primary storage site and/or secondary storage site, a PFO out of synchronization (OOS) event that is sent to a mediator agent with no indication of a role for serving IO, and starting a configuration independent unplanned failover if a disaster or site failure occurs during the PFO.
G06F 11/16 - Error detection or correction of the data by redundancy in hardware
G06F 11/20 - Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
77.
SYSTEMS AND METHODS FOR NON-DISRUPTIVE PLANNED FAILOVER WITHIN A CROSS-SITE STORAGE SYSTEM HAVING BIDIRECTIONAL SYNCHRONOUS REPLICATION
In one example, the present storage solution provides an order of operations of a computer-implemented method that includes establishing bi-directional synchronous replication between one or more members of a first consistency group (CG1) of a primary storage site and one or more members of a second consistency group (CG2) of a secondary storage site with each storage site having read/write access while maintaining zero recovery point objective (RPO) and Zero recovery time objective (RTO). The method includes initiating a non-disruptive planned failover to change a role for the secondary storage site and change a role for the primary storage site while maintaining in sync status of the bi-directional synchronous replication between the one or more members of the CG1 of the primary storage site and the one or more members of the CG2 of a secondary storage site, and while maintaining zero data loss protection.
G06F 11/20 - Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
Techniques are provided for performing adaptive sampling for data summarization. An insight service may provide monitoring, troubleshooting, optimization, security, and/or other functionality for a computing environment. The insight service may intake millions to billions of events on a monthly basis from the computing environment, which are stored within a database. The insight service may provide data summarization for the events, which may include access patterns (e.g., file access patterns), anomalies, and ransomware detection. Dynamically querying and generating the data summarization may be impractical due to the sheer amount of events. Accordingly, adaptive sampling is provided for merely sampling certain events based upon various thresholds and criteria being met so that an evaluation output can be dynamically and efficiently generated within an acceptable time as the data summarization.
Various embodiments of the present technology generally relate to systems and methods for providing a replication engine for preserving discontiguous and fragmented compressed data extents (CDEs). In an aspect, a replication engine may determine a replication request to replicate one or more data blocks from a source storage system to a destination storage system. Based on the replication request, the replication engine may determine a first CDE containing the one or more data blocks. The replication engine may also determine a transfer map associated with the replication request. Based on the transfer map, the replication engine may determine a replication state associated with the one or more data blocks and initiate replication of the first CDE from the source storage system to the destination storage system based on the replication state associated with the one or more data blocks.
Techniques are provided for incremental backup to an object store. A request may be received from an application to perform a backup from a volume hosted by a node to a backup target within the object store. A set of changed files within the volume since a prior backup of the volume was performed to the backup target is identified, along with metadata associated with the set of changed files. The metadata is utilized to identify changed data blocks comprising data of the set of changed files that was modified since the prior backup. The changed data blocks are backed up to the object store.
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
A system is described including one or more processing resources and a non-transitory computer-readable medium, coupled to the processing resource, having stored therein instructions that when executed by the one or more processing resources cause the one or more processing resources to establish a remote direct memory access (RDMA) connection with a client computer system and remotely access a crash dump file stored at the client computer system, including providing a file offset of interest and size of data to be read from the client computer system and translating the file offset of interest and size of data into a plurality of RDMA messages.
A system is described including one or more processing resources and a non-transitory computer-readable medium, coupled to the processing resource, having stored therein instructions that when executed by the one or more processing resources cause the one or more processing resources to receive a request from a remote computer system to initiate a remote direct memory access (RDMA) connection, establish the RDMA connection with the remote computer system and provide access to a crash dump file via the RDMA connection.
H04L 67/1097 - Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Systems and methods for implementation and use of remote clone volumes within a distributed storage system are provided. In one of various contemplated examples, all nodes of multiple nodes of a cluster representing a distributed storage system are able to access an entirety of a global physical volume block number (PVBN) space of a storage pod. A remote clone volume of a parent volume of a source node may be created for use by a destination node by creating a dummy volume on the destination node. The dummy volume may then be converted into the remote clone volume by copying metadata associated with a backing snapshot of the parent volume to the dummy volume. After completing creation of the remote clone volume, the backing snapshot may then be locked to protect the backing snapshot from deletion.
Systems, methods, and software are disclosed herein for identifying duplicate blocks of a storage system and deduplicating the storage system. In one example, a method of operating a computing device includes scanning first metadata of blocks of a container file of a virtual volume to generate a first log file including records of virtual volume block numbers (VVBNs) and fingerprints of the blocks; scanning second metadata of blocks of an active file system of the virtual volume to generate a second log file including records of VVBNs and file block numbers (FBNs) of the blocks; generating tuples based on merging the records of the first log file and the second records of the second log file according to the VVBNs; identifying duplications among the blocks based on the tuples; and deduplicating the blocks based on the duplications in the active file system identified based on the tuples.
The disclosure describes system, devices, and methods for fan speed control. In an example implementation, a method for operating a computer-implemented service is provided. The method includes obtaining sensor data from one or more sensors in a data storage environment. The sensor data includes temperature data associated with storage devices in the data storage environment. The method also includes providing an input (e.g., the sensor data) to a machine learning model trained to predict fan control settings of a fan in the data storage environment, determining the fan control setting based on an output from the machine learning model, and controlling the fan based on the fan control setting.
The disclosure describes system, devices, and methods for fan speed control. In an example implementation, a method for operating a computer-implemented service is provided. The method includes obtaining sensor data from one or more sensors in a data storage environment. The sensor data includes temperature data associated with storage devices in the data storage environment. The method also includes providing an input (e.g., the sensor data) to a machine learning model trained to predict fan control settings of a fan in the data storage environment, determining the fan control setting based on an output from the machine learning model, and controlling the fan based on the fan control setting.
Various embodiments of the present technology generally relate to systems and methods for providing a data preparation engine for curating secure and compliant data collections from distributed storage systems. In an aspect, a data preparation engine receives a query from a client device and determines files from one or more distributed sources based on the query. The data preparation engine determines sensitive data within the files and anonymizes the sensitive data while preserving context and integrity of the underlying information. The data preparation engine generates a data collection including the files with anonymized sensitive data. The data collection may then be deployed to downstream applications or workflows, such as used to generate curated data sets for training of artificial intelligence applications. Once deployed, the data preparation engine may continuously monitor the distributed sources for changes to data within the files and automatically update data collections in real-time.
Systems and methods are described for performing an instant recovery of data associated with a locked snapshot. In various examples, the amount of time for performing a recovery of data associated with a locked snapshot is performed by making use of volume cloning functionality instead of making an actual copy of the data to be recovered. In one embodiment, the resulting volume clone representing the recovery volume is cleared of all data protection information (e.g., WORM flags and/or lock metafiles) that was previously used to protect the content from being changed when stored on the data protection volume so as allow the recovery volume to be used in read-write mode.
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
G06F 16/176 - Support for shared access to filesFile sharing support
A liveness monitoring architecture to deterministically ascertain the functional status of nodes in a high-availability (HA) environment to more efficiently recover from a split-brain condition is disclosed.
A method, system and computer program product, the method comprising: determining properties of a set of containers that are deployed over a computer infrastructure, wherein the computer infrastructure is provisioned via an infrastructure management service; determining properties of one or more headroom containers, wherein the one or more headroom containers are not deployed over the computer infrastructure; simulating the container orchestrator using the properties of the set of container and the properties of the headroom containers, for obtaining an expected deployment of the set of containers together with the one or more head room containers; based on the expected deployment, determining whether the computer infrastructure is sufficient for deploying the set of containers together with the one or more headroom containers; and subject to the computer infrastructure being insufficient, issuing a request to the infrastructure management service to allocate additional computer infrastructure.
The disclosure describes artificial intelligence (AI) data platform that utilizes snapshots obtained from a storage node to update a vector database. The AI data platform compares snapshots to generate differential snapshots that identify changed data in storage volumes. The AI data platform uses the differential snapshots to update vector embeddings in a vector database for retrieval-augmented generation (RAG) workflows.
37 - Construction and mining; installation and repair services
41 - Education, entertainment, sporting and cultural services
42 - Scientific, technological and industrial services, research and design
Goods & Services
Installation, maintenance, and repair of computer hardware for storing, managing, replicating, transferring, securing, retrieving, and restoring data and files; installation, maintenance, and repair of computer hardware for sharing, updating, partitioning, and accessing files over a computer network; installation, maintenance, and repair of computer hardware for optimizing the accessibility, delivery, backup, security, restoration, and replication of data; installation, maintenance, and repair of computer hardware for managing, monitoring, and securing networks, computer disc drives, electronic data storage systems, databases and other storage devices across computer networks; installation, maintenance, and repair of computer hardware using artificial intelligence for the purpose of managing business data; consulting services related to installation, repair, and maintenance of computer hardware and artificial intelligence (AI) in the fields of artificial intelligence, computers, computer software and hardware, computer storage networks, computer storage devices, data and computer file storage, management, replication, security, retrieval, restoration and distribution, and management, monitoring and security of computer networks, computer disc drives, electronic data storage systems, databases and other storage devices across computer networks Educational services, namely, arranging and conducting training courses, lectures, classes, workshops, seminars and conferences in the fields of artificial intelligence, computers, computer software and hardware, computer storage networks, computer storage devices, data and computer file storage, management, replication, security, retrieval, restoration and distribution, and management, monitoring and security of computer networks, computer disc drives, electronic data storage systems, databases and other storage devices across computer networks; providing educational testing to determine professional skills in the fields of computers, computer software and hardware, computer storage networks, computer storage devices, data and computer file storage, management, replication, security, retrieval, restoration and distribution, and management, monitoring and security of computer networks, computer disc drives, electronic data storage systems, databases and other storage devices across computer networks Providing online and cloud-based non-downloadable software for monitoring, securing, optimizing, analyzing, and managing electronic and cloud data storage, computer servers, databases, and other storage devices and systems across computer networks; technical support services, namely, monitoring and troubleshooting for problems with computers, computer software and hardware and computer storage networks, and providing back-up computer programs and facilities; software as a Service (SAAS) services featuring software for management, monitoring, modeling, troubleshooting, optimization, reporting and analysis of network storage infrastructure and data; cloud computing featuring software for use in controlling, monitoring, and managing cloud infrastructure, virtual services, and networking configuration; design, deployment and management of software and hardware using artificial intelligence for the purpose of managing business data; technical consulting in the field of artificial intelligence (AI) software customization; technical consulting in the fields of artificial intelligence, computers, computer software and hardware, computer storage networks, computer storage devices, data and computer file storage, management, replication, security, retrieval, restoration and distribution, and management, monitoring and security of computer networks, computer disc drives, electronic data storage systems, databases and other storage devices across computer networks; artificial Intelligence as a Service (AIAAS) services featuring software using artificial intelligence (AI) for use in database management; providing online and cloud-based non-downloadable software for deploying, implementing, monitoring, securing, optimizing, analyzing, storing, managing, and troubleshooting artificial intelligence (AI) platforms and internal processes; providing temporary use of online non-downloadable software for use in connection with data management, data storage, data protection, security, and compliance; providing temporary use of online non-downloadable software for managing, accessing, modifying, updating, organizing, delivering, synchronizing, processing, transmitting, storing, and restoring data
93.
SYSTEM AND METHOD FOR EFFICIENT BLOCK LEVEL GRANULAR REPLICATION
A system and method for efficiently restoring one or more data containers is provided. A common persistent consistency point image (PCPI) is identified between a source and a destination storage systems prior to the destination storage system performing a rollback operation to the commonly identified PCPI. Differential data is then transmitted from the source storage system in a line efficient manner to the destination storage system.
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
G06F 16/11 - File system administration, e.g. details of archiving or snapshots
94.
SERVERLESS TIEBREAKER FOR SHARED-NOTHING ARCHITECTURE
Systems and methods for a serverless tiebreaker for a shared-nothing architecture are provided. In some examples, a cloud-native service that supports serialization of writes (or write fencing), for example, via atomic operations with persistent locking and/or reservations, is used to support HA mediation instead of a separate server operating as a tiebreaker, thereby reducing costs and complexity as well as increasing availability and durability of the HA mediation functionality. For example, a fast, fully managed, serverless, key-value noSQL database service (e.g., the Amazon DynamoDB) may be used to perform one or more of maintaining the authoritative source of information regarding which node of an HA pair currently represents the primary node for serving data from a particular dataset, persisting HA metadata, and/or assisting in the failover and failback processes.
H04L 67/1095 - Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
H04L 67/1097 - Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
95.
Methods and Systems for Raid Protection in Zoned Solid-State Drives
Methods and systems for a storage environment are provided. One method includes splitting storage of a plurality of zoned solid-state drives (ZNS SSDs) into a plurality of physical zones (PZones) across a plurality of independent media units of each ZNS SSD, the PZones visible to a first tier RAID (redundant array of independent disks) layer; generating a plurality of RAID zones (RZones), each RZone having a plurality of PZones; presenting one or more RZones to a second tier RAID layer by the first tier RAID layer for processing read and write requests using the plurality of ZNS SSDs; and utilizing, by the first tier RAID layer, a parity PZone at each ZNS SSD for storing parity information corresponding to data written in one or more PZone corresponding to a RZone presented to the second tier RAID layer and storing the parity information in a single parity ZNS SSD.
Systems and methods for providing a file system with object versioning support are provided. Rather than adding object records for each version of an object to a chapter database, in one example, the chapter database may be limited to a single object record for a given object including: (i) a name of the object; (ii) an object file handle containing information regarding a file containing data of a current version of multiple versions of the object; and (iii) a version table file handle containing information regarding a file containing a version table. In this manner, enumeration of objects associated with a given chapter may be performed more efficiently and prior versions of objects may be maintained separately within the version table without causing disproportionate growth of object records and without increasing the search depth with objects that are not referenced by the search at issue.
Techniques are provided for providing a storage abstraction layer for a composite aggregate architecture. A storage abstraction layer is utilized as an indirection layer between a file system and a storage environment. The storage abstraction layer obtains characteristic of a plurality of storage providers that provide access to heterogeneous types of storage of the storage environment (e.g., solid state storage, high availability storage, object storage, hard disk drive storage, etc.). The storage abstraction layer generates storage bins to manage storage of each storage provider. The storage abstraction layer generates a storage aggregate from the heterogeneous types of storage as a single storage container. The storage aggregate is exposed to the file system as the single storage container that abstracts away from the file system the management and physical storage details of data of the storage aggregate.
Techniques are provided for compressing weights of models during training of the models. A model is trained for execution on a target device. As part of training, weights of the model are compressed utilizing palettes to represent weight values using bits. A coding procedure, such as Huffman coding, is used to remove or modify the bit representations of infrequently utilized palettes. The model may be iteratively trained to compress the weights of the model in order to reduce the amount of storage consumed by the model without unduly sacrificing quality of the model. Reducing the size of the model provides the ability to deploy the model on devices that would otherwise lack storage and compute resources for storing and running an uncompressed version of the model.
The disclosure describes system, devices, and methods for dual-stage vector search. In an example implementation, a method for operating a computer-implemented service is provided. The method includes receiving a context request for content with which to augment a prompt, generating a base vector based on input data in the context request and quantizing the base vector to produce a quantized vector. The method also includes searching a vector database to identify content items based at least on the quantized vector and obtaining the content items and generating base vectors for the content items. The method further includes selecting a subset of the content items based on at least on the base vector generated for the input data and the base vectors for the content items.
The disclosure describes systems, devices, and methods for managing data storage environments. In an example implementation, a method of operating a controller in a data storage environment is provided. In performing the method, the controller identifies a change to a layout of drives in the data storage environment, and in response to identifying the change, takes a lock on instances of layout metadata stored on the drives. The controller then updates the instances of the layout metadata to reflect the change to the layout and releases the lock.
