The present application discloses a method, system, and computer system for building a model associated with a dataset. The method includes receiving a data set, the dataset comprising a plurality of keys and a plurality of key-value relationships, determining a plurality of models to build based at least in part on the dataset, wherein determining the plurality of models to build comprises using the dataset format information to identify the plurality of models, building the plurality of models, and optimizing at least one of the plurality of models.
G06F 18/2132 - Feature extraction, e.g. by transforming the feature spaceSummarisationMappings, e.g. subspace methods based on discrimination criteria, e.g. discriminant analysis
A data processing service performs a rebalancing process for rebalancing stateful tasks on a cluster computing system. In one instance, the method for rebalancing stateful tasks is performed such that the per-operator partitions are spread across available executors of a cluster of the cluster computing system with respect to one or more statistics of the tasks. In one instance, the method for rebalancing stateful tasks is also performed such that the total number of stateful tasks are balanced per executor as long as this rebalancing does not imbalance the per-operator placements. In this way, the processing of stateful tasks can be spread across multiple executors in a relatively uniform manner, even though there may be an upfront cost of breaking the local caching on an executor.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database systemDistributed database system architectures therefor
A system performs efficient startup of executors of a distributed computing engine used for processing queries, for example, database queries. The system starts an executor node and processes a set of queries using the executor node to warm up the executor node. The system performs a checkpoint of the warmed-up executor node to create an image. The image is restored in the target executor nodes. The system may store a checkpoint image for each configuration of an executor node. The configuration is determined based on various factors including the hardware of the executor node, memory allocation of the processes, and so on. The user or restore based on checkpoint images improves efficiency of execution of the startup of executor nodes.
The disclosed configurations provide a method (and/or a computer-readable medium or system) for determining, from a table schema describing keys of a data table, one or more clustering keys that can be used to cluster data files of a data table. The method includes generating features for the data table, generating tokens from the features, generating a prediction for each token by applying to the token a machine-learned transformer model trained to predict a likelihood that the key associated with the token is a clustering key for the data table, determining clustering keys based on the predictions, and clustering data records of the data table into data files based on key-values for the clustering keys.
A device for using message identifiers for Publish/subscribe messaging deduplication is described. The system may fetch one or more sets of data records from a data source, and each data record is associated with a message identifier. The system may store the one or more sets of data records in a data file, which is associated with a metadata comprising the message identifier, a file path and a row number for each data record. The system may determine whether one or more of the data records are duplicated based on the associated message identifiers. In response to determining that the one or more data records are duplicated, the system may generate a second metadata comprising the file paths and row numbers associated with the duplicated data records.
A system includes an interface, a processor, and a memory. The interface is configured to receive a version of a model from a model registry. The processor is configured to store the version of the model, start a process running the version of the model, and update a proxy with version information associated with the version of the model, wherein the updated proxy indicates to redirect an indication to invoke the version of the model to the process. The memory is coupled to the processor and configured to provide the processor with instructions.
A data processing service receives a request from a first collaborator to create a clean room for data sharing collaboration with at least a second collaborator. In response, the data processing service creates an execution environment separate from the data environment of the first collaborator and the second collaborator. The first and second collaborators can then add content into the clean room in the form of data tables and executable notebooks. Approval from each collaborator is required before a notebook can be executed using any data table shared into the clean room. Upon receiving notebook approval from each collaborator, the data processing service creates a notebook job to execute the notebook on one or more cluster computing resources of the data processing service to generate an output.
A method, system, and computer system for performing an operation with respect to a target table are disclosed. The method includes performing first and second jobs, obtaining one or more other resulting files based at least in part on unmatched rows, and obtaining a set of processed files based at least in part on performing a post-processing operation with respect to the set of resulting files. The set of processed files has less files than the set of resulting files. Performing the first job includes determining a set of matching target table files and storing target table information indicating for each of the set of matching target table files, a particular set of rows having matching rows. Performing the second job includes performing a matching action based on matched rows and obtaining the second job resulting file(s).
A data processing service generates a data classifier tree for managing data files of a data table. The data classifier tree may be configured as a KD-classifier tree and includes a plurality of nodes and edges. A node of the data classifier tree may represent a splitting condition with respect to key-values for a respective key. A node of the data classifier tree may be associated with one or more data files assigned to the node. The data files assigned to the node each include a subset of records having key-values that satisfy the conditions represented by the node and parent nodes of the node. The data processing service may efficiently cluster the data in the data table while reducing the number of data files that are rewritten when data is modified or added to the data table.
A data tree for managing data files of a data table and performing one or more transaction operations to the data table is described. The data tree is configured as a KD-epsilon tree and includes a plurality of nodes and edges. A node of the data tree may represent a splitting condition with respect to key-values for a respective key. A leaf node of the data tree may correspond to a data file for a data table that includes a subset of records having key-values that satisfy the condition for the node and conditions associated with parent nodes of the node. A parent node may correspond to a file including a buffer that stores changes to data files reachable by this parent node, and also includes dedicated storage to pointers of the child nodes. By using the data tree, the data processing system may efficiently cluster the data in the data table while reducing the number of data files that are rewritten.
Disclosed herein provides enhancements for operating a data access application service executing on a data access server system and an external computing system. In the data access server system, a request is received from a client device executing at least one of multiple application services for a dataset from one or more of multiple storage systems. In the data access server system, a data retrieval instruction is generated for the client device to access the dataset from the one or more of the multiple storage systems. The data retrieval instruction comprises task descriptions and a temporary credential. The data retrieval instruction is transferred to the external computing system via the client device and the requested dataset is retrieved and deployed based on the task descriptions and the temporary credential from the one or more of the multiple storage systems.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database systemDistributed database system architectures therefor
The present application discloses a method, system, and computer system for providing access to data. The method includes receiving, by a data manager service from a data requesting service, a request using an identifier for a high-level data object to access a set of data associated with the high-level data object, determining, by the data manager service, low-level data object(s) corresponding to the set of data based on the identifier for the high-level data object, determining whether a user associated with the request has permission to access at least a subset of the low-level data object(s), and in response to determining that the user associated has permission to access the at least the subset of the low-level data object(s), generating, by the data manager service, a uniform resource locator (URL) via which the at least the subset of the one or more low-level data objects is accessible by the user.
A system performs training and execution of machine learning models that use on-demand features using feature functions. The system receives commands for registering metadata associated with a machine learning model. The machine learning model may process a set of features including on-demand features as well as other features such as batch features. The system executes the command by storing an association between the machine learning model and the feature functions associated with any on-demand features processed by the machine learning model. The feature functions are executed using an end point of a data asset service. The use of the data asset service for invoking the feature functions ensures that the same set of instructions is executed during model training and model inferencing, thereby avoiding model skew.
The system is configured to: 1) receive a client request; 2) determine executor(s) to generate a response to the user request; 3) provide each of the executor(s) with an indication; 4) receive for each indication a response including an output of either a cloud output or an in-line output to generate a group of in-line outputs and a group of cloud outputs; 5) determine whether the group of in-line outputs comprises all outputs; and 6) in response to the group of in-line outputs not comprising all the outputs for the client request: a) convert the group of in-line outputs to a converted group of cloud outputs; b) generate metadata for the converted group of cloud outputs and the group of cloud outputs; and c) provide response to the client request including the metadata for the converted group of cloud outputs and the group of cloud outputs.
A system includes a plurality of computing units. A first computing unit of the plurality of computing units comprises: a communication interface configured to receive an indication to roll up data in a data table; and a processor coupled to the communication interface and configured to: build a preaggregation hash table based at least in part on a set of columns and the data table by aggregating input rows of the data table; for each preaggregated hash table entry of the preaggregated hash table: provide the preaggregated hash table entry to a second computing unit of the plurality of computing units based at least in part on a distribution hash value; receive a set of received entries from computing units of the plurality of computing units; and build an aggregation hash table based at least in part on the set of received entries by aggregating the set of received entries.
The present application discloses a method, system, and computer system for providing access to data. The method includes receiving, by a data manager service from a data requesting service, a request using an identifier for a high-level data object to access a set of data associated with the high-level data object, determining, by the data manager service, low-level data object(s) corresponding to the set of data based on the identifier for the high-level data object, determining whether a user associated with the request has permission to access at least a subset of the low-level data object(s), and in response to determining that the user associated has permission to access the at least the subset of the low-level data object(s), generating, by the data manager service, a uniform resource locator (URL) via which the at least the subset of the one or more low-level data objects is accessible by the user.
Disclosed is a configuration for managing the organization of data tables in cloud-based storage. The configuration receives metrics for data processing operations on the data table. Metrics include at least one of a size of the data table, a size of each file in the data table, and metadata describing the data table. The configuration automatically executes a cost-benefit analysis based on the one or more metrics for each candidate maintenance operation in a plurality of candidate maintenance operations. The configuration automatically selects a maintenance operation from the candidate maintenance operations to automate based on the cost-benefit analysis of the one or more candidate maintenance operations. The selected maintenance operation is automated and scheduled on the data table.
Van Hövell Tot Westerflier, Herman Rudolf Petrus Catharina
Nakandala, Supun Chathuranga
Abstract
A cluster computing system maintains a first set of queues for short queries and a set second set for longer queries. The first set is allocated a majority of the cluster's processing resources and processes queries on a first in first out basis. The second set is allocated a minority of the cluster's processing resources which are shared among queries in the second set. Accordingly, the system assigns each query to the first set of queues for a fixed amount of resource time. While a query is processing, the system monitors the query's resource time and reassigns the query to the second set of queues if the query has not completed within the allotted amount of resource time. Thus, short queries receive the necessary resources to complete quickly without getting stuck behind longer queries while ensuring that longer queries continue to make progress.
A system for retrieving and caching metadata from a remote data source is described. The system may receive a request from a client device. The request is to perform a query operation on a set of data objects stored in the remote data source. The system may access a metadata cache storing metadata information on one or more data objects of the remote data source and identify metadata corresponding to the set of data objects for the query operation in the metadata cache. The system may determine whether the identified metadata for the set of data objects meets an update condition. In response to the identified metadata meeting the update condition, the system may fetch updated metadata for at least the set of data objects from the remote data source, and store the updated metadata in the metadata cache.
A system for multipass sort includes a communication interface and a processor. The communication interface is configured to receive from a client device a request to sort a dataset that includes a plurality of rows. The processor is configured to perform a first sort pass on the dataset in part by: extracting prefixes associated with a first schema element associated with the dataset for the plurality of rows; and sorting the extracted prefixes utilizing an integer sort algorithm based on a sort order included in the request to sort the dataset, where sorting the extracted prefixes includes utilizing NULL values to resolve a tied range that includes at least two rows of the plurality of rows having a same extracted prefix.
The interface is to receive an indication to execute an optimize command. The processor is to receive a file name; determine whether adding a file of the file name to a current bin causes the current bin to exceed a threshold; associate the file with the current bin in response to determining that adding the file does not cause the current bin to exceed the bin threshold; in response to determining that adding the file to the current bin causes the current bin to exceed the bin threshold: associate the file with a next bin, indicate that the current bin is closed, and add the current bin to a batch of bins; determine whether a measure of the batch of bins exceeds a batch threshold; and in response to determining that the measure exceeds the batch threshold, provide the batch of bins for processing.
A data tree for managing data files of a data table and performing one or more transaction operations to the data table is described. The data tree is configured as a KD-epsilon tree and includes a plurality of nodes and edges. A node of the data tree may represent a splitting condition with respect to key-values for a respective key. A leaf node of the data tree may correspond to a data file for a data table that includes a subset of records having key-values that satisfy the condition for the node and conditions associated with parent nodes of the node. A parent node may correspond to a file including a buffer that stores changes to data files reachable by this parent node, and also includes dedicated storage to pointers of the child nodes. By using the data tree, the data processing system may efficiently cluster the data in the data table while reducing the number of data files that are rewritten.
The present application discloses a method, system, and computer system for managing a data in a storage system. The method includes receiving a first transaction that modifies or deletes first data stored in a storage system, determining that the first data is subject to an intervening re-arrangement transaction, and in response to determining that the first data is subject to the intervening re-arrangement transaction, rolling back the re-arrangement transaction at least with respect to the first data and committing the first transaction.
A multi-cluster computing system which includes a query result caching system is presented. The multi-cluster computing system may include a data processing service and client devices communicatively coupled over a network. The data processing service may include a control layer and a data layer. The control layer may be configured to receive and process requests from the client devices and manage resources in the data layer. The data layer may be configured to include instances of clusters of computing resources for executing jobs. The data layer may include a data storage system, which further includes a remote query result cache Store. The query result cache store may include a cloud storage query result cache which stores data associated with results of previously executed requests. As such, when a cluster encounters a previously executed request, the cluster may efficiently retrieve the cached result of the request from the in-memory query result cache or the cloud storage query result cache.
A multi-cluster computing system which includes a query result caching system is presented. The multi-cluster computing system may include a data processing service and client devices communicatively coupled over a network. The data processing service may include a control layer and a data layer. The control layer may be configured to receive and process requests from the client devices and manage resources in the data layer. The data layer may be configured to include instances of clusters of computing resources for executing jobs. The data layer may include a data storage system, which further includes a remote query result cache Store. The query result cache store may include a cloud storage query result cache which stores data associated with results of previously executed requests. As such, when a cluster encounters a previously executed request, the cluster may efficiently retrieve the cached result of the request from the in-memory query result cache or the cloud storage query result cache.
A system executes database queries specified using a declarative database query language such as the structured query language (SQL). The system determines whether a runtime error is encountered during execution of a query, for example, a division by zero error, resource usage errors such as out of memory error, time out error, and so on. The system reports such runtime errors encountered during execution of a database query. The system identifies one or more origins of the runtime error in the database query. The origin identifies a portion of the database query that represents a cause of the runtime error. Reporting the origin of a runtime error in the database query simplifies the task of development and testing of database queries.
Disclosed herein is a method for determining whether to apply a lazy materialization technique to a query run. The method includes receiving a request to perform a new query in a columnar database containing a plurality of columns. A step in the method includes accessing a set of data in a column of the plurality of columns based on the query. The method includes generating an input to a machine-learned model comprising characteristics of the set of data in the column. From the machine-learned model, the method includes generating a likelihood value indicative of whether a filter of a first portion of the set of data in the column has greater efficiency than a download followed by a filter of the set of data in the column. The method further includes comparing the likelihood value to a threshold value. Based on the comparison, the method includes filtering the first portion of the set of data before downloading the set of data if the likelihood value is equal to or above the threshold value.
Disclosed herein is a method for determining whether to apply a lazy materialization technique to a query run. A data processing service receives a request to perform a query identifying a filter column and a non-filter column in a columnar database. The data processing service accesses a first task of contiguous rows in the filter column from a cloud-based object storage. The data processing service applies a filter defined by the query to the first task. The data processing service generates filter results for the first task that may include a percentage of the first task discarded and a run-time. The data processing service determines, based on the filter results for the first task, a likelihood value that indicates a likelihood of gaining a performance benefit by applying the lazy materialization technique to a second task of the query.
Disclosed herein is a method, system, or non-transitory computer readable medium for evaluating a query on a columnar dataset comprising one or more dictionaries associated with columns in the dataset. The method includes receiving a request to perform a query comprising at least an operator for a columnar dataset on cloud storage. At least one column in the dataset is based on a dictionary, and the dictionary maps one or more values for a column to one or more respective identifiers. The method evaluates the operator on one or more values of the dictionary to generate an updated dictionary comprising updated values. The method may decode the updated dictionary into an updated column comprising updated data values.
Disclosed herein is a method, system, or non-transitory computer readable medium for evaluating a query on a columnar dataset comprising one or more dictionaries associated with columns in the dataset. The method includes receiving a request to perform a query comprising at least a operator and a request to return information about a value of interest in a columnar dataset stored on cloud storage. At least one column in the columnar dataset is based on a dictionary. The dictionary maps one or more values for a column to one or more respective identifiers. The method determines whether to perform dictionary filtering for the query by calculating a metric based on one or more factors. Responsive to the metric being below a threshold, which may be predetermined, the method performs the dictionary filtering.
A system executes database queries specified using a declarative database query language such as the structured query language (SQL). The system determines whether a runtime error is encountered during execution of a query, for example, a division by zero error, resource usage errors such as out of memory error, time out error, and so on. The system reports such runtime errors encountered during execution of a database query. The system identifies one or more origins of the runtime error in the database query. The origin identifies a portion of the database query that represents a cause of the runtime error. Reporting the origin of a runtime error in the database query simplifies the task of development and testing of database queries.
Van Hövell Tot Westerflier, Herman Rudolf Petrus Catharina
Leone, Stefania
Abstract
A system executes user defined functions (UDFs) invoked by database queries. The UDF includes UDF code specified using a programing language distinct from a database query language. A hash value from the UDF code provided by a client application for creating the UDF is compared with a hash value generated from UDF code invoked by database queries to determine whether the two UDF codes match. If the two hash values fail to match, the system takes an action, for example, storing an indication of UDF code mismatch or disabling subsequent executions of the database queries invoking the UDF. The system may use encoded UDF code that is decoded by the system at runtime using a key obtained from a separate system such as the client application. The client application can disable execution of database queries executing the UDF code by refusing to provide the key.
Disclosed herein is a method for resource management in a web-based container orchestrating environment. A disclosed method includes initializing a set of micro-virtual machines (VMs) within a macro-VM environment. The method each container within a micro-VM based sandbox. The method assigns a virtual central processing unit (vCPU) to a micro-VM based on an estimated memory required by the micro-VM and the estimated available memory associated with the vCPU. The method pins the vCPU with a physical CPU based on the pod location of the physical CPU and an estimated available memory associated with the vCPU and an available local memory of the physical CPU. The method maintains a state of the vCPU and the physical CPU in a resource manager.
A disclosed configuration receives a first indication that a first transaction is committed to update a first subset of records in a data table at a first version to generate a second version of the data table and receiving a second indication to commit a second transaction to update a second subset of records in a data file of the data table at the first version. The configuration determines a logical prerequisite based on whether the first subset of records changes content of one or more records in the second subset of records and determining a physical prerequisite on whether the second subset of records corresponds to respective data records in data files of the second version of the data table. The configuration commits the second transaction to generate a third version of the data table by updating elements of the deletion vector if the prerequisites are satisfied.
A data processing service performs a rebalancing process for rebalancing stateful tasks on a cluster computing system. In one instance, the method for rebalancing stateful tasks is performed such that the per-operator partitions are spread across available executors of a cluster of the cluster computing system with respect to one or more statistics of the tasks. In one instance, the method for rebalancing stateful tasks is also performed such that the total number of stateful tasks are balanced per executor as long as this rebalancing does not imbalance the per-operator placements. In this way, the processing of stateful tasks can be spread across multiple executors in a relatively uniform manner, even though there may be an upfront cost of breaking the local caching on an executor.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database systemDistributed database system architectures therefor
Various embodiments of the present technology generally relate to management of big data storage and data access control systems. In some embodiments, a data access system for use in multiple application service and multiple storage service environments comprises a sandbox database for users, wherein the sandbox database is a virtual database environment via which a user may access datasets according to one or more access policies. In some embodiments, the data access system receives a user request to access a dataset stored in a database into the sandbox environment, wherein the database is associated with the data access system. In response to the request, the data access system may retrieve the corresponding data from the database, determine any associated sandbox access policies, and generate an anonymized data table in the sandbox environment.
G06F 21/53 - Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems during program execution, e.g. stack integrity, buffer overflow or preventing unwanted data erasure by executing in a restricted environment, e.g. sandbox or secure virtual machine
A system includes an interface, a processor, and a memory. The interface is configured to receive a version of a model from a model registry. The processor is configured to store the version of the model, start a process running the version of the model, and update a proxy with version information associated with the version of the model, wherein the updated proxy indicates to redirect an indication to invoke the version of the model to the process. The memory is coupled to the processor and configured to provide the processor with instructions.
A data processing service generates for iteratively applying a geospatial function to geospatial data. The generated code includes at least a first iterative loop and a second iterative loop. The data processing service compiles the generated code to generate compiled code that vectorized at least the second iterative loop. The data processing service receives a request from a client device to perform one or more data processing operations including applying the geospatial function to a data table of geospatial cell indices. The data processing service compiles the request into one or more tasks including at least a vectorized operation based on the compiled code and executes the one or more tasks by at least invoking the vectorized operation on the set of worker nodes.
The system is configured to: 1) receive a client request; 2) determine executor(s) to generate a response to the user request; 3) provide each of the executor(s) with an indication; 4) receive for each indication a response including an output of either a cloud output or an in-line output to generate a group of in-line outputs and a group of cloud outputs; 5) determine whether the group of in-line outputs comprises all outputs; and 6) in response to the group of in-line outputs not comprising all the outputs for the client request: a) convert the group of in-line outputs to a converted group of cloud outputs; b) generate metadata for the converted group of cloud outputs and the group of cloud outputs; and c) provide response to the client request including the metadata for the converted group of cloud outputs and the group of cloud outputs.
A function creation method is disclosed. The method comprises defining one or more database function inputs, defining cluster processing information, defining a deep learning model, and defining one or more database function outputs. A database function is created based at least in part on the one or more database function inputs, the cluster set-up information, the deep learning model, and the one or more database function outputs. In some embodiments, the database function enables a non-technical user to utilize deep learning models.
A method, system, and computer system for performing an operation with respect to a target table are disclosed. The method includes performing first, second and a third jobs, and obtaining a resulting table based at least in part on the second job resulting file(s) and third job resulting file(s). Performing the first job includes determining a set of matching target table files and storing target table information indicating for each of the set of matching target table files, a particular set of rows having matching rows. Performing the second job includes performing a matching action based on matched rows and obtaining the second job resulting file(s). Performing the third job includes determining unmatched rows for target table files and storing the unmatched rows in third job resulting file(s).
G06F 7/14 - Merging, i.e. combining at least two sets of record carriers each arranged in the same ordered sequence to produce a single set having the same ordered sequence
G06F 16/14 - Details of searching files based on file metadata
G06F 16/16 - File or folder operations, e.g. details of user interfaces specifically adapted to file systems
42.
Efficient merging of tabular data with post-processing compaction
A method, system, and computer system for performing an operation with respect to a target table are disclosed. The method includes performing first and second jobs, obtaining one or more other resulting files based at least in part on unmatched rows, and obtaining a set of processed files based at least in part on performing a post-processing operation with respect to the set of resulting files. The set of processed files has less files than the set of resulting files. Performing the first job includes determining a set of matching target table files and storing target table information indicating for each of the set of matching target table files, a particular set of rows having matching rows. Performing the second job includes performing a matching action based on matched rows and obtaining the second job resulting file(s).
A method, system, and computer system for performing an operation with respect to a target table are disclosed. The method includes performing first and second jobs, and obtaining other resulting files based at least in part on a second set of unmatched rows among the target table and the source table that results from the first set of unmatched rows having been processed in the second job, and obtaining a resulting table based on (i) second job resulting file(s), and (ii) other resulting files. Performing the first job includes determining a set of matching target table files and storing target table information indicating for each of the set of matching target table files, a particular set of rows having matching rows. Performing the second job includes performing a first matching action based on matched rows and a second matching action based on a subset of unmatched rows.
A method, system, and computer system for performing an operation with respect to a target table are disclosed. The method includes performing first and second jobs, and persist, in one or more deletion vector files, one or more deletion vectors for corresponding rows of the one or more target table files, and obtaining a resulting table based at least in part on the second job resulting file(s). Performing the first job includes determining a set of matching target table files and storing target table information indicating for each of the set of matching target table files, a particular set of rows having matching rows. Performing the second job includes performing a matching action based on matched rows and one or more deletion of vectors associated with previously removed rows of the matching target table files and obtaining the second job resulting file(s).
The present application discloses a method, system, and computer system for parsing files. The method includes receiving an indication that a first file is to be processed, determining to begin processing the first file using a first processing engine based at least in part on one or more predefined heuristics, indicating to process the first file using a first processing engine, determining whether a particular error in processing the first file using the first processing engine has been detected, in response to determining that the particular error has been detected, indicate to stop processing the first file using the first processing engine and indicate to continue processing using a second processing engine, and storing in memory information obtained based on processing the first file by one or more of the first processing engine and the second processing engine.
The present application discloses a method, system, and computer system for parsing files. The method includes receiving an indication that a first file is to be processed, determining to begin processing the first file using a first processing engine based at least in part on one or more predefined heuristics, indicating to process the first file using a first processing engine, determining whether a particular error in processing the first file using the first processing engine has been detected, in response to determining that the particular error has been detected, indicate to stop processing the first file using the first processing engine and indicate to continue processing using a second processing engine, and storing in memory information obtained based on processing the first file by one or more of the first processing engine and the second processing engine.
A system includes an interface and a processor. The interface is configured to receive a table indication of a data table and to receive a transaction indication to perform a transaction. The processor is configured to determine a current position N in a transaction log, determine a current state of the metadata; determine a read set associated with a transaction; attempt to write an update to the transaction log associated with a next position N+1; in response to a transaction determination that a simultaneous transaction associated with the next position N+1 already exists, determine a set of updated files; and in response to a determination that there is not an overlap between the read set associated with the current transaction and the set of updated files associated with the simultaneous transaction, attempt to write the update to the transaction to the transaction log associated with a further position N+2.
A system for clustering data into corresponding files comprises one or more processors and a memory. The one or more processors is/are configured to: 1) determine to cluster a set of data into a set of files; 2) determine a set of split points in a corresponding set of dimensions of the set of data to determine the set of files, wherein each file of the set of files has an approximate target size; and 3) store one or more items of the set of data into a corresponding file of the set of files based at least in part on the set of split points. The memory is coupled to the one or more processors and configured to provide the processor with instructions.
A system for monitoring job execution includes an interface and a processor. The interface is configured to receive an indication to start a cluster processing job. The processor is configured to determine whether processing a data instance associated with the cluster processing job satisfies a watchdog criterion; and in the event that processing the data instance satisfies the watchdog criterion, cause the processing of the data instance to be killed.
The present application discloses a method, system, and computer system for building a model associated with a dataset. The method includes receiving a data set, the dataset comprising a plurality of keys and a plurality of key-value relationships, determining a plurality of models to build based at least in part on the dataset, wherein determining the plurality of models to build comprises using the dataset format information to identify the plurality of models, building the plurality of models, and optimizing at least one of the plurality of models.
G06F 18/2132 - Feature extraction, e.g. by transforming the feature spaceSummarisationMappings, e.g. subspace methods based on discrimination criteria, e.g. discriminant analysis
51.
OPTIMIZATION OF TUNING FOR MODELS USED FOR MULTIPLE PREDICTION GENERATION
The present application discloses a method, system, and computer system for tuning a set of models. The method includes determining a set of one or more models to optimize, determining a plurality of optimizer modules with which to optimize the set of one or more models, causing the plurality of optimizer modules to respectively perform a respective optimizing process with respect to at least one model of the set of one or more models, and deploying an optimized model obtained based at least in part on optimizing metrics of the set of the one or more models.
The present application discloses a method, system, and computer system for querying a model associated with a dataset. The method includes providing an input interface via which a first entity inputs a dataset, receiving the dataset, and providing a selection interface that exposes to a second entity the plurality of models determined for the dataset and/or the plurality of results corresponding to the plurality of models using the index entries. The dataset comprises a plurality of keys and a plurality of key-value relationships, and the dataset is formatted according to a predefined format, wherein index entries are generated for a plurality of models and a plurality of results corresponding to the plurality of models.
The present application discloses a method, system, and computer system for building a model associated with a dataset. The method includes receiving a data set, the dataset comprising a plurality of keys and a plurality of key- value relationships, determining a plurality of models to build based at least in part on the dataset, wherein determining the plurality of models to build comprises using the dataset format information to identify the plurality of models, building the plurality of models, and optimizing at least one of the plurality of models.
Various embodiments of the present technology generally relate to management of big data storage and data access control systems. In some embodiments, a data access system for use in multiple application service and multiple storage service environments comprises a sandbox database for users, wherein the sandbox database is a virtual database environment via which a user may access datasets according to one or more access policies. In some embodiments, the data access system receives a user request to access a dataset stored in a database into the sandbox environment, wherein the database is associated with the data access system. In response to the request, the data access system may retrieve the corresponding data from the database, determine any associated sandbox access policies, and generate an anonymized data table in the sandbox environment.
G06F 21/53 - Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems during program execution, e.g. stack integrity, buffer overflow or preventing unwanted data erasure by executing in a restricted environment, e.g. sandbox or secure virtual machine
A system includes a plurality of computing units. A first computing unit of the plurality of computing units comprises: a communication interface configured to receive an indication to roll up data in a data table; and a processor coupled to the communication interface and configured to: build a preaggregation hash table based at least in part on a set of columns and the data table by aggregating input rows of the data table; for each preaggregated hash table entry of the preaggregated hash table: provide the preaggregated hash table entry to a second computing unit of the plurality of computing units based at least in part on a distribution hash value; receive a set of received entries from computing units of the plurality of computing units; and build an aggregation hash table based at least in part on the set of received entries by aggregating the set of received entries.
A system includes an interface, a processor, and a memory. The interface is configured to receive a version of a model from a model registry. The processor is configured to store the version of the model, start a process running the version of the model, and update a proxy with version information associated with the version of the model, wherein the updated proxy indicates to redirect an indication to invoke the version of the model to the process. The memory is coupled to the processor and configured to provide the processor with instructions.
The present application discloses a method, system, and computer system for managing a plurality of features and storing lineage information pertaining to the features. The method includes obtaining one or more datasets, determining a first feature, wherein the first feature is determined based at least in part on the one or more datasets, and storing the first feature in a feature store. The first feature is stored in association with a dataset indication of the one or more datasets from which the first feature is determined. The feature store comprises a plurality of features.
G06F 16/28 - Databases characterised by their database models, e.g. relational or object models
G06F 30/27 - Design optimisation, verification or simulation using machine learning, e.g. artificial intelligence, neural networks, support vector machines [SVM] or training a model
58.
Integrated native vectorized engine for computation
Van Hovell Tot Westerflier, Herman Rudolf Petrus Catharina
Abstract
A system comprises an interface, a processor, and a memory. The interface is configured to receive a query. The processor is configured to: determine a set of nodes for the query; determine whether a node of the set of nodes comprises a first engine node type or a second engine node type, wherein determining whether the node of the set of nodes comprises the first engine node type or the second engine node type is based at least in part on determining whether the node is able to be executed in a second engine; and generate a plan based at least in part on the set of nodes. The memory is coupled to the processor and is configured to provide the processor with instructions.
A system for executing a streaming query includes an interface and a processor. The interface is configured to receive a logical query plan. The processor is configured to determine a physical query plan based at least in part on the logical query plan. The physical query plan comprises an ordered set of operators. Each operator of the ordered set of operators comprises an operator input mode and an operator output mode. The processor is further configured to execute the physical query plan using the operator input mode and the operator output mode for each operator of the query.
A system for dataflow graph processing comprises a communication interface and a processor. The communication interface is configured receive an indication to generate a dataflow graph, wherein the indication includes a set of queries and/or commands. The processor is coupled to the communication interface and configured to: determine dependencies of each query in the set of queries on another query; determine a DAG of nodes based at least in part on the dependencies; determine the dataflow graph by determining in-line expressions for tables of the dataflow graph aggregating calculations associated with a subset of dataflow graph nodes designated as view nodes; and provide the dataflow graph.
A function creation method is disclosed. The method comprises defining one or more database function inputs, defining cluster processing information, defining a deep learning model, and defining one or more database function outputs. A database function is created based at least in part on the one or more database function inputs, the cluster set-up information, the deep learning model, and the one or more database function outputs. In some embodiments, the database function enables a non-technical user to utilize deep learning models.
The interface is to receive an indication to execute an optimize command. The processor is to receive a file name; determine whether adding a file of the file name to a current bin causes the current bin to exceed a threshold; associate the file with the current bin in response to determining that adding the file does not cause the current bin to exceed the bin threshold; in response to determining that adding the file to the current bin causes the current bin to exceed the bin threshold: associate the file with a next bin, indicate that the current bin is closed, and add the current bin to a batch of bins; determine whether a measure of the batch of bins exceeds a batch threshold; and in response to determining that the measure exceeds the batch threshold, provide the batch of bins for processing.
The present application discloses a method, system, and computer system for providing access to information stored on system for data storage. The method includes receiving a data request from a user, determining data corresponding to the data request, determining whether the user has requisite permissions to access the data, and in response to determining that the user has requisite permissions to access the data: determining a manner by which to provide access to the data, wherein the data comprises a filtered subset of stored data, and generating a token based at least in part on the user and the manner by which access to the data is to be provided.
The present application discloses a method, system, and computer system for managing a plurality of features and storing lineage information pertaining to the features. The method includes obtaining one or more datasets, determining a first feature, wherein the first feature is determined based at least in part on the one or more datasets, and storing the first feature in a feature store. The first feature is stored in association with a dataset indication of the one or more datasets from which the first feature is determined. The feature store comprises a plurality of features.
G06F 16/28 - Databases characterised by their database models, e.g. relational or object models
G06F 30/27 - Design optimisation, verification or simulation using machine learning, e.g. artificial intelligence, neural networks, support vector machines [SVM] or training a model
The present application discloses a method, system, and computer system for managing a plurality of features and storing lineage information pertaining to the features. The method includes obtaining one or more datasets, determining a first feature, wherein the first feature is determined based at least in part on the one or more datasets, and storing the first feature in a feature store. The first feature is stored in association with a dataset indication of the one or more datasets from which the first feature is determined. The feature store comprises a plurality of features.
A system for spilling comprises an interface and a processor. The interface is configured to receive an indication to perform a GROUP BY operation, wherein the indication comprises an input table and a grouping column. The processor is configured to: for each input table entry of the input table, determine a key, wherein the key is based at least in part on the input table entry and the grouping column; add the key to a grouping hash table, wherein adding the key to the grouping hash table comprises last-in, first-out (LIFO) spilling when necessary; create an output table based at least in part on the grouping hash table; and provide the output table.
The present application discloses a method, system, and computer system for building a model associated with a dataset. The method includes receiving a data set, the dataset comprising a plurality of keys and a plurality of key-value relationships, determining a plurality of models to build based at least in part on the dataset, wherein determining the plurality of models to build comprises using the dataset format information to identify the plurality of models, building the plurality of models, and optimizing at least one of the plurality of models.
A system for dataflow graph processing comprises a communication interface and a processor. The communication interface is configured receive an indication to generate a dataflow graph, wherein the indication includes a set of queries and/or commands. The processor is coupled to the communication interface and configured to: determine dependencies of each query in the set of queries on another query; determine a DAG of nodes based at least in part on the dependencies; determine the dataflow graph by determining in-line expressions for tables of the dataflow graph aggregating calculations associated with a subset of dataflow graph nodes designated as view nodes; and provide the dataflow graph.
A system for dataflow graph processing comprises a communication interface and a processor. The communication interface is configured receive an indication to generate a dataflow graph, wherein the indication includes a set of queries. The processor is coupled to the communication interface and is configured to: determine dependencies of each query in the set of queries on another query; determine a DAG of nodes based at least in part on the dependencies; insert a node in the DAG of nodes to generate an updated DAG to enforce an expectation; determine a dataflow graph based on the updated DAG; and provide the dataflow graph.
A system for dataflow graph processing comprises a communication interface and a processor. The communication interface is configured receive an indication to generate a dataflow graph, wherein the indication includes a set of queries. The processor is coupled to the communication interface and is configured to: determine dependencies of each query in the set of queries on another query; determine a DAG of nodes based at least in part on the dependencies; insert a node in the DAG of nodes to generate an updated DAG to enforce an expectation; determine a dataflow graph based on the updated DAG; and provide the dataflow graph.
A system includes an interface and a processor. The interface is configured to receive a table indication of a data table and to receive a transaction indication to perform a transaction. The processor is configured to determine a current position N in a transaction log; determine a current state of the metadata; determine a read set associated with a transaction; attempt to write an update to the transaction log associated with a next position N+1; in response to a transaction determination that a simultaneous transaction associated with the next position N+1 already exists, determine a set of updated files; and in response to a determination that there is not an overlap between the read set associated with the current transaction and the set of updated files associated with the simultaneous transaction, attempt to write the update to the transaction to the transaction log associated with a further position N+2.
Van Hovell Tot Westerflier, Herman Rudolf Petrus Catharin
Abstract
A system comprises an interface, a processor, and a memory. The interface is configured to receive a query. The processor is configured to: determine a set of nodes for the query; determine whether a node of the set of nodes comprises a first engine node type or a second engine node type, wherein determining whether the node of the set of nodes comprises the first engine node type or the second engine node type is based at least in part on determining whether the node is able to be executed in a second engine; and generate a plan based at least in part on the set of nodes. The memory is coupled to the processor and is configured to provide the processor with instructions.
G06F 12/126 - Replacement control using replacement algorithms with special data handling, e.g. priority of data or instructions, handling errors or pinning
Van Hövell Tot Westerflier, Herman Rudolf Petrus Catharina
Abstract
A system comprises an interface, a processor, and a memory. The interface is configured to receive a query. The processor is configured to: determine a set of nodes for the query; determine whether a node of the set of nodes comprises a first engine node type or a second engine node type, wherein determining whether the node of the set of nodes comprises the first engine node type or the second engine node type is based at least in part on determining whether the node is able to be executed in a second engine; and generate a plan based at least in part on the set of nodes. The memory is coupled to the processor and is configured to provide the processor with instructions.
A system includes an interface, a processor, and a memory. The interface is configured to receive a version of a model from a model registry. The processor is configured to store the version of the model, start a process running the version of the model, and update a proxy with version information associated with the version of the model, wherein the updated proxy indicates to redirect an indication to invoke the version of the model to the process. The memory is coupled to the processor and configured to provide the processor with instructions.
A system for monitoring job execution includes an interface and a processor. The interface is configured to receive an indication to start a cluster processing job. The processor is configured to determine whether processing a data instance associated with the cluster processing job satisfies a watchdog criterion; and in the event that processing the data instance satisfies the watchdog criterion, cause the processing of the data instance to be killed.
Various embodiments of the present technology generally relate to management of big data storage and data access control systems. In some embodiments, a data access system for use in multiple application service and multiple storage service environments comprises a sandbox database for users, wherein the sandbox database is a virtual database environment via which a user may access datasets according to one or more access policies. In some embodiments, the data access system receives a user request to access a dataset stored in a database into the sandbox environment, wherein the database is associated with the data access system. In response to the request, the data access system may retrieve the corresponding data from the database, determine any associated sandbox access policies, and generate an anonymized data table in the sandbox environment.
G06F 21/53 - Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems during program execution, e.g. stack integrity, buffer overflow or preventing unwanted data erasure by executing in a restricted environment, e.g. sandbox or secure virtual machine
Disclosed herein provides enhancements for operating a data access application service executing on a data access server system and an external computing system. In the data access server system, a request is received from a client device executing at least one of multiple application services for a dataset from one or more of multiple storage systems. In the data access server system, a data retrieval instruction is generated for the client device to access the dataset from the one or more of the multiple storage systems. The data retrieval instruction comprises task descriptions and a temporary credential. The data retrieval instruction is transferred to the external computing system via the client device and the requested dataset is retrieved and deployed based on the task descriptions and the temporary credential from the one or more of the multiple storage systems.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database systemDistributed database system architectures therefor
Various embodiments of the present technology generally relate to management of big data storage and the physical removal of data via data access systems for large data processing environments having multiple application services and multiple storage services. In some embodiments, a method of physically removing data from a storage system provides for identifying one or more files needing data removal treatment, determining that a file needing data removal treatment should be queued, and populating a queue with the file. Determining that a file should be queued is based, at least in part, on a staleness tolerance. The method further provides for treating the file and replacing a previous version of the file in storage with the updated file. In some implementations, treating the file includes removing data from the file to create an updated file and may further include additional changes to the file.
42 - Scientific, technological and industrial services, research and design
Goods & Services
Software as a service (SAAS) services; software as a service
(SAAS) services featuring cloud-based software for
performing data queries; software as a service (SAAS)
services featuring cloud-based software for use in data
optimization, data processing, data analytics, data
integration, data warehousing, data mining, data sharing,
data collection, data interpretation, and data
visualization.
09 - Scientific and electric apparatus and instruments
42 - Scientific, technological and industrial services, research and design
Goods & Services
Downloadable software for use in data visualization, data
queries, data analytics, data processing, data integration,
data warehousing, data mining, data sharing, data
collection, and data interpretation. Software as a service (SAAS) services featuring cloud-based
software for use in data visualization, data queries, data
analytics, data processing, data integration, data
warehousing, data mining, data sharing, data collection, and
data interpretation.
42 - Scientific, technological and industrial services, research and design
Goods & Services
Software as a service (SAAS) services featuring cloud-based software for building data pipelines; Software as a service (SAAS) services featuring cloud-based software for data transformation (ETL); Software as a service (SAAS) services featuring cloud-based software for use in data optimization, data processing, data analytics, data integration, data warehousing, data mining, data sharing, data collection, data interpretation, and data visualization
42 - Scientific, technological and industrial services, research and design
Goods & Services
Software as a service (SAAS) services featuring cloud-based software for performing data queries; Software as a service (SAAS) services featuring cloud-based software for use in data optimization, data processing, data analytics, data integration, data warehousing, data mining, data sharing, data collection, data interpretation, and data visualization
83.
Update and query of a large collection of files that represent a single dataset stored on a blob store
A system includes an interface and a processor. The interface is configured to receive a table indication of a data table and to receive a transaction indication to perform a transaction. The processor is configured to determine a current position N in a transaction log; determine a current state of the metadata; determine a read set associated with a transaction; attempt to write an update to the transaction log associated with a next position N+1; in response to a transaction determination that a simultaneous transaction associated with the next position N+1 already exists, determine a set of updated files; and in response to a determination that there is not an overlap between the read set associated with the current transaction and the set of updated files associated with the simultaneous transaction, attempt to write the update to the transaction to the transaction log associated with a further position N+2.
The allocation system comprises an interface and a processor. The interface is configured to receive an indication to deactivate idle cluster machines of a set of cluster machines. The processor is configured to determine a list of cluster machines storing one or more intermediate data files of a set of intermediate data files; determine a set of idle cluster machines of the set of cluster machines that are neither running one or more tasks of a set of tasks executing or pending on the set of cluster machines nor storing the one or more intermediate data files of the set of intermediate data files, where the set of intermediate data files is associated with the set of tasks executing or pending on the cluster machines; and deactivate each cluster machine of the set of idle cluster machines.
09 - Scientific and electric apparatus and instruments
42 - Scientific, technological and industrial services, research and design
Goods & Services
Downloadable computer software for big data analysis;
downloadable computer software for use in data integration,
data warehousing, data mining, data processing, data
sharing, data collection, data interpretation, data queries,
data visualization, and data analytics; downloadable
computer software platforms for data integration, data
warehousing, data mining, data processing, data sharing,
data collection, data interpretation, data queries, data
visualization, and data analytics; downloadable cloud
computer software for data integration, data warehousing,
data mining, data processing, data sharing, data collection,
data interpretation, data queries, data visualization, and
data analytics; downloadable computer software for
application database integration; desktop and mobile
computing and operating platforms consisting of data
transceivers, wireless networks and gateways, for
collection, analysis, sharing, interpretation and management
of data. Data mining; software as a service (SAAS) services, namely,
hosting software for use by others for big data processing;
software as a service (SAAS) services, namely, hosting
software for use by others for data integration, data
warehousing, data mining, data processing, data sharing,
data collection, data interpretation, data queries, data
visualization, and data analytics; providing temporary use
of nondownloadable analytics software for data importing,
data wrangling, data mining, data processing, data sharing,
data collection, data interpretation, data queries, and data
visualization; custom design and development of computer
software; software as a service (SAAS) services featuring
software for big data processing and analytics; development
and creation of computer programs for data processing and
analysis; software as a service (SAAS) services featuring
software for data storage, data computation, data analysis,
data processing, and database management; computer services,
namely, hosting of search platforms on the Internet to allow
users to index, integrate, warehouse, mine, process, share,
collect, interpret, research, query, visualize, and analyze
data; platform as a services (PAAS) featuring computer
software platforms for use in data management, integration,
warehousing, mining, interpretation, processing, sharing,
collecting, research, queries, visualization, and analysis;
providing temporary use of on-line non-downloadable cloud
computing software for big data processing; providing
temporary use of on-line non-downloadable cloud computing
software for data integration, data warehousing, data
mining, data processing, data sharing, data collection, data
interpretation, data queries, data visualization, and data
analytics; application service provider, namely, hosting,
managing, developing and maintaining applications, and
software of others in the fields of data importing, data
storage, data management, data queries, data processing,
data interpretation, data analytics, and data visualization.
86.
Autoscaling using file access or cache usage for cluster machines
The allocation system comprises an interface and a processor. The interface is configured to receive an indication to deactivate idle cluster machines of a set of cluster machines. The processor is configured to determine a set of tasks executing or pending on the set of cluster machines; determine a set of idle cluster machines of the set of cluster machines that are neither running one or more tasks of the set of tasks nor storing one or more intermediate data files of a set of intermediate data files, where the set of intermediate data files is associated with a set of tasks executing or pending on the cluster machines; and deactivate each cluster machine of the set of idle cluster machines.
09 - Scientific and electric apparatus and instruments
42 - Scientific, technological and industrial services, research and design
Goods & Services
(1) Downloadable computer software for big data analysis; downloadable computer software for use in data integration, data warehousing, data mining, and data visualization; downloadable computer software for use in data processing, data sharing, data collection, data interpretation, data queries, and data analytics, namely, computer software that provides real-time, integrated business management intelligence by combining information from various databases, for database management, and for software for the integration of artificial intelligence and machine learning in the field of Big Data; downloadable computer software platforms for data integration, data warehousing, data mining, and data visualization; downloadable computer software platforms for data processing, data sharing, data collection, data interpretation, data queries, and data analytics, namely, computer software that provides real-time, integrated business management intelligence by combining information from various databases, for database management, and for software for the integration of artificial intelligence and machine learning in the field of Big Data; downloadable cloud computer software for data integration, data warehousing, data mining, and data visualization; downloadable cloud computer software for data processing, data sharing, data collection, data interpretation, data queries, and data analytics, namely, computer software that provides real-time, integrated business management intelligence by combining information from various databases, for database management, and for software for the integration of artificial intelligence and machine learning in the field of Big Data; downloadable computer software for application database integration; desktop and mobile computing and operating platforms consisting of data transceivers, wireless networks and gateways, for collection, analysis, sharing, interpretation and management of data, namely, database management software. (1) Data mining; software as a service (SAAS) services, namely, hosting software for use by others for big data processing; software as a service (SAAS) services, namely, web hosting of computer software applications of others for data integration, data warehousing, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, and data analytics; providing temporary use of non-downloadable analytics software for data wrangling, data mining and data visualization; providing temporary use of non-downloadable analytics software for data importing, data processing, data sharing, data collection, data interpretation, and data queries, namely, for providing real-time integrated business management intelligence by combining information from various databases, for database management, and for software for the integration of artificial intelligence and machine learning in the field of Big Data; custom design and development of computer software; software as a service (SAAS) services featuring software for big data processing and analytics; development and creation of computer programs for data processing and analysis; software as a service (SAAS) services featuring software for data storage, namely, providing cloud storage facilities for use as a data center for others, data computation, data analysis, and data processing, namely, database management; computer services, namely, providing search platforms to allow users to index, integrate, warehouse, mine, process, share, collect, interpret, research, query, visualize, and analyze data; platform as a services (PAAS) featuring computer software platforms for use in data integration, warehousing, mining, and visualization; platform as a services (PAAS) featuring computer software platforms for use in data management, interpretation, processing, sharing, collecting, research, queries, and analysis, namely, for providing real-time integrated business management intelligence by combining information from various databases, for database management, and for software for the integration of artificial intelligence and machine learning in the field of Big Data; providing temporary use of on-line non-downloadable cloud computing software for big data processing; providing temporary use of on-line non-downloadable cloud computing software for data integration, data warehousing, data mining, and data visualization; providing temporary use of on-line non-downloadable cloud computing software for data processing, data sharing, data collection, data interpretation, data queries, and data analytics, namely, for providing real-time integrated business management intelligence by combining information from various databases, for database management, and for software for the integration of artificial intelligence and machine learning in the field of Big Data; application service provider, namely, hosting, managing, developing and maintaining applications, and software of others in the fields of data importing, data storage, data management, data queries, data processing, data interpretation, data analytics, and data visualization.
88.
Split front end for flexible back end cluster processing
A system for code development and execution includes a client interface and a client processor. The client interface is configured to receive user code for execution and receive an indication of a server that will perform the execution. The client processor is configured to parse the user code to identify one or more data items referred to during the execution. The client processor is also configured to provide the server with an inquiry for metadata regarding the one or more data items, receive the metadata regarding the one or more data items, determine a logical plan based at least in part on the metadata regarding the one or more data items; and provide the logical plan to the server for execution.
A system includes an interface and a processor. The interface is configured to receive a table indication of a data table and to receive a transaction indication to perform a transaction. The processor is configured to determine a current position N in a transaction log; determine a current state of the metadata; determine a read set associated with a transaction; attempt to write an update to the transaction log associated with a next position N+1; in response to a transaction determination that a simultaneous transaction associated with the next position N+1 already exists, determine a set of updated files; and in response to a determination that there is not an overlap between the read set associated with the current transaction and the set of updated files associated with the simultaneous transaction, attempt to write the update to the transaction to the transaction log associated with a further position N+2.
A system for executing a streaming query includes an interface and a processor. The interface is configured to receive a logical query plan. The processor is configured to determine a physical query plan based at least in part on the logical query plan. The physical query plan comprises an ordered set of operators. Each operator of the ordered set of operators comprises an operator input mode and an operator output mode. The processor is further configured to execute the physical query plan using the operator input mode and the operator output mode for each operator of the query.
A system for monitoring job execution includes an interface and a processor. The interface is configured to receive an indication to start a cluster processing job. The processor is configured to determine whether processing a data instance associated with the cluster processing job satisfies a watchdog criterion; and in the event that processing the data instance satisfies the watchdog criterion, cause the processing of the data instance to be killed.
09 - Scientific and electric apparatus and instruments
42 - Scientific, technological and industrial services, research and design
Goods & Services
Downloadable software for use in data visualization, data queries, data analytics, data processing, data integration, data warehousing, data mining, data sharing, data collection, and data interpretation Software as a service (SAAS) services featuring cloud-based software for use in data visualization, data queries, data analytics, data processing, data integration, data warehousing, data mining, data sharing, data collection, and data interpretation
09 - Scientific and electric apparatus and instruments
42 - Scientific, technological and industrial services, research and design
Goods & Services
Downloadable computer software for big data analysis; downloadable computer software for use in data integration, data warehousing, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, and data analytics; downloadable computer software platforms for data integration, data warehousing, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, and data analytics; downloadable cloud computer software for data integration, data warehousing, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, and data analytics; downloadable computer software for application database integration; desktop and mobile computing and operating platforms consisting of data transceivers, wireless networks and gateways, for collection, analysis, sharing, interpretation and management of data Data mining; software as a service (SAAS) services, namely, hosting software for use by others for big data processing; software as a service (SAAS) services, namely, hosting software for use by others for data integration, data warehousing, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, and data analytics; providing temporary use of nondownloadable analytics software for data importing, data wrangling, data mining, data processing, data sharing, data collection, data interpretation, data queries, and data visualization; custom design and development of computer software; software as a service (SAAS) services featuring software for big data processing and analytics; development and creation of computer programs for data processing and analysis; software as a service (SAAS) services featuring software for data storage, data computation, data analysis, data processing, and database management; computer services, namely, providing search platforms to allow users to index, integrate, warehouse, mine, process, share, collect, interpret, research, query, visualize, and analyze data; platform as a services (PAAS) featuring computer software platforms for use in data management, integration, warehousing, mining, interpretation, processing, sharing, collecting, research, queries, visualization, and analysis; providing temporary use of on-line non-downloadable cloud computing software for big data processing; providing temporary use of on-line non-downloadable cloud computing software for data integration, data warehousing, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, and data analytics; application service provider, namely, hosting, managing, developing and maintaining applications, and software of others in the fields of data importing, data storage, data management, data queries, data processing, data interpretation, data analytics, and data visualization
09 - Scientific and electric apparatus and instruments
42 - Scientific, technological and industrial services, research and design
Goods & Services
Downloadable computer software for big data analysis; downloadable computer software for use in data integration, data warehousing, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, and data analytics; downloadable computer software platforms for data integration, data warehousing, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, and data analytics; downloadable cloud computer software for data integration, data warehousing, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, and data analytics; downloadable computer software for application database integration; desktop and mobile computing and operating platforms consisting of data transceivers, wireless networks and gateways, for collection, analysis, sharing, interpretation and management of data Data mining; software as a service (SAAS) services, namely, hosting software for use by others for big data processing; software as a service (SAAS) services, namely, hosting software for use by others for data integration, data warehousing, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, and data analytics; providing temporary use of nondownloadable analytics software for data importing, data wrangling, data mining, data processing, data sharing, data collection, data interpretation, data queries, and data visualization; custom design and development of computer software; software as a service (SAAS) services featuring software for big data processing and analytics; development and creation of computer programs for data processing and analysis; software as a service (SAAS) services featuring software for data storage, data computation, data analysis, data processing, and database management; computer services, namely, providing search platforms to allow users to index, integrate, warehouse, mine, process, share, collect, interpret, research, query, visualize, and analyze data; platform as a services (PAAS) featuring computer software platforms for use in data management, integration, warehousing, mining, interpretation, processing, sharing, collecting, research, queries, visualization, and analysis; providing temporary use of on-line non-downloadable cloud computing software for big data processing; providing temporary use of on-line non-downloadable cloud computing software for data integration, data warehousing, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, and data analytics; application service provider, namely, hosting, managing, developing and maintaining applications, and software of others in the fields of data importing, data storage, data management, data queries, data processing, data interpretation, data analytics, and data visualization
A system for monitoring job execution includes an interface and a processor. The interface is configured to receive an indication to start a cluster processing job. The processor is configured to determine whether processing a data instance associated with the cluster processing job satisfies a watchdog criterion; and in the event that processing the data instance satisfies the watchdog criterion, cause the processing of the data instance to be killed.
A system for code development and execution includes a client interface and a client processor. The client interface is configured to receive user code for execution and receive an indication of a server that will perform the execution. The client processor is configured to parse the user code to identify one or more data items referred to during the execution. The client processor is also configured to provide the server with an inquiry for metadata regarding the one or more data items, receive the metadata regarding the one or more data items, determine a logical plan based at least in part on the metadata regarding the one or more data items; and provide the logical plan to the server for execution.
A system for code development and execution includes a client interface and a client processor. The client interface is configured to receive user code for execution and receive an indication of a server that will perform the execution. The client processor is configured to parse the user code to identify one or more data items referred to during the execution. The client processor is also configured to provide the server with an inquiry for metadata regarding the one or more data items, receive the metadata regarding the one or more data items, determine a logical plan based at least in part on the metadata regarding the one or more data items; and provide the logical plan to the server for execution.
G06F 15/16 - Combinations of two or more digital computers each having at least an arithmetic unit, a program unit and a register, e.g. for a simultaneous processing of several programs
A system for multiple views for a notebook includes an input interface and a processor. The input interface to receive a notebook. The processor is to load the notebook into a shell, wherein the shell executes the notebook using a cluster, to receive an indication to view a dashboard associated with the notebook, and to provide dashboard display information. The dashboard includes a page layout display.
G06F 17/22 - Manipulating or registering by use of codes, e.g. in sequence of text characters
G06F 3/0481 - Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
A system for processing a notebook includes an input interface and a processor. The input interface is to receive a first notebook. The notebook comprises code for interactively querying and viewing data. The processor is to load the first notebook into a shell. The shell receives one or more parameters associated with the first notebook. The shell executes the first notebook using a cluster.
Disclosed herein provides enhancements for operating a data access system for large data processing environments. In one implementation, a method provides for receiving a data query from at least one of the multiple application services and identifying metadata that defines policies for deploying the queried data. The method further provides retrieving the queried data from at least one of the multiple storage services, generating a data configuration containing the retrieved data based on standardized parameters and the policies defined by the metadata, and deploying the data configuration to the at least one of the multiple application services.
