A system receives ETL specification for processing stream data, including a transform operation represented using a database query specification. The system generates a dataflow graph of a sequence of database queries by decomposing the database query into a first database query that generates an intermediate results table, and a second database query that receives as input the intermediate results table and outputs data used for performing the transform operation. The system executes the sequence of database queries for performing the transform operation on stream data received from the source. When receiving an incremental data set, the system determines an output change set based on the received incremental data set by traversing an execution plan and processing each operator in the execution plan, and computing a change set of a particular operator from the change sets output by the one or more other operators based on the incremental data set.
A system receives ETL specification for processing stream data, including a transform operation represented using a database query specification. The system generates a dataflow graph of a sequence of database queries by decomposing the database query into a first database query that generates an intermediate results table, and a second database query that receives as input the intermediate results table and outputs data used for performing the transform operation. The system executes the sequence of database queries for performing the transform operation on stream data received from the source. When receiving an incremental data set, the system determines an output change set based on the received incremental data set by traversing an execution plan and processing each operator in the execution plan, and computing a change set of a particular operator from the change sets output by the one or more other operators based on the incremental data set.
The present application discloses a method, system, and computer system for starting up and maintaining a cluster in a warmed up state, and/or allocating clusters from a warmed up state. The method includes instantiating a set of virtual machines, wherein instantiating the set of virtual machines includes setting a temporary security credential for each virtual machine of the set of virtual machines, receiving a virtual machine allocation request associated with a workspace, a customer, or a tenant, in response to the virtual machine allocation request: allocating a virtual machine, wherein allocating the virtual machine comprises replacing the temporary security credential with a security credential associated with the workspace, the customer, or the tenant.
The present application discloses a method, system, and computer system for providing access to information stored on system for data storage. The method includes receiving a data request from a user, determining data corresponding to the data request, determining whether the user has requisite permissions to access the data, and in response to determining that the user has requisite permissions to access the data: determining a manner by which to provide access to the data, wherein the data comprises a filtered subset of stored data, and generating a token based at least in part on the user and the manner by which access to the data is to be provided.
The present application discloses a method, system, and computer system for processing data. The method includes obtaining a query plan for processing input data in response to a query, obtaining the input data, selecting a batch of the input data, creating a metadata structure for the batch, allocating one or more contiguous parts of a memory for processing the batch, processing the batch in accordance with the metadata structure to generate resulting data, and storing each array of the resulting data for the batch in one of the one or more contiguous parts of the memory.
A data processing service builds a container for a customer to run a trained large language model (LLM). The data processing service receives a trained LLM and a desired configuration from a user of a client device. Based on the desired configuration, the data processing service selects a hardware configuration and structures weights of the trained LLM based on the hardware configuration. The data processing service generates a container image reflecting the hardware configuration, registers the container image to a container registry, and generates a container from the container image as well as an application programming interface (API) endpoint for the container. The data processing service deploys the trained LLM in the API endpoint using the container such that the trained LLM is accessible through API calls.
A data processing service facilitates the creation and processing of data processing pipelines that process data processing jobs defined with respect to a set of tasks in a sequence and with data dependencies associated with each separate task such that the output from one task is used as input for a subsequent task. In various embodiments, the set of tasks include at least one cleanroom task that is executed in a cleanroom station and at least one non-cleanroom task executed in an execution environment of a user where each task is configured to read one or more input datasets and transform the one or more input datasets into one or more output datasets.
A database system performs pipelined execution of queries that process batches of streaming data. The database system compiles a database query to generate an execution plan and determines a set of stages based on the execution plan. The database query processes streaming data comprising batches. A scheduler schedules pipelined execution stages of the database query. Accordingly, the database system performs execution of a particular stage processing a batch of the streaming data in parallel with subsequent stages of the database query processing previous batches of the streaming data. The system further maintains watermarks for different stages of the database query.
A system for monitoring job execution includes an interface and a processor. The interface is configured to receive an indication to start a cluster processing job. The processor is configured to determine whether processing a data instance associated with the cluster processing job satisfies a watchdog criterion; and in the event that processing the data instance satisfies the watchdog criterion, cause the processing of the data instance to be killed.
09 - Scientific and electric apparatus and instruments
41 - Education, entertainment, sporting and cultural services
42 - Scientific, technological and industrial services, research and design
Goods & Services
Database management; database management consultancy; business consulting in the field of artificial intelligence, machine learning, deep learning, large language models (LLMs), natural language generation, statistical learning, supervised learning, un-supervised learning, predictive analytics and business intelligence; business data analysis; business data consultancy; providing an online marketplace featuring downloadable software applications, data sets, data notebooks, artificial intelligence models, machine learning models; data management and processing, namely, data analytics, data migration, data importing, data wrangling, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, data integration, data warehousing, data processing, data governance, data science, and building, design and management of data lakes Software; downloadable software; downloadable software for big data processing and analytics; downloadable software for accessing, managing and connecting to data lakes, data warehouses, data assets, data files, data sources; downloadable software for application database integration; downloadable software for use in ETL (extract, transform, load) data processing; downloadable software for compiling, organizing, visualizing, sharing and analyzing business intelligence; desktop and mobile computing and operating platforms consisting of data transceivers, wireless networks and gateways, for collection, analysis, sharing, interpretation and management of data; downloadable software featuring libraries for data science training; downloadable software featuring libraries for data analytics; downloadable software featuring libraries for machine learning training; downloadable software featuring libraries for artificial intelligence, machine learning, deep learning, large language models (LLMs), natural language generation, statistical learning, supervised learning, un-supervised learning, predictive analytics and business intelligence; downloadable software featuring artificial intelligence and machine learning technology; downloadable software for artificial intelligence, machine learning, deep learning, large language models (LLMs), natural language generation, statistical learning, supervised learning, un-supervised learning, predictive analytics and business intelligence; downloadable software for collecting, managing, editing, organizing, modifying, transmitting, sharing, and storing of data; downloadable software for modifying and enabling transmission of images, audio, audio visual and video content and data; downloadable software for processing images, graphics, audio, video, and text; downloadable software for transmitting, sharing, receiving, downloading, displaying, interacting with and transferring content, text, visual works, audio works, audiovisual works, literary works, data, files, documents and electronic works; downloadable software for use in data centers and data storage; downloadable software for use as an application programming interface (API); downloadable software development tools; downloadable software development tools for building, designing, deploying, and monitoring applications for use in data integration, data warehousing, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, and data analytics; downloadable software development tools for building designing, deploying, and monitoring applications featuring artificial intelligence, machine learning, deep learning, large language models (LLMs), natural language generation, statistical learning, supervised learning, un-supervised learning, predictive analytics and business intelligence; downloadable software development kits (SDKs); downloadable software for use in data analytics, data migration, data integration, data warehousing, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, data governance, data science; downloadable software for use in data governance, namely, software featuring artificial intelligence that allows users to define and manage policies for gathering, querying, storing, processing, sharing, accessing and disposing of data, create data clean rooms, and track data lineage; downloadable software for retrieval augmented generation (RAG); downloadable software for retrieval augmented generation (RAG) for use in generative AI applications; downloadable software for retrieval augmented generation (RAG) for use in data analytics, data migration, data integration, data warehousing, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, data governance, data science; downloadable podcasts in the field of data analytics, data migration, data importing, data wrangling, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, data integration, data warehousing, data processing, data governance, data science, artificial intelligence, machine learning, deep learning, large language models (LLMs), natural language generation, statistical learning, supervised learning, un-supervised learning, predictive analytics and business intelligence Educational services; conducting conferences, seminars, classes workshops, courses, and webinars; training services; providing training for certification; non-downloadable podcasts; conducting conferences, seminars, classes workshops, courses, and webinars in field of data analytics, data migration, data importing, data wrangling, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, data integration, data warehousing, data processing, data governance, data science, artificial intelligence, machine learning, deep learning, large language models (LLMs), natural language generation, statistical learning, supervised learning, un-supervised learning, predictive analytics and business intelligence; training services in the field of data analytics, data migration, data importing, data wrangling, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization. data integration, data warehousing, data processing, data governance, data science, artificial intelligence, machine learning, deep learning, large language models (LLMs), natural language generation, statistical learning, supervised learning, un-supervised learning, predictive analytics and business intelligence; providing training for certification in the field of data analytics, data migration, data importing, data wrangling, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization. data integration, data warehousing, data processing, data governance, data science, artificial intelligence, machine learning, deep learning, large language models (LLMs), natural language generation, statistical learning, supervised learning, un-supervised learning, predictive analytics and business intelligence; non-downloadable publications and blogs in the field of data analytics, data migration, data importing, data wrangling, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, data integration, data warehousing, data processing, data governance, data science, artificial intelligence, machine learning, deep learning, large language models (LLMs), natural language generation, statistical learning, supervised learning, un-supervised learning, predictive analytics and business intelligence; non-downloadable podcasts in the field of data analytics, data migration, data importing, data wrangling, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, data integration, data warehousing, data processing, data governance, data science, artificial intelligence, machine learning, deep learning, large language models (LLMs), natural language generation, statistical learning, supervised learning, un-supervised learning, predictive analytics and business intelligence; non-downloadable publications and blogs Providing online non-downloadable software; providing online non-downloadable software for big data processing and analytics; providing online non-downloadable software for accessing, managing and connecting to data lakes, data warehouses, data assets, data files, data sources; providing online non-downloadable software for application database integration; providing online non-downloadable software for use in ETL (extract, transform, load) data processing; providing online non-downloadable software for compiling, organizing, visualizing, sharing and analyzing business intelligence; providing online non-downloadable desktop and mobile computing and operating platforms consisting of data transceivers, wireless networks and gateways, for collection, analysis, sharing, interpretation and management of data; providing online non-downloadable software featuring libraries for data science training; providing online non-downloadable software featuring libraries for data analytics; providing online non-downloadable software featuring libraries for machine learning training; providing online non-downloadable software featuring libraries for artificial intelligence, machine learning, deep learning, large language models (LLMs), natural language generation, statistical learning, supervised learning, un-supervised learning, , predictive analytics and business intelligence; providing online non-downloadable software featuring artificial intelligence and machine learning technology; providing online non-downloadable software for artificial intelligence, machine learning, deep learning, large language models (LLMs), natural language generation, statistical learning, supervised learning, un-supervised learning, predictive analytics and business intelligence; providing online non-downloadable software for collecting, managing, editing, organizing, modifying, transmitting, sharing, and storing of data; providing online non-downloadable software for modifying and enabling transmission of images, audio, audio visual and video content and data; providing online non-downloadable software for processing images, graphics, audio, video, and text; providing online non-downloadable software for transmitting, sharing, receiving, downloading, displaying, interacting with and transferring content, text, visual works, audio works, audiovisual works, literary works, data, files, documents and electronic works; providing online non-downloadable software for use in data centers and data storage; providing online non-downloadable software for use as an application programming interface (API); providing online non-downloadable software development tools; providing online non-downloadable software development tools for building, designing, deploying, and monitoring applications for use in data integration, data warehousing, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, and data analytics; providing online non-downloadable software development tools for building designing, deploying, and monitoring applications featuring artificial intelligence, machine learning, deep learning, large language models (LLMs), natural language generation, statistical learning, supervised learning, un-supervised learning, predictive analytics and business intelligence; providing online non-downloadable software development kits (SDKs); software as a service (SAAS) services; software as a service (SAAS) services featuring software for big data processing and analytics; software as a service (SAAS) services, namely, featuring software for accessing, managing and connecting to data lakes, data warehouses, data assets, data files, data sources; software as a service (SAAS) services featuring software for application database integration; software as a service (SAAS) services featuring software for use in ETL (extract, transform, load) data processing; software as a service (SAAS) services featuring software for compiling, organizing, visualizing, sharing and analyzing business intelligence; software as a service (SAAS) services featuring desktop and mobile computing and operating platforms consisting of data transceivers, wireless networks and gateways, for collection, analysis, sharing, interpretation and management of data; software as a service (SAAS) services featuring software featuring libraries for data science training; software as a service (SAAS) services featuring software featuring libraries for data analytics; software as a service (SAAS) services featuring software featuring libraries for machine learning training; software as a service (SAAS) services featuring software featuring libraries for artificial intelligence, machine learning, deep learning, large language models (LLMs), natural language generation, statistical learning, supervised learning, un-supervised learning, predictive analytics and business intelligence; software as a service (SAAS) services featuring software featuring artificial intelligence and machine learning technology; software as a service (SAAS) services featuring software for artificial intelligence, machine learning, deep learning, large language models (LLMs), natural language generation, statistical learning, supervised learning, un-supervised learning, predictive analytics and business intelligence; software as a service (SAAS) services featuring software for collecting, managing, editing, organizing, modifying, transmitting, sharing, and storing of data; software as a service (SAAS) services featuring software for modifying and enabling transmission of images, audio, audio visual and video content and data; software as a service (SAAS) services featuring software for processing images, graphics, audio, video, and text; software as a service (SAAS) services featuring software for transmitting, sharing, receiving, downloading, displaying, interacting with and transferring content, text, visual works, audio works, audiovisual works, literary works, data, files, documents and electronic works; software as a service (SAAS) services featuring software for use in data centers and data storage; software as a service (SAAS) services featuring software for use as an application programming interface (API); software as a service (SAAS) services featuring software development tools; software as a service (SAAS) services featuring software development tools for building, designing, deploying, and monitoring applications for use in data integration, data warehousing, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, and data analytics; software as a service (SAAS) services featuring software development tools for building designing, deploying, and monitoring applications featuring artificial intelligence, machine learning, deep learning, large language models (LLMs), natural language generation, statistical learning, supervised learning, un-supervised learning, predictive analytics and business intelligence; software as a service (SAAS) services featuring software development kits (SDKs); platform as a services (PAAS) services; platform as a services (PAAS) services featuring software for big data processing and analytics; platform as a services (PAAS) services, namely, featuring software for accessing, managing and connecting to data lakes, data warehouses, data assets, data files, data sources; platform as a services (PAAS) services featuring software for use in ETL (extract, transform, load) data processing; platform as a services (PAAS) services featuring software for compiling, organizing, visualizing, sharing and analyzing business intelligence; platform as a services (PAAS) services featuring desktop and mobile computing and operating platforms consisting of data transceivers, wireless networks and gateways, for collection, analysis, sharing, interpretation and management of data; platform as a services (PAAS) services featuring software featuring libraries for data science training; platform as a services (PAAS) services featuring software featuring libraries for data analytics; platform as a services (PAAS) services featuring software featuring libraries for machine learning training; platform as a services (PAAS) services featuring software featuring libraries for artificial intelligence, machine learning, deep learning, large language models (LLMs), natural language generation, statistical learning, supervised learning, un-supervised learning, predictive analytics and business intelligence; platform as a services (PAAS) services featuring software featuring artificial intelligence and machine learning technology; platform as a services (PAAS) services featuring software for artificial intelligence, machine learning, deep learning, large language models (LLMs), natural language generation, statistical learning, supervised learning, un-supervised learning, predictive analytics and business intelligence; platform as a services (PAAS) services featuring software for collecting, managing, editing, organizing, modifying, transmitting, sharing, and storing of data; platform as a services (PAAS) services featuring software for modifying and enabling transmission of images, audio, audio visual and video content and data; platform as a services (PAAS) services featuring software for processing images, graphics, audio, video, and text; platform as a services (PAAS) services featuring software for transmitting, sharing, receiving, downloading, displaying, interacting with and transferring content, text, visual works, audio works, audiovisual works, literary works, data, files, documents and electronic works; platform as a services (PAAS) services featuring software for use in data centers and data storage; platform as a services (PAAS) services featuring software for use as an application programming interface (API); platform as a services (PAAS) services featuring software development tools; platform as a services (PAAS) services featuring software development tools for building, designing, deploying, and monitoring applications for use in data integration, data warehousing, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, and data analytics; platform as a services (PAAS) services featuring software development tools for building designing, deploying, and monitoring applications featuring artificial intelligence, machine learning, deep learning, large language models (LLMs), natural language generation, statistical learning, supervised learning, un-supervised learning, predictive analytics and business intelligence; platform as a services (PAAS) services featuring software development kits (SDKs); data mining; computer services, namely, hosting of search platforms on the Internet to allow users to index, integrate, warehouse, mine, process, share, collect, interpret, research, query, visualize, and analyze data; custom design and development of computer software; software engineering services for data processing; development and creation of computer programs for data processing and analysis; providing a website featuring information in the fields of technology, computers, computer software, computer networks, web services, mobile computing and artificial intelligence; technology consulting in the field of artificial intelligence, machine learning, deep learning, large language models (LLMs), natural language generation, statistical learning, supervised learning, un-supervised learning, predictive analytics and business intelligence; providing of online non-downloadable software for use in data analytics, data migration, data integration, data warehousing, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, data governance, data science; providing online non-downloadable software featuring artificial intelligence for use in data governance, namely, software that allows users to define and manage policies for gathering, querying, storing, processing, sharing, accessing and disposing of data, create data clean rooms, and track data lineage; providing online non-downloadable software for retrieval augmented generation (RAG); providing online non-downloadable software for retrieval augmented generation (RAG) for use in generative AI applications; providing online non-downloadable software for retrieval augmented generation (RAG) for use in data analytics, data migration, data integration, data warehousing, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, data governance, data science; software as a service (SAAS) services, namely, featuring software use in data analytics, data migration, data integration, data warehousing, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, data governance, • providing online non-downloadable software for retrieval augmented generation (RAG) for use in data analytics, data migration, data integration, data warehousing, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, data governance, data science; platform as a services (PAAS) services, namely, featuring software for data analytics, data migration, data integration, data warehousing, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, data governance, • providing online non-downloadable software for retrieval augmented generation (RAG) for use in data analytics, data migration, data integration, data warehousing, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, data governance, data science; application service provider services, namely, hosting, managing, developing and maintaining applications, and software of others in the fields of big data processing and analytics, data analytics, data migration, data importing, data wrangling, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, data integration, data warehousing, data processing, data governance, data science; computer services, namely, hosting of artificial intelligence models, machine learning models, and large language models (LLMs); computer services, namely, hosting of artificial intelligence models, machine learning models, and large language models (LLMs) to allow users to perform search queries and develop inference; technology consulting in the field of data analytics, data migration, data importing, data wrangling, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, data integration, data warehousing, data processing, data governance, data science, and building, design and management of data lakes
11.
SHORT QUERY PRIORITIZATION FOR DATA PROCESSING SERVICE
Van Hövell Tot Westerflier, Herman Rudolf Petrus Catharina
Nakandala, Supun Chathuranga
Abstract
A cluster computing system maintains a first set of queues for short queries and a set second set for longer queries. The first set is allocated a majority of the cluster's processing resources and processes queries on a first in first out basis. The second set is allocated a minority of the cluster's processing resources which are shared among queries in the second set. Accordingly, the system assigns each query to the first set of queues for a fixed amount of resource time. While a query is processing, the system monitors the query's resource time and reassigns the query to the second set of queues if the query has not completed within the allotted amount of resource time. Thus, short queries receive the necessary resources to complete quickly without getting stuck behind longer queries while ensuring that longer queries continue to make progress.
A system for retrieving and caching metadata from a remote data source is described. The system may receive a request from a client device. The request is to perform a query operation on a set of data objects stored in the remote data source. The system may access a metadata cache storing metadata information on one or more data objects of the remote data source and identify metadata corresponding to the set of data objects for the query operation in the metadata cache. The system may determine whether the identified metadata for the set of data objects meets an update condition. In response to the identified metadata meeting the update condition, the system may fetch updated metadata for at least the set of data objects from the remote data source, and store the updated metadata in the metadata cache.
A system includes an interface and a processor. The interface is configured to receive a table indication of a data table and to receive a transaction indication to perform a transaction. The processor is configured to determine a current position N in a transaction log; determine a current state of the metadata; determine a read set associated with a transaction; attempt to write an update to the transaction log associated with a next position N+1; in response to a transaction determination that a simultaneous transaction associated with the next position N+1 already exists, determine a set of updated files; and in response to a determination that there is not an overlap between the read set associated with the current transaction and the set of updated files associated with the simultaneous transaction, attempt to write the update to the transaction to the transaction log associated with a further position N+2.
Disclosed herein is a method, system, or non-transitory computer readable medium for evaluating a query on a columnar dataset comprising one or more dictionaries associated with columns in the dataset. The method includes receiving a request to perform a query comprising at least an operator for a columnar dataset on cloud storage. At least one column in the dataset is based on a dictionary, and the dictionary maps one or more values for a column to one or more respective identifiers. The method evaluates the operator on one or more values of the dictionary to generate an updated dictionary comprising updated values. The method may decode the updated dictionary into an updated column comprising updated data values.
The disclosed configurations provide a method (and/or a computer-readable medium or system) for determining, from a table schema describing keys of a data table, one or more clustering keys that can be used to cluster data files of a data table. The method includes generating features for the data table, generating tokens from the features, generating a prediction for each token by applying to the token a machine-learned transformer model trained to predict a likelihood that the key associated with the token is a clustering key for the data table, determining clustering keys based on the predictions, and clustering data records of the data table into data files based on key-values for the clustering keys.
A system for multipass sort with subsplitting includes a communication interface and a processor. The communication interface is configured to receive from a client device a request to sort a dataset that includes a plurality of rows, where the size of the dataset is greater than a threshold size. The processor is configured to: subdivide the dataset into a plurality of data subsets; sort each of the plurality of data subsets; merge the plurality of sorted data subsets utilizing a binary merge tree to generate a sorted dataset; and provide the sorted dataset to the client device.
The present application discloses a method, system, and computer system for providing access to data. The method includes receiving, by a data manager service from a data requesting service, a request using an identifier for a high-level data object to access a set of data associated with the high-level data object, determining, by the data manager service, low-level data object(s) corresponding to the set of data based on the identifier for the high-level data object, determining whether a user associated with the request has permission to access at least a subset of the low-level data object(s), and in response to determining that the user associated has permission to access the at least the subset of the low-level data object(s), generating, by the data manager service, a uniform resource locator (URL) via which the at least the subset of the one or more low-level data objects is accessible by the user.
A data processing service receives indication that a recipient will request access to data assets of a provider and provides a request for credentials from a recipient governance module. The recipient governance module stores a recipient metastore including an object for a provider metastore. In response to determining that the assets are associated with the provider metastore, the service provides a request for credentials to a provider governance module. The provider governance module stores the provider metastore describing data assets of the provider and permissions for accessing data assets. The provider metastore includes a recipient object attached to the data assets with an identifier for the recipient metastore. In response to verifying that the recipient was provided access to the data assets, the service provides a token to the recipient governance module. The service then provides the token to a computing resource to provide access to the data assets.
Disclosed is a configuration for managing the organization of data tables in cloud-based storage. The configuration receives metrics for data processing operations on the data table. Metrics include at least one of a size of the data table, a size of each file in the data table, and metadata describing the data table. The configuration automatically executes a cost-benefit analysis based on the one or more metrics for each candidate maintenance operation in a plurality of candidate maintenance operations. The configuration automatically selects a maintenance operation from the candidate maintenance operations to automate based on the cost-benefit analysis of the one or more candidate maintenance operations. The selected maintenance operation is automated and scheduled on the data table.
09 - Scientific and electric apparatus and instruments
35 - Advertising and business services
41 - Education, entertainment, sporting and cultural services
42 - Scientific, technological and industrial services, research and design
Goods & Services
Software; downloadable software; downloadable software for big data processing and analytics; downloadable software for use in data analytics, data migration, data integration, data warehousing, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, data governance, data science; downloadable software for accessing, managing and connecting to data lakes, data warehouses, data assets, data files, data sources; downloadable software for application database integration; downloadable software for use in ETL (extract, transform, load) data processing; downloadable software for compiling, organizing, visualizing, sharing and analyzing business intelligence; downloadable software for use in data governance, namely, software featuring artificial intelligence that allows users to define and manage policies for gathering, querying, storing, processing, sharing, accessing and disposing of data, create data clean rooms, and track data lineage; desktop and mobile computing and operating platforms consisting of data transceivers, wireless networks and gateways, for collection, analysis, sharing, interpretation and management of data; downloadable software featuring libraries for data science training; downloadable software featuring libraries for data analytics; downloadable software featuring libraries for machine learning training; downloadable software featuring libraries for artificial intelligence, machine learning, deep learning, large language models (LLMs), natural language generation, statistical learning, supervised learning, un-supervised learning, predictive analytics and business intelligence; downloadable software featuring artificial intelligence and machine learning technology; downloadable software for artificial intelligence, machine learning, deep learning, large language models (LLMs), natural language generation, statistical learning, supervised learning, un-supervised learning, predictive analytics and business intelligence; downloadable software for retrieval augmented generation (RAG); downloadable software for retrieval augmented generation (RAG) for use in generative AI applications; downloadable software for retrieval augmented generation (RAG) for use in data analytics, data migration, data integration, data warehousing, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, data governance, data science; downloadable software for collecting, managing, editing, organizing, modifying, transmitting, sharing, and storing of data; downloadable software for modifying and enabling transmission of images, audio, audio visual and video content and data; downloadable software for processing images, graphics, audio, video, and text; downloadable software for transmitting, sharing, receiving, downloading, displaying, interacting with and transferring content, text, visual works, audio works, audiovisual works, literary works, data, files, documents and electronic works; downloadable software for use in data centers and data storage; downloadable software for use as an application programming interface (API); downloadable software development tools; downloadable software development tools for building, designing, deploying, and monitoring applications for use in data integration, data warehousing, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, and data analytics; downloadable software development tools for building designing, deploying, and monitoring applications featuring artificial intelligence, machine learning, deep learning, large language models (LLMs), natural language generation, statistical learning, supervised learning, un-supervised learning, predictive analytics and business intelligence; downloadable software development kits (SDKs); downloadable podcasts in the field of data analytics, data migration, data importing, data wrangling, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization data integration, data warehousing, data processing, data governance, data science, artificial intelligence, machine learning, deep learning, large language models (LLMs), natural language generation, statistical learning, supervised learning, un-supervised learning, predictive analytics and business intelligence. Database management; Database management consultancy; data management and processing, namely, data analytics, data migration, data importing, data wrangling, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, data integration, data warehousing, data processing, data governance, data science, and building, design and management of data lakes; business consulting in the field of artificial intelligence, machine learning, deep learning, large language models (LLMs), natural language generation, statistical learning, supervised learning, un-supervised learning, predictive analytics and business intelligence; Business data analysis; Business data consultancy; providing an online marketplace featuring downloadable software applications, data sets, data notebooks, artificial intelligence models, machine learning models. Educational services; Conducting conferences, seminars, classes workshops, courses, and webinars; Conducting conferences, seminars, classes workshops, courses, and webinars in field of data analytics, data migration, data importing, data wrangling, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, data integration, data warehousing, data processing, data governance, data science, artificial intelligence, machine learning, deep learning, large language models (LLMs), natural language generation, statistical learning, supervised learning, un-supervised learning, predictive analytics and business intelligence; Training services; Training services in the field of data analytics, data migration, data importing, data wrangling, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, data integration, data warehousing, data processing, data governance, data science, artificial intelligence, machine learning, deep learning, large language models (LLMs), natural language generation, statistical learning, supervised learning, un-supervised learning, predictive analytics and business intelligence; Providing training for certification; Providing training for certification in the field of data analytics, data migration, data importing, data wrangling, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, data integration, data warehousing, data processing, data governance, data science, artificial intelligence, machine learning, deep learning, large language models (LLMs), natural language generation, statistical learning, supervised learning, un-supervised learning, predictive analytics and business intelligence; Non-downloadable publications and blogs; Non-downloadable publications and blogs in the field of data analytics, data migration, data importing, data wrangling, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, data integration, data warehousing, data processing, data governance, data science, artificial intelligence, machine learning, deep learning, large language models (LLMs), natural language generation, statistical learning, supervised learning, un-supervised learning, predictive analytics and business intelligence; Non-downloadable podcasts; Non-downloadable podcasts in the field of data analytics, data migration, data importing, data wrangling, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, data integration, data warehousing, data processing, data governance, data science, artificial intelligence, machine learning, deep learning, large language models (LLMs), natural language generation, statistical learning, supervised learning, un-supervised learning, predictive analytics and business intelligence. providing online non-downloadable software; providing online non-downloadable software for big data processing and analytics; providing of online non-downloadable software for use in data analytics, data migration, data integration, data warehousing, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, data governance, data science; providing online non-downloadable software for accessing, managing and connecting to data lakes, data warehouses, data assets, data files, data sources; providing online non-downloadable software for application database integration; providing online non-downloadable software for use in ETL (extract, transform, load) data processing; providing online non-downloadable software for compiling, organizing, visualizing, sharing and analyzing business intelligence; providing online non-downloadable software featuring artificial intelligence for use in data governance, namely, software that allows users to define and manage policies for gathering, querying, storing, processing, sharing, accessing and disposing of data, create data clean rooms, and track data lineage; providing online non-downloadable desktop and mobile computing and operating platforms consisting of data transceivers, wireless networks and gateways, for collection, analysis, sharing, interpretation and management of data; providing online non-downloadable software featuring libraries for data science training; providing online non-downloadable software featuring libraries for data analytics; providing online non-downloadable software featuring libraries for machine learning training; providing online non-downloadable software featuring libraries for artificial intelligence, machine learning, deep learning, large language models (LLMs), natural language generation, statistical learning, supervised learning, un-supervised learning, predictive analytics and business intelligence; providing online non-downloadable software featuring artificial intelligence and machine learning technology; providing online non-downloadable software for artificial intelligence, machine learning, deep learning, large language models (LLMs), natural language generation, statistical learning, supervised learning, un-supervised learning, predictive analytics and business intelligence; providing online non-downloadable software for retrieval augmented generation (RAG); providing online non-downloadable software for retrieval augmented generation (RAG) for use in generative AI applications; providing online non-downloadable software for retrieval augmented generation (RAG) for use in data analytics, data migration, data integration, data warehousing, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, data governance, data science; providing online non-downloadable software for collecting, managing, editing, organizing, modifying, transmitting, sharing, and storing of data; providing online non-downloadable software for modifying and enabling transmission of images, audio, audio visual and video content and data; providing online non-downloadable software for processing images, graphics, audio, video, and text; providing online non-downloadable software for transmitting, sharing, receiving, downloading, displaying, interacting with and transferring content, text, visual works, audio works, audiovisual works, literary works, data, files, documents and electronic works; providing online non-downloadable software for use in data centers and data storage; providing online non-downloadable software for use as an application programming interface (API); providing online non-downloadable software development tools; providing online non-downloadable software development tools for building, designing, deploying, and monitoring applications for use in data integration, data warehousing, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, and data analytics; providing online non-downloadable software development tools for building designing, deploying, and monitoring applications featuring artificial intelligence, machine learning, deep learning, large language models (LLMs), natural language generation, statistical learning, supervised learning, un-supervised learning, predictive analytics and business intelligence; providing online non-downloadable software development kits (SDKs); software as a service (SAAS) services; software as a service (SAAS) services featuring software for big data processing and analytics; software as a service (SAAS) services, namely, featuring software use in data analytics, data migration, data integration, data warehousing, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, data governance, data science; software as a service (SAAS) services, namely, featuring software for accessing, managing and connecting to data lakes, data warehouses, data assets, data files, data sources; software as a service (SAAS) services featuring software for application database integration; software as a service (SAAS) services featuring software for use in ETL (extract, transform, load) data processing; software as a service (SAAS) services featuring software for compiling, organizing, visualizing, sharing and analyzing business intelligence; software as a service (SAAS) services featuring software featuring artificial intelligence for use in data governance, namely, software that allows users to define and manage policies for gathering, querying, storing, processing, sharing, accessing and disposing of data, create data clean rooms, and track data lineage; software as a service (SAAS) services featuring desktop and mobile computing and operating platforms consisting of data transceivers, wireless networks and gateways, for collection, analysis, sharing, interpretation and management of data; software as a service (SAAS) services featuring software featuring libraries for data science training; software as a service (SAAS) services featuring software featuring libraries for data analytics; software as a service (SAAS) services featuring software featuring libraries for machine learning training; software as a service (SAAS) services featuring software featuring libraries for artificial intelligence, machine learning, deep learning, large language models (LLMs), natural language generation, statistical learning, supervised learning, un-supervised learning, , predictive analytics and business intelligence; software as a service (SAAS) services featuring software featuring artificial intelligence and machine learning technology; software as a service (SAAS) services featuring software for artificial intelligence, machine learning, deep learning, large language models (LLMs), natural language generation, statistical learning, supervised learning, un-supervised learning, predictive analytics and business intelligence; software as a service (SAAS) services featuring software for retrieval augmented generation (RAG); software as a service (SAAS) services featuring software for retrieval augmented generation (RAG) for use in generative AI applications; software as a service (SAAS) services featuring software for retrieval augmented generation (RAG) for use in data analytics, data migration, data integration, data warehousing, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, data governance, data science; software as a service (SAAS) services featuring software for collecting, managing, editing, organizing, modifying, transmitting, sharing, and storing of data; software as a service (SAAS) services featuring software for modifying and enabling transmission of images, audio, audio visual and video content and data; software as a service (SAAS) services featuring software for processing images, graphics, audio, video, and text; software as a service (SAAS) services featuring software for transmitting, sharing, receiving, downloading, displaying, interacting with and transferring content, text, visual works, audio works, audiovisual works, literary works, data, files, documents and electronic works; software as a service (SAAS) services featuring software for use in data centers and data storage; software as a service (SAAS) services featuring software for use as an application programming interface (API); software as a service (SAAS) services featuring software development tools; software as a service (SAAS) services featuring software development tools for building, designing, deploying, and monitoring applications for use in data integration, data warehousing, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, and data analytics; software as a service (SAAS) services featuring software development tools for building designing, deploying, and monitoring applications featuring artificial intelligence, machine learning, deep learning, large language models (LLMs), natural language generation, statistical learning, supervised learning, un-supervised learning, predictive analytics and business intelligence; software as a service (SAAS) services featuring software development kits (SDKs); platform as a services (PAAS) services; platform as a services (PAAS) services featuring software for big data processing and analytics; platform as a services (PAAS) services, namely, featuring software for data analytics, data migration, data integration, data warehousing, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, data governance, data science; platform as a services (PAAS) services, namely, featuring software for accessing, managing and connecting to data lakes, data warehouses, data assets, data files, data sources; platform as a services (PAAS) services featuring software for use in ETL (extract, transform, load) data processing; platform as a services (PAAS) services featuring software for compiling, organizing, visualizing, sharing and analyzing business intelligence; platform as a services (PAAS) services featuring desktop and mobile computing and operating platforms consisting of data transceivers, wireless networks and gateways, for collection, analysis, sharing, interpretation and management of data; platform as a services (PAAS) services featuring software featuring libraries for data science training; platform as a services (PAAS) services featuring software featuring libraries for data analytics; platform as a services (PAAS) services featuring software featuring libraries for machine learning training; platform as a services (PAAS) services featuring software featuring libraries for artificial intelligence, machine learning, deep learning, large language models (LLMs), natural language generation, statistical learning, supervised learning, un-supervised learning, , predictive analytics and business intelligence; platform as a services (PAAS) services featuring software featuring artificial intelligence and machine learning technology; platform as a services (PAAS) services featuring software for artificial intelligence, machine learning, deep learning, large language models (LLMs), natural language generation, statistical learning, supervised learning, un-supervised learning, predictive analytics and business intelligence; platform as a services (PAAS) services featuring software for collecting, managing, editing, organizing, modifying, transmitting, sharing, and storing of data; platform as a services (PAAS) services featuring software for modifying and enabling transmission of images, audio, audio visual and video content and data; platform as a services (PAAS) services featuring software for processing images, graphics, audio, video, and text; platform as a services (PAAS) services featuring software for transmitting, sharing, receiving, downloading, displaying, interacting with and transferring content, text, visual works, audio works, audiovisual works, literary works, data, files, documents and electronic works; platform as a services (PAAS) services featuring software for use in data centers and data storage; platform as a services (PAAS) services featuring software for use as an application programming interface (API); platform as a services (PAAS) services featuring software development tools; platform as a services (PAAS) services featuring software development tools for building, designing, deploying, and monitoring applications for use in data integration, data warehousing, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, and data analytics; platform as a services (PAAS) services featuring software development tools for building designing, deploying, and monitoring applications featuring artificial intelligence, machine learning, deep learning, large language models (LLMs), natural language generation, statistical learning, supervised learning, un-supervised learning, predictive analytics and business intelligence; platform as a services (PAAS) services featuring software development kits (SDKs); Data mining; computer services, namely, hosting of search platforms on the Internet to allow users to index, integrate, warehouse, mine, process, share, collect, interpret, research, query, visualize, and analyze data; computer services, namely, hosting of artificial intelligence models, machine learning models, and large language models (LLMs); computer services, namely, hosting of artificial intelligence models, machine learning models, and large language models (LLMs) to allow users to perform search queries and develop inference; application service provider services, namely, hosting, managing, developing and maintaining applications, and software of others in the fields of big data processing and analytics, data analytics, data migration, data importing, data wrangling, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, data integration, data warehousing, data processing, data governance, data science; custom design and development of computer software; Software engineering services for data processing; development and creation of computer programs for data processing and analysis; Providing a website featuring information in the fields of technology, computers, computer software, computer networks, web services, mobile computing and artificial intelligence; Technology consulting in the field of data analytics, data migration, data importing, data wrangling, data mining, data processing, data sharing, data collection, data interpretation, data queries, data visualization, data integration, data warehousing, data processing, data governance, data science and building, design and management of data lakes; Technology consulting in the field of artificial intelligence, machine learning, deep learning, large language models (LLMs), natural language generation, statistical learning, supervised learning, un-supervised learning, predictive analytics and business intelligence.
21.
USING LLM FUNCTIONS TO EVALUATE AND COMPARE LARGE TEXT OUTPUTS OF LLMS
A method for evaluating textual output of one or more machine-learned language models is presented. The method includes receiving, from a user of a client device, a first prompt for input to one or more machine-learned language models, providing the first prompt to the one or more models for execution, and receiving a set of generated responses to the first prompt from the one or more models. The method further includes generating a user interface (UI) on the client device displaying the first prompt and generated responses as a table user interface element. The method applies a selected evaluation function to the generated response to evaluate the response with respect to an evaluation objective and identifies words that influence the evaluation. The method generates one or more UI elements on the UI to display the results of the evaluation for the generated responses.
A disclosed configuration receives a first indication that a first transaction is committed to update a first subset of records in a data table at a first version to generate a second version of the data table and receiving a second indication to commit a second transaction to update a second subset of records in a data file of the data table at the first version. The configuration determines a logical prerequisite based on whether the first subset of records changes content of one or more records in the second subset of records and determining a physical prerequisite on whether the second subset of records corresponds to respective data records in data files of the second version of the data table. The configuration commits the second transaction to generate a third version of the data table by updating elements of the deletion vector if the prerequisites are satisfied.
A data processing service facilitates the creation and processing of data processing pipelines that process data processing jobs defined with respect to a set of tasks in a sequence and with data dependencies associated with each separate task such that the output from one task is used as input for a subsequent task. In various embodiments, the set of tasks include at least one cleanroom task that is executed in a cleanroom station and at least one non-cleanroom task executed in an execution environment of a user where each task is configured to read one or more input datasets and transform the one or more input datasets into one or more output datasets.
A resource management configuration may receive an API request from an API server. The API request specifies task information from a plurality of tenants. The configuration transmits status information of a plurality of VMs to the API server to assign tasks to one or more VMs based on the task information and the status information. Tasks assigned to a VM of the plurality of VMs are for one tenant of the plurality of tenants. The configuration configures on an untrusted network, network security groups for managing communications of tenants such that a network security group configured for a tenant permits communications between VMs assigned to the same tenant but prevents communications between VMs assigned to different tenants. The configuration pins each assigned VM of the one or more assigned VMs to perform the task based on the task information of the corresponding tenant.
A system for executing a streaming query includes an interface and a processor. The interface is configured to receive a logical query plan. The processor is configured to determine a physical query plan based at least in part on the logical query plan. The physical query plan comprises an ordered set of operators. Each operator of the ordered set of operators comprises an operator input mode and an operator output mode. The processor is further configured to execute the physical query plan using the operator input mode and the operator output mode for each operator of the query.
A system for clustering data into corresponding files comprises one or more processors and a memory. The one or more processors is/are configured to: 1) determine to cluster a set of data into a set of files; 2) determine a set of split points in a corresponding set of dimensions of the set of data to determine the set of files, wherein each file of the set of files has an approximate target size; and 3) store one or more items of the set of data into a corresponding file of the set of files based at least in part on the set of split points. The memory is coupled to the one or more processors and configured to provide the processor with instructions.
The present application discloses a method, system, and computer system for starting up and maintaining a cluster in a warmed up state, and/or allocating clusters from a warmed up state. The method includes instantiating a set of virtual machines, wherein instantiating the set of virtual machines includes setting a temporary security credential for each virtual machine of the set of virtual machines, receiving a virtual machine allocation request associated with a workspace, a customer, or a tenant, in response to the virtual machine allocation request: allocating a virtual machine, wherein allocating the virtual machine comprises replacing the temporary security credential with a security credential associated with the workspace, the customer, or the tenant.
The present application discloses a method, system, and computer system for managing lineage data for data entities. The method includes generating lineage data, wherein generating the lineage data, and storing and indexing, in a data structure, the lineage data in association with the selected data entity. The generating the lineage data includes selecting a selected data entity, obtaining a query tree that was used to generate the selected data entity, and determining lineage data for the selected data entity based at least in part on the query tree.
The present application discloses a method, system, and computer system for building a model associated with a dataset. The method includes receiving a data set, the dataset comprising a plurality of keys and a plurality of key-value relationships, determining a plurality of models to build based at least in part on the dataset, wherein determining the plurality of models to build comprises using the dataset format information to identify the plurality of models, building the plurality of models, and optimizing at least one of the plurality of models.
G06F 18/2132 - Feature extraction, e.g. by transforming the feature spaceSummarisationMappings, e.g. subspace methods based on discrimination criteria, e.g. discriminant analysis
A data processing service performs a rebalancing process for rebalancing stateful tasks on a cluster computing system. In one instance, the method for rebalancing stateful tasks is performed such that the per-operator partitions are spread across available executors of a cluster of the cluster computing system with respect to one or more statistics of the tasks. In one instance, the method for rebalancing stateful tasks is also performed such that the total number of stateful tasks are balanced per executor as long as this rebalancing does not imbalance the per-operator placements. In this way, the processing of stateful tasks can be spread across multiple executors in a relatively uniform manner, even though there may be an upfront cost of breaking the local caching on an executor.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database systemDistributed database system architectures therefor
A system performs efficient startup of executors of a distributed computing engine used for processing queries, for example, database queries. The system starts an executor node and processes a set of queries using the executor node to warm up the executor node. The system performs a checkpoint of the warmed-up executor node to create an image. The image is restored in the target executor nodes. The system may store a checkpoint image for each configuration of an executor node. The configuration is determined based on various factors including the hardware of the executor node, memory allocation of the processes, and so on. The user or restore based on checkpoint images improves efficiency of execution of the startup of executor nodes.
The disclosed configurations provide a method (and/or a computer-readable medium or system) for determining, from a table schema describing keys of a data table, one or more clustering keys that can be used to cluster data files of a data table. The method includes generating features for the data table, generating tokens from the features, generating a prediction for each token by applying to the token a machine-learned transformer model trained to predict a likelihood that the key associated with the token is a clustering key for the data table, determining clustering keys based on the predictions, and clustering data records of the data table into data files based on key-values for the clustering keys.
A device for using message identifiers for Publish/subscribe messaging deduplication is described. The system may fetch one or more sets of data records from a data source, and each data record is associated with a message identifier. The system may store the one or more sets of data records in a data file, which is associated with a metadata comprising the message identifier, a file path and a row number for each data record. The system may determine whether one or more of the data records are duplicated based on the associated message identifiers. In response to determining that the one or more data records are duplicated, the system may generate a second metadata comprising the file paths and row numbers associated with the duplicated data records.
A system includes an interface, a processor, and a memory. The interface is configured to receive a version of a model from a model registry. The processor is configured to store the version of the model, start a process running the version of the model, and update a proxy with version information associated with the version of the model, wherein the updated proxy indicates to redirect an indication to invoke the version of the model to the process. The memory is coupled to the processor and configured to provide the processor with instructions.
A data processing service receives a request from a first collaborator to create a clean room for data sharing collaboration with at least a second collaborator. In response, the data processing service creates an execution environment separate from the data environment of the first collaborator and the second collaborator. The first and second collaborators can then add content into the clean room in the form of data tables and executable notebooks. Approval from each collaborator is required before a notebook can be executed using any data table shared into the clean room. Upon receiving notebook approval from each collaborator, the data processing service creates a notebook job to execute the notebook on one or more cluster computing resources of the data processing service to generate an output.
A method, system, and computer system for performing an operation with respect to a target table are disclosed. The method includes performing first and second jobs, obtaining one or more other resulting files based at least in part on unmatched rows, and obtaining a set of processed files based at least in part on performing a post-processing operation with respect to the set of resulting files. The set of processed files has less files than the set of resulting files. Performing the first job includes determining a set of matching target table files and storing target table information indicating for each of the set of matching target table files, a particular set of rows having matching rows. Performing the second job includes performing a matching action based on matched rows and obtaining the second job resulting file(s).
A data processing service generates a data classifier tree for managing data files of a data table. The data classifier tree may be configured as a KD-classifier tree and includes a plurality of nodes and edges. A node of the data classifier tree may represent a splitting condition with respect to key-values for a respective key. A node of the data classifier tree may be associated with one or more data files assigned to the node. The data files assigned to the node each include a subset of records having key-values that satisfy the conditions represented by the node and parent nodes of the node. The data processing service may efficiently cluster the data in the data table while reducing the number of data files that are rewritten when data is modified or added to the data table.
A data tree for managing data files of a data table and performing one or more transaction operations to the data table is described. The data tree is configured as a KD-epsilon tree and includes a plurality of nodes and edges. A node of the data tree may represent a splitting condition with respect to key-values for a respective key. A leaf node of the data tree may correspond to a data file for a data table that includes a subset of records having key-values that satisfy the condition for the node and conditions associated with parent nodes of the node. A parent node may correspond to a file including a buffer that stores changes to data files reachable by this parent node, and also includes dedicated storage to pointers of the child nodes. By using the data tree, the data processing system may efficiently cluster the data in the data table while reducing the number of data files that are rewritten.
Disclosed herein provides enhancements for operating a data access application service executing on a data access server system and an external computing system. In the data access server system, a request is received from a client device executing at least one of multiple application services for a dataset from one or more of multiple storage systems. In the data access server system, a data retrieval instruction is generated for the client device to access the dataset from the one or more of the multiple storage systems. The data retrieval instruction comprises task descriptions and a temporary credential. The data retrieval instruction is transferred to the external computing system via the client device and the requested dataset is retrieved and deployed based on the task descriptions and the temporary credential from the one or more of the multiple storage systems.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database systemDistributed database system architectures therefor
The present application discloses a method, system, and computer system for providing access to data. The method includes receiving, by a data manager service from a data requesting service, a request using an identifier for a high-level data object to access a set of data associated with the high-level data object, determining, by the data manager service, low-level data object(s) corresponding to the set of data based on the identifier for the high-level data object, determining whether a user associated with the request has permission to access at least a subset of the low-level data object(s), and in response to determining that the user associated has permission to access the at least the subset of the low-level data object(s), generating, by the data manager service, a uniform resource locator (URL) via which the at least the subset of the one or more low-level data objects is accessible by the user.
A system performs training and execution of machine learning models that use on-demand features using feature functions. The system receives commands for registering metadata associated with a machine learning model. The machine learning model may process a set of features including on-demand features as well as other features such as batch features. The system executes the command by storing an association between the machine learning model and the feature functions associated with any on-demand features processed by the machine learning model. The feature functions are executed using an end point of a data asset service. The use of the data asset service for invoking the feature functions ensures that the same set of instructions is executed during model training and model inferencing, thereby avoiding model skew.
The system is configured to: 1) receive a client request; 2) determine executor(s) to generate a response to the user request; 3) provide each of the executor(s) with an indication; 4) receive for each indication a response including an output of either a cloud output or an in-line output to generate a group of in-line outputs and a group of cloud outputs; 5) determine whether the group of in-line outputs comprises all outputs; and 6) in response to the group of in-line outputs not comprising all the outputs for the client request: a) convert the group of in-line outputs to a converted group of cloud outputs; b) generate metadata for the converted group of cloud outputs and the group of cloud outputs; and c) provide response to the client request including the metadata for the converted group of cloud outputs and the group of cloud outputs.
A system includes a plurality of computing units. A first computing unit of the plurality of computing units comprises: a communication interface configured to receive an indication to roll up data in a data table; and a processor coupled to the communication interface and configured to: build a preaggregation hash table based at least in part on a set of columns and the data table by aggregating input rows of the data table; for each preaggregated hash table entry of the preaggregated hash table: provide the preaggregated hash table entry to a second computing unit of the plurality of computing units based at least in part on a distribution hash value; receive a set of received entries from computing units of the plurality of computing units; and build an aggregation hash table based at least in part on the set of received entries by aggregating the set of received entries.
The present application discloses a method, system, and computer system for providing access to data. The method includes receiving, by a data manager service from a data requesting service, a request using an identifier for a high-level data object to access a set of data associated with the high-level data object, determining, by the data manager service, low-level data object(s) corresponding to the set of data based on the identifier for the high-level data object, determining whether a user associated with the request has permission to access at least a subset of the low-level data object(s), and in response to determining that the user associated has permission to access the at least the subset of the low-level data object(s), generating, by the data manager service, a uniform resource locator (URL) via which the at least the subset of the one or more low-level data objects is accessible by the user.
Disclosed is a configuration for managing the organization of data tables in cloud-based storage. The configuration receives metrics for data processing operations on the data table. Metrics include at least one of a size of the data table, a size of each file in the data table, and metadata describing the data table. The configuration automatically executes a cost-benefit analysis based on the one or more metrics for each candidate maintenance operation in a plurality of candidate maintenance operations. The configuration automatically selects a maintenance operation from the candidate maintenance operations to automate based on the cost-benefit analysis of the one or more candidate maintenance operations. The selected maintenance operation is automated and scheduled on the data table.
Van Hövell Tot Westerflier, Herman Rudolf Petrus Catharina
Nakandala, Supun Chathuranga
Abstract
A cluster computing system maintains a first set of queues for short queries and a set second set for longer queries. The first set is allocated a majority of the cluster's processing resources and processes queries on a first in first out basis. The second set is allocated a minority of the cluster's processing resources which are shared among queries in the second set. Accordingly, the system assigns each query to the first set of queues for a fixed amount of resource time. While a query is processing, the system monitors the query's resource time and reassigns the query to the second set of queues if the query has not completed within the allotted amount of resource time. Thus, short queries receive the necessary resources to complete quickly without getting stuck behind longer queries while ensuring that longer queries continue to make progress.
A system for retrieving and caching metadata from a remote data source is described. The system may receive a request from a client device. The request is to perform a query operation on a set of data objects stored in the remote data source. The system may access a metadata cache storing metadata information on one or more data objects of the remote data source and identify metadata corresponding to the set of data objects for the query operation in the metadata cache. The system may determine whether the identified metadata for the set of data objects meets an update condition. In response to the identified metadata meeting the update condition, the system may fetch updated metadata for at least the set of data objects from the remote data source, and store the updated metadata in the metadata cache.
A system for multipass sort includes a communication interface and a processor. The communication interface is configured to receive from a client device a request to sort a dataset that includes a plurality of rows. The processor is configured to perform a first sort pass on the dataset in part by: extracting prefixes associated with a first schema element associated with the dataset for the plurality of rows; and sorting the extracted prefixes utilizing an integer sort algorithm based on a sort order included in the request to sort the dataset, where sorting the extracted prefixes includes utilizing NULL values to resolve a tied range that includes at least two rows of the plurality of rows having a same extracted prefix.
The interface is to receive an indication to execute an optimize command. The processor is to receive a file name; determine whether adding a file of the file name to a current bin causes the current bin to exceed a threshold; associate the file with the current bin in response to determining that adding the file does not cause the current bin to exceed the bin threshold; in response to determining that adding the file to the current bin causes the current bin to exceed the bin threshold: associate the file with a next bin, indicate that the current bin is closed, and add the current bin to a batch of bins; determine whether a measure of the batch of bins exceeds a batch threshold; and in response to determining that the measure exceeds the batch threshold, provide the batch of bins for processing.
A data tree for managing data files of a data table and performing one or more transaction operations to the data table is described. The data tree is configured as a KD-epsilon tree and includes a plurality of nodes and edges. A node of the data tree may represent a splitting condition with respect to key-values for a respective key. A leaf node of the data tree may correspond to a data file for a data table that includes a subset of records having key-values that satisfy the condition for the node and conditions associated with parent nodes of the node. A parent node may correspond to a file including a buffer that stores changes to data files reachable by this parent node, and also includes dedicated storage to pointers of the child nodes. By using the data tree, the data processing system may efficiently cluster the data in the data table while reducing the number of data files that are rewritten.
The present application discloses a method, system, and computer system for managing a data in a storage system. The method includes receiving a first transaction that modifies or deletes first data stored in a storage system, determining that the first data is subject to an intervening re-arrangement transaction, and in response to determining that the first data is subject to the intervening re-arrangement transaction, rolling back the re-arrangement transaction at least with respect to the first data and committing the first transaction.
A multi-cluster computing system which includes a query result caching system is presented. The multi-cluster computing system may include a data processing service and client devices communicatively coupled over a network. The data processing service may include a control layer and a data layer. The control layer may be configured to receive and process requests from the client devices and manage resources in the data layer. The data layer may be configured to include instances of clusters of computing resources for executing jobs. The data layer may include a data storage system, which further includes a remote query result cache Store. The query result cache store may include a cloud storage query result cache which stores data associated with results of previously executed requests. As such, when a cluster encounters a previously executed request, the cluster may efficiently retrieve the cached result of the request from the in-memory query result cache or the cloud storage query result cache.
A multi-cluster computing system which includes a query result caching system is presented. The multi-cluster computing system may include a data processing service and client devices communicatively coupled over a network. The data processing service may include a control layer and a data layer. The control layer may be configured to receive and process requests from the client devices and manage resources in the data layer. The data layer may be configured to include instances of clusters of computing resources for executing jobs. The data layer may include a data storage system, which further includes a remote query result cache Store. The query result cache store may include a cloud storage query result cache which stores data associated with results of previously executed requests. As such, when a cluster encounters a previously executed request, the cluster may efficiently retrieve the cached result of the request from the in-memory query result cache or the cloud storage query result cache.
A system executes database queries specified using a declarative database query language such as the structured query language (SQL). The system determines whether a runtime error is encountered during execution of a query, for example, a division by zero error, resource usage errors such as out of memory error, time out error, and so on. The system reports such runtime errors encountered during execution of a database query. The system identifies one or more origins of the runtime error in the database query. The origin identifies a portion of the database query that represents a cause of the runtime error. Reporting the origin of a runtime error in the database query simplifies the task of development and testing of database queries.
Disclosed herein is a method for determining whether to apply a lazy materialization technique to a query run. The method includes receiving a request to perform a new query in a columnar database containing a plurality of columns. A step in the method includes accessing a set of data in a column of the plurality of columns based on the query. The method includes generating an input to a machine-learned model comprising characteristics of the set of data in the column. From the machine-learned model, the method includes generating a likelihood value indicative of whether a filter of a first portion of the set of data in the column has greater efficiency than a download followed by a filter of the set of data in the column. The method further includes comparing the likelihood value to a threshold value. Based on the comparison, the method includes filtering the first portion of the set of data before downloading the set of data if the likelihood value is equal to or above the threshold value.
Disclosed herein is a method for determining whether to apply a lazy materialization technique to a query run. A data processing service receives a request to perform a query identifying a filter column and a non-filter column in a columnar database. The data processing service accesses a first task of contiguous rows in the filter column from a cloud-based object storage. The data processing service applies a filter defined by the query to the first task. The data processing service generates filter results for the first task that may include a percentage of the first task discarded and a run-time. The data processing service determines, based on the filter results for the first task, a likelihood value that indicates a likelihood of gaining a performance benefit by applying the lazy materialization technique to a second task of the query.
Disclosed herein is a method, system, or non-transitory computer readable medium for evaluating a query on a columnar dataset comprising one or more dictionaries associated with columns in the dataset. The method includes receiving a request to perform a query comprising at least an operator for a columnar dataset on cloud storage. At least one column in the dataset is based on a dictionary, and the dictionary maps one or more values for a column to one or more respective identifiers. The method evaluates the operator on one or more values of the dictionary to generate an updated dictionary comprising updated values. The method may decode the updated dictionary into an updated column comprising updated data values.
Disclosed herein is a method, system, or non-transitory computer readable medium for evaluating a query on a columnar dataset comprising one or more dictionaries associated with columns in the dataset. The method includes receiving a request to perform a query comprising at least a operator and a request to return information about a value of interest in a columnar dataset stored on cloud storage. At least one column in the columnar dataset is based on a dictionary. The dictionary maps one or more values for a column to one or more respective identifiers. The method determines whether to perform dictionary filtering for the query by calculating a metric based on one or more factors. Responsive to the metric being below a threshold, which may be predetermined, the method performs the dictionary filtering.
Van Hövell Tot Westerflier, Herman Rudolf Petrus Catharina
Leone, Stefania
Abstract
A system executes user defined functions (UDFs) invoked by database queries. The UDF includes UDF code specified using a programing language distinct from a database query language. A hash value from the UDF code provided by a client application for creating the UDF is compared with a hash value generated from UDF code invoked by database queries to determine whether the two UDF codes match. If the two hash values fail to match, the system takes an action, for example, storing an indication of UDF code mismatch or disabling subsequent executions of the database queries invoking the UDF. The system may use encoded UDF code that is decoded by the system at runtime using a key obtained from a separate system such as the client application. The client application can disable execution of database queries executing the UDF code by refusing to provide the key.
Disclosed herein is a method for resource management in a web-based container orchestrating environment. A disclosed method includes initializing a set of micro-virtual machines (VMs) within a macro-VM environment. The method each container within a micro-VM based sandbox. The method assigns a virtual central processing unit (vCPU) to a micro-VM based on an estimated memory required by the micro-VM and the estimated available memory associated with the vCPU. The method pins the vCPU with a physical CPU based on the pod location of the physical CPU and an estimated available memory associated with the vCPU and an available local memory of the physical CPU. The method maintains a state of the vCPU and the physical CPU in a resource manager.
A system executes database queries specified using a declarative database query language such as the structured query language (SQL). The system determines whether a runtime error is encountered during execution of a query, for example, a division by zero error, resource usage errors such as out of memory error, time out error, and so on. The system reports such runtime errors encountered during execution of a database query. The system identifies one or more origins of the runtime error in the database query. The origin identifies a portion of the database query that represents a cause of the runtime error. Reporting the origin of a runtime error in the database query simplifies the task of development and testing of database queries.
A disclosed configuration receives a first indication that a first transaction is committed to update a first subset of records in a data table at a first version to generate a second version of the data table and receiving a second indication to commit a second transaction to update a second subset of records in a data file of the data table at the first version. The configuration determines a logical prerequisite based on whether the first subset of records changes content of one or more records in the second subset of records and determining a physical prerequisite on whether the second subset of records corresponds to respective data records in data files of the second version of the data table. The configuration commits the second transaction to generate a third version of the data table by updating elements of the deletion vector if the prerequisites are satisfied.
A data processing service performs a rebalancing process for rebalancing stateful tasks on a cluster computing system. In one instance, the method for rebalancing stateful tasks is performed such that the per-operator partitions are spread across available executors of a cluster of the cluster computing system with respect to one or more statistics of the tasks. In one instance, the method for rebalancing stateful tasks is also performed such that the total number of stateful tasks are balanced per executor as long as this rebalancing does not imbalance the per-operator placements. In this way, the processing of stateful tasks can be spread across multiple executors in a relatively uniform manner, even though there may be an upfront cost of breaking the local caching on an executor.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database systemDistributed database system architectures therefor
Various embodiments of the present technology generally relate to management of big data storage and data access control systems. In some embodiments, a data access system for use in multiple application service and multiple storage service environments comprises a sandbox database for users, wherein the sandbox database is a virtual database environment via which a user may access datasets according to one or more access policies. In some embodiments, the data access system receives a user request to access a dataset stored in a database into the sandbox environment, wherein the database is associated with the data access system. In response to the request, the data access system may retrieve the corresponding data from the database, determine any associated sandbox access policies, and generate an anonymized data table in the sandbox environment.
G06F 21/53 - Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems during program execution, e.g. stack integrity, buffer overflow or preventing unwanted data erasure by executing in a restricted environment, e.g. sandbox or secure virtual machine
A system includes an interface, a processor, and a memory. The interface is configured to receive a version of a model from a model registry. The processor is configured to store the version of the model, start a process running the version of the model, and update a proxy with version information associated with the version of the model, wherein the updated proxy indicates to redirect an indication to invoke the version of the model to the process. The memory is coupled to the processor and configured to provide the processor with instructions.
A data processing service generates for iteratively applying a geospatial function to geospatial data. The generated code includes at least a first iterative loop and a second iterative loop. The data processing service compiles the generated code to generate compiled code that vectorized at least the second iterative loop. The data processing service receives a request from a client device to perform one or more data processing operations including applying the geospatial function to a data table of geospatial cell indices. The data processing service compiles the request into one or more tasks including at least a vectorized operation based on the compiled code and executes the one or more tasks by at least invoking the vectorized operation on the set of worker nodes.
The system is configured to: 1) receive a client request; 2) determine executor(s) to generate a response to the user request; 3) provide each of the executor(s) with an indication; 4) receive for each indication a response including an output of either a cloud output or an in-line output to generate a group of in-line outputs and a group of cloud outputs; 5) determine whether the group of in-line outputs comprises all outputs; and 6) in response to the group of in-line outputs not comprising all the outputs for the client request: a) convert the group of in-line outputs to a converted group of cloud outputs; b) generate metadata for the converted group of cloud outputs and the group of cloud outputs; and c) provide response to the client request including the metadata for the converted group of cloud outputs and the group of cloud outputs.
A function creation method is disclosed. The method comprises defining one or more database function inputs, defining cluster processing information, defining a deep learning model, and defining one or more database function outputs. A database function is created based at least in part on the one or more database function inputs, the cluster set-up information, the deep learning model, and the one or more database function outputs. In some embodiments, the database function enables a non-technical user to utilize deep learning models.
A method, system, and computer system for performing an operation with respect to a target table are disclosed. The method includes performing first and second jobs, obtaining one or more other resulting files based at least in part on unmatched rows, and obtaining a set of processed files based at least in part on performing a post-processing operation with respect to the set of resulting files. The set of processed files has less files than the set of resulting files. Performing the first job includes determining a set of matching target table files and storing target table information indicating for each of the set of matching target table files, a particular set of rows having matching rows. Performing the second job includes performing a matching action based on matched rows and obtaining the second job resulting file(s).
A method, system, and computer system for performing an operation with respect to a target table are disclosed. The method includes performing first, second and a third jobs, and obtaining a resulting table based at least in part on the second job resulting file(s) and third job resulting file(s). Performing the first job includes determining a set of matching target table files and storing target table information indicating for each of the set of matching target table files, a particular set of rows having matching rows. Performing the second job includes performing a matching action based on matched rows and obtaining the second job resulting file(s). Performing the third job includes determining unmatched rows for target table files and storing the unmatched rows in third job resulting file(s).
G06F 16/00 - Information retrievalDatabase structures thereforFile system structures therefor
G06F 7/14 - Merging, i.e. combining at least two sets of record carriers each arranged in the same ordered sequence to produce a single set having the same ordered sequence
G06F 16/14 - Details of searching files based on file metadata
G06F 16/16 - File or folder operations, e.g. details of user interfaces specifically adapted to file systems
A method, system, and computer system for performing an operation with respect to a target table are disclosed. The method includes performing first and second jobs, and obtaining other resulting files based at least in part on a second set of unmatched rows among the target table and the source table that results from the first set of unmatched rows having been processed in the second job, and obtaining a resulting table based on (i) second job resulting file(s), and (ii) other resulting files. Performing the first job includes determining a set of matching target table files and storing target table information indicating for each of the set of matching target table files, a particular set of rows having matching rows. Performing the second job includes performing a first matching action based on matched rows and a second matching action based on a subset of unmatched rows.
A method, system, and computer system for performing an operation with respect to a target table are disclosed. The method includes performing first and second jobs, and persist, in one or more deletion vector files, one or more deletion vectors for corresponding rows of the one or more target table files, and obtaining a resulting table based at least in part on the second job resulting file(s). Performing the first job includes determining a set of matching target table files and storing target table information indicating for each of the set of matching target table files, a particular set of rows having matching rows. Performing the second job includes performing a matching action based on matched rows and one or more deletion of vectors associated with previously removed rows of the matching target table files and obtaining the second job resulting file(s).
The present application discloses a method, system, and computer system for parsing files. The method includes receiving an indication that a first file is to be processed, determining to begin processing the first file using a first processing engine based at least in part on one or more predefined heuristics, indicating to process the first file using a first processing engine, determining whether a particular error in processing the first file using the first processing engine has been detected, in response to determining that the particular error has been detected, indicate to stop processing the first file using the first processing engine and indicate to continue processing using a second processing engine, and storing in memory information obtained based on processing the first file by one or more of the first processing engine and the second processing engine.
The present application discloses a method, system, and computer system for parsing files. The method includes receiving an indication that a first file is to be processed, determining to begin processing the first file using a first processing engine based at least in part on one or more predefined heuristics, indicating to process the first file using a first processing engine, determining whether a particular error in processing the first file using the first processing engine has been detected, in response to determining that the particular error has been detected, indicate to stop processing the first file using the first processing engine and indicate to continue processing using a second processing engine, and storing in memory information obtained based on processing the first file by one or more of the first processing engine and the second processing engine.
A system includes an interface and a processor. The interface is configured to receive a table indication of a data table and to receive a transaction indication to perform a transaction. The processor is configured to determine a current position N in a transaction log, determine a current state of the metadata; determine a read set associated with a transaction; attempt to write an update to the transaction log associated with a next position N+1; in response to a transaction determination that a simultaneous transaction associated with the next position N+1 already exists, determine a set of updated files; and in response to a determination that there is not an overlap between the read set associated with the current transaction and the set of updated files associated with the simultaneous transaction, attempt to write the update to the transaction to the transaction log associated with a further position N+2.
A system for clustering data into corresponding files comprises one or more processors and a memory. The one or more processors is/are configured to: 1) determine to cluster a set of data into a set of files; 2) determine a set of split points in a corresponding set of dimensions of the set of data to determine the set of files, wherein each file of the set of files has an approximate target size; and 3) store one or more items of the set of data into a corresponding file of the set of files based at least in part on the set of split points. The memory is coupled to the one or more processors and configured to provide the processor with instructions.
A system for monitoring job execution includes an interface and a processor. The interface is configured to receive an indication to start a cluster processing job. The processor is configured to determine whether processing a data instance associated with the cluster processing job satisfies a watchdog criterion; and in the event that processing the data instance satisfies the watchdog criterion, cause the processing of the data instance to be killed.
The present application discloses a method, system, and computer system for building a model associated with a dataset. The method includes receiving a data set, the dataset comprising a plurality of keys and a plurality of key-value relationships, determining a plurality of models to build based at least in part on the dataset, wherein determining the plurality of models to build comprises using the dataset format information to identify the plurality of models, building the plurality of models, and optimizing at least one of the plurality of models.
G06F 18/2132 - Feature extraction, e.g. by transforming the feature spaceSummarisationMappings, e.g. subspace methods based on discrimination criteria, e.g. discriminant analysis
79.
OPTIMIZATION OF TUNING FOR MODELS USED FOR MULTIPLE PREDICTION GENERATION
The present application discloses a method, system, and computer system for tuning a set of models. The method includes determining a set of one or more models to optimize, determining a plurality of optimizer modules with which to optimize the set of one or more models, causing the plurality of optimizer modules to respectively perform a respective optimizing process with respect to at least one model of the set of one or more models, and deploying an optimized model obtained based at least in part on optimizing metrics of the set of the one or more models.
The present application discloses a method, system, and computer system for querying a model associated with a dataset. The method includes providing an input interface via which a first entity inputs a dataset, receiving the dataset, and providing a selection interface that exposes to a second entity the plurality of models determined for the dataset and/or the plurality of results corresponding to the plurality of models using the index entries. The dataset comprises a plurality of keys and a plurality of key-value relationships, and the dataset is formatted according to a predefined format, wherein index entries are generated for a plurality of models and a plurality of results corresponding to the plurality of models.
The present application discloses a method, system, and computer system for building a model associated with a dataset. The method includes receiving a data set, the dataset comprising a plurality of keys and a plurality of key- value relationships, determining a plurality of models to build based at least in part on the dataset, wherein determining the plurality of models to build comprises using the dataset format information to identify the plurality of models, building the plurality of models, and optimizing at least one of the plurality of models.
Various embodiments of the present technology generally relate to management of big data storage and data access control systems. In some embodiments, a data access system for use in multiple application service and multiple storage service environments comprises a sandbox database for users, wherein the sandbox database is a virtual database environment via which a user may access datasets according to one or more access policies. In some embodiments, the data access system receives a user request to access a dataset stored in a database into the sandbox environment, wherein the database is associated with the data access system. In response to the request, the data access system may retrieve the corresponding data from the database, determine any associated sandbox access policies, and generate an anonymized data table in the sandbox environment.
G06F 21/53 - Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems during program execution, e.g. stack integrity, buffer overflow or preventing unwanted data erasure by executing in a restricted environment, e.g. sandbox or secure virtual machine
A system includes a plurality of computing units. A first computing unit of the plurality of computing units comprises: a communication interface configured to receive an indication to roll up data in a data table; and a processor coupled to the communication interface and configured to: build a preaggregation hash table based at least in part on a set of columns and the data table by aggregating input rows of the data table; for each preaggregated hash table entry of the preaggregated hash table: provide the preaggregated hash table entry to a second computing unit of the plurality of computing units based at least in part on a distribution hash value; receive a set of received entries from computing units of the plurality of computing units; and build an aggregation hash table based at least in part on the set of received entries by aggregating the set of received entries.
A system includes an interface, a processor, and a memory. The interface is configured to receive a version of a model from a model registry. The processor is configured to store the version of the model, start a process running the version of the model, and update a proxy with version information associated with the version of the model, wherein the updated proxy indicates to redirect an indication to invoke the version of the model to the process. The memory is coupled to the processor and configured to provide the processor with instructions.
The present application discloses a method, system, and computer system for managing a plurality of features and storing lineage information pertaining to the features. The method includes obtaining one or more datasets, determining a first feature, wherein the first feature is determined based at least in part on the one or more datasets, and storing the first feature in a feature store. The first feature is stored in association with a dataset indication of the one or more datasets from which the first feature is determined. The feature store comprises a plurality of features.
G06F 16/28 - Databases characterised by their database models, e.g. relational or object models
G06F 30/27 - Design optimisation, verification or simulation using machine learning, e.g. artificial intelligence, neural networks, support vector machines [SVM] or training a model
86.
Integrated native vectorized engine for computation
Van Hovell Tot Westerflier, Herman Rudolf Petrus Catharina
Abstract
A system comprises an interface, a processor, and a memory. The interface is configured to receive a query. The processor is configured to: determine a set of nodes for the query; determine whether a node of the set of nodes comprises a first engine node type or a second engine node type, wherein determining whether the node of the set of nodes comprises the first engine node type or the second engine node type is based at least in part on determining whether the node is able to be executed in a second engine; and generate a plan based at least in part on the set of nodes. The memory is coupled to the processor and is configured to provide the processor with instructions.
A system for executing a streaming query includes an interface and a processor. The interface is configured to receive a logical query plan. The processor is configured to determine a physical query plan based at least in part on the logical query plan. The physical query plan comprises an ordered set of operators. Each operator of the ordered set of operators comprises an operator input mode and an operator output mode. The processor is further configured to execute the physical query plan using the operator input mode and the operator output mode for each operator of the query.
A system for dataflow graph processing comprises a communication interface and a processor. The communication interface is configured receive an indication to generate a dataflow graph, wherein the indication includes a set of queries and/or commands. The processor is coupled to the communication interface and configured to: determine dependencies of each query in the set of queries on another query; determine a DAG of nodes based at least in part on the dependencies; determine the dataflow graph by determining in-line expressions for tables of the dataflow graph aggregating calculations associated with a subset of dataflow graph nodes designated as view nodes; and provide the dataflow graph.
A function creation method is disclosed. The method comprises defining one or more database function inputs, defining cluster processing information, defining a deep learning model, and defining one or more database function outputs. A database function is created based at least in part on the one or more database function inputs, the cluster set-up information, the deep learning model, and the one or more database function outputs. In some embodiments, the database function enables a non-technical user to utilize deep learning models.
The interface is to receive an indication to execute an optimize command. The processor is to receive a file name; determine whether adding a file of the file name to a current bin causes the current bin to exceed a threshold; associate the file with the current bin in response to determining that adding the file does not cause the current bin to exceed the bin threshold; in response to determining that adding the file to the current bin causes the current bin to exceed the bin threshold: associate the file with a next bin, indicate that the current bin is closed, and add the current bin to a batch of bins; determine whether a measure of the batch of bins exceeds a batch threshold; and in response to determining that the measure exceeds the batch threshold, provide the batch of bins for processing.
The present application discloses a method, system, and computer system for providing access to information stored on system for data storage. The method includes receiving a data request from a user, determining data corresponding to the data request, determining whether the user has requisite permissions to access the data, and in response to determining that the user has requisite permissions to access the data: determining a manner by which to provide access to the data, wherein the data comprises a filtered subset of stored data, and generating a token based at least in part on the user and the manner by which access to the data is to be provided.
The present application discloses a method, system, and computer system for managing a plurality of features and storing lineage information pertaining to the features. The method includes obtaining one or more datasets, determining a first feature, wherein the first feature is determined based at least in part on the one or more datasets, and storing the first feature in a feature store. The first feature is stored in association with a dataset indication of the one or more datasets from which the first feature is determined. The feature store comprises a plurality of features.
G06F 16/28 - Databases characterised by their database models, e.g. relational or object models
G06F 30/27 - Design optimisation, verification or simulation using machine learning, e.g. artificial intelligence, neural networks, support vector machines [SVM] or training a model
The present application discloses a method, system, and computer system for managing a plurality of features and storing lineage information pertaining to the features. The method includes obtaining one or more datasets, determining a first feature, wherein the first feature is determined based at least in part on the one or more datasets, and storing the first feature in a feature store. The first feature is stored in association with a dataset indication of the one or more datasets from which the first feature is determined. The feature store comprises a plurality of features.
A system for spilling comprises an interface and a processor. The interface is configured to receive an indication to perform a GROUP BY operation, wherein the indication comprises an input table and a grouping column. The processor is configured to: for each input table entry of the input table, determine a key, wherein the key is based at least in part on the input table entry and the grouping column; add the key to a grouping hash table, wherein adding the key to the grouping hash table comprises last-in, first-out (LIFO) spilling when necessary; create an output table based at least in part on the grouping hash table; and provide the output table.
The present application discloses a method, system, and computer system for building a model associated with a dataset. The method includes receiving a data set, the dataset comprising a plurality of keys and a plurality of key-value relationships, determining a plurality of models to build based at least in part on the dataset, wherein determining the plurality of models to build comprises using the dataset format information to identify the plurality of models, building the plurality of models, and optimizing at least one of the plurality of models.
A system for dataflow graph processing comprises a communication interface and a processor. The communication interface is configured receive an indication to generate a dataflow graph, wherein the indication includes a set of queries and/or commands. The processor is coupled to the communication interface and configured to: determine dependencies of each query in the set of queries on another query; determine a DAG of nodes based at least in part on the dependencies; determine the dataflow graph by determining in-line expressions for tables of the dataflow graph aggregating calculations associated with a subset of dataflow graph nodes designated as view nodes; and provide the dataflow graph.
A system for dataflow graph processing comprises a communication interface and a processor. The communication interface is configured receive an indication to generate a dataflow graph, wherein the indication includes a set of queries. The processor is coupled to the communication interface and is configured to: determine dependencies of each query in the set of queries on another query; determine a DAG of nodes based at least in part on the dependencies; insert a node in the DAG of nodes to generate an updated DAG to enforce an expectation; determine a dataflow graph based on the updated DAG; and provide the dataflow graph.
A system for dataflow graph processing comprises a communication interface and a processor. The communication interface is configured receive an indication to generate a dataflow graph, wherein the indication includes a set of queries. The processor is coupled to the communication interface and is configured to: determine dependencies of each query in the set of queries on another query; determine a DAG of nodes based at least in part on the dependencies; insert a node in the DAG of nodes to generate an updated DAG to enforce an expectation; determine a dataflow graph based on the updated DAG; and provide the dataflow graph.
A system includes an interface and a processor. The interface is configured to receive a table indication of a data table and to receive a transaction indication to perform a transaction. The processor is configured to determine a current position N in a transaction log; determine a current state of the metadata; determine a read set associated with a transaction; attempt to write an update to the transaction log associated with a next position N+1; in response to a transaction determination that a simultaneous transaction associated with the next position N+1 already exists, determine a set of updated files; and in response to a determination that there is not an overlap between the read set associated with the current transaction and the set of updated files associated with the simultaneous transaction, attempt to write the update to the transaction to the transaction log associated with a further position N+2.
Van Hovell Tot Westerflier, Herman Rudolf Petrus Catharin
Abstract
A system comprises an interface, a processor, and a memory. The interface is configured to receive a query. The processor is configured to: determine a set of nodes for the query; determine whether a node of the set of nodes comprises a first engine node type or a second engine node type, wherein determining whether the node of the set of nodes comprises the first engine node type or the second engine node type is based at least in part on determining whether the node is able to be executed in a second engine; and generate a plan based at least in part on the set of nodes. The memory is coupled to the processor and is configured to provide the processor with instructions.
G06F 12/126 - Replacement control using replacement algorithms with special data handling, e.g. priority of data or instructions, handling errors or pinning
