Systems and method are provided for a driving run performed by a sensor-equipped robot in a driving scene comprising at least one dynamic signalling agent (DSA). DSA data indicating, signalling states of the at least one DSA as a function of time is received, and a graphical user interface (GUI), comprising a schematic representation of the run and at least one DSA state timeline showing a visual indicator of the current signalling state of a corresponding one of the at least one DSA, is rendered on the GUI.
The present disclosure relates to a computer-implemented method of training a generative model to insert an object in spatial sensor data. The method comprises receiving a training sample of spatial sensor data and receiving an indication of a 3D geometric property of an object captured in the training sample. A portion of spatial sensor data corresponding to the object is removed from the training sample, resulting a cropped training sample. The generative model is trained to reconstruct the training sample from the cropped training sample by, providing to the generative model: the cropped training sample as a target input, an indication of the object as a reference input, and the 3D geometric property of the object as a conditioning input. This results in a generated output sample of spatial sensor data. Parameters of the generative model are tuned to reduce a reconstruction error between the training sample and the generated output sample, resulting in a trained generative model configured to insert at inference, in a set of spatial sensor data received as a target input, an object indicated by a reference input with a desired 3D geometric property indicated by a conditioning input.
A method of simulating behaviour of an autonomous vehicle to test a control stack for controlling the autonomous vehicle, the method comprising: providing to a simulator a scenario in which an ego vehicle is under observation; initiating operation of the simulator to execute the scenario and simulating an initial phase of a simulation of the scenario with the ego vehicle under the control of a first control stack implemented by the simulator; detecting that a handover condition is met in the scenario; and relinquishing control of the ego vehicle by the first control stack and causing a second control stack to be engaged to control the ego vehicle in the subsequent phase of the simulation, wherein the second control stack is the control stack under test.
A computer-implemented method of assessing performance of perception component, the perception component for interpreting structure in a scene comprises: receiving a set of multiple computed outputs obtained by applying the perception component to the scene, wherein each computed output comprises a confidence score: generating, from the set of multiple computed outputs, multiple pseudo-ground truth sets, wherein each pseudo-ground truth set comprises, for each computed output, a pseudo-ground truth output sampled from a set of possible ground truth outputs based on a probability distribution defined by the confidence score of the computed output; computing a performance score for the perception component applied to the scene with respect to each pseudo-ground truth set, by comparing the set of multiple outputs with that pseudo-ground truth set; and computing an overall performance score for the perception component applied to the scene, by aggregating the performance scores computed with respect to the multiple pseudo-ground truth sets.
G06V 10/764 - Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
G06V 10/774 - Generating sets of training patternsBootstrap methods, e.g. bagging or boosting
G06V 10/98 - Detection or correction of errors, e.g. by rescanning the pattern or by human interventionEvaluation of the quality of the acquired patterns
The present disclosure relates to techniques for locating and modelling a 3D object captured by a mobile robot. A cost function is defined over a set of variables, and is applied to sensor data. The set of variables comprises shape parameters of a 3D object model and a time sequence of poses of the 3D object model. The cost function penalizes inconsistency between the sensor data and the set of variables. The object belongs to a known object class, and the 3D object model or the cost function encodes expected 3D shape information associated with the known object class. The 3D object is modelled by tuning poses of the object and the shape parameters, to optimize the cost function. A visualization of a location of the robot and an object shape representing the 3D object is rendered in a graphical user interface (GUI)
G06T 7/70 - Determining position or orientation of objects or cameras
G06F 3/04815 - Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
G06F 3/0484 - Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
The present disclosure relates to techniques for training a generative model to insert an object in spatial sensor data. A first training sample of spatial sensor data of a first sensor modality, and a second training sample of spatial sensor data of a second sensor modality are received, the first training sample and the second training sample capture a common object. A first portion of sensor data corresponding to the object is removed from the first training sample, resulting a cropped training sample. A second portion of spatial sensor data corresponding to the common object is extracted from the second training sample. The generative model is trained to reconstruct the first training sample from the cropped training sample by: providing to the generative model: the cropped training sample as a target input, and the second portion of spatial sensor data as a reference input, resulting in a generated output sample of spatial sensor data, and tuning parameters of the generative model to reduce a reconstruction error between the first training sample and the generated output sample. This results in a trained generative model configured to insert at inference, in a first set of spatial sensor data of the first modality received as a target input, an object indicated in a second set of spatial sensor data of the second modality received as a reference input.
The present application relates to techniques for debugging a scenario. A scenario including a static layer and a dynamic layer is received. The static layer includes a road layout defined in a map file, and the dynamic layer defines an intended dynamic behaviour an agent. A visualization of the scenario is rendered, and a behaviour of the agent which deviates from the intended behaviour is exhibited. A user input to an editing interface is received, requesting generation of a scene structure layer that provides a visual indication map file features for the static layer. The scene structure layer is generated and overlaid on visualization. The overlay indicates that the static layer has a map file feature causing a map bug in the scenario. A second user input is received, altering a scenario parameter pertaining to the behaviour of the. Altering the parameter results in a redefined dynamic behaviour of the agent, and removes the map bug from the scenario.
A method of predicting trajectories for agents of a scenario, the method comprising, for each agent generating an agent feature vector based on one or more observed past states of the agent, computing a set of pairwise feature vectors, each computed as a combination of the agent feature vector for that agent with a respective agent feature vector generated for each other agent of the scenario, processing the pairwise feature vectors as independent inputs to one or more interaction layers of a trajectory prediction neural network to generate a pairwise output for each pairwise feature vector, aggregating the pairwise outputs over the other agents of the scenario to generate an interaction-based feature representation for each agent, processing the interaction-based feature representation in one or more prediction layers of the trajectory prediction neural network, and generating, based on the output of the one or more prediction layers, at least one predicted trajectory for each agent.
A computer-implemented method of planning actions for a mobile robot in the presence of occlusion The method comprises determining a first occluded region in a mobile robot field of view at a first time; sampling a first set of agent particles within the first occluded region; planning, using a robotic planner, based on a location and a motion state of a first agent particle of the first set of agent particles a first mobile robot action, the first mobile robot action accounting for predicted motion of the first agent particle; generating a first mobile robot control signal for causing execution of the first mobile robot action; determining a second occluded region in the mobile robot field of view at a second time; computing a propagated set of agent particles for the second time based on the first set of agent particles and a particle motion model; assigning an occlusion weight to each agent particle of the propagated set of agent particles with respect to the second occluded region; sampling a second set of agent particles within the second occluded region based on the propagated set of agent particles and the occlusion weight assigned to each agent particle; planning, using the robotic planner, based on a location and a motion state of a second agent particle of the second set of agent particles a second mobile robot action, the second mobile robot action accounting for predicted motion of the second agent particle; and generating a second mobile robot control signal for causing execution of the second mobile robot action.
A computer system configured to run queries on a static road layout, the computer system comprising: computer storage configured to store the static road layout, the static road layout comprising a section of road having a set of multiple road attributes, each road attribute described throughout the section of road by a describing function that exhibits a change in form at one or more change points along the section of road, the change points of a first of the road attributes exhibiting longitudinal misalignment with respect to the change points of a second of the road attributes; a road partitioning component configured to process the static road layout, and thereby partition the section of road into a sequence of road parts, each road part defined by a longitudinal coordinate interval, in which the describing function of every one of the road attributes has a form that is fixed throughout; a road indexing component configured to generate a road partition index having an entry for each road part, the entry indicating the form of the describing function of each road attribute as fixed throughout the longitudinal coordinate interval of that road part; and a scenario query engine configured to receive a part query, locate the entry in the road partition index for one of the road parts based on the part query, evaluate the describing function of at least one of the road attributes within the road part, and generate a part query response based on the evaluation of the describing function.
The disclosure provides systems and methods for identifying salient test runs involving an autonomous vehicle system. A processor receives sets of run data, each set representative of a driving scenario. For each set, an output set is generated, the output set comprising time-indexed events generated in response to a detected behaviour of at least one challenger agent, and a sequence of decision indicators indicating whether a driving action by an ego agent would be permissible. A data retrieval component is coupled to a results database and retrieves output sets based on the time-indexed events and the sequence of decision indicators. The processor generates the sequence of decision indicators by generating a planned trajectory of the ego agent, and determining whether the predefined driving action by the ego agent would be permissible.
A computer system for generating a scenario to be run in a simulation environment for testing the behaviour of an autonomous vehicle, the computer system comprising: a rendering component configured to: generate display data for causing a display to render a graphical user interface comprising an image of a driving environment and one or more agents within the driving environment; a parameter generator configured to generate in memory a user-defined parameter set responsive to user input defining the parameter set; and an expression manager configured to store in memory a user-defined expression set, responsive to user input defining the expression set, wherein each expression of the expression set is a user-defined function of one or more parameters of the parameter set; and a scenario generator configured to record the scenario in a scenario database; wherein the graphical user interface is configured to provide multiple agent fields for controlling the behaviour of the one or more agents when the scenario is run in a simulation environment, wherein each agent field is modifiable to associate therewith either a parameter of the user-defined parameter set or an expression of the user-defined expression set; and wherein the recorded scenario comprises the driving environment, the one or more agents, the user-defined parameter set, the user-defined expression set, and any user-defined associations between (i) the multiple agent fields and the user-defined parameter set and (ii) the multiple agent fields and the user-defined expression set, wherein each parameter associated with an agent field is controllable to directly modify an agent behaviour, and each parameter that is included in expression associated with an agent field is controllable to indirectly modify an agent behaviour.
A computer-implemented method of defining a scenario to be run in a simulation environment for testing the behaviour of a runtime stack of a sensor-equipped robot is disclosed The method comprises rendering a code editing interface on a display of a computer device, the code editing interface configured to receive an abstract scenario definition in a scenario definition language, the abstract scenario definition defining a scenario to be run in a first simulation environment for testing the behaviour of the runtime stack of the sensor-equipped robot having a role of an ego agent in the scenario; generating a concrete scenario description from the abstract scenario definition and a selected physical context compatible with the abstract scenario definition; providing the concrete scenario description to a second simulation environment; executing the concrete scenario description in the second simulation environment using a lightweight planner not forming part of the runtime stack to act as the ego agent; receiving from the second simulation environment an ego trace representative of actions of the ego agent during execution of the concrete scenario description; rendering, on the display of the computer device, a visualisation interface displaying a visual representation of the selected physical context; and rendering the actions of the ego agent on the visual representation of the selected physical context.
A computer-implemented method of generating lane detector outputs, the method comprising receiving a ground truth lane image containing one or more ground truth lane borders, each ground truth lane border comprising multiple border points; and generating a lane detector output image, by applying horizontal perturbations to the multiple border points of each ground truth lane border, the horizontal perturbations determined using a learned perturbation model, the learned perturbation model constructed to impose mutual correlation in the horizontal perturbations between vertically neighbouring border points of each ground truth lane border, and comprising parameters learned by performing a statistical analysis of lane detector errors computed between computed output images of a modelled lane detector and ground truth lane border annotations corresponding to the computed output images.
A directed search method is applied to a parameter space of a scenario for testing the performance of a robotic system in simulation. The directed search method is applied based multiple performance evaluation rules. A performance predictor is trained to probabilistically predict a pass or fail result for each rule at each point in the parameter space. An overall acquisition function is determined as follows: if a pass outcome is predicted at a given, the performance evaluation rule having the highest probability of an incorrect outcome prediction at determines the acquisition function; whereas, if a fail outcome is predicted at a given point for at least one rule, then the acquisition function is determined by the performance evaluation rule for which a fail outcome is predicted with the lowest probability of an incorrect outcome prediction.
G06F 11/34 - Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation
G05B 13/04 - Adaptive control systems, i.e. systems automatically adjusting themselves to have a performance which is optimum according to some preassigned criterion electric involving the use of models or simulators
A directed search method is applied to a parameter space of a scenario for testing the performance of a robotic system in simulation. The directed search method is applied based on at least one performance evaluation rule that returns a performance score that can be numerical or non-numerical. A hierarchical score prediction model is constructed as follows. A score classification model is trained to probabilistically predict whether a point in the parameter space will result in a numerical or non-numerical outcome. A score regression model is trained to probabilistically predict a performance score for a given point, given that the score is numerical. The score classification and regression models are used to guide a directed search of the parameter space towards the most salient instances of the scenario.
G06F 30/27 - Design optimisation, verification or simulation using machine learning, e.g. artificial intelligence, neural networks, support vector machines [SVM] or training a model
A computer-implemented method of perceiving structure in an environment comprises steps of: receiving at least one structure observation input pertaining to the environment; processing the at least one structure observation input in a perception pipeline to compute a perception output; determining one or more uncertainty source inputs pertaining to the structure observation input; and determining for the perception output an associated uncertainty estimate by applying, to the one or more uncertainty source inputs, an uncertainty estimation function learned from statistical analysis of historical perception outputs.
G06T 7/593 - Depth or shape recovery from multiple images from stereo images
G06T 7/285 - Analysis of motion using a sequence of stereo image pairs
G06T 7/73 - Determining position or orientation of objects or cameras using feature-based methods
G06V 20/56 - Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
G06V 20/58 - Recognition of moving objects or obstacles, e.g. vehicles or pedestriansRecognition of traffic objects, e.g. traffic signs, traffic lights or roads
A computer-implemented method is provided for generating a trajectory for a first agent of a plurality of agents navigating a mapped area, the method comprising: receiving an observed state of each of the plurality of agents, and map data of the mapped area; generating an initial estimated trajectory for each of the plurality of agents based on the observed state of each agent and the map data; performing a first collision assessment to determine a likelihood of collision between the first agent and each other agent, based on the initial estimated trajectory for the first agent and the initial estimated trajectory for each other agent; and generating a second estimated trajectory for the first agent based on the observed states of each of the plurality of agents, the map data, and the results of the first collision assessment.
Performance testing of mobile robot planners is considered. A particular aim is herein to find a perception error or set of perception errors for a scenario that yields 'good' performance on a predetermined perception performance metric but has an unexpected impact on downstream performance (such as planner performance), in a reduced number of simulated runs. Of particular interest are instances of good perception performance that nevertheless lead to planner failure (e.g. breaking of a safety rule) or some other salient planning outcome. A core aim is to identify perception errors that are salient in this sense in a computationally efficient manner, by maximizing the saliency of perception errors for a given number of simulated runs. This is achieved through a systematic search for salient perception error based on a predetermined perception performance metric in combination with selectively performed simulation rollouts and rules-based planning performance evaluation.
A method of generating at least one trajectory in scenario comprising an agent navigating a mapped area, the method comprising: receiving an observed state of the agent and map data of the mapped area; generating a set of multiple trajectory basis elements from the observed state of the agent based on the map data; processing one or more scenario inputs in a neural network to generate a set of weights, each weight corresponding to one of the trajectory basis elements; and generating a trajectory for the agent by weighting each trajectory basis element by its corresponding weights and combining the weighted trajectory basis elements.
In a first aspect, a computer-implemented method of fusing visual features obtained with multiple sensor modalities comprises extracting from a camera image using a visual feature extractor a plurality of camera feature vectors at respective camera plane locations in a camera image plane; receiving sensor data of a non-camera sensor modality; processing the sensor data to compute a plurality of second feature vectors in a target plane lying non- parallel to the camera image plane in 3D space; and computing a plurality of fused feature vectors at a plurality of target plane locations in the target plane, without applying monocular depth detection to the camera image. In a second aspect, the fused feature vectors are computed using a transformer neural network cross-attention between the camera feature vectors and the second feature vectors.
G06V 10/80 - Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
G06V 10/82 - Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
G06V 20/56 - Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
A computer-implemented method of generating black-box adversarial inputs to a perception component using a surrogate model of the perception component comprises receiving an initial input to the perception component and repeatedly perturbing the initial input until an adversarial input is found that satisfies an attack objective by: performing a primary attack process by perturbing the initial input based on a computed gradient of a surrogate attack loss function of the surrogate model that encodes the attack objective; wherein, if the primary attack process terminates without finding any perturbed input satisfying the promising attack condition, a backup attack process is performed to perform a randomized search of the input space of the perception component, guided by the surrogate model, until a perturbed input satisfying the promising attack condition is found; wherein the primary attack process is repeated based on the perturbed input found by the primary attack process or backup process.
A computer system for rendering a graphical user interface for visualising runs of a driving scenario in which an ego agent navigates a road layout, comprising an input configured to receive a map of the road layout and run data comprising a sequence of timestamped ego agent states and a time-varying numerical score quantifying the performance of the ego agent with respect a set of run evaluation rules; and a rendering component configured to cause a graphical user interface to display, for each rule: a plot of the time-varying numerical score, and a marker denoting a selected time index of the plot, the marker movable along the time axis to change the selected time index; and a scenario visualization comprising a visualization of the run at the selected time index, whereby moving the marker along the time axis causes the scenario visualisation to update as the time index is changed.
G06F 11/3698 - Environments for analysis, debugging or testing of software
G06N 3/006 - Artificial life, i.e. computing arrangements simulating life based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
G07C 5/00 - Registering or indicating the working of vehicles
G07C 5/02 - Registering or indicating driving, working, idle, or waiting time only
A computer-implemented method for assessing autonomous vehicle performance comprising receiving, at an input, performance data of at least one autonomous driving run, the performance data comprising at least one time series of perception errors and at least one time series of driving performance results; and generating, at a rendering component, rendering data for rendering a graphical user interface, the graphical user interface for visualizing the performance data and comprising: a perception error timeline, and a driving assessment timeline, wherein the timelines are aligned in time, and divided into multiple time steps of the at least one driving run, wherein, for each time step: the perception timeline comprises a visual indication of whether a perception error occurred at that time step, and the driving assessment timeline comprises a visual indication of driving performance at that time step.
G07C 5/08 - Registering or indicating performance data other than driving, working, idle, or waiting time, with or without registering driving, working, idle, or waiting time
25.
MOTION PREDICTION AND TRAJECTORY GENERATION FOR MOBILE AGENTS
One aspect herein pertains to a computer-implemented method of predicting agent motion comprises receiving a first observed agent state corresponding to a first time instant; determining a set of agent goals; for each agent goal, planning an agent trajectory based on the agent goal and the first observed agent state; receiving a second observed agent state corresponding to a second time instant later than the first time instant; for each goal, comparing the second observed agent state with the at least one agent trajectory planned for the goal, and thereby computing a likelihood of the goal and/or the planned agent trajectory for the goal. Another aspect pertains to trajectory generation, e.g., within a motion planner.
A computer-implemented method of planning a trajectory for an ego agent in the presence of one or more further agents, the method comprising: receiving an observed state of each agent; and, for each agent, generating a predicted distribution over trajectories and adapting the predicted distribution over trajectories to generate a planned distribution over trajectories, the adaptation based on a planner confidence score and a trajectory evaluation score; and selecting an estimated optimal trajectory from the planned distribution over trajectories for the ego agent; wherein the trajectory evaluation score is based on the interaction of each of a set of possible trajectories of that agent with one or more possible trajectories of each other agent; and wherein the planner confidence score is based on a comparison of the observed state of the agent with a previous estimated optimal trajectory of the agent determined by the planner at a previous planning iteration.
The problem of domain shift error in computer vision models and other perception components is addressed. In a label approximation phase, an approximate label distribution is computed for each input of a target batch using a trained machine learning (ML) perception component. In an online label optimization phase, a modified label distribution is assigned to each input of the target batch, via optimization of an unsupervised loss function that (i) penalizes divergence between the approximate label distribution and the modified label distribution for each input of the target batch (ii) penalizes deviation between the modified label distributions assigned to input pairs of the target batch having similar features.
G06V 10/778 - Active pattern-learning, e.g. online learning of image or video features
G06V 10/75 - Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video featuresCoarse-fine approaches, e.g. multi-scale approachesImage or video pattern matchingProximity measures in feature spaces using context analysisSelection of dictionaries
G06V 10/774 - Generating sets of training patternsBootstrap methods, e.g. bagging or boosting
A computer-implemented method of generating a test scenario for performance testing a robotic system in a simulation environment, the method comprising: receiving a dynamic element graph comprising a plurality of dynamic nodes representative of dynamic scenario agents, the dynamic element graph including one or more attachment edges connected at one end to one of the plurality of dynamic nodes and unconnected at the other end; searching a static element graph comprising a plurality of static nodes representative of static scenario elements, to identify locations at which the attachment edges can be connected to the static element graph; and storing data representative of the identified locations.
A computer-implemented method of testing an autonomous vehicle stack, comprising: storing a scenario datastore comprising a plurality of scenarios and first metadata describing each scenario, the first metadata having been generated by a first autonomous vehicle planner; retrieving a set of scenarios from the scenario datastore; for each retrieved scenario: executing a test run of the retrieved scenario using a second autonomous vehicle planner different from the first autonomous vehicle planner; generating second metadata describing the retrieved scenario based on the test run; and storing the second metadata in the scenario database.
For generating driving scenarios for testing an autonomous vehicle planner in a simulation environment, a scenario model comprises a scenario variable and a distribution associated with the scenario variable. The scenario variable is a road layout variable. Multiple sampled values of the scenario variable are computed based on the distribution associated the scenario variable. Based on the scenario model, multiple driving scenarios are generated for testing an autonomous vehicle planner in a simulation environment, each driving scenario comprising a road layout generated using a sampled value of said multiple sampled values of the scenario variable.
A computer-implemented method of generating a static layer and/or a dynamic layer for a test scenario for performance testing a robotic system in a simulation environment, the method comprising: inputting to a geometric constraint solver a scenario skeleton defining geometric elements and geometric constraints on the geometric elements, the geometric elements being modifiable based on a set of geometric variables, and the geometric constraints being characterised by one or more constraint parameters, thereby causing the geometric constraint solver to determine one or more functions of the constraint parameters for returning values of the geometric variables that satisfy the geometric constraints; receiving value(s) of the constraint parameter(s); using the received value(s) of the constraint parameter(s) and the one or more functions to compute values of the geometric variables satisfying the geometric constraints; and configuring based on the received value(s) of the constraint parameter(s), the computed value(s) of the geometric variables and the scenario skeleton, the static layer for the test scenario and/or the dynamic layer for the test scenario.
A computer implemented method of generating a scenario to be run in a simulation environment for testing the behaviour of an autonomous vehicle includes rendering, on a display of a computer device, an image of a static scene topology; and rendering on the display an object editing node comprising a set of input fields for receiving user input. The object editing node is for parameterizing an interaction of a challenger object relative to an ego object; and the method includes receiving into the input fields of the object editing node user input defining at least one temporal or relational constraint of the challenger object relative to the ego object. The at least one temporal or relational constraints define an interaction point of a defined interaction stage between the ego object and the challenger object. The method includes storing the set of constraints and defined interaction stage in an interaction container in a computer memory of the computer system; and generating a scenario to be run in a simulation environment, the scenario comprising the defined interaction stage executed on the static scene topology at the interaction point.
A computer system comprising: computer storage configured to store a static road layout; a topological indexing component configured to generate an in-memory topological index of the static road layout, the topological index in the form of a graph of nodes and edges, wherein each node corresponds to a road structure element of the static road layout, and the edges encode topological relationships between the road structure elements; a geometric indexing component configured to generate at least one in-memory geometric index of the static road layout for mapping geometric constraints to road structure elements of the static road layout; and a scenario query engine configured to receive a geometric query, search the geometric index to locate at least one static road element satisfying one or more geometric constraints of the geometric query, and return a descriptor of the at least one road structure element(s), wherein the scenario query engine is configured to receive a topological query comprising a descriptor of at least one road element, search the topological index to locate the corresponding node(s), identify at least one other node satisfying the topological query based on the topological relationships encoded in the edges of the topological index, and return a descriptor of the other node(s) satisfying the topological query.
A computer-implemented method of predicting an external actor trajectory comprises receiving, at a computer, sensor inputs for detecting and tracking an external actor; applying object tracking to the sensor inputs, in order track the external actor, and thereby determine an observed trace of the external actor over a time interval; determining a set of available goals for the external actor; for each of the available goals, determining an expected trajectory model; and comparing the observed trace of the external actor with the expected trajectory model for each of the available goals, to determine a likelihood of that goal.
B60W 50/00 - Details of control systems for road vehicle drive control not related to the control of a particular sub-unit
G05B 13/02 - Adaptive control systems, i.e. systems automatically adjusting themselves to have a performance which is optimum according to some preassigned criterion electric
G05B 13/04 - Adaptive control systems, i.e. systems automatically adjusting themselves to have a performance which is optimum according to some preassigned criterion electric involving the use of models or simulators
G06V 10/84 - Arrangements for image or video recognition or understanding using pattern recognition or machine learning using probabilistic graphical models from image or video features, e.g. Markov models or Bayesian networks
G06V 20/54 - Surveillance or monitoring of activities, e.g. for recognising suspicious objects of traffic, e.g. cars on the road, trains or boats
G06V 20/56 - Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
G08G 1/01 - Detecting movement of traffic to be counted or controlled
H04N 7/18 - Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
35.
PERFORMANCE TESTING FOR MOBILE ROBOT TRAJECTORY PLANNERS
A computer-implemented method of evaluating the performance of a trajectory planner for a mobile robot in a scenario, in which the trajectory planner is used to control the mobile robot responsive to at least one other agent of the scenario the method comprising: determining a scenario parameter set for the scenario and a likelihood of the scenario parameter set; computing an impact score for a failure event or near failure event between the mobile robot and the other agent occurring in the scenario instance, the impact score quantifying severity of the failure event or near failure event; and computing a risk score for the instance of the scenario based on the impact score and the likelihood of the scenario parameter set.
One aspect herein provides a method of analysing driving behaviour in a data processing computer system, the method comprising: receiving at the data processing computer system driving behaviour data to be analysed, wherein the driving behaviour data records vehicle movements within a monitored driving area; analysing the driving behaviour data to determine a normal driving behaviour model for the monitored driving area; using object tracking to determine driving trajectories of vehicles driving in the monitored driving area; comparing the driving trajectories with the normal driving behaviour model to identify at least one abnormal driving trajectory; and extracting a portion of the driving behaviour data corresponding to a time interval associated with the abnormal driving trajectory.
B60W 50/00 - Details of control systems for road vehicle drive control not related to the control of a particular sub-unit
B60W 60/00 - Drive control systems specially adapted for autonomous road vehicles
G05B 13/02 - Adaptive control systems, i.e. systems automatically adjusting themselves to have a performance which is optimum according to some preassigned criterion electric
G05B 13/04 - Adaptive control systems, i.e. systems automatically adjusting themselves to have a performance which is optimum according to some preassigned criterion electric involving the use of models or simulators
G06V 10/84 - Arrangements for image or video recognition or understanding using pattern recognition or machine learning using probabilistic graphical models from image or video features, e.g. Markov models or Bayesian networks
G06V 20/54 - Surveillance or monitoring of activities, e.g. for recognising suspicious objects of traffic, e.g. cars on the road, trains or boats
G06V 20/56 - Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
H04N 7/18 - Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
A computer-implemented method of evaluating a candidate trajectory for an ego agent in the presence of at least one external agent, comprising: determining a predicted state of the at least one external agent at a future timestep and the corresponding state of the ego vehicle along the candidate trajectory at the future timestep; determining a headway region for each of the ego agent and the at least one external agent, the headway region for each agent defining the region occupied by that agent within a predefined time duration from the predicted state, based on the predicted state of the agent; performing a first intersection check to determine if the predicted state of the at least one external agent intersects with the headway region of the ego agent; and performing a second intersection check to determine if the predicted state of the ego agent intersects with the headway region of the at least one external agent; wherein, if either of the first or second intersection checks determine that an intersection exists, the candidate trajectory is evaluated as invalid.
The present disclosure provides methods and systems for automatically annotating a second map using annotation data from a first map. A source piece of road structure associated with a first piece of annotation data is identified in the first map, the source piece of road structure is matched with a target piece of road structure in the second map, and a second piece of annotation data for annotating the target piece of road structure in the second map is generated using the first piece of annotation data.
Computer systems and methods for planning a spatial path and a motion profile for an ego agent. The system comprises: an input configured to receive a state of the ego agent, and a trajectory generator configured to determine, based on the state of the ego agent, a first trajectory via a constrained optimization of a cost function encoding one or more soft planning constraints, subject to one or more hard planning constraints; the first trajectory comprising a spatial path and a motion component, the spatial path determined via the constrained optimization in dependence on at least one of a hard planning constraint and a soft planning constraint on the motion component; a motion planner configured to compute, based on the spatial path of the first trajectory, a motion profile for the ego agent satisfying one or more constraints of the motion planner.
A method of detecting one or more objects in a scene using multiple calibrated object detectors, is disclosed. Each calibrated object detector is applied to the scene, resulting in one or more predicted object regions for each calibrated object detector and an uncalibrated confidence score for each predicted object region. For each calibrated detector, its score calibration function is used to determine a calibrated confidence score for each predicted object region. A reduction algorithm is applied to a combined prediction set comprising the one or more predicted object regions for each object detector and their calibrated confidence scores, resulting in a reduced prediction set.
A method of evaluating the performance of a target planner for an ego robot in a scenario. Evaluation data is generated by applying the target planner from an initial scenario state to generate an actual ego trajectory taken by the ego robot. The actual ego trajectory is defined by a target trajectory parameter. Comparison data is generated by applying a comparison planner from the same initial scenario state to generate a comparison ego trajectory for a comparison ego robot, the comparison ego trajectory comprising at least one comparison trajectory parameter. A juncture point at which the comparison trajectory parameter differs from the actual trajectory parameter is determined and a difference between the actual trajectory parameter and the comparison trajectory parameter at the juncture point is determined. A comparison between the determined difference and a threshold value indicates whether the juncture point is significant.
A computer-implemented method of generating black-box adversarial inputs to a perception component comprises computing an adversarial input by applying a perturbation to an original input, the adversarial input satisfying an attack objective when inputted to the perception component. The perturbation is determined by selectively combining component perturbations selected from a predetermined set of component perturbations. Inputs correspond to respective points in an input vector space, and the component perturbations encode principal attack directions in the input vector space for satisfying said attack objective, the principal attack directions having been determined by analyzing: (i) a set of sample attack directions, or (ii) a set of input samples.
G06F 21/57 - Certifying or maintaining trusted computer platforms, e.g. secure boots or power-downs, version controls, system software checks, secure updates or assessing vulnerabilities
A computer-implemented method of predicting an agent trajectory in a world space, the method comprising: receiving a visual input comprising an image of the agent in an image space, extracting from the visual input using a visual feature extractor a set of visual features, receiving a trajectory history input comprising an observed trajectory of the agent in the world space, extracting from the trajectory history input a set of trajectory features, and generating, based on the set of visual features and the set of trajectory features, a prediction output comprising a predicted trajectory for the agent in the world space.
G06V 10/62 - Extraction of image or video features relating to a temporal dimension, e.g. time-based feature extractionPattern tracking
B60W 50/00 - Details of control systems for road vehicle drive control not related to the control of a particular sub-unit
B60W 60/00 - Drive control systems specially adapted for autonomous road vehicles
G06V 10/82 - Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
G06V 20/58 - Recognition of moving objects or obstacles, e.g. vehicles or pedestriansRecognition of traffic objects, e.g. traffic signs, traffic lights or roads
Performance of a substitute upstream processing component is tested, in order to determine whether that performance is sufficient to support a downstream processing component, within an autonomous driving system, in place of an existing upstream processing component. The existing upstream processing component and the substitute upstream processing component are mutually interchangeable in so far as they provide the same form of outputs interpretable by the downstream processing component, such that either upstream processing component may be used without modification to the downstream processing component. A direct or indirect metric-based comparison is formulated in terms of the resulting performance of the downstream processing component.
To locate and model a 3D object captured in multiple time-series of sensor data of multiple sensor modalities, a cost function applied to the multiple time-series of sensor data is optimized. The cost function aggregates over time and the multiple sensor modalities, and is defined over a set of variables comprising one or more shape parameters of a 3D object model and a time sequence of poses of the 3D object model. The cost function penalizes inconsistency between the multiple time-series of sensor data and the set of variables. The object belongs to a known object class, and the 3D object model or the cost function encodes expected 3D shape information associated with the known object class, whereby the 3D object is located at multiple time instants and modelled by tuning each pose and the shape parameters with the objective of optimizing the cost function.
G06T 19/20 - Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
G06T 7/70 - Determining position or orientation of objects or cameras
G06V 10/764 - Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
A computer-implemented object detection method, comprising: receiving a point cloud and a 2D image having an image plane located at a known location in the point cloud; applying one or more machine learning models to the 2D image to determine a position of a detected object in the 2D image and to determine estimated dimensions and orientation of at least two sides of a bounding box corresponding to the detected object in the point cloud; and generating an initial estimated location of the at least two sides of the bounding box in the point cloud based at least in part on the estimated dimensions and orientation; optimising the initial estimated location of the at least two sides of the bounding box with respect to the point cloud to determine a detected location of the bounding box corresponding to the detected object.
G06V 20/58 - Recognition of moving objects or obstacles, e.g. vehicles or pedestriansRecognition of traffic objects, e.g. traffic signs, traffic lights or roads
G06V 10/82 - Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
A computer-implemented method of perceiving structure in a radar point cloud comprises: generating a discretised image representation of the radar point cloud having (i) an occupancy channel indicating whether or not each pixel of the discretised image representation corresponds to a point in the radar point cloud and (ii) a Doppler channel containing, for each occupied pixel, a Doppler velocity of the corresponding point in the radar point cloud; and inputting the discretised image representation to a machine learning (ML) perception component, which has been trained extract information about structure exhibited in the radar point cloud from the occupancy and Doppler channels.
G01S 13/42 - Simultaneous measurement of distance and other coordinates
G01S 7/41 - Details of systems according to groups , , of systems according to group using analysis of echo signal for target characterisationTarget signatureTarget cross-section
G01S 13/58 - Velocity or trajectory determination systemsSense-of-movement determination systems
G01S 13/89 - Radar or analogous systems, specially adapted for specific applications for mapping or imaging
48.
Implementing slowdown manoeuvres in autonomous vehicles
A computer-implemented method of determining control signals for controlling an autonomous vehicle to implement a slowdown manoeuvre, comprising: detecting an obstacle at a distance ahead of the autonomous vehicle; comparing the distance with a threshold value and implementing a slowdown manoeuvre in dependence on the comparison, the slowdown manoeuvre selected from: a first slowdown manoeuvre carried out by a kinematic function, which is a time derivative of acceleration, in which a constraint optimisation has been applied to optimise a cost function of the slowdown manoeuvre subject to a set of hard constraints that require a final acceleration, speed and position to satisfy respective acceleration, speed and position targets, given an initial speed and acceleration of the vehicle, and impose a jerk magnitude upper limit; and a second slowdown manoeuvre implemented in an adaptive cruise control mode which aims to reach a target headway between the autonomous vehicle and the obstacle.
A computer-implemented method of tracking detected objects across multiple image sensors having overlapping fields of view, the method comprising: receiving, for a first image sensor of the multiple image sensors, a first detected object region in a first 2D image plane; receiving, for a second image sensor of the multiple image sensors, a second detected object region in a second 2D image plane, the second detected object region synchronous with the first detected object region; computing, for at least two edge sections of the first detected object region, (i) an outward projection of each edge section into 3D space and (ii) an outward projection into 3D space of a corresponding edge section of the second detected object region; identifying at least four reference points in 3D space, by determining an intersection between the outward projection of each edge section of the first detected object region with the outward projection of the corresponding edge section of the second detected object region; calculating a first extent of overlap in the first 2D image plane between the first detected object region and a first inferred object region defined by the at least four reference points projected into the first image plane; calculating a second extent of overlap in the second 2D image plane between the second detected object region and a second inferred object region defined by the at least four reference points projected into the second image plane; and generating based on the first extent of overlap and the second extent of overlap an object tracking output indicating whether the first detected object region and the second detected object region correspond to the same object.
A computer-implemented method of generating road annotation data for annotating an electronic map, the method comprising accessing from persistent storage an electronic map defining a road network; generating a road topology graph encoding a topology of the road network, the road topology graph comprising nodes representing road structure elements and edges representing links between road structure elements; performing a search of the road topology graph for a predetermined graph structure; responsive to identifying a subgraph of the road topology graph exhibiting the predetermined graph structure, generating annotation data for marking in the electronic map a portion of the road network corresponding to the subgraph; and generating in persistent storage an augmented map comprising map data defining the portion of the road network and the annotation data.
A computer-implemented method of planning an ego trajectory for an ego robot in an environment in the presence of an occluding object, the method comprising: determining, for a current time step, an occluded region of the environment based on a current state of the ego robot and a current state of the occluding object; determining, based on the occluded region at the current time step, a predicted virtual agent collision region at a future time step, using a virtual agent dynamics model applied to the occluded region; and, computing, by a trajectory planner, a planned ego trajectory for the mobile robot from the current time step to the future time step accounting for the predicted virtual agent collision region of the environment.
A computer-implemented method of planning an ego trajectory for an ego robot in an environment in the presence of an occluding object, the method comprising: determining a trajectory cost function having one or more planning variables that define a planned ego trajectory, the planning variables tuneable to modify the planned ego trajectory, wherein the trajectory cost function is dependent on a predicted occluded region of the environment, as defined by the planned ego trajectory and a predicted state of the occluding object; and tuning the planning variables using an optimizer applied to the trajectory cost function, wherein the trajectory cost function encodes: (i) a progress objective that rewards modifications to the planned ego trajectory that progress the ego robot towards a chosen goal location, and (ii) a visibility objective, dependent on the predicted occluded region, that rewards modifications to the planned ego trajectory that improve visibility of the environment from the perspective of the ego robot, whereby the optimiser is incentivised to tune the planning variables in a manner satisfying the progress objective and the visibility objective.
A computer implemented method of evaluating planner performance for an ego robot in a scenario, the method comprising: receiving for each of a set of runs, run evaluation data, wherein the run evaluation data for each run is generated by applying a planner in a scenario of that run to generate an ego trajectory taken by the ego robot in the scenario; determining for the set of runs, an examination category; generating for each run, an indicator of an examination parameter for that run in the examination category; the indicator selected from a group of different indicators in the examination category; and identifying a cluster of runs of the set of runs sharing the same indicator in the examination category.
A computer implemented method of evaluating the performance of a target planner for an ego robot in a scenario, the method comprising: rendering on a display of a graphical user interface of a computer device, a dynamic visualisation of an ego robot moving along a first path in accordance with a first planned trajectory from the target planner and of a comparison ego robot moving along a second path in accordance with a second planned trajectory from a comparison planner; detecting a juncture point at which the first and second trajectories diverge; rendering the ego robot and the comparison ego robot as a single visual object in motion along a common path shared by the first and second paths prior to the juncture point; and rendering the ego robot and the comparison ego robot as separate visual objects on the display along the respective first and second paths from the juncture point.
A computer implemented method of evaluating the performance of at least one component of a planning stack for an autonomous robot, the method comprising: generating first evaluation data of a first run by operating the autonomous robot under the control of a planning stack under test in a scenario; modifying at least one operating parameter of at least one component of the planning stack by applying a variable modification to the operating parameter; generating second evaluation data of a second run by operating the autonomous robot under the control of the planning stack in which the at least one operating parameter has been modified, in the scenario; and comparing the first evaluation data with the second evaluation data using at least one performance metric for the comparison.
Methods are provided for generating a target speed for an autonomous vehicle. Waypoints on a drivable path are detected, and a curvature of the drivable path at each waypoint is determined. Using the curvature at each waypoint and a lateral acceleration limit of the vehicle, a speed at or below which the autonomous vehicle is permitted to drive is computed for each waypoint. It is then determined, for each waypoint, whether the speed can be met while adhering to predefined slowdown criteria. Upon identifying a limiting waypoint at which an allowed speed cannot be met, a target speed is set so that the vehicle attains the allowed speed at the limiting waypoint while adhering to slowdown criteria.
A computer-implemented method of estimating a 3D object pose, the method comprising: receiving 3D data comprising a full or partial view of a 3D object, the 3D object exhibiting reflective symmetry about an unknown 2D symmetry plane; applying symmetry detection to the 3D data, and thereby calculating, in 3D space, an estimated 2D symmetry plane for the 3D object; and applying 3D pose detection to the 3D data based on the estimated 2D symmetry plane, thereby computing a 3D pose estimate of the 3D object that is informed by the reflective symmetry of the 3D object.
G06V 10/44 - Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersectionsConnectivity analysis, e.g. of connected components
G06V 10/82 - Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
A computer-implemented method of planning ego actions for a mobile robot in the presence of at least one agent, comprising: searching for an optimal ego action in multiple search steps, each comprising: selecting an ego action from a set of possible ego actions, selecting an agent behaviour from a set of possible agent behaviours, running a simulation based on the selected ego action and agent behaviour, determining a possible outcome, and assigning a reward to the selected ego action, based on a reward metric, wherein selection of the ego action in later search steps is biased towards higher reward ego action(s) but selection of the agent behaviour in later search steps is biased towards riskier agent behaviour(s), a risky agent behaviour being, according to earlier search steps, more likely to result in a lower reward outcome and choosing an ego action based on the rewards computed in the search steps.
Methods and systems are provided for evaluating performance of a runtime stack of a sensor-equipped robot in a run of a scenario comprising two or more agents. For each timestep in the run, an extent of compliance of run data with one or more assessment criterion is displayed in a global assessment timeline of a graphical user interface (GUI). A user input selecting one of the agents is received at the GUI. A subset of the run data pertaining to the selected agent is then selected and an agent-specific timeline showing an indication of extent of compliance of the data subset over time is provided on the GUI.
Systems and method are provided for a driving run performed by a sensor-equipped robot in a driving scene comprising at least one dynamic signalling agent (DSA). DSA data indicating, signalling states of the at least one DSA as a function of time is received, and a graphical user interface (GUI), comprising a schematic representation of the run and at least one DSA state timeline showing a visual indicator of the current signalling state of a corresponding one of the at least one DSA, is rendered on the GUI.
A method for testing performance of a stack for planning ego vehicle trajectories in real or simulated driving scenarios applying driving rules to the scenario ground truth for evaluating the performance of the stack in the scenario, and providing output indicating whether each driving rule has been complied with; wherein the driving rules include at least one ODD-based response rule, wherein applying the ODD-based response rule includes processing the scenario ground truth over multiple time steps, to determine whether the scenario is within the defined ODD at each time step, and thereby detecting a change in the scenario that takes the scenario outside the defined ODD, and processing the internal state data, to determine whether a state change occurred within the stack, within a time interval, the output for the at least one ODD-based response rule indicating whether the state change occurred within the time interval.
G07C 5/08 - Registering or indicating performance data other than driving, working, idle, or waiting time, with or without registering driving, working, idle, or waiting time
63.
PERFORMANCE TESTING FOR MOBILE ROBOT TRAJECTORY PLANNERS
A computer-implemented method of evaluating the performance of a trajectory planner for a mobile robot in a real or simulated scenario, comprises receiving scenario ground truth of the scenario, the scenario ground truth generated using the trajectory planner to control an ego agent of the scenario responsive to at least one scenario element of the scenario. One or more performance evaluation rules for the scenario and at least one activation condition for each performance evaluation rule are received. A test oracle processes the scenario ground truth to determine whether the activation condition of each performance evaluation rule is satisfied over multiple time steps of the scenario. Each performance evaluation rule is evaluated by the test oracle, to provide at least one test result, only when its activation condition is satisfied.
A computer implemented method of training an encoder to extract features from sensor data comprises generating a plurality of training examples, each training example comprising at least two data representations of a set of sensor data, the at least two data representations related by a transformation parameterized by at least one numerical transformation value; and training the encoder based on a self-supervised regression loss function applied to the training examples. The encoder extracts respective features from the at least two data representations of each training example, and at least one numerical output value is computed from the extracted features. The self-supervised regression loss function encourages the at least one numerical output value to match the at least one numerical transformation value parameterizing the transformation.
G06V 10/774 - Generating sets of training patternsBootstrap methods, e.g. bagging or boosting
G06V 10/44 - Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersectionsConnectivity analysis, e.g. of connected components
G06V 10/766 - Arrangements for image or video recognition or understanding using pattern recognition or machine learning using regression, e.g. by projecting features on hyperplanes
A method of predicting actions of one or more actor agent in a scenario is implemented by an ego agent in the scenario. A plurality of agent models are used to generate a set of candidate futures, each candidate future providing an expected action of the actor agent. A weighting function is applied to each candidate future to indicate its relevance in the scenario. A group of candidate futures is selected for each actor agent based on the indicated relevance, wherein the plurality of agent models comprises a first model representing a rational goal directed behaviour inferable from the vehicular scene, and at least one second model representing an alternate behaviour not inferable from the vehicular scene.
A method of assessing lateral stability of a moving vehicle in a real or simulated driving scenario comprises: determining a time-varying lateral position signal for the moving vehicle; computing an evolving frequency spectrum of the time-varying lateral position signal over a moving window across the time-varying lateral position signal; and analysing the evolving frequency spectrum to extract a lateral stability signal that indicates an extent to which the moving vehicle is maintaining a stable lateral position.
A computer-implemented method of computer-implemented method of perceiving structure in a point cloud comprises: applying clustering to the point cloud, and thereby identifying at least one moving object cluster within the point cloud, the point cloud comprising time-stamped points captured over a non-zero accumulation window; determining a motion model for the moving object cluster, by fitting one or more parameters of the motion model to the time-stamped points of that cluster; using the motion model to transform the time-stamped points of the moving object cluster to a common reference time; and applying a perception component to the transformed points of the moving object cluster to extract information about structure exhibited in the transformed points.
A computer system receives scenario data generated using a trajectory planner to control an ego agent responsive to at least one other agent in a real or simulated scenario. A test oracle provides predetermined extractor functions for extracting time-varying numerical signals from the scenario data and predetermined assessor functions for assessing the extracted time-varying signals. The test oracle applies, to the scenario data, a rule graph comprising extractor nodes and assessor nodes. Each extractor node applies one of the predetermined extractor functions to the scenario data to extract an output in the form of a time-varying numerical signal. Each assessor node has one or more child nodes, each child node being one of the extractor nodes or another of the assessor nodes, and the assessor node applies one of the predetermined assessor functions to the output(s) of its child node(s). The test oracle provides an output graph comprising the output of at least one of the assessor nodes and the output(s) of at least one of its child node(s).
A computer-implemented method of assessing performance of perception component, the perception component for interpreting structure in a scene comprises: receiving a set of multiple computed outputs obtained by applying the perception component to the scene, wherein each computed output comprises a confidence score; generating, from the set of multiple computed outputs, multiple pseudo-ground truth sets, wherein each pseudo-ground truth set comprises, for each computed output, a pseudo-ground truth output sampled from a set of possible ground truth outputs based on a probability distribution defined by the confidence score of the computed output; computing a performance score for the perception component applied to the scene with respect to each pseudo-ground truth set, by comparing the set of multiple outputs with that pseudo-ground truth set; and computing an overall performance score for the perception component applied to the scene, by aggregating the performance scores computed with respect to the multiple pseudo-ground truth sets.
G06V 20/56 - Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
G06V 10/98 - Detection or correction of errors, e.g. by rescanning the pattern or by human interventionEvaluation of the quality of the acquired patterns
G06F 18/21 - Design or setup of recognition systems or techniquesExtraction of features in feature spaceBlind source separation
A computer-implemented method of modelling a perception system for perceiving objects captured in sensor data comprises: receiving a plurality of training examples, each comprising a ground truth scene for a set of sensor data and a corresponding perceived scene obtained by applying the perception system to the set of sensor data; fitting to the training examples noise model parameters, encoding a noise distribution over perceived scenes given a misdetection scene, and misdetection model parameters, encoding a misdetection distribution over misdetection scenes given a ground truth scene; computing a perception distribution over perceived scenes for a given ground truth scene by marginalizing the product of noise and misdetection distributions over multiple misdetection scenes, wherein individual objects in the ground truth scene are not associated with individual objects in the perceived scenes; fitting the noise and misdetection model parameters to match the perception distribution to the perceived scene for each training example.
A method of predicting trajectories for agents of a scenario, the method comprising, for each agent generating an agent feature vector based on one or more observed past states of the agent, computing a set of pairwise feature vectors, each computed as a combination of the agent feature vector for that agent with a respective agent feature vector generated for each other agent of the scenario, processing the pairwise feature vectors as independent inputs to one or more interaction layers of a trajectory prediction neural network to generate a pairwise output for each pairwise feature vector, aggregating the pairwise outputs over the other agents of the scenario to generate an interaction-based feature representation for each agent, processing the interaction-based feature representation in one or more prediction layers of the trajectory prediction neural network, and generating, based on the output of the one or more prediction layers, at least one predicted trajectory for each agent.
A computer system for generating a scenario to be run in a simulation environment for testing the behaviour of an autonomous vehicle, the computer system comprising: a rendering component configured to: generate display data for causing a display to render a graphical user interface comprising an image of a driving environment and one or more agents within the driving environment; a parameter generator configured to generate in memory a user-defined parameter set responsive to user input defining the parameter set; and an expression manager configured to store in memory a user-defined expression set, responsive to user input defining the expression set, wherein each expression of the expression set is a user-defined function of one or more parameters of the parameter set; and a scenario generator configured to record the scenario in a scenario database; wherein the graphical user interface is configured to provide multiple agent fields for controlling the behaviour of the one or more agents when the scenario is run in a simulation environment, wherein each agent field is modifiable to associate therewith either a parameter of the user-defined parameter set or an expression of the user-defined expression set; and wherein the recorded scenario comprises the driving environment, the one or more agents, the user-defined parameter set, the user-defined expression set, and any user- defined associations between (i) the multiple agent fields and the user-defined parameter set and (ii) the multiple agent fields and the user-defined expression set, wherein each parameter associated with an agent field is controllable to directly modify an agent behaviour, and each parameter that is included in expression associated with an agent field is controllable to indirectly modify an agent behaviour.
G06F 30/12 - Geometric CAD characterised by design entry means specially adapted for CAD, e.g. graphical user interfaces [GUI] specially adapted for CAD
73.
IDENTIFYING SALIENT TEST RUNS INVOLVING MOBILE ROBOT TRAJECTORY PLANNERS
The disclosure provides systems and methods for identifying salient test runs involving an autonomous vehicle system. A processor receives sets of run data, each set representative of a driving scenario. For each set, an output set is generated, the output set comprising time-indexed events generated in response to a detected behaviour of at least one challenger agent, and a sequence of decision indicators indicating whether a driving action by an ego agent would be permissible. A data retrieval component is coupled to a results database and retrieves output sets based on the time-indexed events and the sequence of decision indicators. The processor generates the sequence of decision indicators by generating a planned trajectory of the ego agent, and determining whether the predefined driving action by the ego agent would be permissible.
A computer-implemented method of processing images for extracting information about known objects comprises the steps of receiving an image containing a view of a known object at a scale dependent on an object distance of the known object from an image capture location of the image; determining, from a world model representing one or more known objects in the vicinity of the image capture location, an object location of the known object, the object location and the image capture location defined in a world frame of reference; and based on the image capture location and the object location in the world frame of reference, applying image scaling to the image, to extract a rescaled image containing a rescaled view of the known object at a scale that is substantially independent of the object distance from the image capture location.
A computer implemented method of path verification in a computer system is described. A user provides input to mark a displayed image of a scenario. A path is generated representing the trajectory of a vehicle. Control points along the path are recorded, each control point associated with a vehicle position and target speed. A vehicle position end target speed of two control points is used to calculate at least one path verification parameter which defines how a vehicle travelling along the path would behave. The at least one verification parameter is compared with a corresponding threshold value; and an alert is generated to the user at the user interface when the at least one path verification parameter exceeds the corresponding threshold value.
B60W 60/00 - Drive control systems specially adapted for autonomous road vehicles
B60W 50/14 - Means for informing the driver, warning the driver or prompting a driver intervention
G01C 21/36 - Input/output arrangements for on-board computers
G05B 13/02 - Adaptive control systems, i.e. systems automatically adjusting themselves to have a performance which is optimum according to some preassigned criterion electric
A computer-implemented method of generating lane detector outputs, the method comprising receiving a ground truth lane image containing one or more ground truth lane borders, each ground truth lane border comprising multiple border points; and generating a lane detector output image, by applying horizontal perturbations to the multiple border points of each ground truth lane border, the horizontal perturbations determined using a learned perturbation model, the learned perturbation model constructed to impose mutual correlation in the horizontal perturbations between vertically neighbouring border points of each ground truth lane border, and comprising parameters learned by performing a statistical analysis of lane detector errors computed between computed output images of a modelled lane detector and ground truth lane border annotations corresponding to the computed output images.
A directed search method is applied to a parameter space of a scenario for testing the performance of a robotic system in simulation. The directed search method is applied based on at least one performance evaluation rule that returns a performance score that can be numerical or non-numerical. A hierarchical score prediction model is constructed as follows. A score classification model is trained to probabilistically predict whether a point in the parameter space will result in a numerical or non-numerical outcome. A score regression model is trained to probabilistically predict a performance score for a given point, given that the score is numerical. The score classification and regression models are used to guide a directed search of the parameter space towards the most salient instances of the scenario.
G05B 13/02 - Adaptive control systems, i.e. systems automatically adjusting themselves to have a performance which is optimum according to some preassigned criterion electric
G05B 13/04 - Adaptive control systems, i.e. systems automatically adjusting themselves to have a performance which is optimum according to some preassigned criterion electric involving the use of models or simulators
G06F 30/20 - Design optimisation, verification or simulation
A directed search method is applied to a parameter space of a scenario for testing the performance of a robotic system in simulation. The directed search method is applied based multiple performance evaluation rules. A performance predictor is trained to probabilistically predict a pass or fail result for each rule at each point in the parameter space. An overall acquisition function is determined as follows: if a pass outcome is predicted at a given, the performance evaluation rule having the highest probability of an incorrect outcome prediction at determines the acquisition function; whereas, if a fail outcome is predicted at a given point for at least one rule, then the acquisition function is determined by the performance evaluation rule for which a fail outcome is predicted with the lowest probability of an incorrect outcome prediction.
G06F 11/36 - Prevention of errors by analysis, debugging or testing of software
G05B 13/04 - Adaptive control systems, i.e. systems automatically adjusting themselves to have a performance which is optimum according to some preassigned criterion electric involving the use of models or simulators
G05B 13/02 - Adaptive control systems, i.e. systems automatically adjusting themselves to have a performance which is optimum according to some preassigned criterion electric
A computer system for analysing driving scenes in relation to an autonomous vehicle (AV) operational design domain (ODD), the computer system comprising: an input configured to receive a definition of the ODD in a formal ontology language; a scene processor configured to receive data of a driving scene and extract a scene representation therefrom, the data comprising an ego trace, at least one agent trace, and environmental data about an environment in which the traces were captured or generated, wherein the scene representation is an ontological representation of both static and dynamic elements of the driving scene extracted from the traces and the environmental data, and expressed in the same formal ontology language as the ODD; and a scene analyzer configured to match the static and dynamic elements of the scene representation with corresponding elements of the ODD, and thereby determine whether or not the driving scene is within the defined ODD.
G06F 30/20 - Design optimisation, verification or simulation
B60W 60/00 - Drive control systems specially adapted for autonomous road vehicles
B60W 40/02 - Estimation or calculation of driving parameters for road vehicle drive control systems not related to the control of a particular sub-unit related to ambient conditions
Abstract: A driving scenario is extracted from real-world driving data captured within a road layout. A simulation is run based on the extracted driving scenario, in which an ego agent and a simulated non-ego agent each exhibit closed-loop behaviour. The closed-loop behaviour of the ego agent is determined by autonomous decisions taken in an AV stack under testing in response to simulated inputs, reactive to the simulated agent. The closed-loop behaviour of the non-ego agent is determined by implementing an inferred goal or behaviour, reactive to the ego agent. The goal or behaviour is inferred from an observed trace of a real-world agent extracted from the real-world driving data.
G06F 11/36 - Prevention of errors by analysis, debugging or testing of software
G06F 30/27 - Design optimisation, verification or simulation using machine learning, e.g. artificial intelligence, neural networks, support vector machines [SVM] or training a model
81.
GENERATING SIMULATION ENVIRONMENTS FOR TESTING AV BEHAVIOUR
A computer implemented method of generating a scenario to be run in a simulation environment for testing the behaviour of an autonomous vehicle is described. An image is rendered on a display. A user can mark multiple locations to create at least one path for an agent vehicle in the rendered image. A path is generated which passes through the locations and rendered on the display. A user can define at least one behavioural parameter for controlling behaviour of the agent vehicle associated with the at least one path when the scenario is run in a simulation environment. The scenario is recorded for future use.
A computer-implemented method is provided for generating a trajectory for a first agent of a plurality of agents navigating a mapped area, the method comprising: receiving an observed state of each of the plurality of agents, and map data of the mapped area; generating an initial estimated trajectory for each of the plurality of agents based on the observed state of each agent and the map data; performing a first collision assessment to determine a likelihood of collision between the first agent and each other agent, based on the initial estimated trajectory for the first agent and the initial estimated trajectory for each other agent; and generating a second estimated trajectory for the first agent based on the observed states of each of the plurality of agents, the map data, and the results of the first collision assessment.
A method of generating at least one trajectory in scenario comprising an agent navigating a mapped area, the method comprising: receiving an observed state of the agent and map data of the mapped area; generating a set of multiple trajectory basis elements from the observed state of the agent based on the map data; processing one or more scenario inputs in a neural network to generate a set of weights, each weight corresponding to one of the trajectory basis elements; and generating a trajectory for the agent by weighting each trajectory basis element by its corresponding weights and combining the weighted trajectory basis elements.
A computer-implemented method of evaluating the performance of a full or partial autonomous vehicle (AV) stack in simulation, the method comprising: applying an optimization algorithm to a numerical performance function defined over a scenario space, wherein the numerical performance function quantifies the extent of success or failure of the AV stack as a numerical score, and the optimization algorithm searches the scenario space for a driving scenario in which the extent of failure of the AV stack is substantially maximized, wherein the optimization algorithm evaluates multiple driving scenarios in the search space over multiple iterations, by running a simulation of each driving scenario in a simulator, in order to provide perception inputs to the AV stack, and thereby generate at least one simulated agent trace and a simulated ego trace reflecting autonomous decisions taken in the AV stack in response to the simulated perception inputs, wherein later iterations of the multiple iterations are guided by the results of previous iterations of the multiple iterations, with the objective of finding the driving scenario for which the extent of failure of the AV stack is maximized.
An occlusion metric is computed for a target object in a 3D multi-object simulation. The target object is represented in 3D space by a collision surface and a 3D bounding box. In a reference surface defined in 3D space, a bounding box projection is determined for the target object with respect to an ego location. The bounding box projection is used to determine a set of reference points in 3D space. For each reference point of the set of reference points, a corresponding ray is cast based on the ego location, and it is determined whether the ray is an object ray that intersects the collision surface of the target object. For each such object ray, it is determined whether the object ray is occluded. The occlusion metric conveys an extent to which the object rays are occluded.
A method of annotating known objects in road images captured from a sensor-equipped vehicle, the method implemented in an annotation system and comprising: receiving at the annotation system a road image containing a view of a known object; receiving ego localization data, as computed in a map frame of reference, via localization applied to sensor data captured by the sensor-equipped vehicle, the ego localization data indicating an image capture pose of the road image in the map frame of reference; determining, from a predetermined road map, an object location of the known object in the map frame of reference, the predetermined road map representing a road layout the map frame of reference, wherein the known object is one of: a piece of road structure, and an object on or adjacent a road; computing, in an image plane defined by the image capture pose, an object projection, by projecting an object model of the known object from the object location into the image plane; and storing, in an image database, image data of the road image, in association with annotation data of the object projection for annotating the image data with a location of the known object in the image plane.
G06V 20/56 - Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
G06T 7/70 - Determining position or orientation of objects or cameras
G06V 10/44 - Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersectionsConnectivity analysis, e.g. of connected components
G06V 20/70 - Labelling scene content, e.g. deriving syntactic or semantic representations
87.
MOTION PREDICTION AND TRAJECTORY GENERATION FOR MOBILE AGENTS
One aspect herein pertains to a computer-implemented method of predicting agent motion comprises receiving a first observed agent state corresponding to a first time instant; determining a set of agent goals; for each agent goal, planning an agent trajectory based on the agent goal and the first observed agent state; receiving a second observed agent state corresponding to a second time instant later than the first time instant; for each goal, comparing the second observed agent state with the at least one agent trajectory planned for the goal, and thereby computing a likelihood of the goal and/or the planned agent trajectory for the goal. Another aspect pertains to trajectory generation, e.g., within a motion planner.
A computer-implemented method of evaluating the performance of a target planner for an ego robot comprises receiving evaluation data for evaluating the performance of the target planner in the scenario, generated by applying the target planner at incrementing planning steps, to compute a series of ego plans that respond to changes in the scenario and are implemented in the scenario to cause changes in an ego state. The evaluation data includes the ego plan computed by the target planner at one of the planning steps, and a scenario state at a time instant of the scenario. The evaluation data is used to evaluate the target planner by computing a reference plan for said time instant based on the scenario state, the scenario state including the ego state at that time instant, and computing at least one evaluation score for comparing the ego plan with the reference plan.
B60W 50/00 - Details of control systems for road vehicle drive control not related to the control of a particular sub-unit
B60W 50/06 - Improving the dynamic response of the control system, e.g. improving the speed of regulation or avoiding hunting or overshoot
B60W 60/00 - Drive control systems specially adapted for autonomous road vehicles
G05B 13/02 - Adaptive control systems, i.e. systems automatically adjusting themselves to have a performance which is optimum according to some preassigned criterion electric
89.
GENERATING SIMULATION ENVIRONMENTS FOR TESTING AUTONOMOUS VEHICLE BEHAVIOUR
For generating driving scenarios for testing an autonomous vehicle planner in a simulation environment, a scenario model comprises a scenario variable and a distribution associated with the scenario variable. The scenario variable is a road layout variable. Multiple sampled values of the scenario variable are computed based on the distribution associated the scenario variable. Based on the scenario model, multiple driving scenarios are generated for testing an autonomous vehicle planner in a simulation environment, each driving scenario comprising a road layout generated using a sampled value of said multiple sampled values of the scenario variable.
The problem of domain shift error in computer vision models and other perception components is addressed. In a label approximation phase, an approximate label distribution is computed for each input of a target batch using a trained machine learning (ML) perception component. In an online label optimization phase, a modified label distribution is assigned to each input of the target batch, via optimization of an unsupervised loss function that (i) penalizes divergence between the approximate label distribution and the modified label distribution for each input of the target batch (ii) penalizes deviation between the modified label distributions assigned to input pairs of the target batch having similar features.
A computer-implemented method of predicting behaviour of an agent for executing an objective of a mobile robot in the vicinity of the agent, in dependence on the predicted behaviour comprises: determining a reference path, wherein multiple actions are available to the agent, and the reference path relates to one of those actions; projecting a measured velocity vector of the agent onto a reference path, thereby determining a projected speed value for the agent along the reference path; computing predicted agent motion data for the agent along the reference path based on the projected speed value; and generating a series of control signals for controlling a mobile robot to fulfil the objective in dependence on the predicted agent motion data.
A computer-implemented method of evaluating the performance of a trajectory planner for a mobile robot in a scenario, in which the trajectory planner is used to control the mobile robot responsive to at least one other agent of the scenario the method comprising: determining a scenario parameter set for the scenario and a likelihood of the scenario parameter set; computing an impact score for a failure event or near failure event between the mobile robot and the other agent occurring in the scenario instance, the impact score quantifying severity of the failure event or near failure event; and computing a risk score for the instance of the scenario based on the impact score and the likelihood of the scenario parameter set.
In one aspect, hierarchical image segmentation is applied to an image formed of a plurality of pixels, by classifying the pixels according to a hierarchical classification scheme, in which at least some of those pixels are classified by a parent level classifier in relation to a set of parent classes, each of which is associated with a subset of child classes, and each of those pixels is also classified by at least one child level classifier in relation to one of the subsets of child classes, wherein each of the parent classes corresponds to a category of visible structure, and each of the subset of child classes associated with it corresponds to a different type of visible structure within that category.
G06K 9/62 - Methods or arrangements for recognition using electronic means
G06F 18/214 - Generating training patternsBootstrap methods, e.g. bagging or boosting
G06F 18/243 - Classification techniques relating to the number of classes
G06K 9/46 - Extraction of features or characteristics of the image
G06N 3/084 - Backpropagation, e.g. using gradient descent
G06V 10/40 - Extraction of image or video features
G06V 10/44 - Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersectionsConnectivity analysis, e.g. of connected components
G06V 10/75 - Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video featuresCoarse-fine approaches, e.g. multi-scale approachesImage or video pattern matchingProximity measures in feature spaces using context analysisSelection of dictionaries
G06V 10/764 - Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
G06V 10/771 - Feature selection, e.g. selecting representative features from a multi-dimensional feature space
G06V 10/82 - Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
G06V 20/56 - Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
A computer system comprising: computer storage configured to store a static road layout; a topological indexing component configured to generate an in-memory topological index of the static road layout, the topological index in the form of a graph of nodes and edges, wherein each node corresponds to a road structure element of the static road layout, and the edges encode topological relationships between the road structure elements; a geometric indexing component configured to generate at least one in-memory geometric index of the static road layout for mapping geometric constraints to road structure elements of the static road layout; and a scenario query engine configured to receive a geometric query, search the geometric index to locate at least one static road element satisfying one or more geometric constraints of the geometric query, and return a descriptor of the at least one road structure element(s), wherein the scenario query engine is configured to receive a topological query comprising a descriptor of at least one road element, search the topological index to locate the corresponding node(s), identify at least one other node satisfying the topological query based on the topological relationships encoded in the edges of the topological index, and return a descriptor of the other node(s) satisfying the topological query.
A computer system configured to run queries on a static road layout, the computer system comprising: computer storage configured to store the static road layout, the static road layout comprising a section of road having a set of multiple road attributes, each road attribute described throughout the section of road by a describing function that exhibits a change in form at one or more change points along the section of road, the change points of a first of the road attributes exhibiting longitudinal misalignment with respect to the change points of a second of the road attributes; a road partitioning component configured to process the static road layout, and thereby partition the section of road into a sequence of road parts, each road part defined by a longitudinal coordinate interval, in which the describing function of every one of the road attributes has a form that is fixed throughout; a road indexing component configured to generate a road partition index having an entry for each road part, the entry indicating the form of the describing function of each road attribute as fixed throughout the longitudinal coordinate interval of that road part; and a scenario query engine configured to receive a part query, locate the entry in the road partition index for one of the road parts based on the part query, evaluate the describing function of at least one of the road attributes within the road part, and generate a part query response based on the evaluation of the describing function.
A computer-implemented method of generating black-box adversarial inputs to a perception component using a surrogate model of the perception component comprises receiving an initial input to the perception component and repeatedly perturbing the initial input until an adversarial input is found that satisfies an attack objective by: performing a primary attack process by perturbing the initial input based on a computed gradient of a surrogate attack loss function of the surrogate model that encodes the attack objective; wherein, if the primary attack process terminates without finding any perturbed input satisfying the promising attack condition, a backup attack process is performed to perform a randomized search of the input space of the perception component, guided by the surrogate model, until a perturbed input satisfying the promising attack condition is found; wherein the primary attack process is repeated based on the perturbed input found by the primary attack process or backup process.
A computer system for planning mobile robot trajectories, the computer system comprising: an input configured to receive a set of scenario description parameters describing a scenario and a desired goal for the mobile robot therein; a runtime optimizer configured to compute a final mobile robot trajectory that substantially optimizes a cost function for the scenario, subject to a set of hard constraints that the final mobile robot trajectory is guaranteed to satisfy; and a trained function approximator configured to compute, from the set of scenario description parameters, initialization data defining an initial mobile robot trajectory. The computer system is configured to initialize the runtime optimizer with the initialization data, in order to guide the optimizer from the initial mobile robot trajectory to the final mobile robot trajectory that satisfies the hard constraints, the function approximator having been trained on example sets of scenario description parameters and ground truth initialization data for the example sets of scenario description parameters.
A computer-implemented method of determining control actions for controlling a mobile robot comprises: receiving a set of scenario description parameters describing a scenario and a desired goal for the mobile robot therein; in a first constrained optimization stage, applying a first optimizer to determine a first series of control actions that substantially globally optimizes a preliminary cost function for the scenario, the preliminary cost function based on a first computed trajectory of the mobile robot, as computed by applying a preliminary robot dynamics model to the first series of control actions, and in a second constrained optimization stage, applying a second optimizer to determine a second series of control actions that substantially globally optimizes a full cost function for the scenario, the full cost function based on a second computed trajectory of the mobile robot, as computed by applying a full robot dynamics model to the second series of control actions; wherein initialization data of at least one of the first computed trajectory and the first series of control actions is used to initialize the second optimizer for determining the second series of control actions, and wherein the preliminary robot dynamic model approximates the full robot dynamics model, the cost functions embody similar objectives to each encourage achievement of the desired goal, and both are optimized with respect to similar hard constraints, such that the initialization data guides the second optimizer to the substantially globally-optimal second series of control actions.
Performance of a substitute upstream processing component is tested, in order to determine whether that performance is sufficient to support a downstream processing component, within an autonomous driving system, in place of an existing upstream processing component. The existing upstream processing component and the substitute upstream processing component are mutually interchangeable in so far as they provide the same form of outputs interpretable by the downstream processing component, such that either upstream processing component may be used without modification to the downstream processing component. A direct or indirect metric-based comparison is formulated in terms of the resulting performance of the downstream processing component.
G06F 11/36 - Prevention of errors by analysis, debugging or testing of software
B60W 40/00 - Estimation or calculation of driving parameters for road vehicle drive control systems not related to the control of a particular sub-unit
G05D 1/00 - Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
A method of testing a candidate perception setup for an autonomous vehicle, comprising: receiving ground truth of a driving scenario run; providing a time-sequence of ground truth snapshots of the driving scenario run, wherein a decision making component independent of the ego agent determines a first time-sequence of decisions for the ground truth snapshots; providing a time-sequence of ablated snapshots of the driving scenario run, each ablated snapshot generated so as to cause a perception error that is representative of the candidate perception setup, wherein the decision making component determines a second time-sequence of decisions for the ablated snapshots; and computing a similarity measure between the first and second time-sequences of decisions denoting an extent to which the candidate perception setup caused a change in one or more decisions points of the second time-sequence relative to the first time-sequence, a decision point occurring when a decision changes between adjacent timesteps.
