A sensor device (100) for capturing images of a scene comprises a sensor substrate (110) having a light receiving surface (112) on which a plurality of light receiving elements is arranged, sensor optics (120) that is configured to guide light from the scene onto the light receiving substrate (112), and a control unit (130) that is configured to control operation of the sensor device (100). Here, when the scene comprises a calibration chart (200) that shows a texture (T) with known geometrical features arranged with a predetermined pitch (P) that lead to a moiré effect when imaged by the plurality of light receiving elements, the control unit (130) is configured to convert the image (S) showing the moiré effect such that it shows the texture (T) of the calibration chart (200) without distortion, and to determine imaging properties of the sensor optics (120) based on the converted image (M).
H04N 17/00 - Diagnosis, testing or measuring for television systems or their details
H04N 17/04 - Diagnosis, testing or measuring for television systems or their details for receivers
H04N 25/13 - Arrangement of colour filter arrays [CFA]Filter mosaics characterised by the spectral characteristics of the filter elements
G06T 7/80 - Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
H04N 25/61 - Noise processing, e.g. detecting, correcting, reducing or removing noise the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4"
H04N 25/615 - Noise processing, e.g. detecting, correcting, reducing or removing noise the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4" involving a transfer function modelling the optical system, e.g. optical transfer function [OTF], phase transfer function [PhTF] or modulation transfer function [MTF]
H04N 25/671 - Noise processing, e.g. detecting, correcting, reducing or removing noise applied to fixed-pattern noise, e.g. non-uniformity of response for non-uniformity detection or correction
2.
OBJECT TRACKING CIRCUITRY AND OBJECT TRACKING METHOD
The present disclosure generally pertains to object tracking circuitry configured to: obtain time-of-flight data, based on a time-of-flight measurement, the time-of-flight data indicating the object; obtain image data indicating the object and at least one light spot, the light spot originating from light emitted for the time-of-flight measurement; and generate object data based on the time-of-flight data, on the image data, and on an association between the time-of-flight data and the image data, the association being based on the at least one light spot detected in the image data.
A sensor device (10) comprises a vision sensor (1010) that comprises a pixel array (1011) having a plurality of pixels (51) each being configured to receive light and to perform photoelectric conversion to generate an electrical signal, based on which electrical signals a data stream is formed and output by the vision sensor (1010), a trigger unit (1020) that is configured to receive portions of the data stream and to generate trigger signals based on said portions of the data stream and in combination or optionally based on a feedback signal from a processing unit (1030), and the processing unit (1030) that is configured to receive the trigger signals and to carry out a predetermined processing of a portion of the data stream for which the predetermined processing was indicated to be allowed by the respective trigger signal. Here, the processing unit (1030) is configured to use a first machine learning module (1035) to carry out the predetermined processing which first machine learning module (1035) has been trained such as to optimize the predetermined processing together with a second machine learning module (1025) used by the trigger unit (1020) to generate the trigger signals and/or a third machine learning module (1015) used by the vision sensor (1010) to generate the data stream.
A sensor device (10) comprises a pixel array (1010) comprising a plurality of pixels (51) each being configured to receive light and to perform photoelectric conversion to generate an electrical signal, a storage unit (1020) that is configured to store fixed pattern noise, FPN, data indicating FPN of the pixel array (1010), and an output unit (1030) that is configured to output the electrical signals generated by the pixels (51) and the FPN data.
H04N 25/47 - Image sensors with pixel address outputEvent-driven image sensorsSelection of pixels to be read out based on image data
G06V 10/82 - Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
H04N 25/67 - Noise processing, e.g. detecting, correcting, reducing or removing noise applied to fixed-pattern noise, e.g. non-uniformity of response
H04N 25/672 - Noise processing, e.g. detecting, correcting, reducing or removing noise applied to fixed-pattern noise, e.g. non-uniformity of response for non-uniformity detection or correction between adjacent sensors or output registers for reading a single image
H04N 25/673 - Noise processing, e.g. detecting, correcting, reducing or removing noise applied to fixed-pattern noise, e.g. non-uniformity of response for non-uniformity detection or correction by using reference sources
H04N 25/677 - Noise processing, e.g. detecting, correcting, reducing or removing noise applied to fixed-pattern noise, e.g. non-uniformity of response for non-uniformity detection or correction for reducing the column or line fixed pattern noise
H04N 25/79 - Arrangements of circuitry being divided between different or multiple substrates, chips or circuit boards, e.g. stacked image sensors
5.
DEPTH SENSOR DEVICE AND METHOD FOR OPERATING A DEPTH SENSOR DEVICE
A sensor device (1000) for generating depth information for an object (O), comprising a projector unit (1010) that is configured to project illumination patterns to the object (O); a receiver unit (1020) that is configured to detect intensities of light reflected from the object (O) stemming from the illumination with the illumination patterns and/or from illumination with ambient light; and a control unit (1030). The control unit (1030) comprises a system parameter unit (1035) that is configured to set system parameters (SP) of the sensor device (1000), a decoder unit (1036) that is configured to generate the depth information based on the illumination patterns reflected from the object (O) and detected by the receiver unit (1020), and an optimizer unit (1037) that is configured to determine system parameter (SP) changes that maximize the depth sensing performance in current environmental conditions.
G01B 11/25 - Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures by projecting a pattern, e.g. moiré fringes, on the object
H04N 25/42 - Extracting pixel data from image sensors by controlling scanning circuits, e.g. by modifying the number of pixels sampled or to be sampled by switching between different modes of operation using different resolutions or aspect ratios, e.g. switching between interlaced and non-interlaced mode
6.
SENSOR DEVICE AND METHOD FOR OPERATING A SENSOR DEVICE
A sensor device (10) comprises a vision sensor (1010) that comprises a pixel array (1011) having a plurality of pixels (51) each being configured to receive light and to perform photoelectric conversion to generate an electrical signal, based on which electrical signals a data stream is formed and output by the vision sensor (1010), a modulation unit (1020) that is configured to receive portions of the data stream and to modulate said portions of the data stream to generate modulated data portions, and a processing unit (1030) that is configured to receive the modulated data portions and to carry out a predetermined processing of the modulated data portions. Here, the processing unit (1030) is configured to use a first machine learning module (1035) to carry out the predetermined processing which first machine learning module (1035) has been trained such as to optimize the predetermined processing together with a second machine learning module (1025) used by the modulation unit (1020) to generate the modulated data portions.
A pixel circuit includes a photoelectric conversion element configured to convert incident radiation into a photoreceptor current IPC. A photoreceptor circuit converts the photoreceptor current IPC into a photoreceptor voltage VPR in a conversion period, wherein an operating point of the photoreceptor circuit is set in an initialization period in response to an active initialization signal INIT. An event detection circuit outputs a digital event signal in response to a predefined change of the photoreceptor voltage VPR.
A radiation sensitive circuit continuously converts a first photocurrent generated by incident radiation into a pixel voltage signal VPR and outputs a gain control signal VGC based on a voltage level of the pixel voltage signal VPR. An active pixel circuit integrates a second photocurrent generated by incident radiation in an exposure period and converts a resulting electric charge into a pixel output voltage VSL at a conversion gain controllable by the gain control signal VGC.
H04N 25/59 - Control of the dynamic range by controlling the amount of charge storable in the pixel, e.g. modification of the charge conversion ratio of the floating node capacitance
H04N 25/706 - Pixels for exposure or ambient light measuring
9.
SENSOR DEVICE AND METHOD FOR OPERATING A SENSOR DEVICE
A sensor device (10) comprises a vision sensor (1010) that comprises a pixel array (1011) having a plurality of event detection pixels (51) each being configured to receive light and to perform photoelectric conversion to generate event data based on the received light, which event data indicate as an event the occurrence of an intensity change of the light above an event detection threshold. The sensor device comprises further an encoding unit (1020) that is configured to compress the event data provided from the pixel array (1011) according to different compression schemes, a control unit (1030) that is configured to determine the compression scheme to be used by the encoding unit (1020) and a processing unit (1040) that is configured to receive the compressed event data from the encoding unit (1020) and to carry out predetermined processing on the compressed event data. Here, the processing unit (1040) provides feedback information about the results of the predetermined processing to the control unit (1030), and the control unit (1030) is configured to use the feedback information to determine the compression scheme to be used.
A sensor module comprising circuitry configured to provide a dual sensing layer (400), the dual sensing layer (400) providing self-registered RGB imaging capabilities and SPAD sensing capabilities, wherein the self-registration is realized by a stacked implementation of sensing layers.
H04N 23/11 - Cameras or camera modules comprising electronic image sensorsControl thereof for generating image signals from different wavelengths for generating image signals from visible and infrared light wavelengths
H04N 25/10 - Circuitry of solid-state image sensors [SSIS]Control thereof for transforming different wavelengths into image signals
H04N 25/13 - Arrangement of colour filter arrays [CFA]Filter mosaics characterised by the spectral characteristics of the filter elements
H04N 25/17 - Colour separation based on photon absorption depth, e.g. full colour resolution obtained simultaneously at each pixel location
A pixel circuit (100) includes a radiation sensitive circuit (110) that converts a change of incident radiation into a pixel voltage signal VPR, wherein the pixel voltage signal VPR increases with increasing radiation intensity. A first capacitive element (121) receives the pixel voltage signal VPR at a first electrode. A second capacitive element (131) receives the pixel voltage signal VPR at a first electrode simultaneously with the first capacitive element (121). A first comparator circuit (125) compares a first resettable voltage at a second electrode of the first capacitive element (121) with a first threshold voltage VTH. A second comparator circuit (135) compares a second resettable voltage at a second electrode of the second capacitive element (121) with a second threshold voltage VTL.
A sensor device (1000) for generating a depth map of a scene comprises a projector unit (1010) that is configured to project a plurality of temporal series of light patches on the scene, with consecutive light patches of one series being either shifted along a first direction (x1) or a second direction (x2) that is opposite to the first direction (x1), a receiver unit (1020) that is arranged at a predetermined distance (d) from the projector unit (1010) and configured to observe reflections of the light patches from the scene, and a control unit (1030) that is configured to temporally correlate the projected light patches with the observed reflections and to use the correlation and the predetermined distance (d) to generate the depth map of the scene. Here, the projector unit (1010) is configured to alternatively project first sweeps and second sweeps, where each first sweep is a series of consecutive light patches that are shifted along the first direction (x1) and each second sweep is a series of consecutive light patches that are shifted along the second direction (x2). The receiver unit (1020) comprises a plurality of event detection pixels (1025) that indicate occurrence of an event when a light intensity that is measured by a respective event detection pixel (1025) changes by more than a predetermined amount in a given time period. The time it takes for an event detection pixel (1025) to indicate the occurrence of an event depends on the absolute magnitude of the light intensity before the change of light intensity, and each event detection pixel (1025) is configured to receive light only from a predetermined solid angle, wherein solid angles observed by adjacent event detection pixels (1025) are adjacent and the observed solid angles of all event detection pixels (1025) span the field of view of the receiver unit (1020). The control unit (1030) is configured to determine during the first sweeps as well as during the second sweeps for each occurrence of an event the predetermined solid angle of the respective event detection pixel and the projection angle under which the projector unit (1010) did project a light patch at the time at which the event occurred, and the control unit (1030) is configured to average for each consecutive first sweep and second sweep and for each consecutive second sweep and first sweep for each respective pair of event occurrences the respectively determined projection angles, and to determine a depth map of the scene based on the pairs of predetermined solid angle and averaged projection angle determined for each event occurrence.
G01B 11/25 - Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures by projecting a pattern, e.g. moiré fringes, on the object
13.
DEPTH SENSOR DEVICE AND METHOD FOR OPERATING A DEPTH SENSOR DEVICE
A sensor device (1000) for generating a depth map of a scene comprises a projector unit (1010) that is configured to project a plurality of temporal series of light patches on the scene, with consecutive light patches of one series being either shifted along a first direction (x1) or a second direction (x2) that is opposite to the first direction (x1), a receiver unit (1020) that is arranged at a predetermined distance (d) from the projector unit (1010) and configured to observe reflections of the light patches from the scene, and a control unit (1030) that is configured to temporally correlate the projected light patches with the observed reflections and to use the correlation and the predetermined distance (d) to generate the depth map of the scene. Here, the receiver unit (1020) comprises a plurality of event detection pixels (1025) that indicate occurrence of an event when a light intensity that is measured by a respective event detection pixel (1025) changes by more than a predetermined amount in a given time period. The time it takes for an event detection pixel (1025) to indicate the occurrence of an event depends on the absolute magnitude of the light intensity before the change of light intensity, and each event detection pixel (1025) is configured to receive light only from a predetermined solid angle, wherein solid angles observed by adjacent event detection pixels (1025) are adjacent and the observed solid angles of all event detection pixels (1025) span the field of view of the receiver unit (1020). The control unit (1030) is configured to determine during a shift of light patches in the first direction (x1) as well as during a shift of light patches in the second direction (x2) for each occurrence of an event the predetermined solid angle of the respective event detection pixel and the projection angle under which the projector unit (1010) did project a light patch at the time at which the event occurred, and the control unit (1030) is configured to average for each given pair of event occurrences the respectively determined projection angles for the shift of light patches in the first direction (x1) and the shift of light patches in the second direction (x2), and to determine a depth map of the scene based on the pairs of predetermined solid angle and averaged projection angle determined for each event occurrence.
G01B 11/25 - Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures by projecting a pattern, e.g. moiré fringes, on the object
A dynamic vision sensor (DVS) or change detection sensor reacts to changes in light intensity and in this way monitors how a scene changes. This disclosure covers both single pixel and array architectures. The DVS may contain one pixel or 2-dimensional or 1-dimensional array of pixels. The change of intensities registered by pixels are compared, and pixel addresses where the change is positive or negative are recorded and processed. Analyzing frames based on just three values for pixels, increase, decrease or unchanged, the proposed DVS can process visual information much faster than traditional computer vision systems, which correlate multi-bit color or gray level pixel values between successive frames.
H04N 25/40 - Extracting pixel data from image sensors by controlling scanning circuits, e.g. by modifying the number of pixels sampled or to be sampled
H04N 25/47 - Image sensors with pixel address outputEvent-driven image sensorsSelection of pixels to be read out based on image data
H04N 25/571 - Control of the dynamic range involving a non-linear response
H04N 25/766 - Addressed sensors, e.g. MOS or CMOS sensors comprising control or output lines used for a plurality of functions, e.g. for pixel output, driving, reset or power
H04N 25/77 - Pixel circuitry, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components
H04N 25/772 - Pixel circuitry, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components comprising A/D, V/T, V/F, I/T or I/F converters
15.
CONTROLLER, CONTROL METHOD AND EVENT-BASED VISION MODULE
A controller for an event-based vision module, the event-based vision module including a light source configured to emit light according to a light emission pattern and an event-based vision sensor configured to acquire event data, wherein the controller includes circuitry configured to: detect, based on obtained event data, a light pattern; compute at least one of a space-division or time division light emission pattern according to the detected light pattern for avoiding interference with the detected light pattern; and control the light source to emit light according to the computed light emission pattern.
An image sensor (80) includes a pixel array (10) with pixel circuits (100), wherein each pixel circuit (100) outputs a request signal REQ in response to a predefined change in illumination. Each pixel circuit (100) includes a floating node (121) configured to temporally store charge as a function of a change of an illumination condition. Each pixel circuit (100) includes a reset switch (126) to set the floating node (121) to a predefined initial potential in response to a reset ramp signal REFR. A ramp signal circuit (20) outputs the reset ramp signal REFR for at least a first group of the pixel circuits (100).
In dynamic vision sensor (DVS) or change detection sensors, the chip or sensor is configured to control or modulate the event rate. For example, this control can be used to keep the event rate close to a desired rate or within desired bounds. Adapting the configuration of the sensor to the scene by changing the ON-event and/or the OFF-event thresholds, allows having necessary amount of data, but not much more than necessary, such that the overall system gets as much information about its state as possible.
H04N 25/40 - Extracting pixel data from image sensors by controlling scanning circuits, e.g. by modifying the number of pixels sampled or to be sampled
H04N 25/47 - Image sensors with pixel address outputEvent-driven image sensorsSelection of pixels to be read out based on image data
H04N 25/571 - Control of the dynamic range involving a non-linear response
H04N 25/766 - Addressed sensors, e.g. MOS or CMOS sensors comprising control or output lines used for a plurality of functions, e.g. for pixel output, driving, reset or power
H04N 25/77 - Pixel circuitry, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components
H04N 25/772 - Pixel circuitry, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components comprising A/D, V/T, V/F, I/T or I/F converters
18.
SENSOR DEVICE AND METHOD FOR OPERATING A SENSOR DEVICE
A sensor device (10) that is configured to capture a video of a scene comprises a plurality of pixels (51) each configured to receive light and perform photoelectric conversion to generate a pixel signal, the plurality of pixels (51) comprising a first subset of pixels and a second subset of pixels, where pixels (51) of the first subset of pixels are capable to generate pixel signals faster and preferably with less spatial resolution than pixels (51) of the second subset of pixels, and wherein color frames can be generated from the pixel signals of the second subset of pixels, a processing unit (40) that is configured to receive and process the pixel signals in order to generate video data, and a control unit (50) that is configured to receive pixel signals from the first subset of pixels and to control operation modes of the processing unit (40) based on the received pixel signals from the first subset of pixels.
H04N 21/414 - Specialised client platforms, e.g. receiver in car or embedded in a mobile appliance
H04N 5/14 - Picture signal circuitry for video frequency region
H04N 19/182 - Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a pixel
H04N 21/44 - Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
H04N 23/667 - Camera operation mode switching, e.g. between still and video, sport and normal or high and low resolution modes
H04N 25/443 - Extracting pixel data from image sensors by controlling scanning circuits, e.g. by modifying the number of pixels sampled or to be sampled by partially reading an SSIS array by reading pixels from selected 2D regions of the array, e.g. for windowing or digital zooming
H04N 25/46 - Extracting pixel data from image sensors by controlling scanning circuits, e.g. by modifying the number of pixels sampled or to be sampled by combining or binning pixels
H04N 25/75 - Circuitry for providing, modifying or processing image signals from the pixel array
H04N 25/77 - Pixel circuitry, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components
H04N 25/42 - Extracting pixel data from image sensors by controlling scanning circuits, e.g. by modifying the number of pixels sampled or to be sampled by switching between different modes of operation using different resolutions or aspect ratios, e.g. switching between interlaced and non-interlaced mode
19.
DEPTH SENSOR DEVICE AND METHOD FOR OPERATING A DEPTH SENSOR DEVICE
A sensor device (1000) for generating depth information for an object comprises a projector unit (1010) that is configured to project in a temporally consecutive manner a plurality of different illumination patterns in a projection solid angle to the object (O), where the projection solid angle consists of a predefined number of predetermined solid angles and each illumination pattern is generated by deciding for each of the predetermined solid angles whether or not to illuminate the respective predetermined solid angle by projecting light into it, a receiver unit (1020) that comprises a plurality of pixels, the receiver unit (1020) being configured to detect on each pixel intensities of light reflected from the object (O) stemming from the illumination with the illumination patterns and/or from illumination with ambient light, and to generate an event at one of the pixels if the intensity detected at the pixel changes by more than a predetermined threshold, and a control unit (1030) that is configured to generate the depth information based on all events generated during a predetermined time period.
G01B 11/25 - Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures by projecting a pattern, e.g. moiré fringes, on the object
20.
Event-based vision sensor with direct memory control
An event-based vision sensor system comprises an event-based pixel array including pixels that detect light, a readout circuit for reading out events associated with light received by each pixel and with the position it occupies in the array, and a memory controller that enables writing of each event, or event-related data, associated with a certain pixel to a certain address in a memory.
H04N 25/44 - Extracting pixel data from image sensors by controlling scanning circuits, e.g. by modifying the number of pixels sampled or to be sampled by partially reading an SSIS array
H04N 25/75 - Circuitry for providing, modifying or processing image signals from the pixel array
H04N 25/77 - Pixel circuitry, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components
H04N 25/771 - Pixel circuitry, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components comprising storage means other than floating diffusion
H04N 25/79 - Arrangements of circuitry being divided between different or multiple substrates, chips or circuit boards, e.g. stacked image sensors
A dynamic vision sensor (DVS) or change detection sensor reacts to changes in light intensity and in this way monitors how a scene changes. This disclosure covers both single pixel and array architectures. The DVS may contain one pixel or 2-dimensional or 1-dimensional array of pixels. The change of intensities registered by pixels are compared, and pixel addresses where the change is positive or negative are recorded and processed. Analyzing frames based on just three values for pixels, increase, decrease or unchanged, the proposed DVS can process visual information much faster than traditional computer vision systems, which correlate multi-bit color or gray level pixel values between successive frames.
H04N 25/40 - Extracting pixel data from image sensors by controlling scanning circuits, e.g. by modifying the number of pixels sampled or to be sampled
H04N 25/47 - Image sensors with pixel address outputEvent-driven image sensorsSelection of pixels to be read out based on image data
H04N 25/571 - Control of the dynamic range involving a non-linear response
H04N 25/75 - Circuitry for providing, modifying or processing image signals from the pixel array
H04N 25/766 - Addressed sensors, e.g. MOS or CMOS sensors comprising control or output lines used for a plurality of functions, e.g. for pixel output, driving, reset or power
H04N 25/772 - Pixel circuitry, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components comprising A/D, V/T, V/F, I/T or I/F converters
23.
SENSOR DEVICE AND METHOD FOR OPERATING A SENSOR DEVICE
A sensor device (10) comprises a plurality of pixels (51) each configured to receive light and perform photoelectric conversion to generate an electrical signal, event detection circuitry (20) that is configured to generate event data (Ev) by detecting as events intensity changes above a predetermined threshold of the light received by each of event detecting pixels (51a) that form a first subset of the pixels (51), pixel signal generating circuitry (30) that is configured to generate for each of a series of frame periods pixel signals constituting a frame image (F) that indicates intensity values of the light received by each of intensity detecting pixels (51b) that form a second subset of the pixels (51) during respective exposure periods (EP), and a control unit (40) that is configured to associate with each other event detecting pixels (51a) and intensity detecting pixels (51b) that have a corresponding field of view and to dynamically change the exposure periods (EP) of the intensity detecting pixels (51b) based on the events detected by the associated event detecting pixels (51a).
H04N 25/47 - Image sensors with pixel address outputEvent-driven image sensorsSelection of pixels to be read out based on image data
H04N 25/533 - Control of the integration time by using differing integration times for different sensor regions
H04N 25/58 - Control of the dynamic range involving two or more exposures
H04N 25/583 - Control of the dynamic range involving two or more exposures acquired simultaneously with different integration times
H04N 25/589 - Control of the dynamic range involving two or more exposures acquired sequentially, e.g. using the combination of odd and even image fields with different integration times, e.g. short and long exposures
H04N 25/60 - Noise processing, e.g. detecting, correcting, reducing or removing noise
24.
IMAGE SENSOR INCLUDING PIXEL CIRCUITS FOR EVENT DETECTION CONNECTED TO A COLUMN SIGNAL LINE
An image sensor includes a plurality of pixel circuits (100) for event detection, wherein each pixel circuit (100) includes at least one output transistor (158) and at least one selection transistor (157) electrically connected in series between at least one pixel output (159) and a first supply potential (VSS). A column signal line (191) electrically connects the pixel outputs (159) of the plurality of pixel circuits (100). A comparator/latch circuit (210) is configured to receive a signal derived from a pixel event signal (EV) transmitted on the column signal line (191) at a first comparator input (211), and to output a latched active column event signal (CEV), when the signal at the first comparator input (211) exceeds or falls below a threshold voltage (VTH) applied to a second comparator input (212) of the comparator/latch circuit (210).
A solid-state imaging device (90) includes pixel circuits (100), wherein each pixel circuit (100) outputs pixel signals (Vout) on a data signal line (VSL) in response to an active row select signal (SEE) in a row selection interval. Each pixel circuit (100) includes a floating diffusion (FD), wherein a floating diffusion potential (Vfd) of the floating diffusion (FD) determines a voltage level of the pixel signals (Vout). For each of the pixel circuits (100), a differencing circuit (200) receives two pixel signals (Vout) successively transmitted from the pixel circuit (100) on the data signal line (VSL) within a same row selection interval. The differencing circuit (200) obtains a difference signal (Vrd) from the two pixel signals (Vout). The solid-state imaging device (90) controls each pixel circuit (100) to output a previous pixel signal and a new pixel signal in a same row selection interval, wherein the previous pixel signal and the new pixel signal contain image information about an imaged scene at different points in time.
H04N 5/14 - Picture signal circuitry for video frequency region
H04N 25/616 - Noise processing, e.g. detecting, correcting, reducing or removing noise involving a correlated sampling function, e.g. correlated double sampling [CDS] or triple sampling
G06T 7/254 - Analysis of motion involving subtraction of images
H04N 25/78 - Readout circuits for addressed sensors, e.g. output amplifiers or A/D converters
An image sensor (90) includes a plurality of pixel clusters (11), wherein each pixel cluster (11) includes a number L of pixel circuits (100). Each pixel circuit (100) outputs pixel event information (EV), wherein the pixel event information (EV) indicates whether or not a change of radiation intensity received by the pixel circuit (100) exceeds a positive threshold value and/or whether or not the change of radiation intensity received by the pixel circuit (100) falls below a negative threshold value. An event processing circuit (210) receives pixel information (EP) based on the pixel event information (EV) of the pixel circuits (100) of a pixel cluster (11) and generates cluster event information (EVC) indicating whether or not the pixel information (EP) of the L pixel circuits (100) fulfills a predefined cluster output condition.
A system (28, 29) comprising circuitry configured to perform a vision-based registration task, the circuitry (28) being coupled with an event-based vision sensor (26) and being configured to be adaptive to inference quality (24) derived from output of the event-based vision sensor (26) and/or to be adaptive to consistency (34) of a task output (33) of the vision-based registration task.
A system comprising circuitry configured to perform an image processing task, the circuitry comprising a multi -vision solution (53) configured to provide APS data (41) and EVS data (42), and the circuitry being configured to generate metadata (58, 66, 68) and to provide the metadata (58, 66, 68) to an image processing pipeline (52, 70) comprising an algorithm (43) configured to perform the image processing task.
A sensor device (10) for observing a scene (S) comprises a plurality of pixels (51) each configured to receive light 5 from the scene and to perform photoelectric conversion to generate an electrical signal, event detection circuitry (20) that is configured to detect as event data intensity changes above a predetermined threshold of infrared light received by each of a first subset (S1) of the pixels (51), pixel signal generating circuitry (30) that is configured to generate pixel signals indicating intensity values of visible light received by each pixel (51) of a second subset (S2) of the pixels (51), and a control unit (40) that is configured to extract additional information from the event data detected within the received infrared light, which additional information differs from two-dimensional intensity information on the observed scene (S).
H04N 23/11 - Cameras or camera modules comprising electronic image sensorsControl thereof for generating image signals from different wavelengths for generating image signals from visible and infrared light wavelengths
H04N 23/12 - Cameras or camera modules comprising electronic image sensorsControl thereof for generating image signals from different wavelengths with one sensor only
H04N 23/21 - Cameras or camera modules comprising electronic image sensorsControl thereof for generating image signals from infrared radiation only from near infrared [NIR] radiation
H04N 13/271 - Image signal generators wherein the generated image signals comprise depth maps or disparity maps
H04N 25/79 - Arrangements of circuitry being divided between different or multiple substrates, chips or circuit boards, e.g. stacked image sensors
H04N 25/705 - Pixels for depth measurement, e.g. RGBZ
H04N 25/131 - Arrangement of colour filter arrays [CFA]Filter mosaics characterised by the spectral characteristics of the filter elements including elements passing infrared wavelengths
H04B 10/112 - Line-of-sight transmission over an extended range
H04N 25/77 - Pixel circuitry, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components
H04N 25/616 - Noise processing, e.g. detecting, correcting, reducing or removing noise involving a correlated sampling function, e.g. correlated double sampling [CDS] or triple sampling
H04N 25/75 - Circuitry for providing, modifying or processing image signals from the pixel array
H04N 25/11 - Arrangement of colour filter arrays [CFA]Filter mosaics
H04N 25/47 - Image sensors with pixel address outputEvent-driven image sensorsSelection of pixels to be read out based on image data
H04N 25/20 - Circuitry of solid-state image sensors [SSIS]Control thereof for transforming only infrared radiation into image signals
The present disclosure is directed to an information processing apparatus comprising circuitry configured to obtain a depth image data stream indicative of at least one depth image data frame representing a target object, to obtain a brightness change event data stream indicative of at least one brightness change event data frame representing the target object, to correlate a most recent depth image data frame with a most recent brightness change event data frame for generating a fused image data frame, and to generate the fused image data frame. Further, the present disclosure is directed to an information processing method.
A sensor device (10) comprises a plurality of pixels (51) each configured to receive light and perform photoelectric conversion to generate an electrical signal, event detection circuitry (20) that is configured to generate event data by detecting as events intensity changes above a predetermined threshold of the light received by each of event detecting pixels (51a) that form a first subset of the pixels (51), and pixel signal generating circuitry that is configured to generate pixel signals indicating intensity values of the light received by each of intensity detecting pixels (51b) that form a second subset of the pixels (51), the intensity detecting pixels (51b) being arranged in a two dimensional array comprising a plurality of pixel rows. The sensor device (10) further comprises a control unit (40) that is configured to associate with each other event detecting pixels (51a) and intensity detecting pixels (51b) that have a corresponding field of view in order to assign to each detected event a corresponding pixel row number, to read out the pixel signals by applying a rolling shutter that starts readout of the pixel signals of adjacent pixel rows at times separated by a predetermined time period and to assign time stamps to each pixel signal that indicate the time of readout of the corresponding pixel row, to assign to the events event time stamps based on the time at which the events were detected, to convert by a conversion function that depends on the pixel row number the time stamps of all pixel signals to a single frame time, where the pixel signals were obtained during an intensity frame acquisition period that is defined as the time required for a single readout of all pixel rows, and to convert the event time stamps by the same conversion function to obtain converted event time stamps for detected events.
The present disclosure provides a circuitry for visual speech processing, configured to determine expressed words based on visual data of a user; present the determined expressed words to the user; and adjust the determination of expressed words based on feedback of the user related to the determined expressed words.
A sensor device (10) comprising a plurality of pixels (51) each configured to receive light and to perform photoelectric conversion to generate a photocurrent, a plurality of current-voltage converting sections (81), each being connected to the output of one corresponding pixel (51) to convert the photocurrent from said pixel (51) to a photovoltage, a plurality of capacitors (40), each being connected to the output of one corresponding current-voltage converting section (81) to receive the photovoltage from said current-voltage converting section (81), a plurality of event detection units (20) each configured to detect based on the photovoltages as an event whether a change in light intensity received by one or several pixels (51) is above a predetermined threshold, and circuitry (30) that connects the plurality of capacitors (40) and the plurality of event detection units (20) such as to allow changing between a first operation mode in which at least one event detection unit (20) receives the photovoltage of only one pixel (51) via the corresponding capacitor (40), and a second operation mode in which the at least one event detection unit (20) receives a combination of photovoltages from the one pixel (51) and at least another pixel (51) via the corresponding capacitors (40).
H04N 25/46 - Extracting pixel data from image sensors by controlling scanning circuits, e.g. by modifying the number of pixels sampled or to be sampled by combining or binning pixels
H04N 25/47 - Image sensors with pixel address outputEvent-driven image sensorsSelection of pixels to be read out based on image data
34.
SENSOR DEVICE AND METHOD FOR OPERATING A SENSOR DEVICE
A sensor device comprises a plurality of pixels (51) each configured to receive light and perform photoelectric conversion to generate an electrical signal, a plurality of processing units (20) each configured to generate output signals based on the electrical signals generated by the pixels (51), and circuitry (30) that connects the plurality of pixels (51) and the plurality of processing units (20) such that information on the electrical signal generated by at least one of the pixels (51) is provided to at least two processing units (20), and such that at least one of said processing units (20) is provided with information on electrical signals from at least two pixels (51).
H04N 25/46 - Extracting pixel data from image sensors by controlling scanning circuits, e.g. by modifying the number of pixels sampled or to be sampled by combining or binning pixels
H04N 25/60 - Noise processing, e.g. detecting, correcting, reducing or removing noise
A method and system for rendering an augmented reality scene on a mobile computing device tracks a real-world scene and/or a viewpoint of a user with one or more event-based vision sensors and blends an augmented reality scene displayed on the mobile computing device based on the viewpoint of the user and a scene map of the real-world scene and on the tracking of the one or more event-based vision sensors. Event-based vision sensors offer many advantages, mainly by intrinsically compressing the data stream and thus reducing the amount of data that a processing unit needs to perform. Furthermore, the event-based vision sensor pixels continuously sense the visual changes in the scene and report them with a very low latency. This makes the event-based vision sensor an ideal sensor for always-on tasks such as visual tracking and smart sensor control or data enhancement of secondary sensing modalities.
In dynamic vision sensor (DVS) or change detection sensors, the chip or sensor is configured to control or modulate the event rate. For example, this control can be used to keep the event rate close to a desired rate or within desired bounds. Adapting the configuration of the sensor to the scene by changing the ON-event and/or the OFF-event thresholds, allows having necessary amount of data, but not much more than necessary, such that the overall system gets as much information about its state as possible.
H04N 25/40 - Extracting pixel data from image sensors by controlling scanning circuits, e.g. by modifying the number of pixels sampled or to be sampled
H04N 25/47 - Image sensors with pixel address outputEvent-driven image sensorsSelection of pixels to be read out based on image data
H04N 25/571 - Control of the dynamic range involving a non-linear response
H04N 25/766 - Addressed sensors, e.g. MOS or CMOS sensors comprising control or output lines used for a plurality of functions, e.g. for pixel output, driving, reset or power
H04N 25/77 - Pixel circuitry, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components
H04N 25/772 - Pixel circuitry, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components comprising A/D, V/T, V/F, I/T or I/F converters
37.
DEPTH SENSOR DEVICE AND METHOD FOR OPERATING A DEPTH SENSOR DEVICE
A depth sensor device (100) for measuring a depth map of an object (200) comprises a projector unit (110) configured to illuminate the object (200) with a series of illumination patterns, a receiver unit (120) comprising a plurality of pixels, the receiver unit (120) being configured to detect on a pixel basis intensities of light reflected from the object (200) while it is illuminated with the illumination patterns, and a control unit (130) configured to determine by triangulation a distance of a point (210) on the object (200) to a single pixel, which point (210) lies on the line of sight of the single pixel, by determining from intensity information obtained from the single pixel for the series of illumination patterns a direction of a pointing line running from the projector unit (110) to said point (210).
G01B 11/25 - Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures by projecting a pattern, e.g. moiré fringes, on the object
G06T 7/521 - Depth or shape recovery from laser ranging, e.g. using interferometryDepth or shape recovery from the projection of structured light
38.
SOLID-STATE IMAGING DEVICE AND METHOD FOR OPERATING A SOLID-STATE IMAGING DEVICE
A solid state imaging device (100) comprises a pixel array (110) that comprises a plurality of imaging pixels (111), each of which being capable to generate an imaging signal depending on the intensity of light falling on the imaging pixel (111), and to detect as an event a positive or negative change of light intensity that is larger than a respective predetermined threshold, and a control unit (115) that is configured to count the number of events occurring within at least one group of imaging pixels (111) and to read out the imaging signals of the imaging pixels (111) within one of the groups, if the according counted number of events is larger than or equal to a predetermined readout threshold.
H04N 5/341 - Extracting pixel data from an image sensor by controlling scanning circuits, e.g. by modifying the number of pixels having been sampled or to be sampled
H04N 5/357 - Noise processing, e.g. detecting, correcting, reducing or removing noise
A circuitry for video image encoding, the circuitry being configured to encode an input video image based on a first set of convolutional kernels of a first neural network convolutional layer and a second set of convolutional kernels of a second neural network convolutional layer, wherein the first set of convolutional kernels is optimized with respect to object representation and the second set of convolutional kernels is optimized with respect to photometric representation.
H04N 19/146 - Data rate or code amount at the encoder output
H04N 19/172 - Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a picture, frame or field
H04N 19/176 - Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a block, e.g. a macroblock
A pixel circuit (100) includes a photoreceptor module (PR). The photoreceptor module (PR) includes a photoelectric conversion element (PD). The photoreceptor module (PR) outputs a photoreceptor signal with a voltage level depending on a detector current generated by the photoelectric conversion element (PD). A voltage memory capacitor (121) receives the detector signal at a first electrode. A latch comparator circuit (130) receives a latch input signal based on a shifted voltage signal tapped from a second electrode of the voltage memory capacitor (121).
H04N 5/3745 - Addressed sensors, e.g. MOS or CMOS sensors having additional components embedded within a pixel or connected to a group of pixels within a sensor matrix, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components
H04N 5/378 - Readout circuits, e.g. correlated double sampling [CDS] circuits, output amplifiers or A/D converters
A circuitry for encoding events, including a plurality of event image pixels, wherein the circuitry is configured to: generate events based on the plurality of event image pixels; determine, for each event, a product of a polarity of the event and dictionary data associated with the event image pixel that generated the event; and generate encoded event image data by summation of the products.
The present disclosure generally pertains to time-of-flight data generation circuitry (1), configured to: acquire a time-of-flight data stream using a ToF camera (2); acquire a brightness change event data stream using an EVS, event-based vision sensor, camera (3); correlate the time-of-flight data stream with the brightness change event data stream in time with each other for generating at least one time-of-flight data frame; and generate the at least one time-of-flight data frame based on the correlation. The ToF data stream is synchronous and the event data stream is asynchronous, such that the data is synchronized based on a clock and transmitted to an image signal processor, ISP, (4). The ToF data generation circuitry (1) is adjusted to picture an object (5) which carries out a high speed movement. High-speed and motion robust depth maps/point clouds can be generated. Correlation may refer to an assigning of data points of the ToF data stream at points of time intrinsic to the ToF measurement to data points of the event data stream at points of time intrinsic to the event measurement. Correlation may include correcting or changing an already existing frame, such that the detected motion may be unblurred in a final depth map. Hence, depth sensing may be improved, e.g., for mobile or handheld device applications with high-speed requirements.
An event based pixel sensor system employs regions of interest or regions of non-interest, which are preferably dynamic, to isolate change events and reduce data from non-interesting events.
H04N 25/443 - Extracting pixel data from image sensors by controlling scanning circuits, e.g. by modifying the number of pixels sampled or to be sampled by partially reading an SSIS array by reading pixels from selected 2D regions of the array, e.g. for windowing or digital zooming
A dynamic vision sensor such as an event based vision sensor employs analog to digital converters (ADC), such as ramp ADCs, that analog to digital converts the signals from photoreceptors. Current and previous light values are then stored and compared digitally. In addition, log compression can be implemented by increasing the count linearly while increasing the reference voltage exponentially, or increasing the count logarithmically while increasing the reference voltage linearly.
H04N 25/771 - Pixel circuitry, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components comprising storage means other than floating diffusion
H04N 25/772 - Pixel circuitry, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components comprising A/D, V/T, V/F, I/T or I/F converters
46.
Event-based vision sensor using on/off event and grayscale detection ramps
A dynamic vision sensor such as an event based vision sensor employs analog to digital converters (ADC), such as ramp ADCs, that analog to digital converts the signals from photoreceptors. Only previous light values need to be stored in the pixels. This is accomplished by generating three ramps. In addition, log compression can be implemented by increasing the count linearly while increasing the reference voltage exponentially, or increasing the count logarithmically while increasing the reference voltage linearly.
H04N 25/772 - Pixel circuitry, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components comprising A/D, V/T, V/F, I/T or I/F converters
H04N 25/40 - Extracting pixel data from image sensors by controlling scanning circuits, e.g. by modifying the number of pixels sampled or to be sampled
H04N 25/75 - Circuitry for providing, modifying or processing image signals from the pixel array
H04N 25/766 - Addressed sensors, e.g. MOS or CMOS sensors comprising control or output lines used for a plurality of functions, e.g. for pixel output, driving, reset or power
An NMOS-only operational transconductance amplifier (OTA) replaces the PMOS transistors with switched capacitor pseudo-resistors in the pixels of an optical sensor such as a dynamic vision sensor or event-based vision sensor. Thus, if a stacked CMOS image sensor (CIS) process is employed, then the upper wafer can be kept free from N-wells, while still having the complete OTA on the upper wafer. Thus, it is possible to have only one wafer-to-wafer connection per pixel. Moreover, by operating the switched capacitor pseudo-resistors as three terminal devices, the gain can further be increased.
H04N 25/771 - Pixel circuitry, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components comprising storage means other than floating diffusion
H04N 25/79 - Arrangements of circuitry being divided between different or multiple substrates, chips or circuit boards, e.g. stacked image sensors
48.
Event-based vision sensor with direct memory control
An event-based vision sensor system comprises an event-based pixel array including pixels that detect light, a readout circuit for reading out events associated with light received by each pixel and with the position it occupies in the array, and a memory controller that enables writing of each event, or event-related data, associated with a certain pixel to a certain address in a memory.
H04N 25/44 - Extracting pixel data from image sensors by controlling scanning circuits, e.g. by modifying the number of pixels sampled or to be sampled by partially reading an SSIS array
H04N 25/75 - Circuitry for providing, modifying or processing image signals from the pixel array
H04N 25/77 - Pixel circuitry, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components
H04N 25/771 - Pixel circuitry, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components comprising storage means other than floating diffusion
H04N 25/79 - Arrangements of circuitry being divided between different or multiple substrates, chips or circuit boards, e.g. stacked image sensors
A method and system for rendering an augmented reality scene on a mobile computing device tracks a real-world scene and/or a viewpoint of a user with one or more event-based vision sensors and blends an augmented reality scene displayed on the mobile computing device based on the viewpoint of the user and a scene map of the real-world scene and on the tracking of the one or more event-based vision sensors. Event-based vision sensors offer many advantages, mainly by intrinsically compressing the data stream and thus reducing the amount of data that a processing unit needs to perform. Furthermore, the event-based vision sensor pixels continuously sense the visual changes in the scene and report them with a very low latency. This makes the event-based vision sensor an ideal sensor for always-on tasks such as visual tracking and smart sensor control or data enhancement of secondary sensing modalities.
A method and system comprising one or more event-based vision sensors for monitoring an environment and one or more depth sensors monitoring the environment as well as potentially additional sensors to monitor the environment such as inertial measurement units and image sensors. The device as a processor assembly will then monitor the event-based vision sensor and the depth sensor and other sensors and determine whether to adapt a control schem, such as activate, the depth sensor to acquire depth information of the environment based on the response of the event-based vision sensor. This approach leverages advantages of event-based sensors to reduce the power consumption but still react quickly to such changes by continuously measuring intervals directly relating to the minimally admitted delays in the application for which depth measurements or inferred data are used.
H04N 5/232 - Devices for controlling television cameras, e.g. remote control
G06F 3/0346 - Pointing devices displaced or positioned by the userAccessories therefor with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
A dynamic vision sensor (DVS) or change detection sensor reacts to changes in light intensity and in this way monitors how a scene changes. This disclosure covers both single pixel and array architectures. The DVS may contain one pixel or 2-dimensional or 1-dimensional array of pixels. The change of intensities registered by pixels are compared, and pixel addresses where the change is positive or negative are recorded and processed. Analyzing frames based on just three values for pixels, increase, decrease or unchanged, the proposed DVS can process visual information much faster than traditional computer vision systems, which correlate multi-bit color or gray level pixel values between successive frames.
H04N 25/77 - Pixel circuitry, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components
H04N 25/571 - Control of the dynamic range involving a non-linear response
H04N 25/772 - Pixel circuitry, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components comprising A/D, V/T, V/F, I/T or I/F converters
H04N 25/47 - Image sensors with pixel address outputEvent-driven image sensorsSelection of pixels to be read out based on image data
H04N 25/75 - Circuitry for providing, modifying or processing image signals from the pixel array
H04N 25/40 - Extracting pixel data from image sensors by controlling scanning circuits, e.g. by modifying the number of pixels sampled or to be sampled
H04N 25/766 - Addressed sensors, e.g. MOS or CMOS sensors comprising control or output lines used for a plurality of functions, e.g. for pixel output, driving, reset or power
According to the present invention there is provided a vision sensor comprising, an array of pixels comprising rows and columns of pixels, wherein each pixel in the array comprises, a photosensor which is configured to output a current proportional to the intensity of light which is incident on the photosensor; a current source which is configured such that it can output a current which has a constant current level which is equal to the current level of the current output by the photosensor at a selected first instant in time, and can maintain that constant current level even if the level of the current output from the photosensor changes after said selected first instant in time; an integrator which is configured to integrate the difference between the level of current output by the current source and the level of current output by the photosensor, after the selected first instant in time; wherein the vision sensor further comprises a counter which can measure time, wherein the counter is configured such that it can begin to measure time at the selected first instant; and wherein each pixel in the array further comprises a storage means which can store the value on the counter at a second instant in time, the second instant in time being the instant when the integration of the difference between the level of current output by the current source and the level of current output by the photosensor of that pixel reaches a predefined threshold level. There is further provided a corresponding method of vision sensing, and a depth sensor assembly which comprises the vision sensor.
H04N 5/365 - Noise processing, e.g. detecting, correcting, reducing or removing noise applied to fixed-pattern noise, e.g. non-uniformity of response
H04N 5/363 - Noise processing, e.g. detecting, correcting, reducing or removing noise applied to reset noise, e.g. KTC noise
G01B 11/25 - Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures by projecting a pattern, e.g. moiré fringes, on the object
H04N 5/3745 - Addressed sensors, e.g. MOS or CMOS sensors having additional components embedded within a pixel or connected to a group of pixels within a sensor matrix, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components
H04N 5/374 - Addressed sensors, e.g. MOS or CMOS sensors
53.
Event-based vision sensor manufactured with 3D-IC technology
An event-based vision sensor is fabricated using an advanced stacking technique, known as Three-Dimensional Integrated Circuit, which stacks more wafers (or dies) and interconnects them vertically. The electronic integrated circuits of the sensor are then distributed between the two or more electrically connected dies.
H01L 29/423 - Electrodes characterised by their shape, relative sizes or dispositions not carrying the current to be rectified, amplified or switched
H04N 5/3745 - Addressed sensors, e.g. MOS or CMOS sensors having additional components embedded within a pixel or connected to a group of pixels within a sensor matrix, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components
According to the present invention there is provided a vision sensor comprising, an array of pixels (101) comprising rows and columns of pixels, wherein each pixel has an address assigned thereto which represents the position of the pixel in the array, wherein each pixel in the array of pixels comprises a photodiode (103) which can receive light, and which can output current having an amplitude proportional to the intensity of the received light; a photoreceptor circuit (104) which is electronically connected to the photodiode, and which is configured to convert current which it receives from the photodiode into a voltage; a first storage capacitor (105), and at least a first switch which (106) is positioned between the first storage capacitor and an output of the photoreceptor circuit, wherein the first switch can be selectively closed to electronically connect the output of the photoreceptor circuit to the first storage capacitor, or selectively opened to electronically disconnect the output of the photoreceptor circuit from the first storage capacitor; and a circuit (102, 201) which is configured so that it can he selectively electronically connected to a pixel in the array, and to determine if the difference between the voltage output from the photoreceptor circuit and the voltage across the first storage capacitor is greater than a predefined threshold voltage, and to output the address of the pixel to a receiver only if the difference between the voltage output from the photoreceptor circuit and the voltage across the first storage capacitor is greater than a predefined threshold voltage. There is further provided a corresponding method of vision sensing using the vision sensor.
H04N 5/351 - Control of the SSIS depending on the scene, e.g. brightness or motion in the scene
H04N 5/3745 - Addressed sensors, e.g. MOS or CMOS sensors having additional components embedded within a pixel or connected to a group of pixels within a sensor matrix, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components
H04N 5/345 - Extracting pixel data from an image sensor by controlling scanning circuits, e.g. by modifying the number of pixels having been sampled or to be sampled by partially reading an SSIS array
H04N 5/378 - Readout circuits, e.g. correlated double sampling [CDS] circuits, output amplifiers or A/D converters
H04N 5/357 - Noise processing, e.g. detecting, correcting, reducing or removing noise
H04N 5/335 - Transforming light or analogous information into electric information using solid-state image sensors [SSIS]
H04N 5/3745 - Addressed sensors, e.g. MOS or CMOS sensors having additional components embedded within a pixel or connected to a group of pixels within a sensor matrix, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components
56.
DYNAMIC REGION OF INTEREST (ROI) FOR EVENT-BASED VISION SENSORS
An event based pixel sensor system employs regions of interest or regions of non-interest, which are preferably dynamic, to isolate change events and reduce data from non-interesting events.
H04N 5/341 - Extracting pixel data from an image sensor by controlling scanning circuits, e.g. by modifying the number of pixels having been sampled or to be sampled
H04N 5/345 - Extracting pixel data from an image sensor by controlling scanning circuits, e.g. by modifying the number of pixels having been sampled or to be sampled by partially reading an SSIS array
An NMOS-only operational transconductance amplifier (OTA) replaces the PMOS transistors with switched capacitor pseudo-resistors in the pixels of an optical sensor such as a dynamic vision sensor or event- based vision sensor. Thus, if a stacked CMOS image sensor (CIS) process is employed, then the upper wafer can be kept free from N- wells, while still having the complete OTA on the upper wafer. Thus, it is possible to have only one wafer-to-wafer connection per pixel. Moreover, by operating the switched capacitor pseudo-resistors as three terminal devices, the gain can further be increased.
H04N 5/3745 - Addressed sensors, e.g. MOS or CMOS sensors having additional components embedded within a pixel or connected to a group of pixels within a sensor matrix, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components
A dynamic vision sensor such as an event based vision sensor employs analog to digital converters (ADC), such as ramp ADCs, that analog to digital converts the signals from photoreceptors. Only previous light values need to be stored in the pixels. This is accomplished by generating three ramps. In addition, log compression can be implemented by increasing the count linearly while increasing the reference voltage exponentially, or increasing the count logarithmically while increasing the reference voltage linearly.
H04N 5/3745 - Addressed sensors, e.g. MOS or CMOS sensors having additional components embedded within a pixel or connected to a group of pixels within a sensor matrix, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components
A dynamic vision sensor such as an event based vision sensor employs analog to digital converters (ADC), such as ramp ADCs, that analog to digital converts the signals from photoreceptors. Current and previous light values are then stored and compared digitally. In addition, log compression can be implemented by increasing the count linearly while increasing the reference voltage exponentially, or increasing the count logarithmically while increasing the reference voltage linearly.
H04N 5/341 - Extracting pixel data from an image sensor by controlling scanning circuits, e.g. by modifying the number of pixels having been sampled or to be sampled
In dynamic vision sensor (DVS) or change detection sensors, the chip or sensor is configured to control or modulate the event rate. For example, this control can be used to keep the event rate close to a desired rate or within desired bounds. Adapting the configuration of the sensor to the scene by changing the ON-event and/or the OFF-event thresholds, allows having necessary amount of data, but not much more than necessary, such that the overall system gets as much information about its state as possible.
H04N 5/30 - Transforming light or analogous information into electric information
H04N 5/378 - Readout circuits, e.g. correlated double sampling [CDS] circuits, output amplifiers or A/D converters
H04N 5/3745 - Addressed sensors, e.g. MOS or CMOS sensors having additional components embedded within a pixel or connected to a group of pixels within a sensor matrix, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components
H04N 5/374 - Addressed sensors, e.g. MOS or CMOS sensors
H04N 5/335 - Transforming light or analogous information into electric information using solid-state image sensors [SSIS]
H04N 5/341 - Extracting pixel data from an image sensor by controlling scanning circuits, e.g. by modifying the number of pixels having been sampled or to be sampled
According to the present invention there is provided a vision sensor comprising, an array of pixels comprising rows and columns of pixels, wherein each pixel in the array comprises, a photosensor which is configured to output a current proportional to the intensity of light which is incident on the photosensor; a current source which is configured such that it can output a current which has a constant current level which is equal to the current level of the current output by the photosensor at a selected first instant in time, and can maintain that constant current level even if the level of the current output from the photosensor changes after said selected first instant in time; an integrator which is configured to integrate the difference between the level of current output by the current source and the level of current output by the photosensor, after the selected first instant in time; wherein the vision sensor further comprises a counter which can measure time, wherein the counter is configured such that it can begin to measure time at the selected first instant; and wherein each pixel in the array further comprises a storage means which can store the value on the counter at a second instant in time, the second instant in time being the instant when the integration of the difference between the level of current output by the current source and the level of current output by the photosensor of that pixel reaches a predefined threshold level. There is further provided a corresponding method of vision sensing, and a depth sensor assembly which comprises the vision sensor.
H04N 5/365 - Noise processing, e.g. detecting, correcting, reducing or removing noise applied to fixed-pattern noise, e.g. non-uniformity of response
H04N 5/363 - Noise processing, e.g. detecting, correcting, reducing or removing noise applied to reset noise, e.g. KTC noise
G01B 11/25 - Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures by projecting a pattern, e.g. moiré fringes, on the object
H04N 5/3745 - Addressed sensors, e.g. MOS or CMOS sensors having additional components embedded within a pixel or connected to a group of pixels within a sensor matrix, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components
H04N 5/374 - Addressed sensors, e.g. MOS or CMOS sensors
In an image sensor, some pixels in an array contain a sampling circuit to sample the light intensity and a capacitor to store an analog value representing the intensity at that pixel. Alternatively, a group of pixel circuits will be equipped with such sampling and capacitor circuits. This allows simple redundancy-reducing computations with a relatively simple pixel architecture.
H04N 5/378 - Readout circuits, e.g. correlated double sampling [CDS] circuits, output amplifiers or A/D converters
H04N 5/3745 - Addressed sensors, e.g. MOS or CMOS sensors having additional components embedded within a pixel or connected to a group of pixels within a sensor matrix, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components
H04N 5/374 - Addressed sensors, e.g. MOS or CMOS sensors
H04N 5/335 - Transforming light or analogous information into electric information using solid-state image sensors [SSIS]
H04N 5/341 - Extracting pixel data from an image sensor by controlling scanning circuits, e.g. by modifying the number of pixels having been sampled or to be sampled
According to the present invention there is provided a vision sensor comprising, an array of pixels (101) comprising rows and columns of pixels, wherein each pixel has an address assigned thereto which represents the position of the pixel in the array, wherein each pixel in the array of pixels comprises a photodiode (103) which can receive light, and which can output current having an amplitude proportional to the intensity of the received light; a photoreceptor circuit (104) which is electronically connected to the photodiode, and which is configured to convert current which it receives from the photodiode into a voltage; a first storage capacitor (105), and at least a first switch which (106) is positioned between the first storage capacitor and an output of the photoreceptor circuit, wherein the first switch can be selectively closed to electronically connect the output of the photoreceptor circuit to the first storage capacitor, or selectively opened to electronically disconnect the output of the photoreceptor circuit from the first storage capacitor; and a circuit (102, 201) which is configured so that it can be selectively electronically connected to a pixel in the array, and to determine if the difference between the voltage output from the photoreceptor circuit and the voltage across the first storage capacitor is greater than a predefined threshold voltage, and to output the address of the pixel to a receiver only if the difference between the voltage output from the photoreceptor circuit and the voltage across the first storage capacitor is greater than a predefined threshold voltage. There is further provided a corresponding method of vision sensing using the vision sensor.
H04N 5/351 - Control of the SSIS depending on the scene, e.g. brightness or motion in the scene
H04N 5/345 - Extracting pixel data from an image sensor by controlling scanning circuits, e.g. by modifying the number of pixels having been sampled or to be sampled by partially reading an SSIS array
H04N 5/3745 - Addressed sensors, e.g. MOS or CMOS sensors having additional components embedded within a pixel or connected to a group of pixels within a sensor matrix, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components
H04N 5/378 - Readout circuits, e.g. correlated double sampling [CDS] circuits, output amplifiers or A/D converters
A dynamic vision sensor (DVS) or change detection sensor reacts to changes in light intensity and in this way monitors how a scene changes. This disclosure covers both single pixel and array architectures. The DVS may contain one pixel or 2-dimensional or 1-dimensional array of pixels. The change of intensities registered by pixels are compared, and pixel addresses where the change is positive or negative are recorded and processed. Analyzing frames based on just three values for pixels, increase, decrease or unchanged, the proposed DVS can process visual information much faster than traditional computer vision systems, which correlate multi-bit color or gray level pixel values between successive frames.
H04N 5/335 - Transforming light or analogous information into electric information using solid-state image sensors [SSIS]
H04N 5/3745 - Addressed sensors, e.g. MOS or CMOS sensors having additional components embedded within a pixel or connected to a group of pixels within a sensor matrix, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components
H04N 5/374 - Addressed sensors, e.g. MOS or CMOS sensors
H04N 5/341 - Extracting pixel data from an image sensor by controlling scanning circuits, e.g. by modifying the number of pixels having been sampled or to be sampled
H04N 5/378 - Readout circuits, e.g. correlated double sampling [CDS] circuits, output amplifiers or A/D converters
An event-based vision sensor is fabricated using an advanced stacking technique, known as Three-Dimensional Integrated Circuit, which stacks more wafers (or dies) and interconnects them vertically. The electronic integrated circuits of the sensor are then distributed between the two or more electrically connected dies.
H01L 29/423 - Electrodes characterised by their shape, relative sizes or dispositions not carrying the current to be rectified, amplified or switched
H04N 5/3745 - Addressed sensors, e.g. MOS or CMOS sensors having additional components embedded within a pixel or connected to a group of pixels within a sensor matrix, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components
66.
Event-based vision sensor with direct memory control
An event-based vision sensor system comprises an event-based pixel array including pixels that detect light, a readout circuit for reading out events associated with light received by each pixel and with the position it occupies in the array, and a memory controller that enables writing of each event, or event-related data, associated with a certain pixel to a certain address in a memory.
H04N 5/3745 - Addressed sensors, e.g. MOS or CMOS sensors having additional components embedded within a pixel or connected to a group of pixels within a sensor matrix, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components
H04N 5/345 - Extracting pixel data from an image sensor by controlling scanning circuits, e.g. by modifying the number of pixels having been sampled or to be sampled by partially reading an SSIS array
According to the present invention there is provided a vision sensor comprising, an array of pixels (101) comprising rows and columns of pixels, wherein each pixel has an address assigned thereto which represents the position of the pixel in the array, wherein each pixel in the array of pixels comprises a photodiode (103) which can receive light, and which can output current having an amplitude proportional to the intensity of the received light; a photoreceptor circuit (104) which is electronically connected to the photodiode, and which is configured to convert current which it receives from the photodiode into a voltage; a first storage capacitor (105), and at least a first switch which (106) is positioned between the first storage capacitor and an output of the photoreceptor circuit, wherein the first switch can be selectively closed to electronically connect the output of the photoreceptor circuit to the first storage capacitor, or selectively opened to electronically disconnect the output of the photoreceptor circuit from the first storage capacitor; and a circuit (102, 201) which is configured so that it can be selectively electronically connected to a pixel in the array, and to determine if the difference between the voltage output from the photoreceptor circuit and the voltage across the first storage capacitor is greater than a predefined threshold voltage, and to output the address of the pixel to a receiver only if the difference between the voltage output from the photoreceptor circuit and the voltage across the first storage capacitor is greater than a predefined threshold voltage. There is further provided a corresponding method of vision sensing using the vision sensor.
H04N 5/351 - Control of the SSIS depending on the scene, e.g. brightness or motion in the scene
H04N 5/345 - Extracting pixel data from an image sensor by controlling scanning circuits, e.g. by modifying the number of pixels having been sampled or to be sampled by partially reading an SSIS array
H04N 5/3745 - Addressed sensors, e.g. MOS or CMOS sensors having additional components embedded within a pixel or connected to a group of pixels within a sensor matrix, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components
H04N 5/378 - Readout circuits, e.g. correlated double sampling [CDS] circuits, output amplifiers or A/D converters
68.
Vision sensor, a method of vision sensing, and a depth sensor assembly
According to the present invention there is provided a vision sensor comprising, an array of pixels comprising rows and columns of pixels, wherein each pixel in the array comprises, a photosensor which is configured to output a current proportional to the intensity of light which is incident on the photosensor; a current source which is configured such that it can output a current which has a constant current level which is equal to the current level of the current output by the photosensor at a selected first instant in time, and can maintain that constant current level even if the level of the current output from the photosensor changes after said selected first instant in time; an integrator which is configured to integrate the difference between the level of current output by the current source and the level of current output by the photosensor, after the selected first instant in time; wherein the vision sensor further comprises a counter which can measure time, wherein the counter is configured such that it can begin to measure time at the selected first instant; and wherein each pixel in the array further comprises a storage means which can store the value on the counter at a second instant in time, the second instant in time being the instant when the integration of the difference between the level of current output by the current source and the level of current output by the photosensor of that pixel reaches a predefined threshold level. There is further provided a corresponding method of vision sensing, and a depth sensor assembly which comprises the vision sensor.
H04N 5/365 - Noise processing, e.g. detecting, correcting, reducing or removing noise applied to fixed-pattern noise, e.g. non-uniformity of response
H04N 5/363 - Noise processing, e.g. detecting, correcting, reducing or removing noise applied to reset noise, e.g. KTC noise
G01B 11/25 - Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures by projecting a pattern, e.g. moiré fringes, on the object
H04N 5/3745 - Addressed sensors, e.g. MOS or CMOS sensors having additional components embedded within a pixel or connected to a group of pixels within a sensor matrix, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components
H04N 5/374 - Addressed sensors, e.g. MOS or CMOS sensors
A dynamic vision sensor (DVS) or change detection sensor reacts to changes in light intensity and in this way monitors how a scene changes. This disclosure covers both single pixel and array architectures. The DVS may contain one pixel or 2-dimensional or 1-dimensional array of pixels. The change of intensities registered by pixels are compared, and pixel addresses where the change is positive or negative are recorded and processed. Analyzing frames based on just three values for pixels, increase, decrease or unchanged, the proposed DVS can process visual information much faster than traditional computer vision systems, which correlate multi-bit color or gray level pixel values between successive frames.
H04N 5/335 - Transforming light or analogous information into electric information using solid-state image sensors [SSIS]
H04N 5/3745 - Addressed sensors, e.g. MOS or CMOS sensors having additional components embedded within a pixel or connected to a group of pixels within a sensor matrix, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components
H04N 5/374 - Addressed sensors, e.g. MOS or CMOS sensors
H04N 5/378 - Readout circuits, e.g. correlated double sampling [CDS] circuits, output amplifiers or A/D converters
In an image sensor, some pixels in an array contain a sampling circuit to sample the light intensity and a capacitor to store an analog value representing the intensity at that pixel. Alternatively, a group of pixel circuits will be equipped with such sampling and capacitor circuits. This allows simple redundancy-reducing computations with a relatively simple pixel architecture.
H04N 5/335 - Transforming light or analogous information into electric information using solid-state image sensors [SSIS]
H04N 5/3745 - Addressed sensors, e.g. MOS or CMOS sensors having additional components embedded within a pixel or connected to a group of pixels within a sensor matrix, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components
H04N 5/374 - Addressed sensors, e.g. MOS or CMOS sensors
H04N 5/341 - Extracting pixel data from an image sensor by controlling scanning circuits, e.g. by modifying the number of pixels having been sampled or to be sampled
In dynamic vision sensor (DVS) or change detection sensors, the chip or sensor is configured to control or modulate the event rate. For example, this control can be used to keep the event rate close to a desired rate or within desired bounds. Adapting the configuration of the sensor to the scene by changing the ON-event and/or the OFF-event thresholds, allows having necessary amount of data, but not much more than necessary, such that the overall system gets as much information about its state as possible.
H04N 5/335 - Transforming light or analogous information into electric information using solid-state image sensors [SSIS]
H04N 5/3745 - Addressed sensors, e.g. MOS or CMOS sensors having additional components embedded within a pixel or connected to a group of pixels within a sensor matrix, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components
H04N 5/374 - Addressed sensors, e.g. MOS or CMOS sensors
H04N 5/341 - Extracting pixel data from an image sensor by controlling scanning circuits, e.g. by modifying the number of pixels having been sampled or to be sampled
patents
