Video frame pipeline

araviq6.videostream provides video pipeline classes to handle QVideoFrame using numpy array processing.

There are two options to handle the pipeline.

1. Frame-based approach

2. Array-based approach

The first approach can be achieved by connecting VideoFrameProcessor with QVideoSink. The second is more low-level, and consists of FrameToArrayConverter, ArrayProcessor and ArrayToFrameConverter

Convenience multimedia classes for array-based pipeline are also provided in this module. These classes can replace the FrameToArrayConverter connected to QVideoSink.

1. NDArrayVideoPlayer (video file -> ndarray)

2. NDArrayMediaCaptureSession (camera -> ndarray)

Pipeline classes

class araviq6.videostream.VideoFrameProcessor(*args: Any, **kwargs: Any)

Bases: QObject

Video pipeline component to process QVideoFrame.

QVideoFrame processing structure

VideoFrameProcessor runs VideoFrameWorker in an internal thread to process the incoming video frame. To perform processing, pass the input frame to processVideoFrame() slot and listen to videoFrameProcessed signal which emits two objects; processed QVideoFrame and processed NDArray.

skipIfRunning() defines whether the incoming video frames should be skipped when the worker is running.

worker() → VideoFrameWorker | None

Worker to process the video frame.

See also setWorker().

setWorker(worker: VideoFrameWorker | None)

Set worker as video frame processor.

See also worker().

skipIfRunning() → bool

If True, incoming frames to processVideoFrame() are skipped if worker() is running.

If False, every incoming frame is queued if worker is not ready. This may consume a significant amount of memory and cause laggy displaying, thus should be used with discretion.

See also setSkipIfRunning().

setSkipIfRunning(flag: bool)

Set skipIfRunning() to flag.

See also skipIfRunning().

processVideoFrame(frame: araviq6.qt_compat.QtMultimedia.QVideoFrame)

Request worker() to process frame.

Processed QVideoFrame and processed array are emitted by videoFrameProcessed.

If worker is running and skipIfRunning() is True, frame is skipped without being emitted. If skipIfRunning() is False, incoming frames are queued when worker is running.

stop()

Stop the worker thread.

class araviq6.videostream.VideoFrameWorker(*args: Any, **kwargs: Any)

Bases: QObject

Worker to process QVideoFrame using numpy.ndarray operation.

To perform processing, pass the input frame to runProcess() and listen to videoFrameProcessed signal which emits two objects; processed QVideoFrame and processed NDArray.

ready() is set to False when the processing is running. This property can be utilized in multithreading.

ready() → bool

Returns true if the worker finished processing and can process the next video frame without being blocked.

runProcess(frame: araviq6.qt_compat.QtMultimedia.QVideoFrame)

Process frame and emit the results to videoFrameProcessed.

When a video frame is passed, it is first converted to QImage by QVideoFrame.toImage() and then to array by imageToArray(). Array processing is done by processArray(), and the result is converted back to QVideoFrame by arrayToVideoFrame().

Processed QVideoFrame and processed array are emitted by videoFrameProcessed.

During the processing ready() is set to False.

Notes

This method must not be Qt Slot to be multithreaded.

imageToArray(image: araviq6.qt_compat.QtGui.QImage) → ndarray[Any, dtype[uint8]]

Convert image to numpy array.

By default, this method uses qimage2ndarray.rgb_view for conversion. Null image is converted to 3D empty array. Subclass can redefine this method.

processArray(array: ndarray[Any, dtype[uint8]]) → ndarray[Any, dtype[uint8]]

Perform image processing on array and return the result.

By default this method does not perform any processing. Subclass can redefine this method.

See also runProcess().

arrayToVideoFrame(array: ndarray[Any, dtype[uint8]], hintFrame: araviq6.qt_compat.QtMultimedia.QVideoFrame) → araviq6.qt_compat.QtMultimedia.QVideoFrame

Convert array to QVideoFrame, using hintFrame as hint.

By default this method uses array2qvideoframe() for conversion. Then, it updates mapMode(), startTime(), and endTime() properties of the new frame with those of hintFrame. For empty array, hintFrame is just returned. Subclass can redefine this method.

class araviq6.videostream.QVideoFrameProperty(mapMode: araviq6.qt_compat.QtMultimedia.QVideoFrame.MapMode = araviq6.qt_compat.QtMultimedia.QVideoFrame.MapMode.NotMapped, startTime: int = -1, endTime: int = -1, mirrored: bool = False, rotationAngle: araviq6.qt_compat.QtMultimedia.QVideoFrame.RotationAngle = araviq6.qt_compat.QtMultimedia.QVideoFrame.RotationAngle.Rotation0, subtitleText: str = '')

Bases: object

Wrapper for the properties of QVideoFrame.

classmethod fromVideoFrame(frame: araviq6.qt_compat.QtMultimedia.QVideoFrame)

Construct QVideoFrameProperty instance from frame.

setToVideoFrame(frame: araviq6.qt_compat.QtMultimedia.QVideoFrame)

Set the properties of frame to the values in self.

class araviq6.videostream.FrameToArrayConverter(*args: Any, **kwargs: Any)

Bases: QObject

Video pipeline component which converts QVideoFrame to numpy array.

FrameToArrayConverter structure

When video frame is passed to convertVideoFrame(), FrameToArrayConverter first converts it to QImage and then to numpy array using imageToArray(). arrayConverted emits resulting array and QVideoFrameProperty.

Invalid video frame is converted to 3D empty array.

convertVideoFrame(frame: araviq6.qt_compat.QtMultimedia.QVideoFrame)

Convert frame to numpy array and emit to arrayConverted.

Video frame is converted using using imageToArray(), and its properties are wrapped by QVideoFrameProperty. Converted array frame property are emitted to arrayConverted().

imageToArray(image: araviq6.qt_compat.QtGui.QImage) → ndarray[Any, dtype[uint8]]

Convert image to numpy array.

image is QImage from QVideoFrame. By default this method uses qimage2ndarray.rgb_view for conversion. Subclass can redefine this method.

class araviq6.videostream.ArrayToFrameConverter(*args: Any, **kwargs: Any)

Bases: QObject

Video pipeline component which converts numpy array to QVideoFrame.

ArrayToFrameConverter structure

Conversion is done by passing an array to convertArray() slot and listening to frameConverted signal. Frame properties can be set by passing optional QVideoFrameProperty with the array.

convertArray(array: ~numpy.ndarray[~typing.Any, ~numpy.dtype[~numpy.uint8]], frameProperty: ~araviq6.videostream.QVideoFrameProperty = <araviq6.videostream.QVideoFrameProperty object>)

Convert array to QvideoFrame and emit to frameConverted.

Valid array is converted using arrayToFrame() with properties defined by frameProperty. Empty array is converted to invalid video frame.

arrayToFrame(array: ndarray[Any, dtype[uint8]]) → araviq6.qt_compat.QtMultimedia.QVideoFrame

Converts array to QVideoFrame.

By default this method uses array2qvideoframe() for conversion. Subclass can redefine this method.

class araviq6.videostream.ArrayProcessor(*args: Any, **kwargs: Any)

Bases: QObject

Video pipeline component to process numpy array.

NDArray processing structure

ArrayProcessor runs ArrayWorker in internal an thread t process the incoming array. To perform processing, pass the input array to processArray() slot and listen to arrayProcessed signal.

skipIfRunning() defines whether the incoming arrays should be skipped when the worker is running.

worker() → ArrayWorker | None

Worker to process the array.

See also setWorker().

setWorker(worker: ArrayWorker | None)

Set worker as array processor.

See also worker().

skipIfRunning() → bool

If True, incoming arrays to processArray() are skipped if worker() is not ready.

If False, incoming arrays are queued if worker is not ready. This may consume a significant amount of memory and cause laggy displaying, thus should be used with discretion.

See also setSkipIfRunning().

setSkipIfRunning(flag: bool)

Set skipIfRunning() to flag.

See also skipIfRunning().

processArray(array: ndarray[Any, dtype[uint8]])

Request worker() to process array.

The result is emitted to arrayProcessed.

If worker is running and skipIfRunning() is True, array is skipped without being emitted. If skipIfRunning() is False, incoming arrays are queued when worker is running.

stop()

Stop the worker thread.

class araviq6.videostream.ArrayWorker(*args: Any, **kwargs: Any)

Bases: QObject

Worker to process numpy array.

To perform processing, pass the input array to runProcess() and listen to arrayProcessed signal.

ready() is set to False when the processing is being run. This property can be utilized in multithreading.

ready() → bool

Returns true if the worker finished processing and can process the next array without being blocked.

runProcess(array: ndarray[Any, dtype[uint8]])

Process array and emit the result to arrayProcessed.

Array processing is done by processArray().

During the processing ready() is set to False.

Notes

This method must not be Qt Slot to be multithreaded.

processArray(array: ndarray[Any, dtype[uint8]]) → ndarray[Any, dtype[uint8]]

Perform image processing on array and return the result.

By default this method does not perform any processing. Subclass can redefine this method.

See also runProcess().

Convenience classes

class araviq6.videostream.NDArrayVideoPlayer(*args: Any, **kwargs: Any)

Bases: QMediaPlayer

Video player which emits numpy array.

When playing, NDArrayVideoPlayer converts the frame to numpy array and QVideoFrameProperty, and emits them to arrayChanged signal.

User may use this class for convenience, or define own pipeline.

class araviq6.videostream.NDArrayMediaCaptureSession(*args: Any, **kwargs: Any)

Bases: QMediaCaptureSession

Capture session which emits numpy array.

When capturing, NDArrayMediaCaptureSession converts the frame to numpy array and QVideoFrameProperty, and emits them to arrayChanged signal.

User may use this class for convenience, or define own pipeline.