Class VideoDecoder

java.lang.Object
org.javagi.base.ProxyInstance
org.gnome.gobject.TypeInstance
org.gnome.gobject.GObject
org.gnome.gobject.InitiallyUnowned
org.freedesktop.gstreamer.gst.GstObject
org.freedesktop.gstreamer.gst.Element
org.freedesktop.gstreamer.video.VideoDecoder
All Implemented Interfaces:
Proxy
Direct Known Subclasses:
VideoDecoder.VideoDecoder$Impl
@Generated("org.javagi.JavaGI") public abstract class VideoDecoder extends Element

This base class is for video decoders turning encoded data into raw video frames.

The GstVideoDecoder base class and derived subclasses should cooperate as follows:

Configuration

  • Initially, GstVideoDecoder calls start when the decoder element is activated, which allows the subclass to perform any global setup.

  • GstVideoDecoder calls setFormat to inform the subclass of caps describing input video data that it is about to receive, including possibly configuration data. While unlikely, it might be called more than once, if changing input parameters require reconfiguration.

  • Incoming data buffers are processed as needed, described in Data Processing below.

  • GstVideoDecoder calls stop at end of all processing.

Data processing

  • The base class gathers input data, and optionally allows subclass to parse this into subsequently manageable chunks, typically corresponding to and referred to as 'frames'.

  • Each input frame is provided in turn to the subclass' handleFrame callback.

  • When the subclass enables the subframe mode with gst_video_decoder_set_subframe_mode, the base class will provide to the subclass the same input frame with different input buffers to the subclass handleFrame callback. During this call, the subclass needs to take ownership of the input_buffer as GstVideoCodecFrame.inputBuffer will have been changed before the next subframe buffer is received. The subclass will call gst_video_decoder_have_last_subframe when a new input frame can be created by the base class. Every subframe will share the same GstVideoCodecFrame.outputBuffer to write the decoding result. The subclass is responsible to protect its access.

  • If codec processing results in decoded data, the subclass should call gstVideoDecoderFinishFrame to have decoded data pushed downstream. In subframe mode the subclass should call gstVideoDecoderFinishSubframe until the last subframe where it should call gstVideoDecoderFinishFrame. The subclass can detect the last subframe using GST_VIDEO_BUFFER_FLAG_MARKER on buffers or using its own logic to collect the subframes. In case of decoding failure, the subclass must call gstVideoDecoderDropFrame or gstVideoDecoderDropSubframe, to allow the base class to do timestamp and offset tracking, and possibly to requeue the frame for a later attempt in the case of reverse playback.

Shutdown phase

  • The GstVideoDecoder class calls stop to inform the subclass that data parsing will be stopped.

Additional Notes

  • Seeking/Flushing

    • When the pipeline is seeked or otherwise flushed, the subclass is informed via a call to its reset callback, with the hard parameter set to true. This indicates the subclass should drop any internal data queues and timestamps and prepare for a fresh set of buffers to arrive for parsing and decoding.

  • End Of Stream

    • At end-of-stream, the subclass parse function may be called some final times with the at_eos parameter set to true, indicating that the element should not expect any more data to be arriving, and it should parse and remaining frames and call gst_video_decoder_have_frame() if possible.

The subclass is responsible for providing pad template caps for source and sink pads. The pads need to be named "sink" and "src". It also needs to provide information about the output caps, when they are known. This may be when the base class calls the subclass' setFormat function, though it might be during decoding, before calling gstVideoDecoderFinishFrame. This is done via gstVideoDecoderSetOutputState

The subclass is also responsible for providing (presentation) timestamps (likely based on corresponding input ones). If that is not applicable or possible, the base class provides limited framerate based interpolation.

Similarly, the base class provides some limited (legacy) seeking support if specifically requested by the subclass, as full-fledged support should rather be left to upstream demuxer, parser or alike. This simple approach caters for seeking and duration reporting using estimated input bitrates. To enable it, a subclass should call gstVideoDecoderSetEstimateRate to enable handling of incoming byte-streams.

The base class provides some support for reverse playback, in particular in case incoming data is not packetized or upstream does not provide fragments on keyframe boundaries. However, the subclass should then be prepared for the parsing and frame processing stage to occur separately (in normal forward processing, the latter immediately follows the former), The subclass also needs to ensure the parsing stage properly marks keyframes, unless it knows the upstream elements will do so properly for incoming data.

The bare minimum that a functional subclass needs to implement is:

  • Provide pad templates

  • Inform the base class of output caps via gstVideoDecoderSetOutputState

  • Parse input data, if it is not considered packetized from upstream Data will be provided to parse which should invoke gstVideoDecoderAddToFrame and gstVideoDecoderHaveFrame to separate the data belonging to each video frame.

  • Accept data in handleFrame and provide decoded results to gstVideoDecoderFinishFrame, or call gstVideoDecoderDropFrame.

  • Constructor Details

    • VideoDecoder

      public VideoDecoder(MemorySegment address)
      Create a VideoDecoder instance for the provided memory address.
      Parameters:
      address - the memory address of the native object

    • VideoDecoder

      public VideoDecoder()
      Create a new VideoDecoder.

  • Method Details

    • getType

      public static @Nullable Type getType()
      Get the GType of the VideoDecoder class.
      Returns:
      the GType

    • getMemoryLayout

      public static MemoryLayout getMemoryLayout()
      The memory layout of the native struct.
      Returns:
      the memory layout

    • asParent

      protected VideoDecoder asParent()
      Return this instance as if it were its parent type. Comparable to the Java super keyword, but ensures the parent typeclass is also used in native code.
      Overrides:
      asParent in class Element
      Returns:
      the instance as if it were its parent type

    • addToFrame

      public void addToFrame(int nBytes)
      Removes next nBytes of input data and adds it to currently parsed frame.
      Parameters:
      nBytes - the number of bytes to add

    • allocateOutputBuffer

      public @Nullable Buffer allocateOutputBuffer()

      Helper function that allocates a buffer to hold a video frame for decoder's current GstVideoCodecState.

      You should use gst_video_decoder_allocate_output_frame() instead of this function, if possible at all.

      Returns:
      allocated buffer, or NULL if no buffer could be allocated (e.g. when downstream is flushing or shutting down)

    • allocateOutputFrame

      public FlowReturn allocateOutputFrame(VideoCodecFrame frame)

      Helper function that allocates a buffer to hold a video frame for decoder's current GstVideoCodecState. Subclass should already have configured video state and set src pad caps.

      The buffer allocated here is owned by the frame and you should only keep references to the frame, not the buffer.

      Parameters:
      frame - a GstVideoCodecFrame
      Returns:
      FlowReturn.OK if an output buffer could be allocated

    • allocateOutputFrameWithParams

      public FlowReturn allocateOutputFrameWithParams(VideoCodecFrame frame, BufferPoolAcquireParams params)
      Same as gst_video_decoder_allocate_output_frame except it allows passing GstBufferPoolAcquireParams to the sub call gst_buffer_pool_acquire_buffer.
      Parameters:
      frame - a GstVideoCodecFrame
      params - a GstBufferPoolAcquireParams
      Returns:
      FlowReturn.OK if an output buffer could be allocated
      Since:
      1.12

    • dropFrame

      public FlowReturn dropFrame(VideoCodecFrame frame)
      Similar to gst_video_decoder_finish_frame(), but drops frame in any case and posts a QoS message with the frame's details on the bus. In any case, the frame is considered finished and released.
      Parameters:
      frame - the GstVideoCodecFrame to drop
      Returns:
      a GstFlowReturn, usually GST_FLOW_OK.

    • dropSubframe

      public FlowReturn dropSubframe(VideoCodecFrame frame)
      Drops input data. The frame is not considered finished until the whole frame is finished or dropped by the subclass.
      Parameters:
      frame - the GstVideoCodecFrame
      Returns:
      a GstFlowReturn, usually GST_FLOW_OK.
      Since:
      1.20

    • finishFrame

      public FlowReturn finishFrame(VideoCodecFrame frame)

      frame should have a valid decoded data buffer, whose metadata fields are then appropriately set according to frame data and pushed downstream. If no output data is provided, frame is considered skipped. In any case, the frame is considered finished and released.

      After calling this function the output buffer of the frame is to be considered read-only. This function will also change the metadata of the buffer.

      Parameters:
      frame - a decoded GstVideoCodecFrame
      Returns:
      a GstFlowReturn resulting from sending data downstream

    • finishSubframe

      public FlowReturn finishSubframe(VideoCodecFrame frame)
      Indicate that a subframe has been finished to be decoded by the subclass. This method should be called for all subframes except the last subframe where gstVideoDecoderFinishFrame should be called instead.
      Parameters:
      frame - the GstVideoCodecFrame
      Returns:
      a GstFlowReturn, usually GST_FLOW_OK.
      Since:
      1.20

    • getAllocator

      public void getAllocator(@Nullable Out<Allocator> allocator, @Nullable AllocationParams params)

      Lets GstVideoDecoder sub-classes to know the memory allocator used by the base class and its params.

      Unref the allocator after use it.

      Parameters:
      allocator - the GstAllocator used
      params - the GstAllocationParams of allocator

    • getBufferPool

      public @Nullable BufferPool getBufferPool()

    • getEstimateRate

      public int getEstimateRate()

    • getFrame

      public @Nullable VideoCodecFrame getFrame(int frameNumber)
      Get a pending unfinished GstVideoCodecFrame
      Parameters:
      frameNumber - system_frame_number of a frame
      Returns:
      pending unfinished GstVideoCodecFrame identified by frameNumber.

    • getFrames

      public List<VideoCodecFrame> getFrames()
      Get all pending unfinished GstVideoCodecFrame
      Returns:
      pending unfinished GstVideoCodecFrame.

    • getInputSubframeIndex

      public int getInputSubframeIndex(VideoCodecFrame frame)
      Queries the number of the last subframe received by the decoder baseclass in the frame.
      Parameters:
      frame - the GstVideoCodecFrame to update
      Returns:
      the current subframe index received in subframe mode, 1 otherwise.
      Since:
      1.20

    • getLatency

      public void getLatency(@Nullable ClockTime minLatency, @Nullable ClockTime maxLatency)
      Query the configured decoder latency. Results will be returned via minLatency and maxLatency.
      Parameters:
      minLatency - address of variable in which to store the configured minimum latency, or null
      maxLatency - address of variable in which to store the configured mximum latency, or null

    • getMaxDecodeTime

      public ClockTimeDiff getMaxDecodeTime(VideoCodecFrame frame)
      Determines maximum possible decoding time for frame that will allow it to decode and arrive in time (as determined by QoS events). In particular, a negative result means decoding in time is no longer possible and should therefore occur as soon/skippy as possible.
      Parameters:
      frame - a GstVideoCodecFrame
      Returns:
      max decoding time.

    • getMaxErrors

      public int getMaxErrors()

    • getNeedsFormat

      public boolean getNeedsFormat()
      Queries decoder required format handling.
      Returns:
      true if required format handling is enabled.
      Since:
      1.4

    • getNeedsSyncPoint

      public boolean getNeedsSyncPoint()
      Queries if the decoder requires a sync point before it starts outputting data in the beginning.
      Returns:
      true if a sync point is required in the beginning.
      Since:
      1.20

    • getOldestFrame

      public @Nullable VideoCodecFrame getOldestFrame()
      Get the oldest pending unfinished GstVideoCodecFrame
      Returns:
      oldest pending unfinished GstVideoCodecFrame.

    • getOutputState

      public @Nullable VideoCodecState getOutputState()
      Get the GstVideoCodecState currently describing the output stream.
      Returns:
      GstVideoCodecState describing format of video data.

    • getPacketized

      public boolean getPacketized()
      Queries whether input data is considered packetized or not by the base class.
      Returns:
      TRUE if input data is considered packetized.

    • getPendingFrameSize

      public long getPendingFrameSize()
      Returns the number of bytes previously added to the current frame by calling gst_video_decoder_add_to_frame().
      Returns:
      The number of bytes pending for the current frame
      Since:
      1.4

    • getProcessedSubframeIndex

      public int getProcessedSubframeIndex(VideoCodecFrame frame)
      Queries the number of subframes in the frame processed by the decoder baseclass.
      Parameters:
      frame - the GstVideoCodecFrame to update
      Returns:
      the current subframe processed received in subframe mode.
      Since:
      1.20

    • getQosProportion

      public double getQosProportion()

    • getSubframeMode

      public boolean getSubframeMode()
      Queries whether input data is considered as subframes or not by the base class. If FALSE, each input buffer will be considered as a full frame.
      Returns:
      TRUE if input data is considered as sub frames.
      Since:
      1.20

    • haveFrame

      public FlowReturn haveFrame()
      Gathers all data collected for currently parsed frame, gathers corresponding metadata and passes it along for further processing, i.e. handleFrame.
      Returns:
      a GstFlowReturn

    • haveLastSubframe

      public FlowReturn haveLastSubframe(VideoCodecFrame frame)
      Indicates that the last subframe has been processed by the decoder in frame. This will release the current frame in video decoder allowing to receive new frames from upstream elements. This method must be called in the subclass handleFrame callback.
      Parameters:
      frame - the GstVideoCodecFrame to update
      Returns:
      a GstFlowReturn, usually GST_FLOW_OK.
      Since:
      1.20

    • mergeTags

      public void mergeTags(@Nullable TagList tags, TagMergeMode mode)

      Sets the audio decoder tags and how they should be merged with any upstream stream tags. This will override any tags previously-set with gst_audio_decoder_merge_tags().

      Note that this is provided for convenience, and the subclass is not required to use this and can still do tag handling on its own.

      MT safe.

      Parameters:
      tags - a GstTagList to merge, or NULL to unset previously-set tags
      mode - the GstTagMergeMode to use, usually GST_TAG_MERGE_REPLACE

    • negotiate

      public boolean negotiate()
      Negotiate with downstream elements to currently configured GstVideoCodecState. Unmark GST_PAD_FLAG_NEED_RECONFIGURE in any case. But mark it again if negotiate fails.
      Returns:
      true if the negotiation succeeded, else false.

    • proxyGetcaps

      public Caps proxyGetcaps(@Nullable Caps caps, @Nullable Caps filter)
      Returns caps that express caps (or sink template caps if caps == NULL) restricted to resolution/format/... combinations supported by downstream elements.
      Parameters:
      caps - initial caps
      filter - filter caps
      Returns:
      a GstCaps owned by caller
      Since:
      1.6

    • releaseFrame

      public void releaseFrame(VideoCodecFrame frame)
      Similar to gst_video_decoder_drop_frame(), but simply releases frame without any processing other than removing it from list of pending frames, after which it is considered finished and released.
      Parameters:
      frame - the GstVideoCodecFrame to release
      Since:
      1.2.2

    • requestSyncPoint

      public void requestSyncPoint(VideoCodecFrame frame, Set<VideoDecoderRequestSyncPointFlags> flags)

      Allows the GstVideoDecoder subclass to request from the base class that a new sync should be requested from upstream, and that frame was the frame when the subclass noticed that a new sync point is required. A reason for the subclass to do this could be missing reference frames, for example.

      The base class will then request a new sync point from upstream as long as the time that passed since the last one is exceeding GstVideoDecoder:min-force-key-unit-interval.

      The subclass can signal via flags how the frames until the next sync point should be handled:

      Parameters:
      frame - a GstVideoCodecFrame
      flags - GstVideoDecoderRequestSyncPointFlags
      Since:
      1.20

    • requestSyncPoint

      public void requestSyncPoint(VideoCodecFrame frame, VideoDecoderRequestSyncPointFlags... flags)

      Allows the GstVideoDecoder subclass to request from the base class that a new sync should be requested from upstream, and that frame was the frame when the subclass noticed that a new sync point is required. A reason for the subclass to do this could be missing reference frames, for example.

      The base class will then request a new sync point from upstream as long as the time that passed since the last one is exceeding GstVideoDecoder:min-force-key-unit-interval.

      The subclass can signal via flags how the frames until the next sync point should be handled:

      Parameters:
      frame - a GstVideoCodecFrame
      flags - GstVideoDecoderRequestSyncPointFlags
      Since:
      1.20

    • setEstimateRate

      public void setEstimateRate(boolean enabled)
      Allows baseclass to perform byte to time estimated conversion.
      Parameters:
      enabled - whether to enable byte to time conversion

    • setInterlacedOutputState

      public @Nullable VideoCodecState setInterlacedOutputState(VideoFormat fmt, VideoInterlaceMode interlaceMode, int width, int height, @Nullable VideoCodecState reference)
      Same as gst_video_decoder_set_output_state() but also allows you to also set the interlacing mode.
      Parameters:
      fmt - a GstVideoFormat
      interlaceMode - A GstVideoInterlaceMode
      width - The width in pixels
      height - The height in pixels
      reference - An optional reference GstVideoCodecState
      Returns:
      the newly configured output state.
      Since:
      1.16.

    • setLatency

      public void setLatency(ClockTime minLatency, ClockTime maxLatency)
      Lets GstVideoDecoder sub-classes tell the baseclass what the decoder latency is. If the provided values changed from previously provided ones, this will also post a LATENCY message on the bus so the pipeline can reconfigure its global latency.
      Parameters:
      minLatency - minimum latency
      maxLatency - maximum latency

    • setMaxErrors

      public void setMaxErrors(int num)

      Sets numbers of tolerated decoder errors, where a tolerated one is then only warned about, but more than tolerated will lead to fatal error. You can set -1 for never returning fatal errors. Default is set to GST_VIDEO_DECODER_MAX_ERRORS.

      The '-1' option was added in 1.4

      Parameters:
      num - max tolerated errors

    • setNeedsFormat

      public void setNeedsFormat(boolean enabled)
      Configures decoder format needs. If enabled, subclass needs to be negotiated with format caps before it can process any data. It will then never be handed any data before it has been configured. Otherwise, it might be handed data without having been configured and is then expected being able to do so either by default or based on the input data.
      Parameters:
      enabled - new state
      Since:
      1.4

    • setNeedsSyncPoint

      public void setNeedsSyncPoint(boolean enabled)

      Configures whether the decoder requires a sync point before it starts outputting data in the beginning. If enabled, the base class will discard all non-sync point frames in the beginning and after a flush and does not pass it to the subclass.

      If the first frame is not a sync point, the base class will request a sync point via the force-key-unit event.

      Parameters:
      enabled - new state
      Since:
      1.20

    • setOutputState

      public @Nullable VideoCodecState setOutputState(VideoFormat fmt, int width, int height, @Nullable VideoCodecState reference)

      Creates a new GstVideoCodecState with the specified fmt, width and height as the output state for the decoder. Any previously set output state on this VideoDecoder will be replaced by the newly created one.

      If the subclass wishes to copy over existing fields (like pixel aspec ratio, or framerate) from an existing GstVideoCodecState, it can be provided as a reference.

      If the subclass wishes to override some fields from the output state (like pixel-aspect-ratio or framerate) it can do so on the returned GstVideoCodecState.

      The new output state will only take effect (set on pads and buffers) starting from the next call to gst_video_decoder_finish_frame().

      Parameters:
      fmt - a GstVideoFormat
      width - The width in pixels
      height - The height in pixels
      reference - An optional reference GstVideoCodecState
      Returns:
      the newly configured output state.

    • setPacketized

      public void setPacketized(boolean packetized)
      Allows baseclass to consider input data as packetized or not. If the input is packetized, then the parse method will not be called.
      Parameters:
      packetized - whether the input data should be considered as packetized.

    • setSubframeMode

      public void setSubframeMode(boolean subframeMode)

      If this is set to TRUE, it informs the base class that the subclass can receive the data at a granularity lower than one frame.

      Note that in this mode, the subclass has two options. It can either require the presence of a GST_VIDEO_BUFFER_FLAG_MARKER to mark the end of a frame. Or it can operate in such a way that it will decode a single frame at a time. In this second case, every buffer that arrives to the element is considered part of the same frame until gst_video_decoder_finish_frame() is called.

      In either case, the same GstVideoCodecFrame will be passed to the GstVideoDecoderClass:handle_frame vmethod repeatedly with a different GstVideoCodecFrame:input_buffer every time until the end of the frame has been signaled using either method. This method must be called during the decoder subclass setFormat call.

      Parameters:
      subframeMode - whether the input data should be considered as subframes.
      Since:
      1.20

    • setUseDefaultPadAcceptcaps

      public void setUseDefaultPadAcceptcaps(boolean use)

      Lets GstVideoDecoder sub-classes decide if they want the sink pad to use the default pad query handler to reply to accept-caps queries.

      By setting this to true it is possible to further customize the default handler with GST_PAD_SET_ACCEPT_INTERSECT and GST_PAD_SET_ACCEPT_TEMPLATE

      Parameters:
      use - if the default pad accept-caps query handling should be used
      Since:
      1.6

    • close

      protected boolean close()
      Optional. Called when the element changes to GST_STATE_NULL. Allows closing external resources.

    • decideAllocation

      protected boolean decideAllocation(Query query)
      Optional. Setup the allocation parameters for allocating output buffers. The passed in query contains the result of the downstream allocation query. Subclasses should chain up to the parent implementation to invoke the default handler.

    • drain

      protected FlowReturn drain()
      Optional. Called to request subclass to decode any data it can at this point, but that more data may arrive after. (e.g. at segment end). Sub-classes should be prepared to handle new data afterward, or seamless segment processing will break. Since: 1.6

    • finish

      protected FlowReturn finish()
      Optional. Called to request subclass to dispatch any pending remaining data at EOS. Sub-classes can refuse to decode new data after.

    • flush

      protected boolean flush()
      Optional. Flush all remaining data from the decoder without pushing it downstream. Since: 1.2

    • getcaps

      protected Caps getcaps(Caps filter)
      Optional. Allows for a custom sink getcaps implementation. If not implemented, default returns gst_video_decoder_proxy_getcaps applied to sink template caps.

    • handleFrame

      protected FlowReturn handleFrame(VideoCodecFrame frame)

    • handleMissingData

      protected boolean handleMissingData(ClockTime timestamp, ClockTime duration)

    • open

      protected boolean open()
      Optional. Called when the element changes to GST_STATE_READY. Allows opening external resources.

    • parse

      protected FlowReturn parse(VideoCodecFrame frame, Adapter adapter, boolean atEos)
      Required for non-packetized input. Allows chopping incoming data into manageable units (frames) for subsequent decoding.

    • proposeAllocation

      protected boolean proposeAllocation(Query query)
      Optional. Propose buffer allocation parameters for upstream elements. Subclasses should chain up to the parent implementation to invoke the default handler.

    • reset

      protected boolean reset(boolean hard)
      Optional. Allows subclass (decoder) to perform post-seek semantics reset. Deprecated.

    • setFormat

      protected boolean setFormat(VideoCodecState state)
      Notifies subclass of incoming data format (caps).

    • sinkEvent

      protected boolean sinkEvent(Event event)
      Optional. Event handler on the sink pad. This function should return TRUE if the event was handled and should be discarded (i.e. not unref'ed). Subclasses should chain up to the parent implementation to invoke the default handler.

    • sinkQuery

      protected boolean sinkQuery(Query query)
      Optional. Query handler on the sink pad. This function should return TRUE if the query could be performed. Subclasses should chain up to the parent implementation to invoke the default handler. Since: 1.4

    • srcEvent

      protected boolean srcEvent(Event event)
      Optional. Event handler on the source pad. This function should return TRUE if the event was handled and should be discarded (i.e. not unref'ed). Subclasses should chain up to the parent implementation to invoke the default handler.

    • srcQuery

      protected boolean srcQuery(Query query)
      Optional. Query handler on the source pad. This function should return TRUE if the query could be performed. Subclasses should chain up to the parent implementation to invoke the default handler. Since: 1.4

    • start

      protected boolean start()
      Optional. Called when the element starts processing. Allows opening external resources.

    • stop

      protected boolean stop()
      Optional. Called when the element stops processing. Allows closing external resources.

    • transformMeta

      protected boolean transformMeta(VideoCodecFrame frame, Meta meta)
      Optional. Transform the metadata on the input buffer to the output buffer. By default this method is copies all meta without tags and meta with only the "video" tag. subclasses can implement this method and return true if the metadata is to be copied. Since: 1.6