# `Membrane.H264.Parser` [🔗](https://github.com/membraneframework/membrane_h26x_plugin/blob/v0.11.2/lib/membrane_h26x_plugin/h264_parser.ex#L1) Membrane element providing parser for H264 encoded video stream. The parser: * prepares and sends the appropriate stream format, based on information provided in the stream and via the element's options * splits the incoming stream into H264 access units - each buffer being output is a `Membrane.Buffer` struct with a binary payload of a single access unit or network abstraction layer unit. * enriches the output buffers with the metadata describing the way the access unit is split into NAL units, type of each NAL unit making up the access unit and the information if the access unit hold a keyframe. * converts the stream's structure (Annex B, avc1 or avc3) to the one provided via the element's options. The parser handles one of three possible input alignments, depending on the structure of the input buffers: * `:bytestream` - each input buffer contains some part of H264 stream's payload, but not necessary a logical H264 unit (like NAL unit or an access unit). Can be used for i.e. for parsing the stream read from the file. * `:nalu` - each input buffer contains a single NAL unit's payload * `:au` - each input buffer contains a single access unit's payload The input alignment is set automatically, based on the input stream format received by that element: * Receiving `Membrane.RemoteStream` results in the input alignment being set to `:bytestream` * Receiving `%Membrane.H264{alignment: :nalu}` results in the input alignment being set to `:nalu` * Receiving `%Membrane.H264{alignment: :au}` results in the input alignment being set to `:au` The distinction between input alignments was introduced to eliminate the redundant operations and to provide a reliable way for rewriting of timestamps: * for the `:bytestream` input alignment: * if option `:framerate` is set to nil, the output buffers have their `:pts` and `:dts` set to nil * if framerate is specified, `:pts` and `:dts` will be generated automatically, based on that framerate, starting from 0 This may only be used with H264 profiles `:baseline` and `:constrained_baseline`, where `PTS==DTS`. * for the `:nalu` input alignment, the output buffers have their `:pts` and `:dts` set to `:pts` and `:dts` of the input buffer that was holding the first NAL unit making up given access unit (that is being sent inside that output buffer). * for the `:au` input alignment, the output buffers have their `:pts` and `:dts` set to `:pts` and `:dts` of the input buffer (holding the whole access unit being output) The parser also allows for conversion between stream structures. The available structures are: * Annex B, `:annexb` - In a stream with this structure each NAL unit is prefixed by three or four-byte start code (`0x(00)000001`) that allows to identify boundaries between them. * avc1, `:avc1` - In such stream a DCR (Decoder Configuration Record) is included in `stream_format` and NALUs lack the start codes, but are prefixed with their length. The length of these prefixes is contained in the stream's DCR. PPSs and SPSs (Picture Parameter Sets and Sequence Parameter Sets) are transported in the DCR. * avc3, `:avc3` - The same as avc1, only that parameter sets may be also present in the stream (in-band). ## Element options Passed via struct `t:Membrane.H264.Parser.t/0` - `spss` ``` [binary()] ``` Default value: `[]` Sequence Parameter Set NAL unit binary payloads - if absent in the stream, should be provided via this option (only available for `:annexb` output stream structure). - `ppss` ``` [binary()] ``` Default value: `[]` Picture Parameter Set NAL unit binary payloads - if absent in the stream, should be provided via this option (only available for `:annexb` output stream structure). - `output_alignment` ``` :au | :nalu ``` Default value: `:au` Alignment of the buffers produced as an output of the parser. If set to `:au`, each output buffer will be a single access unit. Otherwise, if set to `:nalu`, each output buffer will be a single NAL unit. Defaults to `:au`. - `skip_until_keyframe` ``` boolean() ``` Default value: `true` Determines whether to drop the stream until the first key frame is received. - `repeat_parameter_sets` ``` boolean() ``` Default value: `false` Repeat all parameter sets (`sps` and `pps`) on each IDR picture. Parameter sets may be retrieved from: * The stream * `Parser` options. * Decoder Configuration Record, sent in `:acv1` and `:avc3` stream types - `output_stream_structure` ``` nil | stream_structure() | :avc1 | :avc3 ``` Default value: `nil` format of the outgoing H264 stream, if set to `:annexb` NALUs will be separated by a start code (0x(00)000001) or if set to `:avc3` or `:avc1` they will be prefixed by their size. Additionally for `:avc1` and `:avc3` a tuple can be passed containing the atom and `nalu_length_size` that determines the size in bytes of each NALU's field describing their length (by default 4). In avc1 output streams the PPSs and SPSs will be transported in the DCR, when in avc3 they will be present only in the stream (in-band). If not provided or set to nil the stream's structure will remain unchanged. - `generate_best_effort_timestamps` ``` false | %{:framerate => {pos_integer(), pos_integer()}, optional(:add_dts_offset) => boolean()} ``` Default value: `false` Generates timestamps based on given `framerate`. This option works only when `Membrane.RemoteStream` format arrives. Keep in mind that the generated timestamps may be inaccurate and lead to video getting out of sync with other media, therefore h264 should be kept in a container that stores the timestamps alongside. PTS are derived from each frame's presentation order, recovered by sorting the access units by their Picture Order Count (POC). Because PTS are based on the relative POC order rather than the absolute POC values, the timestamps are correct regardless of the step by which POC advances. Recovering the presentation order requires buffering (reordering) a bounded number of access units before emitting them, which introduces a constant latency of that many frame durations. This only happens when the stream can actually reorder frames; when it can't (baseline / constrained baseline profile, or `pic_order_cnt_type == 2`) the buffering is disabled and no latency is added. When the SPS VUI provides `max_num_reorder_frames`, it is used as the exact buffer depth, keeping the latency as low as the stream allows; otherwise a safe maximum of 15 frames (half a second at 30 FPS) is assumed. By default, the parser adds negative DTS offset to the timestamps, so that in case of frame reorder (which always happens when B frames are present) the DTS was always bigger than PTS. If that is not desired, you can set `add_dts_offset: false`. ## Pads ### `:input` Accepted formats: ``` RemoteStream ``` ``` H264 ``` Direction: | `:input` Availability: | `:always` Flow control: | `:auto` ### `:output` Accepted formats: ``` %H264{alignment: alignment, nalu_in_metadata?: true} when alignment in [:nalu, :au] ``` Direction: | `:output` Availability: | `:always` Flow control: | `:auto` # `stream_structure` ```elixir @type stream_structure() :: :annexb | {codec_tag :: :avc1 | :avc3, nalu_length_size :: pos_integer()} ``` Format of the H264 stream, if set to `:annexb` NALUs will be separated by a start code (0x(00)000001) or if set to `:avc3` or `:avc1` they will be prefixed by their size. # `t` ```elixir @type t() :: %Membrane.H264.Parser{ generate_best_effort_timestamps: false | %{ :framerate => {pos_integer(), pos_integer()}, optional(:add_dts_offset) => boolean() }, output_alignment: :au | :nalu, output_stream_structure: nil | stream_structure() | :avc1 | :avc3, ppss: [binary()], repeat_parameter_sets: boolean(), skip_until_keyframe: boolean(), spss: [binary()] } ``` Struct containing options for `Membrane.H264.Parser` # `options` ```elixir @spec options() :: keyword() ``` Returns description of options available for this module --- *Consult [api-reference.md](api-reference.md) for complete listing*