Struct LinkExtractor 

struct LinkExtractor<S: SpanProvider> {
    span_provider: S,
    links: Vec<RawUri>,
    fragments: HashSet<String>,
    include_verbatim: bool,
    current_element: String,
    current_attributes: HashMap<String, Spanned<String>>,
    current_attribute_name: String,
    verbatim_stack: Vec<String>,
    in_style_tag: bool,
    style_content: String,
    style_content_offset: usize,
}

Extract links from HTML documents.

This is the main driver for the html5gum tokenizer. It implements the Emitter trait, which is used by the tokenizer to communicate with the caller.

The LinkExtractor keeps track of the current element being processed, the current attribute being processed, and a bunch of plain characters currently being processed.

The links vector contains all links extracted from the HTML document and the fragments set contains all fragments extracted from the HTML document.

Fields

span_provider: S

The SpanProvider which will be used to compute spans for URIs.

This is generic, since e.g. the markdown parser has already started, so we have to compute the span location in relation to the offset in the outer document.

links: Vec<RawUri>

Links extracted from the HTML document.

fragments: HashSet<String>

Fragments extracted from the HTML document.

include_verbatim: bool

Whether to include verbatim elements in the output.

current_element: String

Current element name being processed. This is called a tag in html5gum.

current_attributes: HashMap<String, Spanned<String>>

Current attributes being processed. This is a list of key-value pairs (in order of appearance), where the key is the attribute name and the value is the attribute value.

current_attribute_name: String

Current attribute name being processed.

verbatim_stack: Vec<String>

Element name of the current verbatim block. Used to keep track of nested verbatim blocks.

in_style_tag: bool

Whether we’re currently inside a <style> tag.

style_content: String

Accumulated CSS content from within a <style> tag.

style_content_offset: usize

Start offset of the style tag content (for span calculation).

Implementations

impl<S: SpanProvider> LinkExtractor<S>

fn new(span_provider: S, include_verbatim: bool) -> Self

Create a new LinkExtractor.

Set include_verbatim to true if you want to include verbatim elements in the output.

fn extract_urls_from_elem_attr(&self) -> Vec<RawUri>

Extract all semantically known links from a given HTML attribute.

fn filter_verbatim_here(&self) -> bool

Check if we should filter out links in the current context due to being inside a verbatim element.

fn flush_links(&mut self)

Flush the current element and attribute values to the links vector.

This function is called whenever a new element is encountered or when the current element is closing. It extracts URLs from the current attribute value and adds them to the links vector.

Here are the rules for extracting links:

• If the current element has a rel=nofollow attribute, the current attribute value is ignored.
• If the current element has a rel=preconnect or rel=dns-prefetch attribute, the current attribute value is ignored.
• If the current attribute value is not a URL, it is treated as plain text and added to the links vector.
• If the current attribute name is id, the current attribute value is added to the fragments set.

The current attribute name and value are cleared after processing.

Trait Implementations

impl<S: SpanProvider> Callback<(), usize> for &mut LinkExtractor<S>

fn handle_event( &mut self, event: CallbackEvent<'_>, span: Span<usize>, ) -> Option<()>

Perform some action on a parsing event, and, optionally, return a value that can be yielded from the crate::Tokenizer iterator.

impl<S: Clone + SpanProvider> Clone for LinkExtractor<S>

fn clone(&self) -> LinkExtractor<S>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<S: Debug + SpanProvider> Debug for LinkExtractor<S>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.