Struct Host 

pub struct Host {
    pub key: HostKey,
    rate_limiter: Option<RateLimiter<NotKeyed, InMemoryState, DefaultClock>>,
    semaphore: Semaphore,
    client: Client,
    stats: Mutex<HostStats>,
    backoff_duration: Mutex<Duration>,
    cache: DashMap<Uri, CacheableResponse>,
}

Represents a single host with its own rate limiting, concurrency control, HTTP client configuration, and request cache.

Each host maintains:

• A token bucket rate limiter using governor
• A semaphore for concurrency control
• A dedicated HTTP client with host-specific headers and cookies
• Statistics tracking for adaptive behavior
• A per-host cache to prevent duplicate requests

Fields

key: HostKey

The hostname this instance manages

rate_limiter: Option<RateLimiter<NotKeyed, InMemoryState, DefaultClock>>

Rate limiter using token bucket algorithm

semaphore: Semaphore

Controls maximum concurrent requests to this host

client: Client

HTTP client configured for this specific host

stats: Mutex<HostStats>

Request statistics and adaptive behavior tracking

backoff_duration: Mutex<Duration>

Current backoff duration for adaptive rate limiting

cache: DashMap<Uri, CacheableResponse>

Per-host cache to prevent duplicate requests during a single link check invocation. Note that this cache has no direct relation to the inter-process persistable crate::CacheStatus.

Implementations

impl Host

pub fn new( key: HostKey, host_config: &HostConfig, global_config: &RateLimitConfig, client: ReqwestClient, ) -> Self

Create a new Host instance for the given hostname

fn get_cached_status( &self, uri: &Uri, needs_body: bool, ) -> Option<CacheableResponse>

Check if a URI is cached and returns the cached response if it is valid and satisfies the needs_body requirement.

fn record_cache_hit(&self)

fn record_cache_miss(&self)

fn cache_result(&self, uri: &Uri, response: CacheableResponse)

Cache a request result

pub(crate) async fn execute_request( &self, request: Request, needs_body: bool, ) -> Result<CacheableResponse>

Execute a request with rate limiting, concurrency control, and caching

Errors

Returns an error if the request fails or rate limiting is exceeded

Panics

Panics if the statistics mutex is poisoned

pub(crate) const fn get_client(&self) -> &ReqwestClient

async fn perform_request( &self, request: Request, uri: Uri, needs_body: bool, ) -> Result<CacheableResponse>

async fn await_backoff(&self)

Await adaptive backoff if needed

async fn acquire_semaphore(&self) -> SemaphorePermit<'_>

fn update_backoff(&self, status: StatusCode)

fn update_stats(&self, status: StatusCode, request_time: Duration)

fn handle_rate_limit_headers(&self, response: &ReqwestResponse)

Parse rate limit headers from response and adjust behavior

fn handle_common_rate_limit_header_fields(&self, headers: &HeaderMap)

Handle the common “X-RateLimit” header fields.

fn handle_retry_after_header(&self, headers: &HeaderMap)

Handle the “Retry-After” header

fn increase_backoff(&self, increased_backoff: Duration)

pub fn stats(&self) -> HostStats

Get host statistics

Panics

Panics if the statistics mutex is poisoned

pub(crate) fn record_persistent_cache_hit(&self)

Record a cache hit from the persistent disk cache. Cache misses are tracked internally, so we don’t expose such a method.

pub fn cache_size(&self) -> usize

Get the current cache size (number of cached entries)

Trait Implementations

impl Debug for Host

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.