Skip to main content

Go SDK

kloakd-go — one API for everything, idiomatic Go client with full module coverage

4 min read

Install

go get github.com/kloakd/kloakd-go

Go 1.21+ | 71/71 tests | 82.7% coverage | zero dependencies (stdlib net/http)

Client

import kloakd "github.com/kloakd/kloakd-go"

client := kloakd.MustNew(kloakd.Config{
    APIKey:         os.Getenv("KLOAKD_API_KEY"),
    OrganizationID: os.Getenv("KLOAKD_ORG_ID"),
    Timeout:        30 * time.Second,
    MaxRetries:     3,
})

Quickstart — one call does everything

result, _ := client.Crawl(ctx, "https://example.com",
    &kloakd.CrawlOptions{
        MaxPages: 50,
        ExtractSchema: map[string]string{"title": "css:h1", "content": "css:article"},
    })

for _, page := range result.Pages {
    if page.Success {
        fmt.Printf("%s → %v\n", page.URL, page.StructuredData)
    }
}

That's it. Crawl() internally:

  1. Discovers all pages on the site (Webgrph BFS)
  2. Fetches each page through the 4-tier anti-bot engine (auto-escalates to headless browser for Cloudflare-protected sites — no ForceBrowser needed)
  3. Extracts structured data from each page if ExtractSchema is provided (Kolektr, using cached fetch artifacts — no second HTTP round-trip)

Per-page failures are caught and marked Success: false — the crawl never aborts on a single page error.

Primary method

client.Crawl()

result, _ := client.Crawl(ctx, "https://example.com",
    &kloakd.CrawlOptions{
        MaxDepth:      3,
        MaxPages:      100,
        ExtractSchema: map[string]string{"title": "css:h1", "price": "css:.price"},
    })

fmt.Printf("Discovered: %d pages\n", result.TotalPagesDiscovered)
fmt.Printf("Fetched:    %d\n", result.PagesFetched)
fmt.Printf("Failed:     %d\n", result.PagesFailed)

for _, page := range result.Pages {
    if page.Success {
        fmt.Printf("  %s [tier %d] %v\n", page.URL, page.TierUsed, page.StructuredData)
    } else {
        fmt.Printf("  %s FAILED: %s\n", page.URL, *page.Error)
    }
}

Parameters

| Parameter | Type | Default | Description | |---|---|---|---| | url | string | required | Seed URL to start crawling from | | MaxDepth | int | 3 | Maximum BFS depth | | MaxPages | int | 100 | Maximum pages to crawl | | ExtractSchema | map[string]string | nil | CSS selector schema for structured extraction | | IncludeExternalLinks | bool | false | Follow off-domain links | | SessionArtifactID | string | "" | Reuse an authenticated session artifact (from Fetchyr) |

Returns: SiteCrawlResult

| Field | Type | Description | |---|---|---| | Success | bool | Whether the crawl completed | | URL | string | Seed URL | | TotalPagesDiscovered | int | Pages found during BFS | | PagesFetched | int | Pages successfully fetched | | PagesFailed | int | Pages that failed (included with Success: false) | | Pages | []CrawlPage | All pages with HTML and optional structured data | | CrawlArtifactID | *string | Artifact ID for the site hierarchy | | Error | *string | Error message if crawl failed entirely |

client.CrawlStream() — streaming

For long-running crawls, use the streaming version to receive real-time progress events via a channel:

ch, _ := client.CrawlStream(ctx, "https://example.com",
    &kloakd.CrawlStreamOptions{
        MaxPages:      500,
        ExtractSchema: map[string]string{"title": "css:h1"},
    })

for event := range ch {
    if event.Err != nil { break }
    switch event.Type {
    case "page_fetched":
        fmt.Printf("[%d/%d] %s OK\n", event.Page, event.Total, event.URL)
    case "page_failed":
        fmt.Printf("[%d/%d] %s FAIL: %s\n", event.Page, event.Total, event.URL, event.Error)
    case "crawl_complete":
        fmt.Println("Done")
    }
}

Event types

| Type | Description | |---|---| | discovery_started | Crawl discovery has begun | | discovery_progress | Pages found during BFS (PagesFound field) | | discovery_complete | Discovery finished, fetch phase starting | | page_fetching | About to fetch page N of total | | page_fetched | Page fetched successfully | | page_failed | Page fetch failed (crawl continues) | | crawl_complete | All pages processed |

Low-level methods

Need fine-grained control? The individual modules are still available:

Evadr.Fetch()

page, _ := client.Evadr.Fetch(ctx, "https://example.com", nil)
fmt.Printf("Status: %d, Tier: %d\n", page.StatusCode, page.TierUsed)
fmt.Printf("Artifact ID: %s\n", page.ArtifactID)

Webgrph.Crawl()

crawl, _ := client.Webgrph.Crawl(ctx, "https://example.com",
    &kloakd.CrawlOptions{MaxDepth: 3, MaxPages: 100})
fmt.Printf("Crawl started: %s\n", crawl.CrawlID)

Kolektr.Page()

result, _ := client.Kolektr.Page(ctx, "https://example.com",
    &kloakd.PageOptions{
        Schema: map[string]string{"title": "css:h1", "price": "css:.price"},
    })
for _, r := range result.Records {
    fmt.Println(r)
}

Artifact chaining

Pass artifact IDs between low-level methods to skip redundant work:

page, _ := client.Evadr.Fetch(ctx, "https://example.com", nil)

data, _ := client.Kolektr.Page(ctx, "https://example.com",
    &kloakd.PageOptions{
        Schema:          map[string]string{"title": "css:h1"},
        FetchArtifactID: page.ArtifactID,
    })

crawl, _ := client.Webgrph.Crawl(ctx, "https://example.com",
    &kloakd.CrawlOptions{
        MaxDepth:         2,
        SessionArtifactID: page.ArtifactID,
    })

Error handling

result, err := client.Crawl(ctx, "https://example.com", nil)
if err != nil {
    if errors.Is(err, kloakd.ErrRateLimit) {
        var rle *kloakd.RateLimitError
        errors.As(err, &rle)
        fmt.Printf("Rate limited. Retry after %ds\n", rle.RetryAfter)
    } else if errors.Is(err, kloakd.ErrAuthentication) {
        fmt.Println("Invalid API key")
    } else {
        fmt.Printf("Error: %v\n", err)
    }
}

Advanced namespaces

client.Skanyr    // API discovery
client.Nexus     // AI strategy engine
client.Parlyr    // Natural language queries
client.Fetchyr   // RPA & authenticated scraping

Next steps

Was this page helpful?
PreviousTypeScript SDKNextJava SDK