{ "datasets": { "agent_keys": { "cycle": "iso_week", "fingerprint_inputs": [ "agent_key_history:METHOD" ], "title": "Agent keys: public keys and card signatures agent hosts publish", "versions": [ { "additions": [ { "basis": "founder request data2-ondemand-census (PR #797)", "date_utc": "2026-10-02T18:00:00Z", "what": "Rows from hosts the Observatory reads because an outside caller asked about them are kept with cohort 'requested_hosts' (their change entries are marked the same way) and are left out of the weekly aggregates and change counts, which report them beside the figures as requested_hosts_not_counted. No such rows existed before, so no earlier row or published figure is read differently and no overlap is needed." }, { "basis": "founder request data4-declared-extras (decision #24)", "date_utc": "2026-10-03T00:00:00Z", "what": "Rows read for A2A cards the weekly standards probes found (source 'standards_probe') are tagged exactly like requested hosts' rows (cohort 'requested_hosts') and left out of the weekly aggregates and change counts. No such rows existed before, so no earlier row or published figure is read differently and no overlap is needed." }, { "basis": "founder decision #26 (LEGAL, delegated to Claude, 3 Oct 2026); data4 audit leftover", "date_utc": "2026-10-03T16:00:00Z", "what": "Part and figure addition, not a change to existing rows, hashes or figures. (1) Web Bot Auth key directories: the weekly standards probes add one GET of /.well-known/http-message-signatures-directory (IETF draft-meunier-http-message-signatures-directory) per Observatory target host per ISO week, robots.txt honoured (RFC 9309 matcher), identifying user agent, through safe_fetch; shop panel hosts and hosts a caller asked about are not included, and no registry list (Cloudflare's CC BY-NC list or any other) is imported. The row's new part bot_auth_directory keeps the status, media type, per key the jwks facts (type, curve, algorithm, use, key id, RFC 7638 thumbprint; never key material beyond the thumbprint) and, from a signature-agent card, only the client name, purpose/description as SHA-256 plus a 300-character excerpt (e-mail and phone masked), expected crawl rate and URL hosts; contact fields are never stored, not even hashed. It enters the hashed facts only when a directory is published, so rows without one hash exactly as before; a row written before the week's probe reads the part as unknown, never as a removal. New change classes bot_auth:directory_appeared (also a host's first reading of a published directory), bot_auth:directory_removed, bot_auth:key_added, bot_auth:key_removed, bot_auth:card_changed, bot_auth:directory_changed. A host whose probe finds a directory gets a row even without a card or 401 (its card, JWKS and protected-resource parts are listed as unread); such hosts are left out of hosts_observed and the earlier shares and counted beside them (web_bot_auth.hosts_with_only_a_directory); their first rows are new first_observation entries. The weekly aggregate web_bot_auth gives rows with a directory reading, hosts publishing (share), keys published and key types (only from 5 publishing hosts), cards, and hosts rotating keys (share), under the small-cell rule. (2) When an MCP server's 401 names /.well-known/oauth-protected-resource on its own host, the standards probe's GET of that document in the same ISO week fills protected_resource and the extra request is no longer sent (the same document read with the same parser, so the facts are identical)." } ], "change_kind": "initial", "fingerprint": "474ae7df429bbdbdf5adb79c4b8426c7b225f9a426283ef19d46557be901b75b", "from_basis": "commit 13b6e28 (four long-run datasets)", "from_utc": "2026-10-01T20:49:42Z", "version": "v1", "what_changed": "First version: A2A agent-card signatures, /.well-known/jwks.json keys (thumbprints, never private material), RFC 9728 protected-resource metadata hosts." } ] }, "agent_prices": { "cycle": "iso_week", "fingerprint_inputs": [ "agent_price_history:METHOD" ], "title": "Agent prices: x402 Bazaar listings (off until switched on)", "versions": [ { "change_kind": "initial", "fingerprint": "2e6e8648cf7152b6d5b04603e41764bba22b972584a8bf95047e1db4a2dd1e53", "from_basis": "commit 086842e (decision #20); no rows until GAIP_SOURCE_X402_BAZAAR=1", "from_utc": "2026-10-02T01:56:25Z", "version": "v1", "what_changed": "First version: per resource host and ISO week, each listed resource's path, type, category and x402 price requirements (never the pay-to address)." } ] }, "agent_terms": { "cycle": "iso_week", "fingerprint_inputs": [ "agent_terms_history:METHOD" ], "title": "Agent terms: what agent services promise and charge", "versions": [ { "additions": [ { "basis": "founder request data2-ondemand-census (PR #797)", "date_utc": "2026-10-02T18:00:00Z", "what": "Rows from hosts the Observatory reads because an outside caller asked about them are kept with cohort 'requested_hosts' (their change entries are marked the same way) and are left out of the weekly aggregates and change counts, which report them beside the figures as requested_hosts_not_counted. No such rows existed before, so no earlier row or published figure is read differently and no overlap is needed." }, { "basis": "founder request data4-declared-extras (decision #24)", "date_utc": "2026-10-03T00:00:00Z", "what": "Rows read for A2A cards the weekly standards probes found (source 'standards_probe') are tagged exactly like requested hosts' rows (cohort 'requested_hosts') and left out of the weekly aggregates and change counts. No such rows existed before, so no earlier row or published figure is read differently and no overlap is needed." } ], "change_kind": "initial", "fingerprint": "24e60dd76f84dc9543837ec023e48ed3747b7a7cef72e145224f8d881e77925d", "from_basis": "commit 13b6e28 (four long-run datasets)", "from_utc": "2026-10-01T20:49:42Z", "version": "v1", "what_changed": "First version: rate limits, uptime/SLA, retention, terms and privacy links, x402 prices declared in documents GAIP reads, beside what GAIP observed of the host that week." } ] }, "ai_permissions": { "cycle": "iso_week", "fingerprint_inputs": [ "ai_permission_history:METHOD", "robots_rfc9309:MATCHER_VERSION" ], "title": "AI permissions: what a site tells AI systems they may do with its content", "versions": [ { "additions": [ { "basis": "commit 086842e (decision #20)", "date_utc": "2026-10-02T01:56:25Z", "what": "RSL licence links and IETF Content-Usage lines and headers added as new fields. Additive: earlier rows lack them and hash exactly as before; not a change to existing fields." } ], "change_kind": "initial", "fingerprint": null, "from_basis": "commit 13b6e28 (four long-run datasets)", "from_utc": "2026-10-01T20:49:42Z", "known_defects": [ { "affects": [ "robots.ai_tokens", "robots.wildcard_home_allowed" ], "detail": "urllib.robotparser ignores * and $ in rules and applies the first matching rule instead of the longest, so whether a named crawler may fetch the home page may be wrong for sites using such rules. Rows observed before the v2 cutover are left out of published shares.", "excluded_from_published_shares": true, "flag": "known defect: wildcard robots rules", "id": "wildcard_robots_rules" } ], "version": "v1", "what_changed": "First version: robots.txt rules for 24 named AI crawlers and assistants (named, home page allowed), * home page allowed, Content-Signal lines, the file fingerprint, TDMRep and TDM-Reservation, llms.txt where read anyway. robots.txt was parsed with Python's urllib.robotparser." }, { "additions": [ { "basis": "founder request data2-ondemand-census (PR #797)", "date_utc": "2026-10-02T18:00:00Z", "what": "Rows from hosts the Observatory reads because an outside caller asked about them are kept with cohort 'requested_hosts' (their change entries are marked the same way) and are left out of the weekly aggregates and change counts, which report them beside the figures as requested_hosts_not_counted. No such rows existed before, so no earlier row or published figure is read differently and no overlap is needed." }, { "basis": "founder request data4-declared-extras (decision #24)", "date_utc": "2026-10-03T00:00:00Z", "what": "Rows read for A2A cards the weekly standards probes found (source 'standards_probe') are tagged exactly like requested hosts' rows (cohort 'requested_hosts') and left out of the weekly aggregates and change counts. No such rows existed before, so no earlier row or published figure is read differently and no overlap is needed." } ], "change_kind": "parser", "fingerprint": "0619516474212b947b85c882a452c9c59f5617cb5937b81a48f2ed649d65ae5f", "from_basis": "founder-set cutover for PR #784 (merged 13:05 UTC); rows observed before 14:00 UTC are read as v1", "from_utc": "2026-10-02T14:00:00Z", "overlap": null, "overlap_note": "Not run: this change predates the overlap rule and fixed a defect; pre-fix rows are flagged and left out of published shares instead.", "version": "v2", "what_changed": "robots.txt parsed by GAIP's RFC 9309 matcher (agent_economy/gaip/robots_rfc9309.py). The first change entry per host whose robots.txt bytes did not change but whose parsed rules did is labelled method_change:robots_rfc9309; entries written before that labelling existed are counted as method_boundary:robots_rfc9309_unverified." }, { "change_kind": "scope", "fingerprint": "132d37b0aa78247222bb4e50665f4143e5bc68c1f04b28f2ac0d2489fb26fbdc", "from_basis": "founder requests data4-robots-reachability and data4-dataset-queue (decision #24); both in force from the end of the same side-by-side week, ISO week 2026-W41 (agent_economy/gaip/ai_permission_history.py READABLE_KEPT_FROM)", "from_utc": "2026-10-12T00:00:00Z", "overlap": { "compared": "(1) In ISO week 2026-W41 the v2 rule still decides each row; whenever it replaces a readable reading by an unreadable one, the row keeps the v3 reading beside it (robots_readable_kept_by_v3, outside the hashed facts), so the weekly robots_read counts and the robots:readable / robots:unreadable change counts can be computed under both rules for the same week. (2) On hosts where both happened in the same week, the panel's own tdmrep.json reading (kept beside the row as tdmrep_panel, not published) against the v2 extra read (which fills tdmrep and the published TDMRep shares until v3): status, published, site reserved. Counted in the weekly snapshot as unique_datasets.ai_permissions.tdmrep_method_overlap (shares, cells under 5 suppressed).", "cycles": 1, "old_and_new_side_by_side_from": "2026-10-05T00:00:00Z", "until": "2026-10-12T00:00:00Z" }, "overlap_note": "Only the tdmrep.json reading can differ by method (another user agent and robots.txt group), so that is what runs side by side. Rows the v2 queue dropped are kept from the deploy because their values are read and parsed exactly as under v2: no per-host value changes, only which hosts are counted, and that cannot be compared with a run that dropped hosts by queue order.", "version": "v3", "what_changed": "Two changes, one version. (1) Parser, data4-robots-reachability: Within one host's week, a readable robots.txt reading is never replaced by an unreadable one (a timeout, a refused connection or any answer that did not give the file); a later readable reading still replaces an earlier one. Under v2 one failed read between two good ones logged the false changes robots:unreadable then robots:readable, and a failed read as the week's last froze the week's row as unreadable. Rows from v3 count the failed reads they ignored (robots_unreadable_reads_ignored: count and last status), outside the hashed facts, so the v2 value can still be told. Effect on published figures: fewer robots:unreadable and robots:readable change entries, and the weekly robots_read counts no longer drop for a host that was read correctly earlier in the week. (2) Scope, data4-dataset-queue: No host GAIP reads is left out any more, and the shop panel reads tdmrep.json itself. Under v2 the facts waited in a 5,000-entry in-memory queue that evicted its oldest entries, and a shop host's row was written only after an extra tdmrep.json read from the Observatory's budget, of which only 20 a tick (about 80 an hour) were possible while the shop panel alone read about 305 shops an hour at its 3,300-an-hour allowance (candidate classification adds more): modelled at 320 an hour, after 30 hours 9,600 hosts had been read and 2,400 rows written, and from then on about three in four were lost. Weekly v2 counts (hosts, robots_read, tdmrep_checked) therefore cover a subset of the hosts read; each row written is correct. From the deploy of this change (on or after 3 Oct 2026, still read as v2 until the overlap ends): an entry is never evicted before it is written; a host whose extra read did not happen within two ticks is written with that part listed under 'unread' (its value null: unknown, never 'absent'), so rows the queue used to drop now appear with identical per-host robots.txt, Content-Signal, llms.txt, RSL and Content-Usage values; the week's extra reads are kept in the store so a deploy does not repeat them. From v3: the tdmrep.json reading of each panel shop comes from the shop panel's own weekly check (Listing Observer user agent and robots.txt group, the panel's own allowance; tdmrep_via 'shop_panel') instead of the Observatory's extra read (tdmrep_via 'extra_read', still used for shop hosts the panel does not read, such as classified candidates). An unread part is never counted as a change in the change log." } ] }, "capability_record": { "cycle": "iso_week", "fingerprint_inputs": [ "capability_record:SCHEMA_VERSION" ], "title": "Capability record: what public MCP servers and A2A agents declared", "versions": [ { "additions": [ { "basis": "founder request data2-ondemand-census (PR #797)", "date_utc": "2026-10-02T18:00:00Z", "what": "Scope addition, not a change to existing rows or figures: hosts an outside caller asked about (source 'requested', cohort 'requested') are observed in the census on the same rules. They are outside the 2026-10 panel, the quarterly random sample, the weekly coverage figure, the standards probes and every per-target public view, so every figure published before keeps its meaning; they are counted beside the figures. No overlap is needed: no earlier row or published aggregate is read differently." }, { "basis": "founder request data3-drift-and-hidden-totals", "date_utc": "2026-10-02T22:00:00Z", "what": "Field and figure addition, not a change to existing rows or figures: new change-log entries carry version_fields_changed (which version fields moved: protocol, declared, server_info, card_version), and the weekly snapshot gains declaration_drift (agents changed by kind, with protocol-version-only changes counted apart from 'agents_changed' and substantive changes made without a declared version change counted as silent; all watched agents and the panel AGENTS-COHORT-2026-10) and declared_totals (tools per server, sign-in kinds, behaviour hints, output schemas, x402 mentions, registry churn). Entries written before this addition are read through their diff's change kinds; where the moved version field cannot be told, the agent is counted as version_change_not_known, never as silent. Earlier snapshots are not rebuilt; the existing change_classes_this_week figure keeps its meaning." }, { "basis": "founder request data3-index-all-agents", "date_utc": "2026-10-02T23:00:00Z", "what": "Published-figure addition, not a change to rows: the GAIP Agent Web Index (/v1/free/agent-web-index) now leads each of its tools, terms, signed-card and AI-preference figures with an all_agents count over every public agent or host GAIP watched or read in the month (requested hosts, GAIP's own agents and unpublishable method rows left out), each with its denominator and ISO-week parts that add up to it; the panel figure (value, denominator) keeps its meaning and is shown beside it as panel. The sample frame's registry figures are one block with four defined counts ((a) entries in the last complete walk, (b) remote, (c) watched, (d) read in the last 7 days); the panel's mcp_servers_from_official_mcp_registry is labelled as part of the panel, not the registry's size. No earlier row or published figure is read differently, so no overlap is needed." }, { "basis": "founder decision #23, founder request data3-watched-agent-list", "date_utc": "2026-10-03T00:00:00Z", "what": "Publication addition, not a change to rows or figures: the list of public agents GAIP watches (/data/watched-agents.csv and .jsonl: target id, source registry, name and identifier as the registry lists them, kind, cohorts, date first watched; no results) is built daily at the Observatory tick, with opted-out, excluded, retired, requested and GAIP's own targets left out, so outsiders can recompute the AGENTS-COHORT-2026-10 membership hash (when no member is withheld) and repeat the quarterly sample. Not under CC BY 4.0. No earlier row or published figure is read differently, so no overlap is needed." }, { "basis": "founder request data4-declared-extras (decision #24)", "date_utc": "2026-10-03T00:00:00Z", "what": "Field and figure addition, not a change to existing rows, hashes or figures: facts the Observatory already received and did not keep (MCP initialize instructions; serverInfo title, description, website and icons; the whole capabilities object with extensions such as MCP Apps, tasks and tools.listChanged; per-tool _meta keys, MCP Apps UI resource, icons, title and execution hints; A2A card extensions such as AP2, x402 and UCP, skill examples, input and output modes, provider and documentation URLs, extended-card support and signature presence; the WWW-Authenticate challenge of a 401 answer) are kept in a separate structure gaip.declared-extras.v1 with its own SHA-256 per section, in side tables, never in the manifest: manifest_sha256 and every capability-record hash are computed exactly as before. Prose is kept as SHA-256, an excerpt of at most 300 characters and the length, personal data redacted; URLs that are not identifiers as host plus SHA-256; extension parameters as key names only. A section that moves appends an internal DECLARED_EXTRAS_CHANGED entry to the change log (classes extras_changed, instructions_changed, extensions_changed, auth_challenge_changed, capabilities_changed, server_info_changed, tool_extras_changed, card_extras_changed), outside every public change feed and count; the first reading creates none. New aggregate blocks only: declared_extras in the weekly snapshot and the summary (counts from 1 to 4 shown as <5). Scope addition: A2A cards the weekly standards probes find on a host that is not yet a card target join the census as source and cohort standards_probe, kept apart like requested hosts (outside the panel, the quarterly sample, the weekly coverage figure, the standards probes and every per-target public view). No earlier row or published figure is read differently, so no overlap is needed." } ], "change_kind": "initial", "fingerprint": "b3e23ba4b9960349f05a83853d656578a433e36cd87dfd28c00c980b01241e4c", "from_basis": "commit c8a6fd6 (capability record, 2026-10 reference panel)", "from_utc": "2026-10-01T16:30:26Z", "version": "v1", "what_changed": "First version: tool names, description hashes and excerpts, input-schema hashes, behaviour hints, A2A card fields, change classes per change-log entry." }, { "additions": [ { "basis": "founder decision #25 (PAYMENT_OR_SPEND, 3 Oct 2026 11:35, to-do observatory-discovery-lists-budget)", "date_utc": "2026-10-03T12:00:00Z", "what": "Request and field addition, not a change to existing rows, hashes or figures: during a scheduled MCP observation GAIP now sends one resources/list (first page only) when the server's initialize result declares the resources capability, after tools/list, in the same session and before the session DELETE, on the same budget, timeouts, size limits and safe_fetch rules. Never resources/read, resources/templates/list or prompts/list; on-demand captures do not send it. What it lists is kept only in declared extras (a fifth section mcp_resources under the unchanged schema gaip.declared-extras.v1, own SHA-256): per resource the URI scheme, ui:// URIs (MCP Apps) in full up to 300 characters, every other URI as host plus SHA-256, media type, bounded and redacted name and title, the description as SHA-256 plus an excerpt of at most 300 characters and its length, _meta key names only and the declared size; 8 KiB cap with a truncated flag and count_read; list_complete false when a next page exists. A failed read is a read status on the latest declared-extras row only: the observation's state, manifest, manifest_sha256 and capability record are built exactly as before, and the four earlier declared-extras sections keep their hashes. A later change of the list appends one internal DECLARED_EXTRAS_CHANGED entry with class resources_changed, outside every public change feed and count; the first reading creates none. New aggregate keys only: declared_extras.mcp_servers.resources (servers declaring resources, servers whose list was read, servers listing a ui:// resource, servers listing an mcp-app media type, read failures; small-cell rule). Published planning figures change once: the plan counts the extra request as a constant +1 a visit (an upper bound: 6 requests a daily visit, 5 a census visit), so the summary's revisit plan, tier capacities, target ceiling and reach.budget.capacity are lower at the same budget; the hourly budget rises from 1,800 (decision #19) to 2,250 (decision #25), at which the census holds about 42,800 a week beside 1,200 daily targets (32,256 at 1,800 with the extra request). No earlier row or published observation figure is read differently, so no overlap is needed." }, { "basis": "data4 efficiency leftovers (branch claude/data4-efficiency-leftovers)", "date_utc": "2026-10-03T16:00:00Z", "what": "Scheduling change for requested hosts only (source 'requested', method doc section 7a), not a change to rows, hashes or how any row is read: a requested host whose /.well-known/agent-card.json has not given a card in 3 census visits in a row is next visited 90 days later (or later, if the census plan or the reachability back-off says so) instead of on the census cadence; the count is kept on the target as no_card_streak and dropped as soon as a card has been read, and a visit the operator held (excluded or paused host) neither counts nor resets it. Requested hosts are outside the 2026-10 panel, the quarterly sample, the weekly coverage figure, the standards probes and every per-target view, so none of those moves. Effect on published figures: the census tier's coverage windows (observed in the last 7, 14 and 30 days), which include requested targets, count such hosts as not visited in those windows, and requested_hosts_not_counted.observed_last_7_days beside the weekly coverage figure falls by them. No earlier row is read differently, so no overlap is needed." }, { "basis": "founder requests data5-data-appetite, data5-dependency-hosts, data5-operator-key (branch claude/data5-declared-derivations)", "date_utc": "2026-10-03T20:00:00Z", "what": "Derived-record addition, not a change to existing rows, hashes or figures, and no new request: from the documents each observation already holds and the registry facts already kept on the target, a separate versioned record gaip.declared-derivations.v1 (side tables gaip_observatory_declared_derivations_v1 and _latest_v1, never the manifest, the targets document or the declared-extras sections, so manifest_sha256, every capability-record hash and every declared-extras hash are computed exactly as before) keeps per target: data appetite (every inputSchema property, nested to 4 levels, classified by the documented keyword rules gaip.data-appetite-rules.v1 into payment_card, bank_account, email, phone, postal_address, precise_location, government_id, health, date_of_birth, credentials, biometric, free_text_large and file_upload; per category the number of fields and of tools asking for it, never a property name or text), dependency hosts (registrable domains of the URLs the declarations point to, by role: auth, provider, interface, documentation, website, remote, repository; the agent's own domain left out; host names only, per source section) and the operator key (SHA-256 of the normalised declared operator: registry namespace, else card provider organisation, else provider URL domain, with the kind of source; the value itself is not kept). A category, dependency or operator that moves appends an internal DECLARED_DERIVATIONS_CHANGED entry to the change log (classes derivations_changed, data_category_added, data_category_removed, dependency_added, dependency_removed, operator_changed), outside every public change feed and count; the first reading creates none. New aggregate blocks only: declared_derivations in the weekly snapshot and the summary (share of agents asking for each category; agents pointing to outside domains and well-known service domains shared by at least 5 agents; operator concentration with no operator named; small-cell rule), beside the Agent Web Index's figures and on /state-of-agents. The per-target view to an asker adds the record (the operator as the agent publishes it, never its hash). No earlier row or published figure is read differently, so no overlap is needed." }, { "basis": "Global A2A Registry operator's written permission of 5 Oct 2026 (founder to-do ask-for-data-access; agent_economy/operations/data_permissions/a2a_registry_org_2026_10_05.md; branch claude/a2a-registry-org-source)", "date_utc": "2026-10-05T12:00:00Z", "what": "Scope addition, not a change to existing rows, hashes or figures: a new source, the Global A2A Registry (a2a-registry.org, source 'global-a2a-registry'). Its public agent list (GET https://api.a2a-registry.org/public/agents?page=N, page the only parameter; 409 agents on 5 Oct 2026) is read at most once a week (a setting may shorten it to daily, never more often), one page at a time, and each public A2A entry's agent card URL (manifestUrl) becomes an A2A_CARD target read on the agent's own host on the same rules and cadence as every card target. A card already held from another source stays one target (de-duplicated by card URL); one both A2A registries list keeps the source that admitted it. The new targets enter the census, the watched-agents list (source_registry global-a2a-registry, identifier the card URL) and the weekly coverage figure like a2aregistry.org's; they are not in the 2026-10 reference panel. No earlier row or published figure is read differently, so no overlap is needed." } ], "change_kind": "scope", "fingerprint": "b3e23ba4b9960349f05a83853d656578a433e36cd87dfd28c00c980b01241e4c", "fixes_defect": { "affects": [ "weekly changes report counts", "change_counts in weekly snapshots" ], "detail": "Until this version the report read every observation since 26 Sep 2026 in one query that stopped at 20,000 rows sorted by target id (reached within about three days at 6,000-7,500 observations a day; targets late in id order then dropped out of targets_observed and the reachability and protocol figures, more each day), and every public change entry since launch oldest first stopping at 5,000 (newer weeks lost manifest changes once the log held more). The rows themselves were complete; only the report undercounted. No weekly snapshot had been frozen with the defect (the first freeze, 2026-W40, is due on 5 Oct 2026).", "excluded_from_published_shares": false, "flag": "known defect: weekly changes report row caps", "id": "weekly_report_row_caps" }, "from_basis": "founder request data4-weekly-views (decision #24), branch claude/data4-weekly-views; the stored weekly changes report is rebuilt once from the record at the first Observatory cycle after deploy", "from_utc": "2026-10-02T22:10:00Z", "overlap": null, "overlap_note": "Not run: a defect fix before the overlap rule's start (3 Oct 2026); running the capped method beside it would only republish counts known to be too low.", "version": "v2", "what_changed": "Defect fix in how the weekly changes report (/v1/free/changes/weekly.json, /changes, and the change_counts frozen into each weekly snapshot) reads the record; the stored rows are unchanged. Each ISO week is now read on its own: that week's observations (only the fields the report uses, in pages with no silent cap) and that week's public change-log entries, each target's state before the week carried from the week before. A safety limit far above real volume (5,000,000 observations or 1,000,000 change entries a week) is reported as truncated: true on the week instead of dropping rows silently. Published counts of targets observed, reachability changes, protocol-version shifts and manifest changes go up for weeks the old caps cut. Finished weeks are frozen: a week's report is final once built an hour after the week ends, and only the current week is rebuilt each cycle; a frozen week's named facts about a target withdrawn later are still removed at the next cycle." }, { "change_kind": "field", "fingerprint": "b3e23ba4b9960349f05a83853d656578a433e36cd87dfd28c00c980b01241e4c", "from_basis": "founder request data4-robots-reachability (decision #24); in force from the end of the side-by-side week (agent_economy/gaip/observatory_reachability.py CUTOVER_UTC)", "from_utc": "2026-10-12T00:00:00Z", "overlap": { "compared": "Every observation refused at robots.txt in ISO week 2026-W41 keeps the old state (ROBOTS_DISALLOWED) and carries the new reading beside it (robots_txt_unreachable: NOT_REACHABLE or REFUSED_NOT_PUBLIC_HTTPS), so the status distribution and panel statuses under both methods can be computed for the same week; no back-off runs in that week.", "cycles": 1, "old_and_new_side_by_side_from": "2026-10-05T00:00:00Z", "until": "2026-10-12T00:00:00Z" }, "version": "v3", "what_changed": "A target whose host did not answer when GAIP read its robots.txt (DNS failure, refused connection, timeout, non-public address) is recorded as NOT_REACHABLE (or REFUSED_NOT_PUBLIC_HTTPS) with reachable false (panel status UNREACHABLE), no longer as ROBOTS_DISALLOWED with reachable empty; GAIP still sends that host nothing further in that observation. ROBOTS_DISALLOWED now means robots.txt was read (or answered 401, 403, 5xx or a redirect) and refused GAIP. After 3 observations in a row that found the host unreachable, a census target's next visit moves out to 7 days, then 30 days while it stays unreachable; a daily-tier target keeps its tier and backs off to at most 72 hours; GAIP's own control targets never back off. Effect on published figures: the status distribution's ROBOTS_DISALLOWED count falls and NOT_REACHABLE rises by the dead hosts; the panel's NOT_OBSERVED_BY_RULE count falls and UNREACHABLE rises; dead hosts are revisited less often, so census coverage windows may count fewer of them as visited." }, { "change_kind": "field", "fingerprint": "b3e23ba4b9960349f05a83853d656578a433e36cd87dfd28c00c980b01241e4c", "from_basis": "founder decision #35 (7 Oct 2026, legal call legal-capability-schema-structure: store input-schema structure, not description strings); in force from the end of the side-by-side week ISO 2026-W42", "from_utc": "2026-10-19T00:00:00Z", "overlap": { "compared": "In ISO week 2026-W42 the structure is stored on every new record while published schema_changed counts still come from input_schema_sha256 alone; for each tool whose schema hash changed that week, the structure is checked to differ too (a hash change with an identical structure means only strings changed), and the counts under both readings are compared before any figure uses the structure.", "cycles": 1, "old_and_new_side_by_side_from": "2026-10-12T00:00:00Z", "until": "2026-10-19T00:00:00Z" }, "version": "v4", "what_changed": "Each stored MCP tool entry gains input_schema_structure (capability_record.schema_structure): parameter names, JSON types, required flags and enum counts, nested at most 3 levels and 64 parameters a level; never a description, title, example, default or enum value. It is derived from the schema whose SHA-256 the record already keeps, so it sits outside record_sha256 and the entry hash (DERIVED_ENTRY_KEYS): no record hash moves and no change is logged because of it. A schema larger than 8 KiB, kept as a hash only, has structure null (unknown, not empty). No published figure uses the field before this version." } ] }, "client_behaviour": { "cycle": "iso_week", "fingerprint_inputs": [ "agent_client_behaviour:METHOD", "robots_rfc9309:MATCHER_VERSION" ], "title": "Client behaviour at GAIP's own surfaces, by client family", "versions": [ { "change_kind": "initial", "fingerprint": null, "from_basis": "commit 13b6e28 (four long-run datasets)", "from_utc": "2026-10-01T20:49:42Z", "known_defects": [ { "affects": [ "requests_to_disallowed_paths", "disallowed_after_reading_robots_txt" ], "detail": "A missing import meant requests to paths GAIP's robots.txt disallows were never counted (always 0) before PR #784.", "excluded_from_published_shares": false, "flag": "known defect: disallowed-path counter not counted", "id": "disallowed_path_counter_missing" } ], "version": "v1", "what_changed": "First version: requests, sessions, robots.txt read first, Retry-After after 429, payment header after 402, MCP protocol versions, retry storms." }, { "change_kind": "parser", "fingerprint": "f0ee38580be31e7be9e473c36e017d3fa03f9d23e92c894bb065aba7d197a01a", "from_basis": "founder-set cutover for PR #784 (merged 13:05 UTC)", "from_utc": "2026-10-02T14:00:00Z", "overlap": null, "overlap_note": "Not run: this change predates the overlap rule and fixed a defect.", "version": "v2", "what_changed": "GAIP's own robots.txt matched with the RFC 9309 matcher, and the disallowed-path counters counted (discovery_surfaces.robots_txt)." } ] }, "collection_ledger": { "cycle": "day", "fingerprint_inputs": [ "collection_ledger:SCHEMA_VERSION", "collection_ledger:METHOD" ], "title": "Collection ledger: GAIP's own daily record of when the Observatory ran, what it collected and its gaps", "versions": [ { "change_kind": "initial", "fingerprint": "3ce91c1abe5ade4bc7932aa9ea518bee0001e2b65194576961c5c5497ba2999e", "from_basis": "founder decision 4 Oct 2026, 09:08 ('I don't want to lose any history that has value or future value'), founder request noloss-collection-ledger (branch claude/noloss-collection-ledger); the first row is the UTC day of the first Observatory tick after the deploy (that day is marked ledger_first_day and never counted as fully covered); earlier gaps are in agent_economy/config/known_gaps_v1.json", "from_utc": "2026-10-04T09:08:00Z", "version": "v1", "what_changed": "First version: one row per UTC day, added to by every Observatory tick (ticks, error ticks by class, targets claimed/observed/deferred, requests and hourly budget, collector counts and states, stored view states, 15-minute windows with a tick, covered seconds, gaps over 2 x 900 seconds); each ended day frozen once and SHA-256 chained; known-gaps register." } ] }, "erc8004_cards": { "cycle": "iso_week", "fingerprint_inputs": [ "erc8004_card_archive:METHOD" ], "title": "ERC-8004 card archive (off until switched on)", "versions": [ { "change_kind": "initial", "fingerprint": "889c553ffed4979224e584a697b30942992f40b8fcec5a03bcbe4757fba7a7aa", "from_basis": "commit 086842e (decision #20); no rows until GAIP_ERC8004_ARCHIVE=1", "from_utc": "2026-10-02T01:56:25Z", "version": "v1", "what_changed": "First version: per ERC-8004 agent and ISO week, agentURI kind, host and hash, the registration file's hash and a short excerpt, declared services; never an owner or wallet address." } ] }, "outcome_ledger": { "cycle": "month", "fingerprint_inputs": [ "outcome_ledger:SCHEMA_VERSION" ], "title": "Outcome ledger: monthly counts of what happened in agent work GAIP saw", "versions": [ { "change_kind": "initial", "fingerprint": "102326364539cfa6a569e0f221664b47bb19b0aae26d07c0fd53371b1f19c3ee", "from_basis": "commit f2ec336 (outcome ledger); the first row covers October 2026", "from_utc": "2026-10-01T18:36:12Z", "version": "v1", "what_changed": "First version: one frozen, chained row per ended month for outside callers (delivery verdicts, dispute bundles, SLA breaches, receipts, failed protocol calls)." } ] }, "outcome_research_ledger": { "cycle": "month", "fingerprint_inputs": [ "outcome_research_ledger:SCHEMA_VERSION" ], "title": "Research ledger: monthly failure types and research-consented outcomes", "versions": [ { "change_kind": "initial", "fingerprint": "5201414637c02deedfb64a6bdd3ea34e35921d2abcd5ac1a4ac9d4210ac77b6b", "from_basis": "commit 086842e (decision #20); the first row covers October 2026", "from_utc": "2026-10-02T01:56:25Z", "version": "v1", "what_changed": "First version: front-door diagnoses by class and protocol, and Witness delivery checks by callers who passed research_consent: true, cells under 5 folded." } ] }, "package_history": { "cycle": "iso_week", "fingerprint_inputs": [ "package_history:METHOD" ], "title": "MCP server package history (npm and PyPI, with OSV advisories)", "versions": [ { "change_kind": "initial", "fingerprint": "0631f0f825b214538a320946b211dfec077117c7777d0ee622de0b68b43de868", "from_basis": "the data5-package-history branch (founder requests data5-package-history and data5-advisories); rows from the first Observatory tick after the deploy", "from_utc": "2026-10-03T16:15:39Z", "version": "v1", "what_changed": "First version: per MCP server package and ISO week, npm and PyPI version facts with publish times, versions added, removed or yanked, install scripts, provenance, maintainer count and a hash of the maintainer set (never names), npm weekly downloads, repository host, and OSV advisories with fixed versions and days to the first fix." } ] }, "reliability_index": { "cycle": "month", "fingerprint_inputs": [ "reliability_index:SCHEMA_VERSION", "reliability_index:METHOD" ], "title": "Agent Reliability Index: monthly reachability and response times of public agents", "versions": [ { "change_kind": "initial", "fingerprint": "8b68cd8b91879916fb410070fe3779c35e7bae28f458c95295657ca322a065a7", "from_basis": "founder request data5-reliability-index (branch claude/data5-reliability-warnings); a new published figure computed from status observations GAIP already stored (reachable, http_status, latency_ms on every visit), so no earlier row is read differently; the first final month is October 2026 (window 5 Oct - 2 Nov 2026)", "from_utc": "2026-10-03T16:30:00Z", "version": "v1", "what_changed": "First version: per month (frozen once its window ends) and for the latest 30 days (rebuilt hourly), by protocol (MCP, A2A, both) and cohort (all public agents, AGENTS-COHORT-2026-10): share of visits reachable, share of agents reachable on at least 95% of their visits, p50 and p95 response time of successful handshakes (HTTP 2xx), nearest rank; cells under 5 agents withheld." } ] }, "shop_panel": { "cycle": "iso_week", "fingerprint_inputs": [ "shop_panel:SCHEMA_VERSION", "shop_panel:METHOD", "robots_rfc9309:MATCHER_VERSION" ], "title": "Agentic-commerce shop panel", "versions": [ { "change_kind": "initial", "fingerprint": null, "from_basis": "commit f2ec336 (outcome ledger and shops-2026-10 panel)", "from_utc": "2026-10-01T18:36:12Z", "known_defects": [ { "affects": [ "robots.ai_bots", "robots.wildcard_home_allowed" ], "detail": "urllib.robotparser ignores * and $ in rules and applies the first matching rule instead of the longest, so the parsed robots facts may be wrong for sites using such rules. The file fingerprint (robots.f) and the other facts are not affected.", "excluded_from_published_shares": true, "flag": "known defect: wildcard robots rules", "id": "wildcard_robots_rules" } ], "version": "v1", "what_changed": "First version: weekly light checks (robots.txt AI-crawler rules for the home page and a fingerprint, discovery documents) and full checks of 2 fixed products. robots.txt was parsed with Python's urllib.robotparser." }, { "change_kind": "parser", "fingerprint": "484ff39c1e6583a98a2f6c16c9dca70325a591cb44e358bfbdc6030d680669e8", "from_basis": "founder-set cutover for PR #784 (merged 13:05 UTC); rows observed before 14:00 UTC are read as v1", "from_utc": "2026-10-02T14:00:00Z", "overlap": null, "overlap_note": "Not run: this change predates the overlap rule and fixed a defect; pre-fix robots facts are flagged and left out of published shares instead.", "version": "v2", "what_changed": "robots.txt parsed by GAIP's RFC 9309 matcher (agent_economy/gaip/robots_rfc9309.py: * and $, longest match, exact product tokens). The first change entry per shop whose robots.txt bytes did not change but whose parsed rules did is labelled method_change:robots_rfc9309." }, { "change_kind": "field", "fingerprint": "05b7de945ee837f6bb61cfa87c0ddcc2c140ecd670f021ca604061c6fda1f237", "from_basis": "founder decision #35 (7 Oct 2026, legal call legal-ucp-shop-endpoints: keep shops' public UCP profile contents; read the tool list on at most 50 shops a week); in force from the end of the side-by-side week ISO 2026-W42", "from_utc": "2026-10-19T00:00:00Z", "overlap": { "compared": "In ISO week 2026-W42 both fields are recorded on every read while all published figures stay on v2; the week's shares of shops whose profile declares the UCP catalog lookup (v2: lookup_advertised from documents.ucp) are compared with the same share computed from ucp_contents.capabilities, and must match before any figure uses the new fields.", "cycles": 1, "old_and_new_side_by_side_from": "2026-10-12T00:00:00Z", "until": "2026-10-19T00:00:00Z" }, "version": "v3", "what_changed": "Two new fields beside each weekly fact set, outside the hashed facts (shop_panel.UNPUBLISHED_FACTS), so no v2 value, change entry or published share moves. (1) ucp_contents: what the shop's /.well-known/ucp profile declares, read from the same request as before (no extra read): version, capability names (at most 40), and per service binding the transport and whether its endpoint is on the shop's own site (ON_SITE, NOT_ALLOWED or NONE); never a URL path or personal data. (2) ucp_tools: on at most 50 panel shops a week across all runs (shop_panel.ToolSample, kept in the panel's meta table), one tools/list request to the shop's declared on-site UCP MCP endpoint with the UCP-Agent header naming GAIP's own profile and no credentials, never tools/call; it keeps the HTTP status, the tool count and, for at most 64 tools, each name and its input-schema structure (capability_record.schema_structure: parameter names, types, required flags and enum counts; never descriptions or enum values). A 429 or 503 stops tool reads for the rest of the week. Read only when GAIP_UCP_AGENT_PROFILE is set (an honest UCP-Agent header); off with GAIP_SHOP_PANEL_UCP_TOOLS=0." } ] } }, "from_basis_note": "Unless stated otherwise, a version's from_utc is the time (UTC) of the commit that added the code; rows were first written after the next deploy.", "license": { "attribution": "Credit the GAIP Agent Observatory and give the dataset or snapshot id (see citation).", "covers": "Aggregate figures only (counts and shares; no per-agent or per-shop result), founder decision #13(c), 1 Oct 2026. It does not cover per-agent or per-shop results (for example the change-log entries), receipts, GAIP's software, its name or logo; GAIP's terms of use apply to those.", "data_page": "https://www.gaipagents.com/data", "id": "CC-BY-4.0", "name": "Creative Commons Attribution 4.0 International", "terms": "https://www.gaipagents.com/terms#reuse-licence", "url": "https://creativecommons.org/licenses/by/4.0/" }, "meaning": "For each long-run dataset GAIP keeps: each method version, when it took effect, what changed and any known defect of an earlier version. Rows are never rewritten or deleted when a method changes; a row is read under the version in force when it was observed, and rows with a defect marked excluded_from_published_shares are left out of published figures. Facts observed at the stated times; not an assessment of any party.", "method_doc": "https://github.com/tjcgraham-rgb/-gaip-agent-to-art-prototype/blob/main/agent_economy/docs/observatory-method-v1.md#2g-method-versions-and-the-overlap-rule-since-2-october-2026", "overlap_rule": { "applies_to_changes_from_utc": "2026-10-03T00:00:00Z", "enforced_by": "agent_economy/gaip/dataset_methods.py (validate, stale_fingerprints) and agent_economy/tests/test_data2_method_changelog_v1.py", "rule": "A parser, field or scope change runs the old and the new method side by side for at least one collection cycle (an ISO week for weekly datasets, a month for monthly ledgers) before the new values are published. The new version declares that overlap (dates, number of cycles, what was compared) here, and its from_utc is not before the overlap ends. Wording-only changes need no overlap." }, "schema_version": "gaip.dataset-methods.v1", "title": "GAIP dataset methods: versions and changes", "url": "/v1/free/observatory/methods" }