Trace Processor Architecture

This document explains how Perfetto's trace processor works, from ingesting raw trace files to providing SQL-queryable data. It covers the key components, data flow, and architectural patterns that enable the trace processor to handle traces from various formats (Proto, JSON, Systrace, etc.) and transform them into a unified analytical database.

Overview

The trace processor is a system that ingests trace files of various formats, parses their contents, sorts events by timestamp, and stores the data in a columnar SQL database for analysis. It processes traces in chunks to efficiently handle large files.

Core Data Pipeline

Raw Trace → ForwardingTraceParser → Format-Specific ChunkedTraceReader →
TraceSorter → TraceStorage → SQL Query Engine

Format Detection and Delegation

ForwardingTraceParser (src/trace_processor/forwarding_trace_parser.cc:95-134)

Detects trace format using GuessTraceType() from first bytes
Creates appropriate reader via TraceReaderRegistry (src/trace_processor/trace_reader_registry.h)
All readers implement ChunkedTraceReader interface (src/trace_processor/importers/common/chunked_trace_reader.h)

Format Registration (src/trace_processor/trace_processor_impl.cc:475-519)

context()->reader_registry->RegisterTraceReader<JsonTraceTokenizer>(kJsonTraceType);
context()->reader_registry->RegisterTraceReader<ProtoTraceReader>(kProtoTraceType);
context()->reader_registry->RegisterTraceReader<SystraceTraceParser>(kSystraceTraceType);

Format-Specific Readers (Diverse Approaches)

1. JSON Traces

JsonTraceTokenizer (src/trace_processor/importers/json/json_trace_tokenizer.h:73)

Data Flow: Raw JSON → Tokenizer → JsonEvent objects → TraceSorter::Stream
Parser: JsonTraceParser processes sorted events → TraceStorage
Architecture: Tokenizer/Parser split with JSON-specific state machine

2. Proto Traces (Complex Modular System)

ProtoTraceReader (src/trace_processor/importers/proto/proto_trace_reader.h:58)

Data Flow: Proto bytes → ProtoTraceTokenizer → ProtoImporterModules → TraceSorter::Stream
Modules: Register for specific packet field IDs (src/trace_processor/importers/proto/proto_importer_module.h:110)
- Tokenization phase: Early processing before sorting
- Parsing phase: Post-sorting detailed processing
Examples: FtraceModule, TrackEventModule, AndroidModule (many files in src/trace_processor/importers/proto/)

3. Systrace (Line-Based Processing)

SystraceTraceParser (src/trace_processor/importers/systrace/systrace_trace_parser.h:34)

Data Flow: Text lines → SystraceLineTokenizer → SystraceLine objects → TraceSorter::Stream
Architecture: State machine for HTML + trace data sections

4. Other Formats

Perf: perf_importer::PerfDataTokenizer (binary perf.data format)
Gecko: gecko_importer::GeckoTraceTokenizer (Firefox traces)
Fuchsia: FuchsiaTraceTokenizer (Fuchsia kernel traces)

Event Sorting and Processing

TraceSorter (src/trace_processor/sorter/trace_sorter.h:43)

Purpose: Multi-stream timestamp-based merge sorting
Architecture: Per-CPU queues for ftrace, windowed sorting for streaming
Streams: Each format creates typed streams (JsonEvent, TracePacketData, SystraceLine, etc.)
Output: Sorted events to format-specific parsers

Storage Layer

TraceStorage (src/trace_processor/storage/trace_storage.h)

Architecture: Columnar storage with specialized table types
Tables: SliceTable, ProcessTable, ThreadTable, CounterTable, etc.
Access: Direct insertion by parsers, SQL queries by engine

Context and Coordination

TraceProcessorContext (src/trace_processor/types/trace_processor_context.h)

Multi-level state management:
- Global state (shared across machines)
- Per-trace state (specific to each trace file)
- Per-machine state (unique to each machine)
- Per-trace-and-machine state (most specific)
Coordination: Central access point for storage, sorter, trackers

Key Architectural Patterns

1. ChunkedTraceReader Interface

All format readers implement same interface but with completely different internal architectures:

JSON: Incremental JSON parsing with state machine
Proto: Modular packet processing with field-based routing
Systrace: Line-by-line text processing
Archives (ZIP/TAR): Container formats that extract and delegate

2. TraceSorter::Stream Pattern

Each format defines its own event types and creates typed streams:

Stream<JsonEvent> for JSON traces
Stream<TracePacketData> for proto events
Stream<SystraceLine> for systrace lines

3. Parser vs Tokenizer Split

Tokenizer: Pre-sorting processing, fast timestamp extraction
Parser: Post-sorting detailed processing into storage
Not all formats use this split (depends on complexity)

File Path Reference

Core Infrastructure:

src/trace_processor/forwarding_trace_parser.{h,cc} - Format detection and delegation
src/trace_processor/trace_reader_registry.{h,cc} - Reader registration
src/trace_processor/sorter/trace_sorter.h - Event sorting
src/trace_processor/storage/trace_storage.h - Columnar storage

Format Readers (examples):

src/trace_processor/importers/json/json_trace_tokenizer.h - JSON processing
src/trace_processor/importers/proto/proto_trace_reader.h - Proto entry point
src/trace_processor/importers/proto/proto_importer_module.h - Proto module system
src/trace_processor/importers/systrace/systrace_trace_parser.h - Systrace processing

Registration:

src/trace_processor/trace_processor_impl.cc:475-519 - Where all readers are registered