Concurrent tracing sessions

Perfetto supports multiple concurrent tracing sessions. Sessions are isolated from each other each and each session can choose a different mix of producers and data sources in its config and, in general, it will only receive events specified by that config. This is a powerful mechanism which allows great flexibility when collecting traces from the lab or field. However there are a few caveats to bear in mind with concurrent tracing sessions:

1. Some data sources do not support concurrent sessions
2. Some settings are per session while others are per producer
3. Due to the way atrace works works if a session requests any atrace category or app it receives all atrace events enabled on the device
4. Various limits apply

Some data sources do not support concurrent sessions

Whilst most data sources implemented with the Perfetto SDK as well as most data sources provided by the Perfetto team, do support concurrent tracing sessions some do not. This can be due to:

• Hardware or driver constraints
• Difficulty of implementing the config muxing
• Perfetto SDK: users may opt-out of multiple session support

Known to work

• traced_probes data sources (linux.ftrace, linux.process_stats, linux.sys_stats, linux.system_info, etc)

Known to work with caveats

• heapprofd supports multiple sessions but each process can only be in one session.
• traced_perf in general supports multiple sessions but the kernel has a limit on counters so may reject a config.

Known not to work

• traced metatracing

Some settings are per session while others are per producer

Most buffer sizes and timings specified in the config are per session. For example the buffer sizes.

However some parameters configure per-producer settings: for example the size and layout of the shmem buffer between the producer and traced. While that is general data source setting the same can apply to data source specific settings. For example the ftrace kernel buffer size and drain period are settings that have to be shared between all users of traced_probes.

Bear in mind that

• Some resources like the shmem buffers are shared by all sessions
• As suggested by the comments in linked code above some settings are best treated as 'hints' since another config may have already set them before you get a chance to.

Atrace

Atrace is an Android specific mechanism for doing userland instrumentation and the only available tracing method prior to the introduction of the Perfetto SDK into Android. It still powers os.Trace (as used by platform and application Java code) and ATRACE_* (as used by platform C++).

Atrace (both prior to Perfetto and via Perfetto) works as follows:

• Configuration:
  • Users choose zero or more 'categories' from a hardcoded list
  • Users choose zero or more package names including globs
• This sets:
  • Some kernel ftrace events
  • A system property bitmask (for the atrace categories)
  • A system property for each package.
• When the Java or C++ tracing APIs are called we examine the system props.
• If the relevant category or package is enabled we write the event to trace_marker

As mentioned, each category may enable a number of kernel ftrace events. For example the 'sched' atrace category enables the sched/sched_switch ftrace event. Kernel ftrace events do not suffer from the current session issues so will not be described further.

For the userland instrumentation:

• Perfetto ensures the union of all atrace packages categories are installed
• However since:
  • the atrace system properties are global
  • we cannot tell which event comes from which category/package Every session that requests any atrace event gets all enabled atrace events.

Various limits

• Perfetto SDK: Max 8 datasource instances per datasource type per producer
• traced: Limit of 15 concurrent sessions
• traced: Limit of 5 (10 for statsd) concurrent sessions per UID