gh-138122: Add differential flame graph#145785
Conversation
Differential flame graphs compare two profiling runs and highlight where performance has changed. This makes it easier to detect regressions introduced by code changes and to verify that optimizations have the intended effect. The visualization renders the current profile with frame widths representing current time consumption. Color is then applied to show the difference relative to the baseline profile: red gradients indicate regressions, while blue gradients indicate improvements. Some call paths may disappear entirely between profiles. These are referred to as elided stacks and occur when optimizations remove code paths or when certain branches stop executing. When elided stacks are present, an "Elided" toggle is displayed, allowing the user to switch between the main differential view and a view showing only the removed paths.
ivonastojanovic
changed the title
|
I’m a bit stuck on what colors we should use for new vs deleted functions. Right now:
But I’m not sure if we should even treat them differently from other functions visually. From a perf perspective it’s kind of confusing:
So I’m not sure what the right visual semantics are here. Curious what you think 🙂 |
Perhaps use purple for both new and elided functions, with different shades to distinguish them: Bright/saturated purple for new functions (present in current, not in baseline): the current color, kept as-is and then muted/desaturated purple for elided functions? This unifies the two "out-of-band" categories under a single visual language that reads as "no direct comparison is available", keeping them distinct from the red/blue performance axis entirely. The legend becomes simpler too: instead of explaining four special cases, you just say "purple = this function has no counterpart in the other profile." |
Use four gradients consistently for both regressions and improvements. Also avoid calling getComputedStyle(document.documentElement) on every frame, cache the values like we do for the heatmap, since doing this per frame can be expensive for large profiles.
Since baseline and current self time are shown in ms, the diff should also be displayed in ms instead of samples.
We were determining the selected data and applying filtering/processing inside each toggle handler, which led to inconsistencies (e.g. thread filtering) and missed cases. This approach also won’t scale well as we add more toggles. Instead, introduce a single centralized function that returns the active flamegraph data and use it consistently for all updates and processing.
Clean up tests by extracting repeated logic (resolving function names and finding child nodes by name) into helper functions. Also add a test that doesn’t mock BinaryCollector to cover the full round trip.
Use purple (with gradients) for both removed and new functions to unify these “out-of-band” cases under a single visual language, meaning no direct comparison is available. This keeps them clearly separate from the red/blue performance axis and simplifies the legend: “purple = this function has no counterpart in the other profile.”
Opcodes from multiple call paths were silently dropped, only the first path's opcodes were kept. Now they're summed correctly when nodes merge.
| } | ||
|
|
||
| // Neutral zone: small percentage change | ||
| if (Math.abs(diff_pct) < 15) { |
There was a problem hiding this comment.
This may be a stupid question but do you plan to have the sections configurable? I can see some worth where people are only interested in improvements above a certain threshold and the notion of what is a deep / medium / light / negligible improvement is quite subjective.
For instance, if I really needed to estimate the cost of something, vs readability for instance, I would be happy to only catch 3x improvements for instance if the change is complex.
There was a problem hiding this comment.
Good question! Let's keep it as-is for this PR and explore making them configurable separately. An interactive slider in the HTML toolbar would probably be the most useful approach since you'd want to experiment without re-running the profiler.
There was a problem hiding this comment.
I like the idea as well, I can experiment with it in a separate PR
There was a problem hiding this comment.
Yes that's what I had in mind. I would suggest a slide bar with multiple markers that you can move left/right to define the regions of what improvements are. I found https://brechtdr.github.io/enhanced-range-slider-poc/ where it shows a rangegroup in Vite but since it's not a standard element nor something that is implemented natively, we'll need to create the pure HTML/CSS alternative. It would look like:
|
@ivonastojanovic Great job. I have pushed a small commit fixing some small stuff:
|
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
|
Amazing job @ivonastojanovic 🖤 |
…ktor * upstream/main: (119 commits) pythongh-144270: Make SubElement parent and tag positional-only (pythonGH-144845) pythongh-146558: JIT optimize dict access for objects with known hash (python#146559) pythongh-139922: always run MSVC 64-bit tail-calling CI (pythonGH-146570) pythongh-126835: Fix _PY_IS_SMALL_INT() macro (python#146631) pythongh-146587: fix type slot assignment incase of multiple slots for same name (python#146593) pythongh-138122: Add differential flame graph (python#145785) pythongh-146416: Emscripten: Improve standard stream handling in node_entry.mjs (python#146417) pythongh-146444: Don't package as part of iOS 'build hosts' target (python#146628) pythongh-138850: Add --disable-epoll to configure (pythonGH-145768) pythongh-146444: Make Platforms/Apple/ compatible with Python 3.9 (python#146624) pythongh-138577: Fix keyboard shortcuts in getpass with echo_char (python#141597) pythongh-146556: Fix infinite loop in annotationlib.get_annotations() on circular __wrapped__ (python#146557) pythongh-146579: _zstd: Fix decompression options dict error message (python#146577) pythongh-146083: Upgrade bundled Expat to 2.7.5 (python#146085) pythongh-146080: fix a crash in SNI callbacks when the SSL object is gone (python#146573) pythongh-146090: fix memory management of internal `sqlite3` callback contexts (python#146569) pythongh-145876: Do not mask KeyErrors raised during dictionary unpacking in call (pythonGH-146472) pythongh-146004: fix test_args_from_interpreter_flags on windows (python#146580) pythongh-139003: Use frozenset for module level attributes in _pyrepl.utils (python#139004) pythonGH-146527: Add more data to GC statistics and add it to PyDebugOffsets (python#146532) ...
3 participants
Differential flame graphs compare two profiling runs and highlight where performance has changed. This makes it easier to detect regressions introduced by code changes and to verify that optimizations have the intended effect.
The visualization renders the current profile with frame widths representing current time consumption. Color is then applied to show the difference relative to the baseline profile: red gradients indicate regressions, while blue gradients indicate improvements.
Some call paths may disappear entirely between profiles. These are referred to as elided stacks and occur when optimizations remove code paths or when certain branches stop executing. When elided stacks are present, an "Elided" toggle is displayed, allowing the user to switch between the main differential view and a view showing only the removed paths.
In the diff view,
slow_functionimproved (blue) andfast_functionregressed (red) compared to the baseline.medium_functionwas removed (shown in the elided stacks toggle) andnew_functionwas added (purple).Differential view
Elided view
CC: @pablogsal @lkollar
📚 Documentation preview 📚: https://cpython-previews--145785.org.readthedocs.build/