Profiler list ordering repair
=============================

Repository
----------

Work in:

  /home/johnny/projects/gldoze

Goal
----

Remove value-based (current milliseconds) reordering from the Profiler so rows
do not jump around as timings change. Use one of these policies:

1. Preferred: alphabetical, case-sensitive ordering by group/category and then
   function name.
2. Alternative: unsorted, stable registration/insertion order.

Do not alter timing collection, displayed millisecond values, filtering,
colouring, the profiler service ABI, or the table widget globally.

Important current-source finding
--------------------------------

There are several independent orderings. Do not assume every profiler list is
controlled by one sort.

The visible Functions / Scopes table is built here:

  apps/plugins/profiler/profiler_plugin.cpp
  DrawScopeTable(), currently around lines 510-584

At the time this handoff was written, that function already contains:

  std::sort(i->Scopes.begin(), i->Scopes.end(),
      [](const ScopeRow &a, const ScopeRow &b)
      {
          if(a.category!=b.category)return a.category<b.category;
          return a.name<b.name;
      });

That is alphabetical by category, then function name. It is not an ms-value
sort. `RefreshSnapshot()` collects rows through `visit_scopes()` and
`DrawScopeTable()` applies the final visible ordering immediately before
`table_set_rows()`.

The host service also returns scopes alphabetically here:

  apps/desktop/src/desktop_profiler_service.cpp
  ServiceTable.visit_scopes, currently around lines 115-151

Its comparator is `a.first < b.first`, where `first` is the scope name.

The remaining descending-ms sort is here:

  apps/desktop/src/desktop_profiler_service.cpp
  DesktopProfilerService::EndFrame(), currently around lines 503-514

It fills `Last.SortedScopes` and sorts with:

  return a.second > b.second;

Here `second` is `total_ms`. This vector is consumed by
`ServiceTable.visit_timeline()` around lines 257-273. Therefore this sort
controls the profiler timeline-bar visitation order, not the visible scope
table in the current source.

There is a second value sort in:

  apps/desktop/src/desktop_profiler_service.cpp
  ServiceTable.visit_categories, currently around lines 152-169

It sorts category totals descending with:

  return a.second.second > b.second.second;

Those categories are collected by the plugin and used in the CPU/category
summary. They do not currently determine the grouped function-table row order.

Before editing
--------------

1. Run:

     git status --short

   The worktree is intentionally broad and dirty. Preserve all unrelated
   changes. `apps/plugins/profiler/` and the desktop profiler service may be
   untracked in this checkout, so do not rely only on `git diff`.

2. Confirm which list the user sees moving:

   - Function / time-slice table: final order is in `DrawScopeTable()`.
   - Timeline bars: order comes from `Last.SortedScopes`.
   - Category summary: order comes from `visit_categories()`.
   - Plugin table: order comes from host `PluginOrder`; it has no local
     millisecond comparator.

3. Check the active executable before claiming the source is live:

     pgrep -a gldoze_desktop
     readlink /proc/<PID>/exe

   If `/proc/<PID>/exe` ends in `(deleted)`, the current process does not prove
   current desktop-service behavior. Also inspect the loaded Profiler plugin
   mapping:

     grep -E 'profiler.*\.so' /proc/<PID>/maps

   A deleted or old plugin mapping means the current `DrawScopeTable()` code is
   not live. Do not restart or relaunch the desktop without explicit user
   permission.

Implementation: alphabetical policy
-----------------------------------

Use this policy unless the user specifically requests unsorted order.

For the visible Functions / Scopes table, retain the existing comparator in
`DrawScopeTable()`:

  category ascending, then name ascending

This preserves contiguous categories, which is required because
`DrawScopeTable()` inserts a group header whenever `category` changes. Sorting
only by name would split or repeat group headings.

For the timeline, rename `Last.SortedScopes` to a neutral name such as
`Last.OrderedScopes` if the scope of the change permits it, then replace the
descending `total_ms` comparator in `EndFrame()` with:

  [](const auto &a, const auto &b)
  {
      return a.first < b.first;
  }

If minimizing churn is more important, keep the existing member name and only
change the comparator, but add a comment explaining that timeline order is
stable alphabetical order and must not follow live timing values.

For category summaries, replace the descending-total comparator in
`visit_categories()` with a name comparator:

  [](const auto &a, const auto &b)
  {
      return a.first < b.first;
  }

Do not sort numeric strings after formatting. Sort typed rows before calling
`table_set_rows()`.

Implementation: unsorted/stable policy
--------------------------------------

Do not simply delete every `std::sort`.

`ScopeByHash` and `CategoryTotals` are maps whose iteration order is an
implementation/container property, not the intended registration order.
`ScopeHashOrder` is the explicit scope registration-order vector and should be
the source of stable unsorted scope order.

For `visit_scopes()`, emit entries directly in `ScopeHashOrder` order and
remove its temporary alphabetical sort.

For `DrawScopeTable()`, stable unsorted rows still must remain grouped because
the renderer inserts category headings on category transitions. Build a
category-order list using first appearance, then emit each category's scopes
in their original visitor order. Do not use a duration value anywhere in this
ordering. A `std::stable_partition` loop or an explicit output vector is
acceptable.

For timeline bars, build the snapshot order from `ScopeHashOrder` rather than
iterating `ScopeByHash`. Removing the sort while retaining unordered/hash-map
iteration can produce unstable rows and does not satisfy the goal.

For categories, add and maintain an explicit first-seen category-order vector
if insertion order is required. Merely iterating `CategoryTotals` is
alphabetical today because it is a `std::map`, but that is alphabetical policy,
not registration policy.

Tests
-----

Update or add focused tests that use deliberately conflicting names and
durations, for example:

  Zebra  = 1.0 ms
  Alpha  = 9.0 ms
  Middle = 4.0 ms

For alphabetical policy, assert the visitor/table order is Alpha, Middle,
Zebra even though the ms order would be Alpha, Middle, Zebra only by chance;
also use a second sample such as Alpha=1, Middle=9, Zebra=4 so timing changes
cannot change row order.

For stable unsorted policy, register Zebra, Alpha, Middle and assert that exact
order across at least two frames with swapped timing values.

Cover independently:

- `visit_scopes()` ordering;
- `visit_timeline()` ordering;
- `visit_categories()` ordering if changed;
- the plugin's grouped Functions table ordering.

Retain assertions for the actual millisecond values to prove only presentation
order changed.

Build and verification
----------------------

Discover the exact configured target names first:

  cmake --build build-cef --target help | grep -E 'profiler|desktop'
  ctest --test-dir build-cef -N | grep -i profiler

Then perform the focused build. Expected targets in this checkout include:

  cmake --build build-cef --target gldoze_profiler profiler_plugin_test -j2

If desktop-profiler service code or its tests changed, also build the desktop:

  cmake --build build-cef --target gldoze_desktop -j2

Run the discovered profiler tests and basic source checks:

  ctest --test-dir build-cef -R 'profiler' --output-on-failure
  git diff --check

Runtime acceptance
------------------

Plugin-only changes can normally be picked up through Plugin Manager Reload.
Host service changes in `desktop_profiler_service.cpp` require a desktop
restart. Do neither without user authorization.

With authorization, verify the intended list for at least 20 seconds while
loads change:

- row positions remain fixed;
- names follow the selected alphabetical or registration policy;
- current/average/peak ms values continue updating;
- group headers occur once and remain adjacent to their functions;
- scrolling and selection do not jump to a different logical row;
- timeline and category order match the requested scope.

Final report
------------

State exactly which list(s) were changed: scope table, timeline, category
summary, or some combination. Cite the comparator/source used for each. Report
build and test commands with their results, and separate source/test success
from authorized live GUI verification.


CEF MemoryInfra SIGILL crash mitigation
========================================

Incident
--------

This GLDoze checkout had a host desktop crash on 2026-08-04 at about 17:40
NZST. The originating process was:

  /home/johnny/projects/gldoze/build-cef/gldoze_desktop --fullscreen

The desktop process received SIGILL. The crashing thread was named
`MemoryInfra`, and its stack was:

  memory_instrumentation::ClientProcessImpl::RequestChromeMemoryDump
  base::trace_event::MemoryDumpManager::CreateProcessDump
  base::trace_event::MemoryDumpManager::InvokeOnMemoryDump
  base::trace_event::MallocDumpProvider::OnMemoryDump

The kernel reported an intentional invalid-opcode trap (`ud2`) in `libcef.so`.
The pinned runtime was CEF 150.0.11 / Chromium 150.0.7871.115. The binary's
`MallocDumpProvider::OnMemoryDump` calls legacy `mallinfo()` and several signed
arithmetic/libc++ range checks converge on the same `ud2`. The exact branch was
not recoverable because systemd retained no core or registers.

Do not misdiagnose this as the NVIDIA profiler. A separate main-thread snapshot
happened to be inside the GLDoze GPU profiling scope, but the faulting thread
was CEF `MemoryInfra`. There was no NVIDIA Xid, OOM, or kernel GPU error. The
later SIGABRT burst was DrKonqi repeatedly failing to connect to display `:1`.

Why the periodic memory path matters
------------------------------------

This CEF binary contains and unconditionally installs
`ChromeBrowserMainExtraPartsMetrics` from `ChromeBrowserMainParts::Create()`.
Its `PostBrowserStart()` immediately calls `RecordMemoryMetricsAfterDelay()`.
That schedules `RecordMemoryMetrics()`, which creates a
`ProcessMemoryMetricsEmitter`, calls
`FetchAndEmitProcessMemoryMetrics()`, and schedules itself again. The resulting
global memory dump reaches the crashing `MallocDumpProvider` path.

The exact disassembly/source mapping observed in the CEF binary was:

  ChromeBrowserMainExtraPartsMetrics::PostBrowserStart()
    -> RecordMemoryMetricsAfterDelay()
  RecordMemoryMetrics()
    -> ProcessMemoryMetricsEmitter::FetchAndEmitProcessMemoryMetrics()
  libcef.so base/trace_event/malloc_dump_provider.cc

Do not add an invented `disable-features` name. Do not assume
`--disable-metrics` is sufficient: the metrics extra part is added before the
periodic memory task and the scheduling call is unconditional in this build.
Verify any switch against the exact CEF binary before treating it as a fix.

Preferred portable mitigation
-----------------------------

Do not overwrite or patch the cached `libcef.so` under
`~/.cache/gldoze-cef/`. That changes only one machine and can be replaced by
the next CEF fetch. Do not use LD_PRELOAD or allocator interposition.

For this exact CEF build, first test an app-level command-line mitigation in
`apps/desktop/src/browser_application.cpp`, inside
`GLDozeCefApp::OnBeforeCommandLineProcessing()` and only when
`ProcessType.empty()`:

  CommandLine->AppendSwitchWithValue(
      "test-memory-log-delay-in-minutes", "2147483647");

This is a Chromium testing switch, not a permanent upstream fix. In this
binary `RecordMemoryMetricsAfterDelay()` reads the switch, parses a positive
integer as minutes, and schedules the next dump that far in the future. The
maximum signed 32-bit value makes periodic memory dumps practically absent
during a desktop session while leaving ordinary CEF browsing, rendering, and
GLDoze's own profiler intact.

Before keeping this mitigation:

1. Confirm the exact CEF binary still contains the switch and the same path:

     strings -a build-cef/libcef.so | grep test-memory-log-delay-in-minutes
     addr2line -Cfipe build-cef/libcef.so \
       0x7834ac0 0x7835970 0x783ac50

   Addresses are build-specific; use `nm -anC` to find the current symbols.

2. Add a focused comment explaining that this suppresses the Chromium periodic
   process-memory dump which crashed in `MallocDumpProvider`, and keep the
   switch browser-process-only.

3. Build `gldoze_desktop`, verify the resulting command line through
   `/proc/<pid>/cmdline` after an explicitly authorized restart, and run a
   multi-hour soak. The default memory-log interval is stochastic, so a short
   startup check is not enough.

4. Check for new GLDoze SIGILL entries:

     coredumpctl list --since "<start time>" --no-pager
     journalctl -k --since "<start time>" --no-pager

   Separate any DrKonqi failures from the originating desktop process.

If the switch is rejected, ignored, or changes behavior on another CEF build,
do not keep it silently. Revert it and either find a supported equivalent in
that CEF/Chromium version or use the CEF-source mitigation below.

Durable fix path
----------------

The durable fix is to update CEF/Chromium to a build whose memory dump provider
uses safe 64-bit allocator statistics (`mallinfo2` or the corresponding modern
allocator provider), or to carry a documented upstream CEF patch that skips
periodic process-memory metrics for this embedded application. That requires
rebuilding and redistributing the CEF runtime; it is not an app-source-only
change.

If a CEF source tree is available on the laptop, inspect
`chrome/browser/metrics/chrome_browser_main_extra_parts_metrics.cc` and
`base/trace_event/malloc_dump_provider.cc`. A source patch should make the
periodic memory-metrics registration conditional for CEF, or replace the
legacy allocator arithmetic with checked 64-bit statistics. Keep the patch
isolated to the CEF build and record the source revision. Do not edit the
downloaded binary in place.

Verification rules on the laptop
---------------------------------

Before changing anything:

  git status --short
  pgrep -a gldoze_desktop
  readlink /proc/<PID>/exe
  grep -E 'libcef.so|gldoze.*\.so' /proc/<PID>/maps

Preserve unrelated dirty work. A deleted executable or plugin mapping means
the live desktop is stale; do not claim runtime success from a fresh build.
Do not restart or launch the GUI without explicit authorization.

For the 1920x1080 laptop desktop, use the normal fullscreen launch for live
acceptance unless a windowed launch is specifically needed for debugging.
Resolution is not implicated in this crash. Validate separately that browser
navigation, OSR rendering, input, GPU acceleration, downloads, and the native
GLDoze UI still work after the mitigation.

Report separately:

  - source/build evidence;
  - whether the mitigation switch is present in the live process;
  - coredump/journal evidence after the soak; and
  - authorized live GUI verification.
