Queue Forecasting Dashboard

Generated 2026-07-09 17:52:06Z · refreshes every 15m · Project Status · Predictions · Aggregations

Today (UTC)

181,431 pending 178,233 resolved 160,118 completed 7,853 failed

Hour (UTC)	Pending	Started	Resolved	Completed	Failed
00:00	10,544	10,465	12,279	11,608	390
01:00	7,949	8,054	6,092	5,585	249
02:00	8,806	8,607	8,215	7,291	429
03:00	6,994	6,925	7,130	6,603	364
04:00	2,972	3,325	6,095	5,470	409
05:00	3,935	3,910	3,660	3,217	310
06:00	9,059	7,716	5,231	4,089	353
07:00	7,631	8,494	10,828	10,237	301
08:00	9,710	7,721	8,457	6,274	226
09:00	7,500	7,452	8,409	7,672	270
10:00	13,420	12,984	10,475	9,899	474
11:00	8,654	7,741	11,200	9,940	423
12:00	14,314	13,765	9,960	9,398	455
13:00	9,097	9,514	13,041	11,730	765
14:00	10,637	10,094	9,718	8,306	423
15:00	19,549	16,130	13,925	12,913	741
16:00	23,680	22,569	20,576	18,869	686
17:00	6,980	8,130	12,942	11,017	585

Data Collection Health

Table Sizes

Table	Rows	Newest	Age
queue_forecast_daily_health	130	2026-07-09 16:44:17Z	1.1h ago
queue_forecast_run_predictions	9,687,938	2026-07-09 17:34:19Z	18m ago
queue_forecast_task_runs	12,872,171	2026-07-09 17:34:19Z	18m ago
queue_forecast_tasks	12,669,759	2026-07-09 17:33:51Z	18m ago
queue_forecast_worker_counts	7,646,468	2026-07-09 17:31:09Z	21m ago
queue_forecast_worker_pools	710	2026-07-09 08:05:35Z	9.8h ago

Stream Freshness

Stream	Latest	Age
pending_at	2026-07-09 17:34:17Z	18m ago	OK
started_at	2026-07-09 17:34:19Z	18m ago	OK
resolved_at	2026-07-09 17:34:19Z	18m ago	OK
enriched_at	2026-07-09 17:33:51Z	18m ago	OK
worker_sample	2026-07-09 17:31:09Z	21m ago	OK
daily_health	2026-07-09 16:44:17Z	1.1h ago	OK

Ingestion Windows

Window	Pending	Started	Resolved
5m	660	847	1,534
15m	2,691	3,097	5,168
1h	16,673	17,456	22,636
6h	87,687	83,221	83,403
24h	254,752	243,935	250,976

Open Issues

Metric	Value
daily_health_rows_last_7d	7
unenriched_tasks	10
unresolved_runs	9,727
unresolved_runs_older_than_2h	2,986
worker_samples_last_30m	6

Recent Resolutions (1h)

Reason	Count (1h)
completed	19,805
worker-shutdown	1,105
failed	875
canceled	495
deadline-exceeded	276
intermittent-task	28
claim-expired	20
internal-error	1

Training Results

Latest: trainer/data/models/2026-06-20/

trained 18.7d ago · as_of 2026-06-20 · lightgbm

train: 5,735,018 rows (2026-05-15..2026-06-14) · val: 99,781 · holdout: 1,194,466

	Model	Baseline	Delta
MAE	2.3m	2.4m	-3.7%
within-2x	91.7%	91.0%	—
p90 coverage	87.7%	—	—

Per-day holdout breakdown

Day	n	MAE	w/in 2x	p90 cov
2026-06-15	237,394	2.5m	91.0%	84.7%
2026-06-16	239,642	2.2m	92.0%	88.4%
2026-06-17	231,833	2.3m	92.3%	88.3%
2026-06-18	244,063	2.4m	91.6%	88.1%
2026-06-19	181,074	2.1m	91.7%	89.3%

	Model	Baseline	Delta
MAE	4.4m	4.7m	-4.9%
within-2x	57.4%	52.5%	—
p90 coverage	88.4%	—	—

Bucket	n	MAE	Base MAE	w/in 2x	Base w/in 2x
<1m	643,154	19.8s	25.7s	46.2%	41.8%
1-5m	264,473	1.2m	1.4m	74.3%	68.1%
5-30m	181,696	5.3m	6.1m	71.8%	66.6%
30m+	44,682	1.3h	1.3h	58.7%	55.5%

Day	n	MAE	w/in 2x	p90 cov
2026-06-15	237,394	6.2m	56.2%	86.6%
2026-06-16	239,641	4.4m	59.7%	90.4%
2026-06-17	231,833	3.9m	56.9%	88.9%
2026-06-18	244,063	4.8m	56.0%	88.8%
2026-06-19	181,074	2.4m	58.2%	86.7%

Walk-Forward Evaluation

Cohort	Baseline MAE	Model MAE	Δ MAE %	Model within-2x	Δ within-2x pp	p90 cov	Hold rows
2026-05-06	2.3m	2.0m	-11.6%	91.8%	+1.1pp	90.6%	879,412
2026-05-07	2.3m	2.0m	-12.1%	91.9%	+1.2pp	90.3%	973,439
2026-05-08	2.2m	2.0m	-11.6%	91.9%	+1.0pp	90.5%	1,102,243
2026-05-09	2.3m	2.1m	-8.4%	91.5%	+1.0pp	90.1%	1,240,472
2026-05-10	2.2m	2.0m	-9.4%	91.8%	+0.8pp	89.6%	1,125,642
2026-05-11	2.1m	1.9m	-8.9%	91.8%	+0.8pp	89.3%	962,369
2026-05-12	2.1m	1.9m	-8.3%	91.3%	+0.9pp	89.0%	878,844
2026-05-13	2.1m	1.9m	-8.2%	91.2%	+0.8pp	88.9%	900,432
2026-05-14	2.1m	2.0m	-7.0%	91.3%	+0.5pp	89.1%	935,697
2026-05-15	2.1m	2.0m	-5.7%	90.9%	+0.5pp	89.9%	1,007,226
2026-05-16	2.1m	2.0m	-6.7%	91.1%	+0.8pp	90.1%	1,134,724
2026-05-17	2.1m	2.0m	-6.9%	91.6%	+0.7pp	90.5%	1,012,628
2026-05-18	2.1m	1.9m	-6.6%	91.5%	+0.4pp	90.7%	858,423
2026-05-19	2.1m	2.0m	-7.8%	90.6%	+0.7pp	89.8%	780,225

Cohort	Baseline MAE	Model MAE	Δ MAE %	Model within-2x	Δ within-2x pp	p90 cov	Hold rows
2026-05-06	9.8m	6.7m	-31.7%	58.4%	+8.0pp	87.6%	906,644
2026-05-07	13.1m	9.1m	-30.5%	57.6%	+7.3pp	85.0%	1,003,977
2026-05-08	13.3m	8.9m	-33.1%	56.9%	+7.2pp	85.1%	1,138,133
2026-05-09	12.6m	8.5m	-32.9%	56.0%	+6.7pp	85.5%	1,283,970
2026-05-10	12.1m	8.5m	-30.0%	56.3%	+5.9pp	86.2%	1,160,669
2026-05-11	9.6m	6.5m	-31.7%	57.4%	+5.4pp	89.6%	990,529
2026-05-12	5.3m	3.3m	-37.3%	58.4%	+6.8pp	90.0%	902,774
2026-05-13	5.1m	3.8m	-25.2%	58.3%	+6.8pp	87.6%	926,158
2026-05-14	7.7m	5.2m	-32.9%	58.1%	+8.1pp	88.2%	955,222
2026-05-15	8.1m	5.1m	-37.2%	58.0%	+8.4pp	87.3%	1,032,009
2026-05-16	7.9m	4.9m	-37.8%	57.4%	+8.2pp	86.9%	1,163,885
2026-05-17	8.0m	5.1m	-36.4%	58.5%	+8.1pp	87.6%	1,037,226
2026-05-18	7.4m	4.5m	-39.0%	59.0%	+7.9pp	88.3%	874,726
2026-05-19	4.5m	3.4m	-23.8%	60.4%	+7.2pp	87.8%	798,490

Cohort	Baseline MAE	Model MAE	Δ MAE %	Model within-2x	Δ within-2x pp	p90 cov	Hold rows
2026-05-12	5.3m	3.4m	-35.2%	56.7%	+4.9pp	90.9%	902,774
2026-05-13	5.1m	3.7m	-26.4%	44.6%	-7.2pp	92.0%	926,158
2026-05-14	8.2m	4.4m	-46.0%	56.0%	+5.2pp	89.8%	955,222

Cohort	Baseline MAE	Model MAE	Δ MAE %	Model within-2x	Δ within-2x pp	p90 cov	Hold rows
2026-05-12	5.3m	3.3m	-36.5%	58.5%	+6.7pp	89.8%	902,774
2026-05-13	5.1m	3.7m	-26.5%	58.2%	+6.4pp	88.8%	926,081
2026-05-14	8.2m	5.2m	-36.4%	57.9%	+7.1pp	87.9%	955,222

Daily Health (last 60d)

Date	Total	Comp%	Exc%	Wait p99	Cap p50	Util	Anomaly
2026-07-08	246,086	90.6%	1.7%	2.8h	2,473	91.4%	OK
2026-07-07	274,613	88.0%	1.7%	9.4h	3,189	91.7%	1 flag
2026-07-06	231,663	91.6%	1.4%	1.9h	2,392	91.1%	OK
2026-07-05	95,793	89.7%	0.8%	42.8m	742	90.1%	2 flags
2026-07-04	68,704	89.2%	0.6%	33.8m	457	89.1%	2 flags
2026-07-03	237,329	87.2%	1.3%	1.6h	2,330	90.8%	OK
2026-07-02	243,832	88.7%	1.6%	2.6h	3,040	87.9%	OK
2026-07-01	243,298	87.8%	1.7%	1.5h	2,954	86.4%	OK
2026-06-30	283,032	87.6%	1.7%	2.4h	3,959	87.3%	OK
2026-06-29	268,060	92.9%	1.7%	57.3m	3,912	87.0%	OK
2026-06-28	142,128	93.1%	0.7%	2.1h	1,764	88.4%	OK
2026-06-27	101,674	86.6%	0.9%	29.8m	840	87.8%	3 flags
2026-06-26	234,954	88.7%	1.9%	1.4h	3,127	86.4%	OK
2026-06-25	262,436	87.3%	1.8%	4.2h	3,807	84.9%	1 flag
2026-06-24	235,100	87.6%	1.7%	52.7m	2,839	85.1%	OK
2026-06-23	235,882	90.6%	2.0%	5.0h	3,361	84.6%	1 flag
2026-06-22	225,703	92.0%	1.6%	2.4h	3,298	86.0%	OK
2026-06-21	88,695	92.9%	1.0%	25.0m	683	86.7%	2 flags
2026-06-20	129,566	86.3%	0.9%	36.9m	1,305	84.5%	3 flags
2026-06-19	206,884	87.5%	1.7%	1.3h	2,083	85.9%	OK
2026-06-18	267,552	91.2%	2.0%	1.4h	4,020	86.2%	OK
2026-06-17	267,900	86.5%	1.8%	1.9h	3,741	85.8%	OK
2026-06-16	266,853	89.8%	1.7%	2.7h	3,467	85.8%	OK
2026-06-15	320,774	74.0%	4.2%	6.1h	4,217	86.7%	2 flags
2026-06-14	113,118	85.0%	5.0%	38.3m	994	85.3%	3 flags
2026-06-13	98,192	88.9%	1.1%	28.1m	1,038	89.2%	2 flags
2026-06-12	258,308	82.0%	2.2%	3.2h	3,150	85.8%	OK
2026-06-11	266,114	79.8%	2.9%	2.0h	3,788	85.3%	OK
2026-06-10	306,794	85.4%	2.6%	5.5h	3,991	85.7%	OK
2026-06-09	285,287	85.8%	2.2%	2.0h	3,822	85.3%	OK
2026-06-08	239,038	84.2%	1.6%	2.0h	3,112	85.7%	OK
2026-06-07	95,770	94.9%	1.0%	24.1m	879	87.4%	2 flags
2026-06-06	99,658	87.6%	1.5%	42.5m	954	86.1%	2 flags
2026-06-05	210,502	83.0%	2.1%	3.1h	2,236	86.9%	1 flag
2026-06-04	257,291	83.6%	2.0%	4.2h	3,213	84.3%	OK
2026-06-03	297,748	84.4%	2.2%	3.5h	4,027	84.8%	OK
2026-06-02	250,210	86.0%	2.1%	2.8h	3,512	85.1%	1 flag
2026-06-01	242,418	89.6%	1.8%	2.1h	3,530	85.8%	OK
2026-05-31	102,261	94.6%	1.0%	41.2m	1,014	87.2%	2 flags
2026-05-30	121,825	83.4%	1.1%	1.1h	1,208	86.8%	2 flags
2026-05-29	297,213	84.8%	2.0%	4.2h	3,671	86.6%	OK
2026-05-28	249,726	89.2%	2.1%	2.9h	3,517	80.7%	OK
2026-05-27	256,661	84.7%	1.5%	1.5h	3,753	74.2%	OK
2026-05-26	222,248	90.9%	1.8%	2.2h	3,673	78.3%	OK
2026-05-25	163,251	95.2%	1.0%	35.1m	2,318	81.2%	OK
2026-05-24	99,096	86.1%	0.7%	29.8m	865	83.2%	3 flags
2026-05-23	168,672	80.4%	0.9%	5.1h	2,351	82.2%	1 flag
2026-05-22	271,701	86.3%	1.7%	10.4h	3,574	78.0%	2 flags
2026-05-21	291,509	86.4%	1.5%	4.0h	4,610	75.1%	OK
2026-05-20	289,557	87.2%	2.0%	2.0h	4,580	75.8%	OK
2026-05-19	211,727	90.4%	2.3%	1.5h	3,447	72.2%	OK
2026-05-18	198,804	91.0%	1.8%	1.7h	3,263	76.2%	OK
2026-05-17	98,974	90.8%	0.6%	50.7m	1,001	79.8%	2 flags
2026-05-16	114,184	88.6%	1.0%	1.1h	1,355	79.4%	1 flag
2026-05-15	223,722	90.0%	2.1%	1.7h	3,171	76.4%	OK
2026-05-14	212,969	81.9%	1.8%	1.6h	2,871	74.9%	OK
2026-05-13	299,409	84.3%	2.2%	4.9h	4,832	75.2%	OK
2026-05-12	287,683	81.2%	2.4%	3.3h	3,924	75.5%	1 flag
2026-05-11	245,675	87.5%	2.0%	3.3h	3,683	73.3%	OK
2026-05-10	84,344	93.5%	0.8%	27.8m	976	80.6%	2 flags