An Invision Community site that has not been given a cron job runs its scheduled tasks during ordinary page views. That is the default, and most self-hosted communities never change it. If your application does anything slow in a queue or a task — and calling an outside API is slow — a random visitor pays for it with their page load, and with the database connection that page load is holding.
This is not a theoretical concern. It took a live community down.
The symptom
The report to look out for, near enough verbatim:
PHP-FPM was completely overloaded, and so MariaDB too. But the MySQL databases haven't gotten significantly larger.
That combination is the whole diagnosis. Workers are being held, not overworked, and no amount of looking at table sizes will explain it:
- Each held PHP-FPM worker keeps its MySQL connection open for as long as it is held, so MySQL saturates on connections while the data is untouched.
- A pool of, say, twenty workers is exhausted by twenty concurrent slow calls, at which point every other request queues and the entire site looks down.
- Because the cause is an outbound call, server CPU and disk look fine, which sends people hunting for a DDoS or a bad query instead.
Why it happens
The setting is task_use_cron, and its default value is normal. Under normal, \IPS\Dispatcher\Standard::__destruct() picks up one queued task and runs it at the end of a page request:
if ( $this->runTasks and Settings::i()->task_use_cron == 'normal' and !Request::i()->isAjax() )
{
$this->inDestructor = true;
$task = Task::queued();
if ( $task )
{
$task->runAndLog();
}
}
So the question is not "is my task slow?" but "who is waiting while it runs?" On a cron-driven site, nobody. On a default site, a visitor.
The fix
Cap how long an outbound call may take when a page request is waiting on it, and keep the generous limit for everything else. Dispatcher::i()->inDestructor tells you which situation you are in.
public static function seconds( ?int $background = NULL ): int
{
return static::blocking() ? 20 : ( $background ?? 90 );
}
public static function blocking(): bool
{
/* defined() first - \IPS\CLI is not declared in every entry point,
and an undefined constant is a fatal Error in PHP 8. */
if ( defined( '\IPS\CLI' ) and \IPS\CLI )
{
return FALSE;
}
try
{
return (bool) Dispatcher::i()->inDestructor;
}
catch ( \Throwable $e )
{
return FALSE;
}
}
Then route every outbound call through it:
$response = Url::external( $endpoint )->request( Timeout::seconds( 90 ) )->post( $payload );
Twenty seconds is chosen to be comfortably below a typical max_execution_time and far below any FPM request_terminate_timeout, so your call gives up before anything above it does.
What not to do
The obvious-looking fix is to detect the destructor and refuse to run there, deferring the work to a proper task run. Do not do this. On a site with no cron, the destructor is the task runner. Skipping it means the feature never runs at all, and you have traded a slow site for a broken application — which the administrator will notice later and understand less.
Bound the wait. Do not skip the work.
Tell the administrator
Capped is not the same as correct. A 20-second ceiling protects the server but can truncate whatever the call was doing. If your application depends on outbound calls, say plainly in the AdminCP when the site is in this mode:
if ( Settings::i()->task_use_cron === 'normal' and !\IPS\CIC )
{
/* warn: tasks are running during visitors' page loads */
}
Note the \IPS\CIC check — Invision Community Cloud runs tasks properly, so the warning would be wrong there.
Audit an existing application
Any application with a Queue extension or a Task, plus an outbound HTTP call, is exposed. Find them:
grep -rn "request(" $(find applications/myapp -name "*.php") | grep -v "Request::i()"
Anything much over 30 seconds needs the guard. When this was audited across a catalogue of two dozen applications, seven were affected — the worst had six calls allowed 120 seconds and one allowed 300.
Related application: AI Assistant — AI Assistant caps every call it makes to an AI provider at 20 seconds when a visitor's page load is waiting on it, and says so in the AdminCP when your community has no cron job set up.
Recommended Comments