A core/FileStorage extension is how an application tells the suite that it owns a bucket of files. Each extension becomes one row on AdminCP → System → Files → Storage Settings, one entry in the upload_settings setting, and one unit of work for the MoveFiles background task. Core calls it in exactly four situations: when the admin repoints that bucket at a different storage configuration (count() then repeated move()), when the admin runs Find orphaned files (isValidFile() for every file in a configuration), when the owning application is uninstalled (delete()), and — for the string form of a storage key — every time anything in your app calls File::create() or File::get().
The contract
IPS\Extensions\FileStorageAbstract declares four abstract methods and nothing else. All four are required.
abstract class FileStorageAbstract
{
/* Progress denominator for the move task. Must be > 0 if you have
files, or nothing will ever be moved. Need NOT be an accurate
file count. Must never throw. */
abstract public function count(): int;
/* Move one "step" of files. $offset starts at 0 and increases by 1
per call, forever, until you throw UnderflowException.
Return type is void — you cannot return a new offset. */
abstract public function move( int $offset, int $storageConfiguration, int $oldConfiguration=NULL ) : void;
/* TRUE if this container-relative path is a file your app still
references. FALSE means "orphan" and the file is queued for
deletion. Called with a string in practice, never a File. */
abstract public function isValidFile( File|string $file ): bool;
/* Delete every file this extension owns. Called on app uninstall,
while your tables still exist. */
abstract public function delete() : void;
}
Three further members are read by core but are not declared on the abstract, so nothing reminds you they exist:
| Member | Read by | Effect |
|---|---|---|
public static bool $isPrivate | IPS\File\Amazon::isPrivate() | When TRUE, objects are PUT with X-Amz-Acl: bucket-owner-read instead of public-read, and API output returns a signed temporary URL. |
public static array $storeGzipExtensions | IPS\File\Amazon::getGzipExtensions() | File extensions for which a second, gzipped copy is stored (core uses array( 'css', 'js' ) for core_Theme). |
public static function settingsUpdated() | MoveFiles and IPS\File::settingsUpdated(), both via method_exists() | Called once when a move to a new configuration completes, so you can flush caches that embed file URLs. |
The abstract writes int $oldConfiguration=NULL, which is an implicitly-nullable parameter. Declaring ?int $oldConfiguration=NULL in your subclass is the identical type and is signature-compatible.
A minimal example
Two files. First, register the class in applications/myapp/data/extensions.json — in IC5 this file is the extension registry; the directory is never scanned.
{
"core": {
"FileStorage": {
"Photos": "IPS\\myapp\\extensions\\core\\FileStorage\\Photos"
}
}
}
Then applications/myapp/extensions/core/FileStorage/Photos.php. The storage key this produces is myapp_Photos (application directory, underscore, extensions.json key), and that is the string you pass to File::create() and File::get() everywhere else in your app.
<?php
namespace IPS\myapp\extensions\core\FileStorage;
use Exception;
use IPS\Db;
use IPS\Extensions\FileStorageAbstract;
use IPS\File;
use UnderflowException;
use function defined;
if ( !defined( '\IPS\SUITE_UNIQUE_KEY' ) )
{
header( ( $_SERVER['SERVER_PROTOCOL'] ?? 'HTTP/1.0' ) . ' 403 Forbidden' );
exit;
}
class Photos extends FileStorageAbstract
{
public function count(): int
{
/* Guard: this runs on a shared AdminCP screen and must not throw */
if ( !Db::i()->checkForTable( 'myapp_photos' ) )
{
return 0;
}
return (int) Db::i()->select( 'COUNT(*)', 'myapp_photos', 'photo_location IS NOT NULL' )->first();
}
public function move( int $offset, int $storageConfiguration, ?int $oldConfiguration=NULL ) : void
{
/* NOTE: deliberately OUTSIDE the try. When the offset runs past the
last row, ->first() throws UnderflowException and that is the ONLY
thing that ends the background task. */
$row = Db::i()->select( '*', 'myapp_photos', 'photo_location IS NOT NULL', 'photo_id', array( $offset, 1 ) )->first();
try
{
$file = File::get( $oldConfiguration ?: 'myapp_Photos', $row['photo_location'] )->move( $storageConfiguration );
if ( (string) $file != $row['photo_location'] )
{
Db::i()->update( 'myapp_photos', array( 'photo_location' => (string) $file ), array( 'photo_id=?', $row['photo_id'] ) );
}
}
catch( Exception $e )
{
/* Logged by IPS\File; skip this row and let the task continue */
}
}
public function isValidFile( File|string $file ): bool
{
try
{
Db::i()->select( 'photo_id', 'myapp_photos', array( 'photo_location=?', (string) $file ) )->first();
return TRUE;
}
catch ( UnderflowException $e )
{
return FALSE;
}
}
public function delete() : void
{
foreach( Db::i()->select( '*', 'myapp_photos', 'photo_location IS NOT NULL' ) as $row )
{
try
{
File::get( 'myapp_Photos', $row['photo_location'] )->delete();
}
catch( Exception $e ){}
}
}
}
My extension never appears in the Storage Settings list
The AdminCP form at applications/core/modules/admin/overview/files.php builds one Select per entry returned by Application::allExtensions( 'core', 'FileStorage', FALSE, NULL, NULL, TRUE ). That walks Application::extensions(), which reads only applications/<app>/data/extensions.json. Dropping a file into extensions/core/FileStorage/ registers nothing.
Two silent filters sit inside that loader. extensions() skips any entry where !is_string( $classname ) or !class_exists( $classname ) — the comment in core is "Switching between branches confuses extensions" — so a namespace typo produces no error at all, just an absent row. And constructExtensionClass() wraps new $class in catch( RuntimeException | OutOfRangeException $e ){} and returns null, so a constructor that throws either of those also just disappears from the list. Neither failure is logged anywhere. If the row is missing, check extensions.json first and your namespace second.
Also note the results are cached: allExtensions() stores the whole map in Store::i()->extensions and extensions() memoises in a static. After editing extensions.json, clear the data store.
RuntimeException: NO_STORAGE_EXTENSION — but only on the dev install
Your storage key is composed from two different sources depending on who is asking, and they must agree.
File::getClass() contains an IN_DEV-only sanity check. When the key you passed has no entry in upload_settings, it loops Application::allExtensions( 'core', 'FileStorage', FALSE ), splits each object's namespace, and rebuilds the key as $extensionNamespace[1] . '_' . $extensionName — that is, application directory plus the class basename. If nothing matches it throws RuntimeException( 'NO_STORAGE_EXTENSION' ). Everywhere else — MoveFiles, Amazon::isPrivate(), Amazon::getGzipExtensions(), File::settingsUpdated() — resolution goes through Application::getExtensionClass(), which looks the key up as a key in extensions.json.
So if you register "Photos": "IPS\\myapp\\extensions\\core\\FileStorage\\PhotoStorage", the key myapp_Photos works on a production install and throws NO_STORAGE_EXTENSION on a dev install, while the key myapp_PhotoStorage does the reverse. Keep the extensions.json key identical to the class basename and the file name.
The dropdown for my extension is greyed out forever: "This setting can not be changed as there is a move in progress."
This one is worth reading twice, because the symptom appears in the AdminCP and the cause is in a background task that reported success.
When you change a storage configuration, the AdminCP does not simply write the new ID. It writes a two-element array into upload_settings — $values[$k] = array( $newId, $oldId ) — as a lock, so that File::getClass() can still find not-yet-moved files. The only code that ever collapses that array back to a plain integer is the UnderflowException branch of IPS\core\extensions\core\Queue\MoveFiles::run(). While the array is present, files.php sets 'disabled' => $disabled on the Select and injects the language string file_storage_move_in_progress as the field warning.
MoveFiles::run() starts like this:
$exploded = explode( '_', $data['storageExtension'] ); // "filestorage__myapp_Photos"
try
{
$classname = Application::getExtensionClass( $exploded[2], 'FileStorage', $exploded[3] );
}
catch( OutOfRangeException )
{
throw new \IPS\Task\Queue\OutOfRangeException;
}
The setting key is filestorage__ + app + _ + extension key, so the naive explode() gives [2] = app and [3] = extension key. Put an underscore anywhere in your extension key (Photos_v2) or your application directory and $exploded[3] is a fragment, getExtensionClass() throws OutOfRangeException, and the handler rethrows it as \IPS\Task\Queue\OutOfRangeException — which Task::runQueue() interprets as "the task is done" and deletes the queue row, logging "completed successfully" to runQueue_log. No files move, no error is raised, and the lock array stays in upload_settings permanently. The admin sees a permanently disabled dropdown and there is nothing in the logs connecting the two.
The same parse also disables the extension when the application is disabled or when RECOVERY_MODE is on, since getExtensionClass() throws OutOfRangeException in both cases. The fix is simply: no underscores in FileStorage extension names. Use PhotosV2, never Photos_V2. Recovering an already-stuck site means editing the upload_settings row in core_sys_conf_settings by hand to replace the two-element array with a single ID.
Files that are supposed to be private are uploaded to S3 as public-read
IPS\File\Amazon::isPrivate() is the only thing that decides the ACL on every PUT:
$bits = explode( '_', $this->storageExtension );
try
{
$class = Application::getExtensionClass( $bits[0], 'FileStorage', $bits[1] );
if ( isset( $class::$isPrivate ) )
{
return $class::$isPrivate;
}
}
catch( OutOfRangeException ){}
return false;
Note $bits[0]/$bits[1] here versus $exploded[2]/$exploded[3] in MoveFiles — same string, two different offsets, because this one has no filestorage__ prefix. The failure mode is the important part: any underscore in your extension name, or a class not present in extensions.json, makes getExtensionClass() throw, the empty catch swallows it, and the method returns false — meaning your private files are written with X-Amz-Acl: public-read. Nothing is logged, nothing fails, and the file uploads perfectly. You only find out by inspecting object ACLs in the bucket. getGzipExtensions() degrades the same way, harmlessly.
Because $isPrivate is read statically off the class name, it must be public static bool $isPrivate = true; — an instance property will not be seen. Core's only example is IPS\downloads\extensions\core\FileStorage\Files.
The move task runs forever and the progress bar passes 100%
MoveFiles::run() has no upper bound derived from count(). It runs a fixed REBUILD_SLOW iterations per cycle, calling move( $offset, ... ) with $offset incremented by one each time, and the only exits are $data['count'] === 0 or an UnderflowException propagating out of your move():
catch ( UnderflowException $e )
{
/* collapse the lock in upload_settings, call settingsUpdated(), then */
throw new \IPS\Task\Queue\OutOfRangeException;
}
catch ( Exception $e )
{
Log::log( $e, 'move_files_bgtask' );
$offset++;
continue;
}
The classic mistake is wrapping the whole body of move() in try { ... } catch ( Exception $e ) {} to be defensive. UnderflowException extends RuntimeException extends Exception, so you swallow your own terminator and the task never finishes — it keeps incrementing the offset every cron run indefinitely, and the lock in upload_settings is never released (see the trap above). Every core implementation puts the terminating query outside its try block for exactly this reason; look at core/Attachment::move().
Ordering matters too: the UnderflowException catch is listed first. If you throw one from deep inside a nested try that catches Exception, it is gone.
Any other exception costs you one row: it is logged to the AdminCP system log under the key move_files_bgtask, the offset advances, and the loop continues. That is where to look when "some of my files moved and some didn't".
Fatal error: A void function must not return a value
In IC4, move() could return an integer to jump the offset. IC5 typed the signature as : void, but the surrounding code was never cleaned up, and the stale documentation is actively misleading. Core's own ClubField, Imageproxycache and Theme extensions carry the docblock line:
* @return void An offset integer to use on the next cycle, or nothing
And MoveFiles::run() still contains:
$return = $extension->move( $offset, (int) $data['newConfiguration'], (int) $data['oldConfiguration'] );
$offset++;
if ( is_numeric( $return ) )
{
$offset = $return;
}
That block is dead code in IC5 — $return is always NULL. If you follow the docblock and return $newOffset;, PHP raises Fatal error: A void function must not return a value inside the background task, which surfaces as a stalled queue row and an uncaught_exception log entry, not as anything the admin can connect to your app. The offset is a plain counter you do not control; design your move() around a LIMIT $offset, 1 cursor or around fixed step numbers, as core/Theme does with its switch ( $offset ).
Nothing moves at all, and the progress bar sits at 100%
count() is not a file count — it is a permission slip and a progress denominator, and both consumers treat zero as "nothing to do".
In files.php: $count = $extension->count(); if ( $count ) { ... Task::queue( 'core', 'MoveFiles', ... ) ... }. No task is queued at all if you return 0. In configurationMove() the test is if( $count > 0 ). And inside MoveFiles::run(), if( $data['count'] === 0 ) { throw new UnderflowException; } ends the task immediately. Return 0 and your files stay exactly where they are, permanently, while the AdminCP reports the move as finished.
Accuracy, on the other hand, does not matter at all. getProgress() computes round( 100 / $data['count'] * $offset, 2 ) and nothing clamps it. Core is wildly inconsistent here and it is harmless: core/Attachment::count() returns MAX(attach_id) while its move() uses LIMIT $offset, 1; core/Theme::count() returns the literal 6 with the comment "it's the number of steps this will take to move them, which is all it's used for"; cms/Pages::count() returns 1; core/ClubField::count() returns a real row count but its move() processes every field on offset 0 and then throws UnderflowException immediately. Any positive integer is a valid answer. Zero is not.
Saving the Storage Settings form dies with my app's database error
In files.php, the settings save loop protects the class lookup but not the call:
try
{
$classname = Application::getExtensionClass( $exploded[2], 'FileStorage', $exploded[3] );
}
catch( OutOfRangeException )
{
continue;
}
$extension = new $classname;
...
$count = $extension->count();
That count() has no try/catch around it. If your table was dropped by a botched upgrade, or your count() queries a column that a migration has not added yet, the exception escapes and takes down the entire Storage Settings save for every application on the site. The admin cannot change any storage setting until your app is disabled. This is why core/Imageproxycache caches a Db::i()->checkForTable( 'core_image_proxy' ) result and returns 0 when the table is missing — treat count() as code that must never throw.
The other AdminCP path, Move all files off this configuration (configurationMove()), inverts the risk: the whole per-extension block including count() sits inside catch( Exception $e ){}. If your count() throws there, the line $settings[ $k ] = $newStorageId; never runs, so your extension keeps pointing at the old configuration — which the same request then queues DeleteMovedFiles against with 'storageToDelete' => $current['id']. Silent skip, followed by a deletion pass over the storage you were skipped onto.
One further hazard in that same loop: it iterates every key in upload_settings testing if ( $v == $current['id'] ), but upload_settings also contains the boolean filestorage_move key, written by the settings form's own json_encode( $values ). On a site whose first storage configuration has id = 1, true == 1 matches, explode( '_', 'filestorage_move' ) has no index 2 or 3, and getExtensionClass( null, 'FileStorage', null ) raises a TypeError — which is an Error, not an Exception, so the surrounding catch( Exception $e ){} does not catch it and the whole action fatals before reaching later extensions. The settings-form loop excludes that key explicitly ($k != 'filestorage_move'); this one does not.
My files were deleted after an admin ran "Find orphaned files"
isValidFile() is the only thing standing between your files and deletion, and the deletion is not reviewed by a human. FindOrphanedFiles calls File::orphanedFiles(), which calls removeOrphanedFiles( $fileIndex, Application::allExtensions( 'core', 'FileStorage', FALSE ) ) on the storage handler. The handler walks every object in the configuration, asks every FileStorage extension in the suite isValidFile( $path ), and logs anything nobody claims via logOrphanedFile() into core_file_logs with log_type = 'orphaned'. As soon as the scan reports _done, FindOrphanedFiles itself queues DeleteOrphanedFiles, which loads each logged row and calls ->delete(). There is no confirmation screen between the scan and the deletion.
The consequences are worth spelling out:
- An empty or wrong
isValidFile()deletes your users' files. A stub that returnsFALSE, or one that compares against the wrong column, is not a no-op. - The argument is a container-relative path string, not a URL and not a
File.FileSystempassesSplFileInfo::getSubPathname(),Databasepasses$file['container'] . '/' . $file['filename'],Ftppasses the pathname with the configured root stripped, andAmazonpasses the S3 key withbucket_pathstripped — as aSimpleXMLElement, not a string. Always(string) $filebefore comparing, exactly as every core implementation does. - If your application is disabled, your extension is not in the list.
allExtensions()skips applications failingappIsEnabled(), and skips all non-IPS applications entirely whenRECOVERY_MODEis on. Running the orphan scan while your app is disabled means nobody claims your files and all of them are queued for deletion. - Throwing from
isValidFile()aborts the scan for everybody. OnlyFileSystemguards the call, and only againstInvalidArgumentException(where it treats the file as valid and moves on).Amazon,DatabaseandFtpcall$engine->isValidFile( ... )bare. One badly written third-party extension takes the whole orphan scan down.
Find orphaned files fails with "Call to undefined method ...::getSubPathname()"
In 5.0.19, IPS\File\FileSystem::removeOrphanedFiles() contains a variable-name bug:
foreach( $engines as $engine )
{
try
{
if( $engine->isValidFile( $engine->getSubPathname() ) )
{
continue 2;
}
}
catch( InvalidArgumentException $e )
{
continue 2;
}
}
$engine is your FileStorage extension object; the intended receiver is $f, the SplFileInfo from the directory iterator. FileStorage extensions have no getSubPathname(), so this raises Error: Call to undefined method IPS\...\FileStorage\<Name>::getSubPathname() on the first file examined. Error is not InvalidArgumentException and not RuntimeException, so neither the local catch nor FindOrphanedFiles's catch( RuntimeException $ex ) intercepts it — it escapes Task::runQueue() and lands in the uncaught_exception log, named after whichever extension happened to be first in the list. The class named in that error is almost never the class at fault; it is simply the first entry of allExtensions( 'core', 'FileStorage' ).
The practical consequence is that on a FileSystem storage configuration the orphan scan cannot run at all in this version, so isValidFile() is effectively exercised only on Amazon, Database and Ftp configurations — which is where the deletion risk described above is real. Do not conclude from a green run on a local filesystem install that your isValidFile() has been tested.
After a storage move, the old copies are never deleted
IPS\File::move() does not delete the source. Unless you pass File::MOVE_DELETE_NOW, it copies the file and then writes a row into core_file_logs with log_type = 'move' and log_configuration_id set to the source configuration. Cleanup is a separate background task, DeleteMovedFiles, which drains those log rows.
That task is queued once, after the settings loop, gated on a single variable:
foreach ( $values as $k => $v )
{
...
if ( ... !$newClass::moveCheck( ... ) ) { $rebuild = FALSE; }
else { $rebuild = TRUE; }
if ( $rebuild ) { /* queue MoveFiles for this extension */ }
}
if( $rebuild )
{
Task::queue( 'core', 'DeleteMovedFiles', ... );
}
$rebuild is reassigned on every iteration and only its final value is tested. If the last changed extension in the loop resolves to "same handler, same configuration, no move needed", $rebuild ends as FALSE and DeleteMovedFiles is never queued — even though earlier extensions were queued and are busy copying files. The move succeeds, the database points at the new locations, and a full duplicate set of files is left behind on the old configuration with rows still sitting in core_file_logs. Nothing reports this. If you are debugging disk usage after a move, check SELECT COUNT(*) FROM core_file_logs WHERE log_type = 'move'.
Files uploaded while a move was running 404 afterwards
Both File::create() and File::get() call static::getClass( $storageExtension, TRUE ) — that second argument is $tryOldFirst. When upload_settings holds the two-element lock array, getClass() does array_pop() and returns the old configuration. That is correct for reads, since unmoved files are still there. It also means new uploads created during the move are written to the storage you are moving away from.
Whether they survive is a race. If your move() uses a LIMIT $offset, 1 cursor over a table, a row inserted before the task reaches the end will still be picked up. A row inserted after it has passed will not, and once the lock collapses to the new configuration ID, File::get() resolves against the new storage and the file is unreachable. Extensions like core/Theme that move fixed sets in numbered steps have no chance of catching late arrivals at all. This is what the AdminCP message filestorage_move_info means by "you may experience broken links/images until the process has completed" — except some of them do not heal. Prefer moving storage during a maintenance window.
settingsUpdated() throws a TypeError, or is never called
This optional hook is discovered by method_exists() and called from two places that disagree about its signature.
/* MoveFiles::run() — passes an int configuration ID */ $extension::settingsUpdated( $data['newConfiguration'] ); /* IPS\File::settingsUpdated() — passes the decoded configuration array */ $classname::settingsUpdated( $this->configuration );
Core's only implementation, core/Theme::settingsUpdated(), declares no parameters at all, which is what makes both call sites work — PHP silently discards extra arguments to userland functions. Declare a typed parameter and one of the two callers will hand you the wrong type: int breaks the File path with a TypeError, array breaks the MoveFiles path. Declare it with no parameters.
It must also be static. Both call sites use the :: form on a class name or an object, and in PHP 8 calling a non-static method that way is a fatal Error. In MoveFiles, that call sits inside the catch ( UnderflowException ) block, after Settings::i()->changeValues() has already released the lock but before throw new \IPS\Task\Queue\OutOfRangeException — so the fatal leaves the queue row undeleted and the task re-runs and re-fatals on every cron cycle.
Finally, note when it does not fire: only on the UnderflowException completion of MoveFiles. If count() returned zero, or the extension key failed to parse, no move task ran and settingsUpdated() is never called, so any cache you flush there stays stale.
Uninstalling my app stops halfway and leaves it half-removed
Application::delete() calls your delete() like this:
foreach( $this->extensions('core', 'FileStorage' ) as $extension )
{
try
{
$extension->delete();
}
catch( PHPException $e ){}
}
PHPException is Exception aliased at the top of the file. A TypeError, an ArgumentCountError, or any other Error is not caught and aborts the uninstall in place — after language strings, theme resources, modules, permissions and settings rows have already been deleted, and before the database tables are dropped. The application is left in a state no screen can finish cleaning up.
Two facts about ordering are useful when writing delete(). Your tables still exist at this point (schema.json is processed roughly sixty lines later), so querying them is correct and expected. But core_sys_conf_settings rows belonging to your app have already been deleted, so do not read your own settings there.
Also be aware that nothing removes your filestorage__<app>_<Name> keys from upload_settings on uninstall. installExtensions() writes them; delete() has no counterpart. They linger as harmless dead keys, invisible in the AdminCP (which lists extensions, not settings keys) but still iterated by configurationMove().
A FileStorage extension added in an app update never gets a setting
Application::installExtensions() — the method that writes $settings['filestorage__' . $this->directory . '_' . $key] = $fileSystem['id'] for each of your extensions — is called from exactly two places: the setup installer and the AdminCP "install application" flow. It does not run on a version upgrade. Add a new FileStorage extension in version 2 of your app and no upload_settings key is ever created for it.
This does not fail loudly. The first time anything calls File::create( 'myapp_NewThing', ... ), File::getClass() notices the missing key, loops core_file_storage, calls testSettings() on each configuration in id order, takes the first that does not throw a LogicException, and persists that choice back into upload_settings. On a single-configuration site the result is right by accident. On a site with two configurations, your new bucket silently lands wherever the first row points — typically the local filesystem — while the rest of your app is on S3, and the admin never chose it. Write the key explicitly in an upgrade step:
$settings = json_decode( Settings::i()->upload_settings, TRUE );
if ( !isset( $settings['filestorage__myapp_NewThing'] ) and isset( $settings['filestorage__myapp_Photos'] ) )
{
$settings['filestorage__myapp_NewThing'] = $settings['filestorage__myapp_Photos'];
Settings::i()->changeValues( array( 'upload_settings' => json_encode( $settings ) ) );
}
While you are there, add the two language strings the AdminCP form needs. The Select is built as new Select( 'filestorage__' . $name, ... ), so it looks for filestorage__myapp_Photos as its label and filestorage__myapp_Photos_desc as its description. Without them the admin sees the raw key.
Verified against
Everything above was read from the source of Invision Community 5.0.19: system/Extensions/FileStorageAbstract.php, system/File/File.php, system/File/FileSystem.php, system/File/Amazon.php, system/File/Database.php, system/File/Ftp.php, system/Application/Application.php, system/Task/Task.php, applications/core/modules/admin/overview/files.php, the MoveFiles, DeleteMovedFiles, FindOrphanedFiles and DeleteOrphanedFiles queue extensions, and all 34 core core/FileStorage implementations across the blog, calendar, cms, core, downloads, forums, gallery and nexus applications. Line-level details — particularly the getSubPathname() bug, the $rebuild overwrite, and the dead is_numeric( $return ) block — are specific to this release and may be fixed without notice.
Recommended Comments