[SOLVED] Dynamically Linked c++ binaries loaded with LoadLib interrupting extension uninstall/update on Windows

Scripts & Extensions
1

I have an extension, Asevoxel, that executes functions from dynamically linked libraries (.dll) on Windows, and Shared Object files (.so) on linux. These files are loaded into memory with the LUA Loadlib command (I couldn’t get another way that worked on both OSs)

Extension runs perfectly on both, but on windows, if Aseprite has been closed and opened at least once after the install session, it fails on uninstall/update with the following error

A problem has occurred. Details: Error deleting file
Access denied

After that, every sub-folder in the extension is empty (properly deleted) except for the one with the dlls. Is there any “unloadlib” or special way to read the uninstall command and then release the binaries before aseprite attempts to delete them? Or to keep the libraries unloaded outside of the extension’s use? Is this even what I have to do?

2

Hi @PixelMatt

Although I am not familiar with your current implementation, I have some ideas that might be worth trying, but I haven’t tested them myself. Maybe you already test them, but I will send you anyway.

  1. I think there is no unload function, but maybe you can create a function that clear the references and then, you can try to call collectgarbage function. Something like:

function unload()
– Clear the references you are using for loading the library.
functionsUsed = {}
libraryLoaded = nil
collectgarbage(“collect”) – Force GC to potentially release references
end

  1. Load each function individually. Maybe there are less problems.

local functions = {}
local loadlib = package.loadlib
functions.libFunction = loadlib(libPath, “libFunction”)

  1. Maybe you can use the exit event of the plugin and use the garbagecolletor as before.

function exit(plugin)
– Clear the references you are using for loading the library.
functionsUsed = {}
libraryLoaded = nil
myLibHandle = nil
collectgarbage(“collect”) – Force GC to potentially release references
end

  1. A different approach is to load from a temporary copy that can be deleted afterwards. The original dll should be deleted without any problem. The temp file will be removed by the OS eventually, but I am not sure what will happen if you install and uninstall several times.

function loadLibrary(plugin)
local originalPath = app.fs.joinPath(app.fs.userConfigPath, “extensions”, “YourExtensionFolder”, “yourlib.dll”)
– Copy to temp with unique name
local tempPath = os.tmpname() .. “.dll”
local success = app.fs.copyFile(originalPath, tempPath)
if success then
return package.loadlib(tempPath, “yourFunction”)
end
end

3

When a .dll is loaded, apparently Windows places a file lock on it. Lua’s garbage collector does not release this lock, it only cleans up Lua objects. This means that no one of your suggestions could solve the problem (it was OS scope, not LUA scope)

The lock stays until the process exits. When Aseprite tries to delete the extension folder on uninstall, it hits that locked file and fails.

I tried making the DLL call FreeLibrary on itself to release the lock. This crashed Aseprite. When FreeLibrary unloads a DLL, Windows removes its code from memory immediately. The function’s own return address is now pointing at unmapped memory, so the process crashes before it can get back to Lua.

A background thread was made to call FreeLibrary after the DLL function had returned, so there would be no code executing inside the DLL when it got unloaded. This did not release the lock. Becauyse every call to package.loadlib increments Windows’ internal reference count on the DLL. The count was 2, so one FreeLibrary brought it to 1 the lock remained (it had to be 0).

Immediately i thought of two calls: count goes 2 then 1 then 0. This crashed Aseprite. The first call drops the count to 1 and the DLL stays loaded. The second call drops it to 0 and Windows unmaps the DLL immediately while the thread is still executing code inside it.

Finally AI suggested this: Since windows will not let you delete a file that is in use, maybe it will let you rename one. MoveFileExW succeeds on a loaded DLL. So instead of unloading the DLL, asevoxel_schedule_unload uses GetModuleFileNameW to find its own path on disk, then calls MoveFileExW to rename it to a temporary name in the same directory. The DLL continues running normally under the new name. The original filename no longer exists in the folder, so Aseprite’s folder deletion succeeds. When Aseprite exits, Windows automatically cleans up the renamed file.

No threads. No reference count arithmetic. No timing dependencies. Simple but kinda dirty tactic with renaming the file

The full Load/release path is this

local NativeLib = {}

local _mod = nil
local _path = nil

-- Detect platform
local isWin = package.config:sub(1,1) == "\\"

--------------------------------------------------------------------------------
-- Load
--------------------------------------------------------------------------------
function NativeLib.load(pluginDir)
  if _mod then return true end

  local sep = isWin and "\\" or "/"
  local libname = isWin and "binaryfile.dll" or "binaryfile.so"
  local fullpath = pluginDir .. sep .. libname

  local loader, err = package.loadlib(fullpath, "luaopen_binaryfile")
  if not loader then
    return false, "could not load " .. fullpath .. ": " .. tostring(err)
  end

  local ok, result = pcall(loader)
  if not ok or type(result) ~= "table" then
    return false, "luaopen_binaryfile failed: " .. tostring(result)
  end

  _mod = result
  _path = fullpath
  package.loaded["binaryfile"] = result
  return true
end

--------------------------------------------------------------------------------
-- Use
--------------------------------------------------------------------------------
function NativeLib.get()
  return _mod
end

--------------------------------------------------------------------------------
-- Unload
--------------------------------------------------------------------------------
function NativeLib.unload()
  if not _mod then return end

  -- Windows: rename the DLL so the extension folder can be deleted.
  -- MoveFileExW succeeds on mapped files; the renamed file is cleaned up
  -- by Windows when the process exits.
  -- On Linux/macOS this is unnecessary: open .so files are not locked.
  if isWin and type(_mod.schedule_unload) == "function" then
    pcall(_mod.schedule_unload)
  end

  _mod = nil
  _path = nil
  package.loaded["binaryfile"] = nil
  collectgarbage("collect")
  collectgarbage("collect")
end

return NativeLib

The corresponding C++ side that schedule_unload must implement:

#include <windows.h>

extern "C" __declspec(dllexport) int schedule_unload(lua_State* L) {
#ifdef _WIN32
    WCHAR dllPath[MAX_PATH] = {0};
    HMODULE hSelf = NULL;

    GetModuleHandleExW(
        GET_MODULE_HANDLE_EX_FLAG_FROM_ADDRESS |
        GET_MODULE_HANDLE_EX_FLAG_UNCHANGED_REFCOUNT,
        (LPCWSTR)(void*)&schedule_unload,
        &hSelf);

    if (hSelf && GetModuleFileNameW(hSelf, dllPath, MAX_PATH)) {
        WCHAR tempPath[MAX_PATH] = {0};
        WCHAR* lastSlash = wcsrchr(dllPath, L'\\');
        if (lastSlash) {
            size_t dirLen = (size_t)(lastSlash - dllPath) + 1;
            wcsncpy_s(tempPath, MAX_PATH, dllPath, dirLen);
            wsprintfW(tempPath + dirLen, L"_unloading_%u.dll", GetTickCount());
            MoveFileExW(dllPath, tempPath, 0);
        }
    }

    lua_pushboolean(L, 1);
#else
    lua_pushboolean(L, 0);
#endif
    return 1;
}

This works for both Linux and Windows, though the release part is not required by linux as, at least that version doesnt crash or fail on uninstall/delete of loaded binaries.