Documentation request: symbol collision with system libsqlite3.so.0 on Linux when used inside a plugin .so (webkit/libsoup)

[rossanoparis]

Hello, and thanks for maintaining SQLite3 Multiple Ciphers - it has been working perfectly on Windows in my project for a long time.

I recently ported my application to Linux and ran into a subtle problem that cost me significant time to diagnose. I'd like to share it here, partly as a possible documentation improvement and partly so other users running into the same issue can find this thread.

Environment

• Linux (Linux Mint / Ubuntu derivative), x86_64
• GCC, wxWidgets 3.3.3 with GTK3
• Application uses wxWebView, which on Linux pulls in libwebkit2gtk-4.0 and libsoup-2.4
• Architecture: main executable loads a plugin shared library (Connectors.Db.Sqlite.so) which statically links a library (libLibraries.Db.Sqlite.a) built from sqlite3mc_amalgamation.c
• Identical source code on Windows works correctly - databases are encrypted as expected

Symptom

On Linux:

• PRAGMA key and sqlite3_key() were silently accepted
• The resulting database file was written as plaintext (SQLite format 3 header visible with a hex dump)
• PRAGMA cipher_version returned no row / was not recognised
• sqlite3_sourceid() returned the stock sqlite.org check-in hash, not an MC-specific one

On Windows, the same source code produced an encrypted database without any special configuration.

Root cause

Linux ELF shared libraries use a flat global symbol namespace by default. libwebkit2gtk and libsoup both declare DT_NEEDED on libsqlite3.so.0, so the system stock SQLite library is loaded into the process before my plugin is loaded.

Because stock libsqlite3.so.0 appears first in the global symbol resolution scope, every sqlite3_* call in the entire process - including those made from inside my plugin which had MC linked in statically - was resolved against the system stock SQLite. MC's code was loaded but never called. Stock SQLite silently ignores unknown pragmas, which is why PRAGMA key appeared to succeed while writing plaintext.

This does not happen on Windows because DLL imports are resolved per-module against explicit DLL names, not against a shared global namespace.

How to diagnose it

For anyone else hitting this, these diagnostics were decisive:

ldd /path/to/app | grep -i sqlite

Showed libsqlite3.so.0 being loaded from /usr/lib/x86_64-linux-gnu even though nothing in my build references it.

LD_DEBUG=bindings /path/to/app 2>/tmp/ld.log
grep "sqlite3_open" /tmp/ld.log

Showed sqlite3_* symbol lookups binding to libsqlite3.so.0 rather than to the MC-containing plugin.

# Confirm who transitively pulls in libsqlite3.so.0
for lib in $(ldd /path/to/app | awk '/=>/ {print $3}'); do
    if [ -f "$lib" ] && ldd "$lib" 2>/dev/null | grep -q libsqlite3; then
        echo "PULLS SQLITE: $lib"
    fi
done

In my case this printed:

PULLS SQLITE: /usr/lib/x86_64-linux-gnu/libwebkit2gtk-4.0.so.37
PULLS SQLITE: /usr/lib/x86_64-linux-gnu/libsoup-2.4.so.1

Running with LD_PRELOAD of an MC-built libsqlite3mc.so successfully routed calls into MC (including webkit's and libsoup's internal calls, which continued working fine - MC is a correct drop-in replacement for stock SQLite when no key is set).

The fix that works for a plugin-only deployment

My constraint was that MC must stay inside the plugin and not be linked into the main executable. The robust solution was to rename all SQLite public symbols inside the plugin so they no longer collide with stock SQLite's symbols in the process:

1. Create a header sqlite3_rename.h with #define lines mapping every sqlite3_* function the plugin uses to a prefixed name:

#ifndef SQLITE3_RENAME_H
#define SQLITE3_RENAME_H

#define sqlite3_open         mc_sqlite3_open
#define sqlite3_open_v2      mc_sqlite3_open_v2
#define sqlite3_close        mc_sqlite3_close
#define sqlite3_prepare_v2   mc_sqlite3_prepare_v2
#define sqlite3_step         mc_sqlite3_step
#define sqlite3_finalize     mc_sqlite3_finalize
#define sqlite3_key          mc_sqlite3_key
#define sqlite3_key_v2       mc_sqlite3_key_v2
#define sqlite3_rekey        mc_sqlite3_rekey
/* ... every sqlite3_* function the plugin uses ... */

#endif

2. Compile both sqlite3mc_amalgamation.c and all plugin source files with -include sqlite3_rename.h. This renames definitions and call sites consistently via the preprocessor, without changing any source code.

3. Link the plugin with -Wl,--exclude-libs=libLibraries.Db.Sqlite.a to prevent the renamed symbols from being exported from the plugin, keeping them fully internal.

After this:

• nm -D on the plugin shows no sqlite3_* exports
• Webkit and libsoup continue to use the system libsqlite3.so.0 untouched
• The plugin uses MC internally via the mc_sqlite3_* names
• PRAGMA cipher_version returns the MC version string as expected
• The database file is correctly encrypted

Verification commands

# Should return nothing (no sqlite3_* exported from the plugin)
nm -D /path/to/plugin.so | grep -E "sqlite3_(open|key|libversion|prepare_v2)$"

# Should find the renamed internal symbols
nm /path/to/plugin.so | grep -E " [tT] mc_sqlite3_(open|key)$"

# First 16 bytes of the DB file should be random (encrypted), not "SQLite format 3"
head -c 16 /path/to/db.db | xxd

Suggestion

It might be worth adding a section to the installation / build documentation covering this scenario, since:

• It is specific to Linux (the Windows path in the docs works out of the box)
• It is very likely to be hit by any application that uses wxWebView, Qt WebEngine, or any webkit/GNOME stack component in the same process as an MC-enabled plugin or shared library
• The symptom (silent plaintext writes with no error) is dangerous - a user could ship a product believing encryption is active when it is not
• The fix (symbol rename via -include + --exclude-libs) is not intuitive and took significant effort to arrive at

If a documentation PR with a "Linux plugin deployment" appendix would be welcome, I'd be happy to contribute one based on what I learned.

Thank you again for the excellent work on this project.

[utelle]

I recently ported my application to Linux and ran into a subtle problem that cost me significant time to diagnose. I'd like to share it here, partly as a possible documentation improvement and partly so other users running into the same issue can find this thread.

In this case I would propose to convert this issue into a discussion item, because after closing an issue (for example, after the documentation will have been updated) it couldn't be found that easily anymore.

Environment

• Linux (Linux Mint / Ubuntu derivative), x86_64
• GCC, wxWidgets 3.3.3 with GTK3
• Application uses wxWebView, which on Linux pulls in libwebkit2gtk-4.0 and libsoup-2.4
• Architecture: main executable loads a plugin shared library (Connectors.Db.Sqlite.so) which statically links a library (libLibraries.Db.Sqlite.a) built from sqlite3mc_amalgamation.c

I'm pretty sure almost any Linux environment would exhibit the same problem, because it is related to how the ELF linker works, when it comes to loading dynamic libraries.

• Identical source code on Windows works correctly - databases are encrypted as expected

Under Windows symbols of statically linked libraries are usually not visible to the outside.

Symptom

On Linux:

• PRAGMA key and sqlite3_key() were silently accepted

That's default behaviour of SQLite, when executing unknown pragmas.

However, for PRAGMA key it is a clear indicator that something went wrong, if this command doesn't return anything. If the pragma is handled by the library, you will get a single line containg the string ok, otherwise nothing is returned.

• The resulting database file was written as plaintext (SQLite format 3 header visible with a hex dump)
• PRAGMA cipher_version returned no row / was not recognised

PRAGMA cipher_version is not supported by SQLite3MC. It is a pragma known in the original SQLCipher library, though. While SQLite3MC supports reading and writing SQLite encrypted with SQLCipher, it uses different pragma names.

• sqlite3_sourceid() returned the stock sqlite.org check-in hash, not an MC-specific one

If you want to verify that your database connection uses SQLite3MC use the following SQL command:

select sqlite3mc_version();

It will return a single row containing the SQLite3MC version string, if you are using a SQLite3MC library, otherwise you will get an error no such function....

Root cause

Linux ELF shared libraries use a flat global symbol namespace by default. libwebkit2gtk and libsoup both declare DT_NEEDED on libsqlite3.so.0, so the system stock SQLite library is loaded into the process before my plugin is loaded.

Because stock libsqlite3.so.0 appears first in the global symbol resolution scope, every sqlite3_* call in the entire process - including those made from inside my plugin which had MC linked in statically - was resolved against the system stock SQLite. MC's code was loaded but never called. Stock SQLite silently ignores unknown pragmas, which is why PRAGMA key appeared to succeed while writing plaintext.

I have to admit that I wasn't aware of this ELF linker behaviour. However, using 2 different SQLite shared libraries in the same application isn't ideal anyway.

How to diagnose it

For anyone else hitting this, these diagnostics were decisive:

ldd /path/to/app | grep -i sqlite

Showed libsqlite3.so.0 being loaded from /usr/lib/x86_64-linux-gnu even though nothing in my build references it.

Well, it is an indirect dependency, because libwebkit2gtk requests it.

Unfortunately, my knowledge about Linux internals is limited. It would be interesting to know, when libwebkit2gtk pulls in libsqlite3.

If it happens already on startup of the application, very strange things could happen - even a crash.

Confirm who transitively pulls in libsqlite3.so.0

for lib in $(ldd /path/to/app | awk '/=>/ {print $3}'); do
if [ -f "$lib" ] && ldd "$lib" 2>/dev/null | grep -q libsqlite3; then
echo "PULLS SQLITE: $lib"
fi
done

In my case this printed:

PULLS SQLITE: /usr/lib/x86_64-linux-gnu/libwebkit2gtk-4.0.so.37
PULLS SQLITE: /usr/lib/x86_64-linux-gnu/libsoup-2.4.so.1

If an application uses both, libsqlite3 and libsqlite3mc, directly or indirectly, then there is a potential problem, because all depends on the order in which these libraries are pulled.

Running with LD_PRELOAD of an MC-built libsqlite3mc.so successfully routed calls into MC (including webkit's and libsoup's internal calls, which continued working fine - MC is a correct drop-in replacement for stock SQLite when no key is set).

Well, actually it depends. If libsqlite3mc was compiled with the same options (or a superset thereof) as libsqlite3, then all is well. Otherwise there may be problems.

The fix that works for a plugin-only deployment

My constraint was that MC must stay inside the plugin and not be linked into the main executable. The robust solution was to rename all SQLite public symbols inside the plugin so they no longer collide with stock SQLite's symbols in the process:

1. Create a header sqlite3_rename.h with #define lines mapping every sqlite3_* function the plugin uses to a prefixed name:

#ifndef SQLITE3_RENAME_H
#define SQLITE3_RENAME_H

#define sqlite3_open mc_sqlite3_open
#define sqlite3_open_v2 mc_sqlite3_open_v2
#define sqlite3_close mc_sqlite3_close
#define sqlite3_prepare_v2 mc_sqlite3_prepare_v2
#define sqlite3_step mc_sqlite3_step
#define sqlite3_finalize mc_sqlite3_finalize
#define sqlite3_key mc_sqlite3_key
#define sqlite3_key_v2 mc_sqlite3_key_v2
#define sqlite3_rekey mc_sqlite3_rekey
/* ... every sqlite3_* function the plugin uses ... */

#endif

Yes, that is certainly a possible workaround. However, it requires to keep track of all used API functions (or to rename all API functions).

Suggestion

It might be worth adding a section to the installation / build documentation covering this scenario, since:

• It is specific to Linux (the Windows path in the docs works out of the box)
• It is very likely to be hit by any application that uses wxWebView, Qt WebEngine, or any webkit/GNOME stack component in the same process as an MC-enabled plugin or shared library
• The symptom (silent plaintext writes with no error) is dangerous - a user could ship a product believing encryption is active when it is not
• The fix (symbol rename via -include + --exclude-libs) is not intuitive and took significant effort to arrive at

Yes, you are absolutely right. Due to how the ELF loader/linker works it is a problem to reference both, the system SQLite library libsqlite3 and libsqlite3mc in an application at the same time.

If a documentation PR with a "Linux plugin deployment" appendix would be welcome, I'd be happy to contribute one based on what I learned.

Adding appropriate documentation for this potential problem on Linux is certainly an important thing to do. And a PR is welcome, of course.

However, IMHO documenting this behaviour is not enough. Most likely, I should add #define statements for optionally renaming the API functions, so that a user of SQLite3MC doesn't need to track himself which functions need to be renamed.

[rossanoparis]

Thank you @utelle
Feel free to turn this issue into a discussion; indeed, it is not a library issue.
I'm sorry I didn't think enough about it before opening this issue.

However, IMHO documenting this behaviour is not enough. Most likely, I should add #define statements for optionally renaming the API functions, so that a user of SQLite3MC doesn't need to track himself which functions need to be renamed.

In my opinion, it would be sufficient to warn the user of this eventuality; perhaps a link to this discussion would be enough.
I've been using your library for the last 5 years, and I had to "fight" with what I'm reporting only because I started using the wxWebView class in my application.
The library is well written in C, unfortunately namespaces can't be used. I forgot this, and it took some time before figuring out the reason for what I was facing.

As soon as I cleared my mind :), despite the long description above, it took only 2 minutes to solve my problem.
Regarding the sqlite3 functions, I simply used your file sqlite3mc.def.
I copied its content into a Calc sheet and generated all the #defines for my sqlite3_rename.h, which I used as a parameter in the GCC configuration described above.
[ mc-fncs-rename.zip ]

The necessary way I used to fix the problem, by using symbol renaming via -include and --exclude-libs for both your library and my plug-in, allows all functions to be used with their original names, so please don't change anything in your code.

[utelle]

Feel free to turn this issue into a discussion; indeed, it is not a library issue.

Well, yes and no. There might be other components/libraries out there that could exhibit the very same problem, when used together with SQLite3MC. That is, the least that has to be done, is to document this potential problem (and to provide information how to overcome it).

I'm sorry I didn't think enough about it before opening this issue.

It was the right thing to open this issue and to bring the problem to my attention. Thanks!

For the time being I will keep it as an issue, but may convert it to a discussion later.

In my opinion, it would be sufficient to warn the user of this eventuality; perhaps a link to this discussion would be enough. I've been using your library for the last 5 years, and I had to "fight" with what I'm reporting only because I started using the wxWebView class in my application. The library is well written in C, unfortunately namespaces can't be used. I forgot this, and it took some time before figuring out the reason for what I was facing.

For sure, I would have heard about this problem earlier, if it would surface regularly. Most of the time it simply works. However, due to the nature of the problem it could happen that a user thinks his database is encrypted, but actually it's not, because the system SQLite3 library was pulled in before SQLite3MC.

As soon as I cleared my mind :), despite the long description above, it took only 2 minutes to solve my problem. Regarding the sqlite3 functions, I simply used your file sqlite3mc.def.

One problem with that approach is that sqlite3mc.def is not in the best shape. And actually the list of names in it depends on how SQLite3MC was compiled. That is, there is a risk of missing one or the other API function name.

I copied its content into a Calc sheet and generated all the #defines for my sqlite3_rename.h, which I used as a parameter in the GCC configuration described above. [ mc-fncs-rename.zip ]

Most likely I will provide a similar generated header file for future releases of SQLite3MC.

The necessary way I used to fix the problem, by using symbol renaming via -include and --exclude-libs for both your library and my plug-in, allows all functions to be used with their original names, so please don't change anything in your code.

I have not yet decided on how to handle this potential problem - except that it has to be documented.

[utelle]

@rossanoparis: I fear that the problem is even bigger than we both were aware of at first glance.

Please take a look at the SQLite documentation about "How To Corrupt An SQLite Database File", chapter 2.3 Multiple copies of SQLite linked into the same application.

In your application you will have 2 instances of the SQLite library, the system library and SQLite3MC. Just renaming the API functions in SQLite3MC doesn't prevent this. Therefore there is a high risk of corrupting your databases (either the ones used by wxWebView internally, or the ones of your application).

The only solution I see is to rename the SQLite3MC library to libsqlite3 and to use LD_LIBRARY_PATH to ensure that this library is found first to resolve the dependency of libwebkit2gtk / libsoup and thus to guarantee that only one instance of the SQLite library will be used in your application.

[rossanoparis]

... wait, wait :)

Regarding their warning:

What I've done avoids the corruption risk in my specific deployment, but it is not the general answer and what they write is real.
But my case describe a diffent context; I have MYAPP without any link to sqlite3 which dynamically loads a PLUGIN with your specific library.
The functions in my plugin, after having applied -include + --exclude-libs, now call symbols which are related only to my loaded library which is MC + all functions renamed by using #define.

Previously it was a mess, because functions in my plugin were using the library loaded within the wxWebView context.
There is no way to prevent a user from intentionally linking multiple copies of SQLite into the same application.
In my case I could achieve reasonable safety because of my architecture.

I think you can cite this specific case: APP (without sqlite3) + wxWebView + PLUGIN (with sqlite3-MC).
Nobody can do anything to prevent what the SQLite developers warn about except the programmer who is working on their own application.
But if a programmer is facing the same situation as mine, they should be led to try the solution I described, which solved my situation; and that scenario is not so remote.

[utelle]

Regarding their warning:

What I've done avoids the corruption risk in my specific deployment,

Sorry to say that, but I fear the corruption risk is still real for your application.

But my case describe a diffent context; I have MYAPP without any link to sqlite3 which dynamically loads a PLUGIN with your specific library. The functions in my plugin, after having applied -include + --exclude-libs, now call symbols which are related only to my loaded library which is MC + all functions renamed by using #define.

It is true that you fixed the symbol collision problem. But that's not the point. wxWebView and your plugin both load a SQLite library - wxWebView loads the system SQLite library, your plugin loads SQLite3MC. And that is just the scenario described in the SQLite documentation: both SQLite instances maintain a global list of open databases, but are not aware of each other's list. And that can lead to database corruption on Linux or MacOS systems.

Previously it was a mess, because functions in my plugin were using the library loaded within the wxWebView context. There is no way to prevent a user from intentionally linking multiple copies of SQLite into the same application. In my case I could achieve reasonable safety because of my architecture.

If the application uses both, wxWebView and your plugin, then there is no safety: you have 2 instances of SQLite in the process context.

I think you can cite this specific case: APP (without sqlite3) + wxWebView + PLUGIN (with sqlite3-MC). Nobody can do anything to prevent what the SQLite developers warn about except the programmer who is working on their own application.

Is the APP developed by you? Or are you just providing a plugin for a third-party APP?

Ultimately, your approach of encapsulating SQLite3MC in your plugin isn't making the situation any better—it's actually making it even worse. Because there is now no way to avoid having 2 instances of SQLite in process memory - except that the plugin runs in a separate process, which is highly unlikely, though.

IMHO the only chance is that the APP takes care of loading the correct SQLite library, in your case a SQLite library with encryption support. And since libwebkit2gtk loads libsqlite3, the name of the SQLite library has to be libsqlite3.

[rossanoparis]

You’re absolutely right, I think I just didn’t want to see what’s now obvious to me, so thanks for that.

I’ll definitely dig deeper into this, run more tests, and think things through more carefully.

That said, at least for now, I can’t really consider dropping that plugin.
The reason is that it’s part of a larger library I’ve built, which is meant to standardize database interactions through a specific API.
The SQLite plugin is just one piece of a bigger architecture.

Over time, since I’ve worked with different databases, I ended up building this system that I use across my applications.
It gives me a single interface to work with multiple DB engines; SQLite, MariaDB, MySQL, FIREBIRD, ADO on Windows, ecc...

All the plugins statically link their respective libraries, so even SQLite is compiled in statically using your library.

Even tough, they explain that each copy keeps its own internal global state (like the list of open database files), so if two different SQLite instances access the same database file, they won’t see each other’s locks. In that case, a close() in one instance might accidentally release locks held by the other, potentially leading to database corruption.

In my case, the situation is a bit different. My plugin links SQLite statically and even renames all symbols, so it’s completely isolated from any other SQLite instance in the process. The main application also uses webkit's (through wxWebView), which internally loads its own system SQLite dynamically.

So yes, there are technically two SQLite copies in the same process, but they operate on completely separate databases. My plugin only touches its own DB, while webkit uses SQLite internally for its own storage (like cache or cookies), in different locations.

Because of that separation, the specific issue described in the SQLite warning (multiple instances unknowingly accessing the same database file) shouldn’t occur here. As long as those database files never overlap, this setup is generally considered safe, even if it’s something to be aware of from a maintenance perspective.

At this point, I’m not entirely sure what the best move is.
The first thing I’ll do is set up a proper test environment to better understand the different scenarios and how much this actually impacts things.

Thanks again, really appreciate it.