  With the alarming rate of security advisories and privacy concerns on
connected devices, there is an urgent need for strong isolation guarantees in
resource-constrained devices that demand very lightweight solutions. However,
the status quo is that Unix-like operating systems do not offer privilege
separation inside a process. Lack of practical fine-grained
compartmentalization inside a shared address space leads to private data
leakage through applications' untrusted dependencies and compromised threads.
To this end, we propose $\mu$Tiles, a lightweight kernel abstraction and set of
security primitives based on mutual distrust for intra-process privilege
separation, memory protection, and secure multithreading. $\mu$Tiles takes
advantage of hardware support for virtual memory tagging (e.g., ARM memory
domains) to achieve significant performance gain while eliminating various
hardware limitations. Our results (based on OpenSSL, the Apache HTTP server,
and LevelDB) show that $\mu$Tiles is extremely lightweight (adds $\approx 10KB$
to kernel image) for IoT use cases. It adds negligible runtime overhead
($\approx 0.5\%-3.5\%$) and is easy to integrate with existing applications for
providing strong privilege separation.
