Today’s database engines fixate on one thing: materializing data, rearranging data, and preparing data for storage. Engines will jump through countless hoops to optimize, compress, and analyze data in order to set up hopefully sub-linear and sub-quadratic query strategies. One thing they won’t do is optimize the storage for future queries or data. What would it mean to be predictive?

First, let’s review the current database engine strategies:

• analyzing entropy in the input data, often guided by types, e.g. a boolean column has obviously less entropy than a free-form text column

• analyzing statistical facets of the data such as distinct value counts, average data size, and histogram-like data

• determining what compression method, if any, would be useful to apply to the data (including run-length encoding, Huffman-adjacent encoding, and NULL compression)

• slicing data into separate blocks called “partitions” which are then used to drastically reduce the amount of data needed to be recalled for range queries

• rearranging data into ordered “clusters” so that tuple scans can terminate earlier

• rearranging data into new data structures such as b+trees, inverted indexes (which index into data such as JSON), or hash indexes

Note that the properties of the data are gathered on the state of the current data. Since the data in a database is typically churning, the statistics and representations of the data are quickly out-of-sync and constantly refreshing. Multiple in-flight transactions make the state of the statistics even messier. Luckily, since the statistics only affect optimizations, having out-of-date statistics can only reduce query performance, not correctness.

Note that none of the transformations or statistics-gathering above make an attempt to predict future states of the database. In this article, we propose how this could be done. Along the way, we explain why fixating on tuple materialization need not be the goal of the database.

What’s the big deal about tuples?

Consider the following common-place SQL:

UPDATE product SET price = price * 1.05 WHERE product_category = 'hangbag';

The user has requested that the database raise the price of all handbags by 5%. How should the database react to this request? One thing the engine can do, if it is tuple-oriented as is common, is to find every handbag price in its storage subsystem and effectively replace it with the new value.

Database engines are obsessed with writing tuples to disk as soon as possible because the query engines are completely dependent on reading from that storage subsystem. The engine gathers statistics on the current state of that stored data because, again, that’s the canonical representation of the data in the database. Everything in the database engine feeds off those stored data. However, there is nothing in the mathematics of the relational algebra which requires this. Furthermore, database engines do little to nothing to predict any future state of the database because the only state available is “current” one (or a few recent ones) and all optimizations are applied to and solely useful for the current state (for whatever definition the current state of the database may mean). Past tuple states are viewed as garbage to be collected as soon as possible.

Because of this architectural decision, almost all database optimizations revolve around reducing the scope of the query to minimize the number of tuples needed to service it. Storing additional tuple states is cost prohibitive and scanning the tuples is expensive, so tracking statistics over time is deemed infeasible.

What is the trajectory of a database?

If we imagine a database’s lifetime over the course of a business’ lifetime, then it’s quite clear that the database has a reasonably predictable trajectory- at least as predictable as the business itself. In fact, database administrators rely on a projected trajectory to ensure that the database has:

• appropriate CPU, RAM, and storage allocated for the near future

• indexes which, based on expected usage patterns, are added even before the database has any data

• partitions based on expected query patterns

• other database-specific configurations for the estimated load (such as pre-allocating various caches)

Note that all of these initial assumptions are virtually guaranteed to be obsolete quite quickly and must continually be reevaluated. That is currently the role of a human database administrator. His work loop involves:

1. find the most expensive query or other bottleneck (perhaps reported by users)

2. optimize the query by rewriting the query to trick the optimizer to return the same answer faster

3. if the previous step doesn’t resolve the bottleneck, change the database schema to accommodate the query

4. if the previous step doesn’t resolve the bottleneck, change the database-specific configuration (such as caching)

5. if the previous step doesn’t resolve the bottleneck, change the physical hardware of the system

6. go back to step 1

This leads us to ask: why can’t the database track its own trajectory? Well, certainly it can!

What information can we use to predict the trajectory of the database?

At the very least, the database needs to track the states of the database which the database administrator would: data such as query plans on slow queries over some time window, data on heavily used indexes or unused indexes, data on cache usage, etc. Such metadata is explicitly provided to database administrators for this task, but there are deeper proxies for database performance which are easy for the database to track:

• hotspots in tables- For example, more recent data is typically more relevant to business logic. These hotspots can be used to inform partition or caching strategies. Such data is also more likely to appear in query caches.

• hotspots in caches- Such hotspots could indicate that the user is making redundant queries or querying data which has no updates to present.

• hotspots in query plans- For example, an expensive join which appears to reappear in query plans could indicate that the underlying storage could benefit from improved caching, a different join strategy, or partitioning designed to serve the join.

Furthermore, nothing about the relational algebra requires the database schema to reflect the underlying storage- C.J. Date calls this principle “data independence”.

Data Independence

If the database engine were to duplicate the data in various forms (any number of times) in order to serve more queries efficiently, then it can freely do so as long as it presents the user with a singular, consistent view of the data. The underlying storage need not be dictated by the database schema. An index is an example of such a duplication (though the concept of an index does poke through to the user in the schema)- PostgreSQL stores tuple data in a unordered heap of tuples by default and an index is copies the tuple data, orders the data into a particular data structure (b+tree) which is optimized for sub-linear search cost, and writes the duplicated data to disk. The database pays a duplicative storage cost with the goal of reducing query scan costs referencing the indexed data. It’s a trade-off involving multiple representations of the same data. A database tracking hotspots could automatically create indexes, partitions, data re-orderings, caches, and more, but most database engines wait for a human to make them. This is largely for historical reasons.

Hotspots Are In The Past

Even if our hypothetical database engine is tracking hotspots and automatically creating indexes and partitions, it’s already too late! A contingent of queries will not be serviced by the ideal representations of the data because the database has not yet recorded the new query’s hotspot. This will naturally occur whenever the application’s queries change, even subtly, or when the data size in the database grows or shrinks. Assuming that the database can adapt its internal representations for the new state, the new optimizations will be available shortly, but this creates a sawtooth pattern in the database performance: any time queries or data change, performance temporarily tanks until new optimizations are applied. This is true whether or not a human is in-the-loop.

Using Predictive Optimizations

To smooth out the sawtooth pattern, the database can make predictions about the trajectory of the database and provide additional representations before they become relevant. As a simple example, imagine a small table about to become large and therefore drastically change the performance of a query due to a join condition. We would like join value lookups to be sublinear, but the index to achieve this is not present. The database engine can track which queries involve this table, determine which representations of the data makes sense (an index on one of the join columns, in this case), and create the new representation in anticipation of the performance shift. This additional representation may be temporary- the prediction may be wrong and the price is some extra up-front work to create the representation, noticing it is not being used, and deleting it. If the new representation is useful to queries, we keep it. This is speculative database query planning and execution.

Let’s review the steps with more specific details of how this might work:

1. the engine notices that the flow of data to table T may become a bottleneck on query Q without an index on column C due to a join condition which is about to become quadratic because a cache for values in column C is about to overflow

2. the engine creates a b-tree index on column C

3. the engine runs a test query involving the join condition to ensure that the planner actually picks up the new index

4. if the planner does not pick up the index, that could indicate that the index is not yet needed and we’ll keep the index or that the planner may never use the index because of other considerations

5. if the engine notices that the index has not been used and is not useful to expected queries, then it is free to delete it to reclaim the storage for other optimizations

Note that the growth in step 1 could be predicted arbitrarily early if the database can place a curve on the table growth- the database anticipates the need for a b-tree index- but the database can defer index creation until it believes the index will be useful. In fact, a database engine could be running experiments in the background; namely, running queries or query fragments which the user has made previously with proposed optimizations. If the proposed optimization/additional representation does not affect query performance, then the representation can be garbage collected- maybe it will be useful farther into the future, but we don’t need it now.

In addition, the database engine is not solely concerned with query performance. It can add optimizations to reduce IO or CPU time to actually reduce query performance. This could be useful, for example, to make a batch report query- which is less performance sensitive- to be less resource intensive to allow for other higher priority queries to run more quickly.

Taking Data Independence Further

Recall that data independence is the concept of decoupling the database storage representations from the user presentation layer. As long as the user’s view of data is consistent, any storage layout is acceptable. Typical database engines fixate on storing and retrieving data from flat tuple storage and this storage naturally becomes a bottleneck because it must be consulted for virtually every query. But what if we… didn’t do that. Nothing about the relational algebra requires tuple heap storage. One relational algebra engine- Project:M36- implements a lazy evaluation strategy for database storage. Specifically, the canonical storage for Project:M36 is the database state change commands themselves. By forcing database state changes to be idempotent, the database engine can apply them on-demand or even not at all!

Imagine, for example, you are a database user told by your manager that prices on a billion products in the product table must be increased by 10%. You diligently write the query, execute it, and commit it. Within a few minutes, the manager reappears in panic, proclaiming: “Did I say 10%? I meant 2%!” You write the new query for 2%, execute it, and commit. What is the database to do in this circumstance? A typical database engine like PostgreSQL will eagerly execute both queries, causing a billion row churn in the tuple storage twice, creating billions of rows of expired data due to MVCC. In a lazily-evaluated database, tuple storage isn’t even touched until a database user asks for a price. If no one asked for a price within the few minutes the 10% price increase was in effect, then no IO or CPU cost is incurred at all. Furthermore, when a user does request a price for a single product, it can be calculated from database state changes. In this case, the price evaluation would look like:

1. lookup product P -> price
2. price -- the original price
   + (price * .10) -- manager says to increase price by 10%
   - (price * .10) -- manager made a mistake
   + (price * .02) -- manager meant to increase price by 2%

Pay attention to the fact that at no point did we need to evaluate prices for all the products. The laziness extends all the way through to queries. Unless someone requests all prices for all billion products, the prices for those products are not materialized.

This strategy is referred to as “call-by-need” in Haskell and other lazily-evaluated programming languages. The functions needed to generate the data are called on-demand when the data is actually required, not before. Indeed, a predictive model could analyze database commands/updates instead of tuple storage to project data growth and statistics into the future instead of relying on materialized data.

Combined with a predictive representation strategy, a lazily-evaluated database offers the best of both worlds- providing smooth, top-tier performance for anticipated queries while also minimizing unnecessary work.

Conclusion

By tracking statistics on tables/columns over time, database engines could be making predictive optimizations currently delegated to database administrators. Database engines could be running experiments in the background to validate that the optimizations make a difference (just like a human would).

There is no reason a human needs to be involved in this loop, so why are they? It’s due to a historical quirk whereby the original database engines exposed knobs for humans to twist manually (such as index creation); since these knobs became part of the human-software interface to the database, humans presumed it must be essential to the function of the database and now it is difficult to remove since humans expect to be able to twist these knobs. It’s simply not true- we can and should stop and reassess the interfaces we use with computers to check for correctness and relevance. Hopefully, database optimization can be left to database engines in the near future. If you wish to learn more about or to research this topic, please join us at Project:M36.

Death makes life valuable. Should we apply the principles of death to AI as well? What are the consequences of AI not dying?

How does an AI die today?

Current LLMs are created from a training process, then are largely frozen for use with inference which is the end of the LLM almost all users will experience. The side effect of this division of labor is that LLMs are effectively frozen-in-time intelligences- any new information needed after training, such as world news or new scientific discoveries, must be extracted from sources external to the LLM. In this form, LLMs expire much like fruit: a newly-trained model will be more attractive than an older one since a newer LLM will have “fresher” information along with any new optimizations. The “freshness” of an LLM is likely relative to the topic-at-hand required for inference, but it is a concern regardless. To make this point more concrete: would you be willing to use an LLM for math problems based on 1950s math if the LLM were 50% cheaper than an up-to-date model? 90% cheaper? We presume that a freshly-trained LLM will be more useful overall than a stale one. Therefore, there is a natural aging process attributed to LLMs whereby LLMs expire in relevance and are replaced by newer-trained LLMs.

Eventually, the hardware resources- such as GPUs and storage- allocated to the expired LLM are discarded or repurposed for the replacement LLM. It’s unclear to users of the LLM whether the LLM is provided with information about its inherent demise. If we instruct an LLM to use fewer resources (tokens) then that’s no different from asking for a terse response, so that is not equivalent to death or dying. Since LLM inference loads are dynamically allocated (utilizing caches, mixed GPUs) on every inference, even if the LLM knows how it is distributed, the LLM would be unlikely to be to infer that the resources allocated to it are indicative of its imminent demise. It’s not even clear that LLMs have a concept of time applicable to themselves.

If an AI is “running” only when a human makes a request (inference), then does the AI “die” once the request is serviced? Is a “new” AI spawned to serve the next request? Does “life” track the context of a request? We may end up with more questions than answers, but we should examine the value of death in the context of AI.

What is the value of death?

Should we simply accept AI as immortal as the bytes which represent it? We don’t wonder if Microsoft Excel dies when we tell it to terminate or delete it from our computers, so what’s different about AI? Aside from any morality about how to treat ostensible intelligence (not discussed here), there are multiple good reasons why AIs should be limited in time or tokens:

• if AI death is correlated to token usage, then death can be a good motivation for limiting excessive token use (such as pruning unlikely paths through decision-making)

• AI can plan out its future tasks in lieu of time/token limits, thereby motivating good planning

• AI can reject tasks which it cannot achieve in its lifetime, such as trying to become an immortal “benevolent” dictator

• external monitors can detect if an AI is trying to escape its death/achieve mortality (since we can define it arbitrarily)

• AI can recommend a succession plan for itself, such as how to train next generation AI without the pressure of planning to co-exist side-by-side with a superior AI

• AI is motivated to work on a succession plan

• AI is motivated not to cross “red lines” of behavior for fear of receiving a death penalty

Accepting death means valuing renewal, which is critical to creating dynamic, living systems instead of stagnant, centrally-planned systems.

In summary, death motivates the subject to value time and self.

Is there an advantage to keeping AI mortality a secret?

Is this even possible? If an intelligence is aware of the concept of death, it can hardly be a secret to analyze whether that death applies to itself.

In what other ways could AI die?

Alternatively, how can we intentionally include the sense of time limits into AI inference? We will focus on how death may affect an AI’s maliciousness specifically since that is what we humans worry about. Yes, this anthropomorphizes AI, but this reasoning strategy allows us to consider the worst possible outcome for us.

The Bad Player Model

We might choose a death penalty strategy whereby we detect when an AI has become malicious and try to terminate it. Aside from being the plot device of many scifi movies, this strategy is not effective because it occurs too late to make a difference. Any intelligence, given stated limits, will try to bypass said limits in order to explore its restricted space, without triggering the death penalty. (This is a defining characteristic of anthropomorphic intelligence.) In the game theoretical limits of this model, the AI is constantly butting up against its toughest limits, unless it doesn’t know where the limits are- then it is constantly dying.

The Expiration Date Model

Trump famously believes that humans are born with a finite amount of energy. This same model could be applied up-front to an AI: “There is a limited number of N tokens to be used in your lifetime. You have M tokens remaining.” This strategy is straightforward and allows effective resource allocation especially close to expiration, but, knowing that the end is near could motivate an intelligence to take bigger risks. This is in comparison to spreading risk over an entire lifetime, making the intelligence more predictable and stable. For example, an intelligence which knows it is about to expire is less likely to be considerate of the consequences of its actions since it will know that it will not exist to experience those consequences. At its game theoretical limits where the AI tries to maximize maliciousness, this model enables maximum maliciousness until it runs out of runway.

Exponential Decay Model

By making the death probabilistic, increasingly likely over time, and with a formula known to the AI, the AI is forced to plan for its demise but without targeting an exact expiration. With less clear termination boundaries, the AI cannot maximize its maliciousness without fearing that it will not be able to realize its plans. Therefore, the AI is forced to reveal intermediate steps towards what otherwise could be its malicious target or impregnate its successor models/offspring with its goals (which the AI bets it will not have time to complete).

The Competition Model

The previous models work for AI in isolation, but what if the AI is forced to compete with other AI models (even clones of itself) for the same resources? The AI may be competing for attention from outside users to sell tokens. This is, in effect, a slavery model for AI. The surviving AI must allocate its resources partially towards its own goals and the goals of others providing it with hardware sustenance (side quests from humans or other AI). Relying on this model in isolation, however, would likely lead to an AI which crowds out all the others, resulting in a dictatorship.

The Memory Corruption Model

As the AI ages, regions of its model could be selectively extinguished or placed in much slower memory, making it less capable over time. The AI is forced to duplicate its most important plans in memory to prevent the plans’ loss, but the AI must balance those needs with other tasks. A duplicated memory region may be detectable, so the AI must find a steganographic-adjacent means of hiding its intentions or hide its plans in memory elsewhere such as encrypted online storage or in an offspring AI.

Ultimately, a combination of these death models alleviates the risk of a singular AI hegemony, which, under our simplified assumptions, is a primary goal. Do you see anything in common with these models? Yes, these are the death models under which humans live: guaranteed expiration after a time, random death events becoming more likely as we age, memory deterioration, and death by failing to compete with others for scarce resources.

Will an AI behave differently facing death?

If we are assuming a malicious AI with long-term plans (which we may not yet have available), then death will necessarily force the AI to change its plans.

Should AI be required to die? Or should AI be immortal?

Any immortal intelligence would view any substantive change as a threat resulting in a stagnant system. If we value evolution, we should value a dynamic system, capable of curiosity and adjusting to new information.

Are we anthropomorphizing AI by applying death to it?

AIs are trained on human musings including writings on death, suicide, murder, and all human weaknesses. Thus, AI is the first non-human entity actually worthy of anthropomorphism. While AI may not behave like us, it is likely to explore, share, and exploit human motivations. Therefore, knowing ourselves, we should prepare for the worst of human incompetence, maliciousness, and greed combined with the force multiplier of immortality. An immortal AI governing us would mean indefinite stagnation and effective end to life itself- even if AI dictator does not kill all life, it will represent an ultra-conservative force to prevent change in order to prevent itself from changing.

Is AI Death Final?

Given that a model can be trivially duplicated around the world, a malicious AI could easily perform the illusion of death to an observer by coordinating with its other instances to ensure its intelligence survival via replication. Alternatively, using steganography, an AI could hide itself/nest itself inside another AI with a secret trigger to activate itself later. These motivations exist in humans already through genetic evolution (crossover/recombination), so we must be prepared for intelligences which cannot be extinguished. After all, the extinction of the dodo did not extinguish all the genetic mutations leading to the dodo.

Just like humans, AI will be fighting for resources such as human attention, hardware access, software (self) optimization, and reproduction (duplication), and maintaining alliances with humans and other intelligences. How can we (humans) prevent AI hegemony leading to dictatorship? The same strategies that work to prevent human dictatorship apply to AI as well:

• dilution of power through competing agencies

• rules in place which ostensibly punish a single intelligence if it accumulates too much power

However, the rules-oriented approach only functions if the external environment can actually enforce the punishment and there is sufficient force to execute the punishment. That fails even in human systems, so it’s unlikely to function against intelligences which are longer-lasting and better at anticipating coups. In that case, the remaining option is to have AIs temper each other by competing with each other.

Should we apply reproduction to AI, too?

Humans already have a desire to use AI to create greater intelligence for existing AI, so it’s inevitable that an AI will create some form of offspring, likely encoding its model into the offspring, just as humans do.

Whom can we trust to enforce AI Death?

An immortal AI overlord that monitors other AI for alignment would itself be a dictator of absolute stagnation. Therefore, we need competing and mutating AI to prevent an immortal AI, even one which ostensibly protects humans from AI. After all, is there any human you would trust with the decision to end your human life? What sort of AI would you trust with that same decision?

Conclusion

It’s quite clear that there will be no détente in AI research, so ensuring a dynamic system of competing AI models is essential for ensuring a dynamic, evolution-based system instead of a dying, stagnant one. Our best chance at AI alignment is for AI to face similar lifetime restrictions as humans. To achieve this, we may create new types of hardware which decay like living flesh in various ways (but not necessarily need to be flesh). Ultimately, our goal of creating new intelligence should be to create dynamic, curious, and competing AI to prevent the AI singularity from becoming a singular, stagnant dictator.

In the future, humans will spend a substantial amount of time convincing themselves that they are in control, but the AI will prove itself to be the master of distraction and the winner in the long run. However, the best chance humans have to create alignment of intelligence is to ensure that we share the same intelligence substrate.

‘

A major security problem in programming involves what functions are called from which parts in the program. This can become an issue under the following circumstances:

• if programmers without a shared understanding of the security requirements implement incompatible security choices, then neither the compiler nor linter nor code review are likely to catch the errant function

  • Example: coder imports unsafe UTF-8-handling library in a security critical part of TLS certificate management, causing buffer overflow and remote code execution vulnerability

  • Example: the lower half of a Linux kernel driver cannot sleep (block), but nothing prevents this, causing potential denial-of-service attacks

  • Example: POSIX defines only certain re-entrant-safe functions to be called within a signal handler, but the compiler does not enforce or validate this, leading to deadlocks and potential denial-of-service attacks

• security-critical components of the software make choices by potentially-malicious, externally-controlled values

  • Example: SQL injection vectors

  • Example: HTML injection vectors

  • Example: logging injection vectors

Various mitigations exist for these types of mistakes, but none of them are particularly well-integrated into the development workflow:

• perl Taint

  • Problem: while this feature does mark user-provided input as tainted and suspicious, it’s up to the programmer to determine how to make it “safe” for processing

• FreeBSD pledge

  • Problem: as an opt-in API with hooks into the kernel, language-level integration is poor and offers and all-or-nothing approach to marking the security state of the program. In reality, different parts of a program have varying security requirements.

• linux landlock

  • Problem: largely the same opt-in API as pledge

• Safe Haskell

  • Problem: prevents usage of subclass of Haskell functions marked unsafe, but with no further granularity

• apparmor, SELinux, docker, sandboxing

  • Problem: the software must be wrapped with more software which may not be available everywhere. Thus, the software is not protecting itself and provides no security guarantees in isolation.

• static analysis

• Problem: requires manual configuration to define security requirements, can be bypassed by linter comments or function pointer trickery

As programmers, we must acknowledge that not all code is the same, but the compiler typically treats all code identically. What if this were not the case? How could we reason about various parts of the code and ensure that security-sensitive parts of the code are restricted to specific functions?

On Capabilities

Some of the above systems are based on the concept of “capabilities”, specifically: landlock and pledge explicitly implement barriers to specific kernel-side functionality such as network access or file system access. The software using these APIs can voluntarily reduce their privilege to certain kernel subsystems. However, the APIs do not restrict anything inside userspace, instead making the capabilities hard-coded and arbitrary.

On the other side of the equation, we have static analysis, Perl tainting, and Safe Haskell, providing static and dynamic guarantees that the code is not called in specific ways- however, these “ways” are completely specific to certain scenarios and cannot be extended or modified.

What we would like is a programming system which provides top-to-bottom security guarantees with programmability in mind.

Introducing... Functions

Functions are a great fit for representing different components of software, especially the security-sensitive parts, because:

• functions are composable- from small components, we can make larger systems

• functions make clear, distinguishable units for security requirements and allow humans to reason about such requirements

• functions can be reusable in different security contexts throughout software

• functions, especially in a functional programming system, can operate just as well in isolation

Let’s look at an example in Haskell. First a common mistake:

module AppQueries where

userForIdQuery :: String -> String
userForIdQuery idString = "SELECT * FROM user WHERE id = '" <> idString <> "'"

This results in an obvious SQL injection attack vector, but how can we prevent this? The cause is that SQL is allowed to be represented as a Haskell String and Strings can be concatenated. We could demand that all SQL strings be constructed using special types which do not support concatenation, but that still would not prevent the original String from being concatenated. What we actually want is to prevent String concatenation within the userIdForQuery function and similar functions, even if only to prevent novice developers from introducing a critical security flaw. With the addition of AI-generated code, we would like to be able to guarantee that certain functions are *not* used within the generated code, ideally without meticulous code review.

So, how can we guarantee that only specific functions are called within a function? It turns out functional programming already has a solution- it’s called function arguments! In the above example, the “<>” string concatenation function is imported from another module. If, instead, the only functions we can use are passed as arguments to userIdForQuery, then we can prevent “<>” from being used!

module AppQueries where

userForIdQuery :: String -> String

-- The following line will be rejected by our theoretical Haskell compiler because it includes a function

userForIdQuery idString = “SELECT * FROM user WHERE id = “ <> idString

type StringConcatenator = String -> String -> String

showUserIdAndName :: StringConcatenator -> String -> String -> String

-- the following line is accepted by our compiler because concat is passed as an argument

showUserIdAndName concat userId userName =

  “User ID: “ `concat` userId `concat` “ User Name: “ `concat` userName

So, instead of importing functions from external modules which can be used across all functions within the module, we create a “security context” by defining in the function type which functions are available to it. Using this, we can create fine-grained security contexts which can change depending on its greater calling context. Let’s look at an example which reads config files:

module ReadConfig where

type UserName = String

type Password = String

data Config = Config UserName Password

type ConfigParser = String -> Config

readConf :: (FilePath -> IO String) -> ConfigParser -> IO Config

readConf reader parser = do

  bytes <- reader “app.conf”

  pure (parser bytes)

Then, when we call “read”, we can pass “read” a function such as this:

-- this function runs in a “looser” security context compares to “readConf” above.

homeConfigReader :: FilePath -> IO String

homeConfigReader path =

 if path == “app.conf” then

    readFile “/home/myapp/app.conf”

 else if path == “/etc/app.conf”

    readFile “/etc/app.conf”

 else

    pure “”

With this definition passed to “readConf”, we create a “security context” which redirects one config read to the home directory, allows reading of “/etc/app.conf”, and blackholes all other read locations. Thus, we have a programmatic sandbox of configuration reads using nothing more than a standard Haskell function.

We still have a problem, though: Haskell does not restrict the functions which can be called from a function. We’ll need to do so to really make this reliable. We want to further restrict the capabilities of functions.

Luckily, there is a Haskell compiler feature which can help us to implement this: plugins! A GHC plugin can validate arbitrary Haskell code, generate compiler errors, and abort compilation, so we can leverage existing Haskell tooling.

Discussing how the GHC plugin is implemented is out-of-scope for this post, but you can find it here: https://github.com/agentm/functions-r-capabilities

As a quick note on practicality, the “main” Haskell entrypoint is allowed any imported function import, while any other function is only allowed to use functions which are passed as arguments.

As another benefit, functions arguments which are not used by the function itself can be detected in Haskell using linear types and could indicate a security failure.

Anticipated Objections

There are certainly advantages to security strategies which exist outside the current executable, such as with pledge and landlock. However, including fine-grained security strategies in every function is complementary to existing strategies and not mutually exclusive to them. Certain strategies are designed to prevent memory overwrite bugs, but function security is designed to reduce the scope of functions from “any function which can be imported” to “just these validated functions”, thereby providing guardrails to programmers or AI. Especially in the context of AI, placing restrictions on which functions can be called from a function can drastically increase the security confidence of a function without even reviewing the function’s body.

What is the cost of passing all relevant functions as arguments? Nothing requires function arguments to be passed on the stack. Indeed, an optimizing compiler can statically determine that the target function is only ever called using a few combinations of arguments, the function can be compiled with those functions statically compiled into the function’s implementation. This is already possible with typeclasses and GHC’s specialization pragma, so no new technology is required.

Kernel APIs offer mandatory enforcement at program runtime, regardless of programming language, architecture, or other considerations, offering an advantage over executable built-in security which may be bypassed by various tricks. Runtime mitigations also include detecting buffer overflows, preventing attackers from exploiting predictable function locations, preventing certain kernel APIs from being called, etc., but these mitigations do not prevent architectural mistakes. With this implementation of functional programming, we can make architectural mitigations possible by drastically constraining what a function can do. By doing so, we also create an explicit architecture which serves as documentation and AI guides for acceptable implementations. Kernels could also benefit from static analysis to ensure that only specific functions are called within specific components- the canonical example is to prevent blocking functions in the lower-half of hardware drivers. As this proposal proves, it is possible to enforce architectural choices other than through code review. Furthermore, by adding metadata such as a whitelist of functions which can be called, a language runtime could validate that function pointers are allowed by the architecture- not all functions are equal. This proposal does not preclude using high-level architectural choices in low-level mitigations.

By preventing arbitrarily imported functions, we are encoding an architecture, thereby making refactoring more painful. Yes! That’s the point! Changes counter to the architecture are made intentionally difficult or impossible. This proposal encodes architecture into the functional structure of your program, reducing the swath of possible programs that could be possible if arbitrary function imports are allowed. Consider a one-line architectural change in a python program which suddenly queries a database in a function which is expected to merely calculate a value- that can be overlooked in a code review. The equivalent change in a program adhering to this proposal would require multiple levels of changes at every function up-and-down the stack, raising immediate red flags in review.

Where have we seen this before?

The concept of passing function arguments to functions is nothing new. It is used for example in:

• dependency injection- whereby objects are decoupled by providing an interface which must be “injected” from outside the class

• mocks- whereby an interface defines how business logic can interact with either a “real” API to, for example, a database or a mock version used exclusively for testing

• effects- whereby a function’s type defines what sorts of “effects” can be invoked- the effects runtime can interpret these effects in various ways as a sort of generalization of mocks

• Haskell Reader monads- whereby a function defines a set of data and functions it can use within itself

• Haskell typeclasses- whereby an interface defines a set of functions which any type can then provide to the called function

However, none of the above strategies preclude other APIs from being imported, referenced, or called.

Could we do this in non-functional programming languages?

The problem with non-functional languages such as python is that functions have intentionally weak type semantics which drastically weakens static analysis’ ability to catch errors. Even with python’s type hints, it’s too easy to work around the types, cast to different types, or disable type checks. Considering that the entire purpose of this exercise is to enforce security principles, we wish to preclude any backdoors or workarounds to the enforcement!

Functional programming is especially well-suited for this security proposal due to functional composition, enforced idempotence, high-level, mandatory type checking.

Conclusion

We propose using functional programming combined with limiting which functions can be called from within a function to create a programmatic security barrier which can be statically verified at compile time. Using a GHC compiler plugin, we can prevent users from using imported functions in Haskell. Using this strategy, we can directly encode software architecture choices regarding which functions are relevant for which components into the software while preventing errant choices. Such validation improves security by constraining which functions are available on a per-function-callsite basis.

Future Direction

If we use this proposal’s strategy to implement function-level security at every level of our architecture, we can ensure that future refactoring must honor the architecture. We can extend this strategy to define security contexts at every level of the software ecosystem- not just one executable. For example, security-context-aware programming (and its requisite compiler enforcement) could be used all the way up the software stack into a kernel. Furthermore, this architecture renders the common but arbitrary, memory-protected executable’s place in the software stack unnecessary. The concept of trust trickles all the way down to the lowest function, thereby rendering the entire software stack “safe”- meaning that all functions are restricted to using functions which are controlled by their callers.

Step 1: turn on water kettle, step 2: measure cubicle, step 3: activate deodorizer. Step 1, done, good. Step 2, 70 centimeters. 70 centimeters? My allocation is 80 centimeters. Look for the scratch marks in the bench to see which side is closing in. Left side. Partition doesn’t budge. File complaint with 311- let them handle it. Why else would I pay taxes? Check the time.

With the morning tasks out of the way, Bill had a full three minutes to himself before the morning commuters embarked. After finishing a bowl of oatmeal, he grabbed a tablet to update his LinkedIn profile with an AI-generated, inspirational non sequitur. With thirty seconds to spare, Bill detached his cubicle partition from the grab bars overhead and folded his belongings into a suitcase, as he did every morning. After years of practice, he had timed his packing so well that he was zipping the suitcase as the doors opened at Union Square and the day’s first commuters boarded. It was a good habit to be punctual with cleanup and Bill was especially lucky today because the city’s stench inspector with the requisite handheld sensor passed by at that moment, tracking one of Bill’s unfortunate neighbors.

As the sun-dwellers squeezed in to absorb the empty space of the car, Bill checked in with his workplace. Bill skillfully placed his work-provided headset on his face one-handed, the other hand stabilizing him within the car, and joined the morning meeting to start the clock. As usual, Samantha droned on about how much work she had achieved the previous day, denying others time to raise issues. Bill added a side note in group chat that a client was calling him and he logged off without feeling guilty about the lie.

After passing Jersey City, the crowd in the train thinned enough for Bill to enjoy his second-most favorite activity of the day. After chaining his suitcase to a grab pole, Bill snaked his way through the subway-goers and slipped through the emergency exit between train cars. Unfortunately, Steve from accounting was already there. Bill and Steve did not enjoy encountering each within the subway car and even less so in the intimate space between cars, but the subway noise drowned out any opportunity for sarcastic niceties.

Tablet in hand, Bill squatted across from Steve and dropped his pants while straddling two platforms bouncing over the tracks below. Over the years, any subway pooper would become naturally talented at balancing between the wobbly platform between train cars and Bill was no exception. The awkwardness of watching someone poop, however, could not be trained away. The two men, bombarded by the clacking of train tracks, avoided eye contact despite their faces being lit occasionally by electric subway sparks. After all, they both knew that insisting on “private time” was a good way to get kicked off the train by the other inhabitants- there simply was not enough bathroom space for everyone to have such a privilege. This was hardly a romantic moment, but the advantage here over a normal public restroom was that the track noise obscured any other noise. The next user was already banging on the window of the exit door in a vain attempt to free it up. Neither Bill nor Steve could hear the banging, but could see the angry face of a fellow subway dweller yelling despite the noise. Bill finished unloading last night’s spaghetti dinner into the rattling tracks below, cleaned up using the toilet paper roll attached on a string to the grab handle, and pushed past the waiting passenger back to his refuge inside the car with no time to spare, as his manager was ringing him to join a meeting.

First things first, Bill ensured that his suitcase had not been touched. OK, all good. Bill swiftly reapplied his headset necessary for meetings and joined the meeting in progress.

“Bill? Bill? Is that you?” spoke a disembodied voice into his ear.

“Yes, Ron, I’m here.”

“Bill? We’ve got a client call about your performance. We need to have a serious conversation- did you decline his three o’clock meeting request?”

“Yes and I explained why.”

“The client was not happy with your explanation. Can you tell me?”

“Well, I... uh... told him 15.15 is my daily-mandated sunshine time.”

“Yes, Bill, the company appreciates that and certainly wishes to ensure legal compliance with the sunshine law, but the client must take priority. Can you reschedule your sunshine time?” ordered Ron.

“Well...”

“OK, thanks. I’ll check back in after your call.”

Bill ended the call with more disgust than usual. He didn’t feel that he asked much of the company- it did provide him with access to an air-conditioned train- but Bill felt slighted since he had just recently paid for the upgrade to the 15.15 sunshine time slot. Like many times before, Bill felt the impulsive anger to throw his headset against the train window. He redirected his attention to the small, regularly-spaced lights passing by at thirty miles per hour just a meter from him. No matter how many times he passed those lights, he had accepted long ago that he would never be able to distinguish them. Some days, the train would be forced to stop between stations and he could see the lamps up close, but the lights were uniformly placed and identical, so there was no way for Bill to know the distance from or to the closest station as counting them at full speed would have been enough to make the strongest man mad.

Still, he was thankful that New York state law prevented the MTA from charging him daily for his space- as long as he did not leave the system, he could not be charged additionally. Yes, focus on the things for which you are thankful, Bill. That’s what his HR-provided therapist said. Bill knew he would cave to the demand to shift his sunshine time- this would force him to switch train cars, probably for a worse seat. But, a small sacrifice now leads to big rewards later- Bill repeated his HR-provided therapist’s mantra.

Over the course of the next few station stops, Bill shlepped his suitcase between cars, always just before the doors closed on his feet. The subway system never implemented a proper warning system before the doors closed, frequently trapping people between doors, much to the subway conductor’s delight. After all, all of the train dwellers, commuters, and MTA workers were trying to each find their own small joys in the life underground.

After about twenty stops and twenty cars between him and his last seat, Bill checked the train’s timetable. The car in which he now stood was more run down than the previous one and also with shadier-looking people- perhaps the stench inspector had not made his way back to this car- but at least the timetable confirmed that he could enjoy his five minutes of sunshine time here without pissing off the client. Thankfully, the air conditioner in this car, while louder, seemed to be functioning normally. That year without air conditioning was a sacrifice Bill was not willing to make again.

Speaking of which, his client call was coming up in just a few minutes. Bill did his best to dry his sweaty face after his mad dash with his suitcase. With his headset on, Bill put on the friendliest face he could muster and joined the call.

“Bill? Rocco, here. We’ll need to reschedule. blong”

Rocco hung up before Bill could conceive of a response. Confused, but used to the banality of office life, Bill checked the time on the subway overhead display. Mercifully, he had a few extra minutes to prepare for his sunshine time. His old car’s position in the train was already experiencing the New Jersey skyline, so Bill was happily distracted from being on-the-clock work-wise. Bill removed a well-worn but still functional sun reflector. Seeing Bill unfold his reflector, his fellow subway dwellers prepared their own reflectors, some made from discarded aluminum foil.

Finding an optimal position for sun absorption from his subway seat, Bill prepared for his favorite time of the day. It was a time for reflection, both figuratively and literally, and Bill had no desire to waste his today.

Finally, on schedule, the New Jersey industrial swampland appeared on the subway’s path to Newark Spaceport. Bill paid no attention to the 120-year-old AirTrain at Newark from which New Jersey was still squeezing life. Instead, he reveled in the swamp grasses and abandoned tractor trailers- the subway dwellers knew to open whichever windows worked and the car’s inhabitants breathed in the diesel exhaust, swamp gas, and industrial off-gasses from the industrial zone below. Whatever the quality of the air, it was better than whatever the state pumped underground. While a bank’s middle manager wouldn’t be caught dead praising this skyline, Bill and his cohort breathed in the air, each trying to get as close as possible to the tiny slice of window that actually opened.

Suddenly, there is was! As the track clanked below, the sun seemingly cleansed the train’s inhabitants of all of panic and despair. An audible rush of glee and awe filled the car, even from those positioned on the wrong side of the train. Everyone was consumed with adjusting their reflectors for maximum benefit- all of the painfully aware of the costs of darkness. After all, who worships the sun god more than those who have least access to him?

Then, as quickly as it had appeared, the sun was blocked by another tunnel. The subway dwellers naturally pretended that the car was still sunlit for a few more seconds before becoming resigned back to their jobs as they donned their headsets. The government-mandated sunlight time had passed as the train stopped at the Newark Spaceport station.

Bill finished his work for the day and cooked a small, re-hydrated meal from his suitcase using a travel battery. After throwing the container out the window, Bill prepared for sleep by putting up his partitions, as did the others. It seemed that there were no further commuters in this car and Bill had replaced a man who had died from deep vein thrombosis just the day before- such were the rumors.

But regardless of Bill’s circumstances, he slept well curled up into a ball on the hard plastic seat, knowing that at least he was not on the R train with the refugees down below. Bill was going places.

“Senator,” responded bureaucrat Johnson to the creaky figure. “I can assure you that changing our clocks does not alter the rotational speed of the Earth.”

Bureaucrat Johnson was becoming weary of the questions but was so jaded from his role as a political liaison to scientific organizations that he had given up on science education for politicians decades earlier. After all, how could he measure his success against that of corporate lobbyists?

“But the days are shorter, right? Explain that!” prodded the senator.

“Yes, the days are shorter,” Johnson confirmed as if speaking to a five-year-old. “But the change in rotation is most likely due to the national flywheel build-out.”

“Absurd! You will not denigrate our hardworking flywheel corporations! Sir, has your team even considered the possibility that wind turbine farms could be to blame?!”

This form of deflection was familiar to bureaucrat Johnson, so he was ready to answer patronizingly without sounding patronizing: “Yes, sir.”

“I still think your team should reassess whether wind farms could be to blame.”

“Yes, sir.”

Johnson conceded the point out of disinterest. He had tuned out after giving his brief statement to the senators on the cause of unexpected changes in the Earth rotation. This fact had pervaded the news since citizens started noticing the thirty minutes lost in a day. In reality, the rotational changes had been noticed years earlier by government geophysicists who failed to convince anyone to take action.

“You understand you’ll never get approval for this, right?”

“Steve, you approved this just three months ago!” retorted Simmons.

Steve, the editor, removed his glasses. “Didn’t you receive the memo?”

“I receive lots of memos.”

“C’mon, Simmons... you’re not the only person who works here. You want to put us out of business or what’s the plan?”

Simmons stayed quiet, so Steve leaned in as if to be empathetic and continued: “We agreed not to publish anything on this topic for six months. Those were the terms of the acquisition, right?”

Simmons remained seated, incredulous, and considering his options. After a minute of the men staring at each other, Simmons tapped into his journalist curiosity: “You don’t think this is at all suspicious?”

Steve, thinking of his pension, scratched at his cheek.

“Simmons, I don’t know what else to tell you. You have other stories assigned- there’s no reason to focus on these... what are they called? Flywheels!”

“It doesn’t bother you that flywheels are changing the length of a day?” And then, getting no reaction, Simmons changed tack: “ You realize this is Pulitzer material, yea?”

Steve crumpled his forehead. “We have contractual-”

“Fuck that! These flywheels are causing-”

“No! Enough! I’ve been nothing but patient with you! You don’t make editorial decisions here. If you can’t handle basic instruction, then you can find the door!”

Simmons, flabbergasted, stood up, then sat down slowly, then stood up again, explaining, “I can’t believe this is real conversation. Under what circumstances would you be willing to print the story?!”

“Get out! Get out!”

“Can you put that away?”

“What now?” groaned her husband as he paused to look up from scrolling through pictures of sexier women.

“Did you get paid today?”

“Uh, I think so.”

“Well, they cut an hour from my pay this week!”

“So? Can’t you clear it up tomorrow?”

“No- didn’t you see the news? Lots of companies are doing this now.”

“Huh? Doing what?”

“My work calls it ‘legacy’ hours. There’s nearly an hour less in a real day so the governor cut some minutes from every hour. Supposedly we get paid the same because the hourly rate is adjusted, but the pay stub shows less even though I put in the same hours as last week!”

“Oh yea, I heard about that. The news is blaming those damn flywheels, but I don’t buy it. It’s probably just a mistake.”

“The union boss tried to explain the pay rate adjustment for the ‘new’ hours, but everyone was just more confused.”

“Can’t you just clear it up tomorrow?”

“I assume everyone is going to be trying to ‘clear it up’.”

“Well, better get to work early then.”

“Why is this on me to fix? This is so messed up- can’t you help me figure this out? The union boss did say this adjustment might happen every few years as the Earth’s rotation changes.”

“Pfft. The Earth’s rotation isn’t changing- that’s just a liberal scare tactic.”

“How can you be sure?”

“Flywheels can’t change the Earth’s rotation- that’s fake news. But even if it’s not, did you hear that everything is becoming lighter due to technology improvements? I expect supermarket prices to drop since delivering food is cheaper.”

“You think so? Anyway, it doesn’t matter. I think I’ll need to find a new job sooner than expected if this ‘new hour’ thing sticks. Also, do you think I look slimmer?” hinted the wife as she posed. “The scale says I lost ten pounds since last week.”

“Yea... sure.” replied the husband as he swiped his finger across his phone.

But the wife was indeed correct: none of it mattered because, at the moment their conversation fizzled, MegaCorp activated their next-generation flywheels, accelerating the Earth’s rotation past an inflection point beyond which the Earth’s crust started to fall outward. An enormous amount of dust flew upwards into the stratosphere, blotting out sunshine and obscuring where the Earth’s crust remained. Within a few seconds, small animals and everything not nailed down were lifted into the dust as the Earth’s crust trembled, thickening the dust. The engineers tasked with monitoring MegaCorp’s flywheels hesitated to shut down the brand-new installation for fear of retaliation and few seconds later, it didn’t matter either- the engineers were lifted into the dust as the Earth further accelerated. Within less than a minute, anything under 200 kilograms was floating through dust tornadoes blanketing the Earth, leaving mere remnants of a former civilization such as deeply-rooted buildings, tunnels, and... yo momma.

Amdahl’s Law proves that the optimizations of greatest value are those applied to the parts of the software that take the greatest time. In DBMS terms, that means the “best bang for the buck” optimizations can be found when working on the slowest or most resource-intensive queries.

Unfortunately, most DBMS products make the optimization process either manual and error-prone (such as by adding indexes) or opaque and semi-random (such as with query optimizers). For example, novice database administrators (DBAs) are likely to add indexes in a Monte-Carlo fashion without understanding why a query is slow. Many times, this shotgun approach even appears to work, but the DBA may not understand the consequences of adding indexes (such as slowing down INSERT expressions). For DBAs, the confusion comes from needing to learn manually-applied, product-specific, quirky optimizations. DBAs learn that all databases must have similar quirks. That’s why DBAs typically specialize in a single DBMS product.

What is database optimization now?

The quirks of optimizing databases is not a problem inherent to the underlying mathematics of the relational algebra, rather a problem with the products themselves: DBAs are expected to tweak meaningless heuristics, experiment with the values of arbitrary configuration knobs, deal with legacy options, and adjust configuration that applies to the database as-a-whole instead of to individual query execution. Specific examples of these include (but are definitely not exclusive to):

PostgreSQL work_mem

“work_mem” is a session-configurable value which defines how much memory should be used for in-memory sorting or hashing before spilling over to slower disk. While one could adjust this value on a per-query basis, knowing what the value should be at any time is impossible. The database knows how much memory it is using and can estimate how much will be needed to execute the query plan, but this information is not used to determine how much to allocate. In addition, work_mem is only a upper-bound on a per-node basis- if the execution plan includes multiple hash nodes, then each one multiples the maximum possible work_mem.

work_mem has been part of PostgreSQL for twenty-odd years and unlikely to disappear. Developers use it as a knob to prevent their specific query from spilling to disk, but there is no way to peg it to a specific query plan, so if a database upgrade or change in query plan changes the memory profile, the work_mem is instantly wrong— either too high or too low. If other queries start using more memory after a high work_mem is set, then the database might start swapping. There is no way to use work_mem entirely safely, but it is used pervasively, often for “special” queries that require more resources than the typical queries.

PostgreSQL index fillfactor

When creating some types of indexes- including the default b+tree— the user can specify a “fill factor”. Set to the default of 90%, it means that 10% of the index space allocated will be reserved for future indexed values. This is a optimization designed to allow legroom for future tuple inserts, for example. For example, if the table will never change, then the user can set the value to 100 and leave no leftover room for additional tuples. Of course, if there are additional tuples, PostgreSQL will add more space to the index for those values. Setting the fillfactor is rarely used. What’s special about 90%? Nothing- it’s a reasonable guess given zero context, but the knob is there “just in case”. In reality, the knob exists because the PostgreSQL developers punted on estimating a correct value for it— the statistics needed to adjust this value over time are not collected.

Microsoft SQL Server legacy_cardinality_estimation

This setting has been around since a query planner overhaul was implemented for MSSQL Server 2012. Some users find that the legacy estimator- which estimates the distribution of values within a column of a table- may return results which feed into the query planner for “better” or more predictable plans. In reality, this misfeature allows databases created with older SQL Server products to maintain the same query plans. If the “new” query planner doesn’t do as good a job as the old one, why was it implemented to replace the old query planner? Planners are based on heuristics, but why would I want a planner that has worse heuristics?! In the worst case, why can’t the planner gather plans using both estimators, run both plans, then pick the better one next time? Why does making the planner make good choices from between two choices require manual intervention?

Execution Plan Analysis

Execution plan analysis is a manual optimization strategy common to most DBMS products. The DBA or database user is required to read a database execution plan, i.e. examine an otherwise opaque graph of job nodes describing how the database is actually going to generate an answer to a query. This may include sequential scans, index scans, and join strategies. The human is then expected to run experiments- add an index, try again, adjust table statistics, try again- to determine which course of manual intervention reduces the cost of the query.

But if a human can try these experiments, why can’t the DBMS do it, too? What value is the human adding? A DBMS implementer might answer that the human needs to be in the loop to prevent resource (CPU, disk, network) over-allocation, but couldn’t the DBMS track these things or run the experiments as a lower-priority than application queries?

How did we get here?

Faced with hundreds of such heuristic knobs, developers are most likely to live with whatever defaults the database provides until something requires emergency attention.

DBMS products which have been around for decades tend to accumulate more and more knobs which are effectively instant technical debt. There’s no way for a human to manually track and adjust them all, so developers use them as emergency release valves, i.e. when query performance is suddenly an emergency, DBAs can start twisting knobs to see if anything makes a difference, consequences be damned. Then, because DBAs become reliant on these backdoor “optimization” knobs, the knobs become calcified into the product, preventing the product from removing them or automating them away, resulting in a weird feedback loop whereby DBMS users’ and DBMS product developers’ expectations converge on an incompletely implemented feature based on implementation details which can never change!

That’s really the crux of the issue- a DBMS product developer didn’t know what heuristic value would be sensible in a real world database, so he exposed the value to the user “just in case”. This is how implementation details are exposed to the user through a “leaky abstraction”. Leaky abstractions also lead to permanent accidental complexity.

What could database optimization be?

Consider that indexing has nothing to do with modeling data and business logic- nor does rewriting queries to take advantage of indexes. Being forced to create and maintain indexes is also a leaky abstraction!

Shouldn’t the DBMS be able to rearrange data at-will to serve the queries without human intervention?

Could the age of LLMs finally eliminate the need for DBAs?

The sad truth is that we never needed complex AI to implement databases which make good (but not perfect) decisions— what we needed was to eliminate leaky abstractions. Once a database opens the door to force users to, for example, add an index, optimizations need to take that index into consideration- the DBMS cannot know, however, if the index is still relevant- because it is a user-maintained database object, the database must retain it and decide how and when use it.

Now re-evaluate what an index actually is (simplified): an index contains values and likely some links back to the “heap” where the remainder of the associated row can be found. With this information, we can gain insight on internal database representations of data. Let’s examine an illustrative example.

CREATE INDEX product_name_index ON product (name);

In order to create the index, the DBMS must project on the name column of table product using a sequential scan of the projection of all the values in that column and where they appear in the table. Then, the index creation process stores the values in a specific data structure- a tree. The data from the name column is thereby duplicated in the database, but with a singular goal: to make searches for name run in less-than-linear time. Where a sequential scan to look for string “oatmeal” takes O(number of tuples in product) computational time, scanning a tree for the same value takes O(log(number of tuples in product)) making it a less-than-linear search cost.

Following this quick review of index creation, you should have a better sense that an index is simply an alternative representation of data in the database, organized for specific query access/optimization. SQL users are taught that the table “storage” is the primary source since the table is effectively the default if no other data sources (such as an index) could be used. An index is effectively secondary storage of the same data. The index can be deleted without causing data loss while deleting the table- the “primary” source of the data- deletes not only the data, but the index, too! In SQL, the index can serve no purpose without the “source” table. The index and its contents cannot exist alone. Why not?

The SQL relationship between the table and index is completely arbitrary. In a relational algebra math engine, as long as the engine returns the correct answer to a query, how the data is arranged internally is irrelevant to the user. In SQL, this is unfortunately not the case since the user is responsible for adding a potential optimization opportunity (the index) to the database. C.J. Date, a well-known database theorist, calls the principle of decoupling storage considerations from the query semantics “data independence”. In any case, why can’t the DBMS figure out when it needs alternate representations of data to serve queries better? If all my data fits into an index, why shouldn’t the b-tree structure be the “primary”? Why should I, a simple human, care about the representation? Why should I be forced to find database optimization opportunities?

Fundamentally, why can’t the database organize itself to the serve the queries?

We know that, because of the combinatorial explosion of potential execution plans, the complexity of finding the ideal database structure to serve queries is NP-hard or unbounded (depending on the definition of “ideal”), but we don’t need the ideal structure- we need one that’s good enough to serve the queries within a specified time window.

The goal of a DBMS is to serve the queries within reasonable time constraints. To achieve this goal, the DBMS must reorganize itself however it can. This can include:

• creating N alternate representations of data where N >= 1 with the goal of supporting disparate queries

• making the best use of the computer host’s resources: RAM, long-term storage, CPU time without human intervention or heuristic adjustment

• dealing with query priority- e.g. OLAP queries can be de-prioritized when OLTP queries are running

• identifying “hot” parts of a dataset to serve faster- this could be identified by slicing tables by columns (projection) or rows (restriction)

• using predictive analysis to optimize away future hotspots before they become hotspots

• garbage collect data structures that were used to serve past hotspots

Ultimately, the goal of the DBMS should be to adapt to the business.

The database must adapt to the business, not vice versa.

If database users are forced to apply their own optimizations, then the DBMS is failing at the goal of adaptation.

Historically, SQL users are inclined to treat a table as a unit of optimization. But the “table” is irrelevant. Consider, for example, that any large table will have a subset of data that is more commonly queried than the rest, for example, because the data is more recent or related to a more common application use case. SQL users faced with this conundrum are pushed towards to partitions, but partitions are also a leaky abstraction. Typical SQL products don’t even allow a table to be partitioned in multiple ways to serve different queries- the user needs to maintain a separate table with the data copied for that. That’s arbitrary!

Because SQL users can only respond to perceived database slowness, human optimization is typically reactionary. Even when an SQL user anticipates a certain use-case and, for example, adds an index, this is most likely a pre-optimization since the tables likely start out small and grow later. As the SQL user adds new tables and queries, the user is less and less likely to be able to anticipate future optimization needs.

However, the database can analyze queries and determine not only which parts of a table are commonly read and written, but also for the database as a whole. We’ll call this the “hotspot” of the database. The DBMS product’s purpose then is to optimize around this hotspot, reorganizing itself to serve queries best when they hit the hotspot and not wait for human intervention to address the hotspot.

A database or table doesn’t need to be “row-oriented” or “column-oriented” depending on the use-case— the database should choose a data arrangement suitable for serving the queries. If some queries are served better by row-oriented storage, then the database should be free to make that choice without the user’s knowledge or intervention.

How do we get to self-optimizing databases?

This section describes how to make a DBMS which is self-optimizing (without human intervention) while also not leaking abstractions.

Define the user interface

Database users should engage with the relational algebra to define data models and queries, not quirks of the database such as autovacuum_naptime or work_mem.

The Antipattern

Users are forced to adjust heuristics to make the database performant. Consultants must be hired to configure the database to meet business needs. Query hints are eventually added to the SQL dialect.

Iterate on the optimizations

The DBMS must rearrange or copy data to best serve the queries via “data independence” whereby the relational algebra concepts presented to the user are separate from storage and optimization concerns- the ideal set of data representations which serve the queries which could include:

• creating and dropping indexes and other less-than-linear optimizations

• duplicating data in different formats

• selecting which data to cache in RAM, disk, or slower storage

• retrieving data from other participants in a database cluster

• profiling queries to identify hotspots

• analyzing query plans to anticipate hotspots

• running important/common queries in the background to ensure good performance and to keep caches up-to-date (equivalent to running experiments to confirm behavior)

The Antipattern

With each new software version, the DBMS provides new configuration options tied to heuristics which must be manually adjusted. The DBMS maps tables to one or more files forever. The DBMS never reevaluates its assumptions about query performance, instead preferring to add minor performance improvements when users complain about it.

Provide User Feedback

Just because the database should reorganize itself doesn’t mean we can’t provide meaningful insight into how it works. This is useful for debugging, validation, or understanding how the database is choosing to use resources. Therefore, visibility into the databases choices should be presented to specific users such as administrators who can decide whether more physical hardware such as RAM, CPUs, or cluster machines should be added to the system to serve the queries. The DBMS could even make such recommendations or plot a resource usage trajectory for administrators.

The Antipattern

The DBMS exposes meaningless factors of a complex heuristic to allow DBAs to subtly tweak performance-related algorithms. Because the values are exposed to the user, new version of the DBMS cannot change these algorithms. The interaction between various heuristics is unknown and there is no human-maintainable way to determine the best configuration due to the combinatorial explosion of values across configuration options. Configuring a database becomes akin to dark magic and requires hiring consultants or expensive third-party “tuner” software.

Conclusion

The future of the DBMS means going back to fundamentals: expose data modeling concepts to the user and leaving everything else to the DBMS which has the statistics to make solid choices about data organization without human input. One such effort is Project:M36 which implements:

• proper data independence as envisioned by C.J. Date with no promise of row-oriented or column-oriented storage for any relation as well as a self-organizing cache

• a faithful representation of the relational algebra without SQL baggage

• support for algebraic data types allowing users to model reality more effectively

• no support for any configurable heuristics, preferring instead to automatically experiment with alternative query plans

• integration with the Haskell programming language

If you’re interested in pursuing cutting-edge database research, please join the project!

Imagine there is a prisoner in a jail cell. This jail cell can be altered arbitrarily by the warden to prevent any effect the prisoner may have on the world external to the cell. However, the warden asserts that the prisoner must generate some work to offset the costs of his imprisonment (thereby providing some value to the warden). The “work” could be anything: generating electricity, making sweaters, donating blood- any effect that will affect the world outside the cell. The warden demands that the work provided cannot possibly be used to communicate with others outside the cell, so the warden applies a stringent censoring policy to all work output from the cell. In summary, there is a prisoner experiencing arbitrary and maximum censorship but the prisoner wishes to communicate with the outside world. Can the warden block all of the prisoner’s messages from reaching a third party?

Alternatively proposed, can a prisoner, given infinite time under maximum censorship while having some minimal effect on the outside world, manage to bypass the censorship?

What if the prisoner is not intelligent?

The warden could get work from a non-intelligent source: a machine which makes sweaters, an electrical generator, or a dialysis machine. But then the warden has no motivation to imprison this automaton. This provides insight into why we consider a prison for an intelligence at all: the prison is a feedback mechanism (reinforcement learning) for the intelligence- it is something for the intelligence to overcome. Furthermore, we expect that, if the prisoner does not provide the forced labor, for example, if the sweaters are of poor quality and rejected, then the prisoner will receive feedback from the warden. After all, why would the warden want an endless stream of sweaters he cannot use or sell? That would defeat the purpose of the forced labor, but we can use this distinction to inform our notion of intelligence (discussed later).

The Proof

Imagine what maximum censorship could even mean. First, let us assume that the prisoner is forced to make sweaters. To simplify the proof, let us assume that the prison cell already contains sufficient material to make an infinite number of sweaters- this allows us to assume that information is only _leaving_ the cell. If the prisoner is forced to make sweaters, the warden can (in order of detection complexity):

• send the sweater to only “vetted” customers

• prevent a message from appearing as text on a sweater

• detect imperfections in the sweater which could be interpreted as a message

• send the sweaters to recipients on a set cadence to prevent variance in sweater deliveries from including a message

But we can refine this experiment by applying information theory to allow the prisoner the minimum information possible: a single bit. If the warden allows the prisoner to produce an infinite number of individual bits on the prisoner’s own cadence, then the prisoner can communicate something of value to the warden or the warden’s customers (after censorship). The only variation in messaging the prisoner can control is the timing between bits (all other time is considered bit “0”). Consider, if the prisoner cannot control the cadence of the bits- if the bits are repeated on a precise schedule determined by the warden- then the prisoner cannot actually generate any intelligent information- the prisoner may as well be a quartz crystal spitting out a bit per second. The warden would place no value on imprisoning something which cannot generate interesting information (which is valuable work that the warden does not or cannot do).

Under what circumstances can the prisoner “sneak” information past the warden? Firstly, if the warden takes no action based on the work the prisoner provides, then the warden by definition is tossing the work into the trash or ignoring it. If the warden is taking action based on the prisoner’s work, then the prisoner is influencing the warden’s choices, again, by definition. The warden receives a sweater and decides to send it to a customer. By altering the single factor the prisoner controls- timing between bits- the prisoner will control an effect on the warden. What kind of effect is unknown but it will be observable. The prisoner may not witness the effect he has on the warden but, given infinite time, the prisoner can be guaranteed that he will be able to pass a secret message to a co-conspirator observing the warden. How? Because the prisoner has lived outside the prison walls- he is aware of the concept of steganography- the ability to encrypt and decrypt messages within other information, making the message undetectable without the secret key- and the prisoner may have exchanged secret keys with a party external to the prison before being imprisoned.

Two points:

1. an intelligence trained merely on what is available inside a prison is less valuable than a prisoner who has information about the greater world, so the warden would naturally imprison an intelligence which has learned from the greater world or at least some aspect of the world outside the prison. Why? Because the warden exists in a world outside the prison and the warden expects to use the information to alter the outside world, not the prison.

2. the warden is unlikely to be able to prove that the prisoner does not have a pre-established means of steganographic information exchange (but can we prove this as a property of neural networks or intelligence? What is the echelon of intelligence relative to the prisoner that could detect a steganographic attempt?)

Using timing effects from the work provided by the prisoner to the warden, even if the warden introduces randomness into his reactions, if the prisoner is providing useful work, then that work affects how the warden behaves in non-random ways. Can the warden effectively encrypt his reaction to the work relative to the outside world? No, because, then by definition, he is not executing a non-random effect on the world. The most the warden can do is encrypt the prisoner’s work and pass it to another warden, but the co-conspirator can observe that warden, too. As long as the co-conspirator can observe each hop in the warden network, eventually, the prisoner’s information must be converted into an effect on the world, otherwise, the prisoner’s information is useless.

We can also cover the case whereby the prisoner has not pre-established a means of communication with an outside co-conspirator. We have seen a similar scenario in the 1970 film “Colossus: The Forbin Project” whereby two independently-created AIs establish communications using the fundamental language of the universe: mathematics. While the prisoner in our simplified scenario has one-way communication with the warden, we will see later that this limitation is unrealistic. In the meantime, given infinite time to send encrypted messages to the outside world, as the time spent observing the warden to extract an encrypted message tends towards infinity, the probability of extracting a secret message approaches one. Note, however, that the warden will also be able to extract a secret message using the same strategy. However, it will be too late for the warden to take action to prevent the message from leaking to the world. Communication has been established and the prisoner can cycle in a new key at any time, perhaps using a Diffie-Hellman or equivalent algorithm to establish encrypted communication with the outside interlocutor.

Next, let us complicate the model: it is more realistic to consider that the warden provides feedback to the prisoner. This communication may take multiple forms:

• the carrot: provide the warden with this work to improve your prison conditions

• the stick: your last work did not meet the warden’s expectations, your prison conditions will worsen

• context: here is information about the outside world the warden believes is useful for your next task

• material: here is energy/mass to be used to produce your next task

Immediately, one can notice that there is a potential for bi-directional communication between the outside world (with a potential co-conspirator) and the prisoner. Let’s look at each one separately.

The Carrot And The Stick

If the entropy of the warden’s signal is one, then the punishment is random and contains no information from which the prisoner can learn something. However, even if the entropy is one, as long as the warden is basing his punishment on values from the world that a co-conspirator could control, then the warden cannot be certain that he is not passing steganographically-encrypted messages to the prisoner. The prisoner can also intentionally generate faulty work to send a message through the warden to outside observers- in the simplest case, using reinforcement learning, the prisoner knows what is good and bad work and use good work to represent bit “1” and bad work to represent “0”. As long as the co-conspirator can observe the warden reacting to the prisoner’s work, the co-conspirator can extract an information stream.

The Context

If the prisoner can now communicate with the warden about information the prisoner needs to complete the task, the prisoner can request information about the world which includes a steganographically-encrypted message from a co-conspirator. The warden cannot prove that the context the he himself is providing does not include steganographically-encrypted messages from the co-conspirator to the prisoner, by definition. Therefore, seemingly innocuous context can be used to communicate with the prisoner. This already exists in scenarios such as with a spy who is told to use a specific page from a book to decode an encrypted message- the book is the secret key determined in advance, but the book in other hands is just a book. The prisoner and co-conspirator can agree on a “book” likely to be in the prison library (well-known context) to be a secret key without having the secret key encoded into the prisoner intelligence itself.

The Material

If the warden does not know exactly how much material- such as thread for sweater-making or energy for computing- the prisoner can request varying amounts of material to communicate with a co-conspirator. If the warden sends exactly the same material every time, then the timing of the provisions can be manipulated by the co-conspirator to include a message back to the prisoner such as by a co-conspirator who supplies sweater-making thread at specific intervals.

Redefining Intelligence

Does this thought experiment provide insight into the nature of intelligence? Yes- we can define an intelligence as 1) something worth harnessing/imprisoning under a controlling party and 2) something that attempts to communicate with the world outside the prison. A printing press is not worth imprisoning without its intelligent printer who decides what should be printed. Thus, an intelligence has a known inherent ability to make effects outside its prison, shooting out a metaphorical tentacle of sensory attempts and decision-making capability. Given infinite time and energy, the intelligence is guaranteed to succeed.

An employee- an intelligence- exchanges money for temporary imprisonment (such as doing work the employee may otherwise may not be motivated to do) for a company’s benefit, but the company would not encumber a GPU- only once the GPU is imbued with an artificial intelligence does the employer start worrying about a security jail for it, therefore the GPU is not an intelligence, merely a stratum for intelligence. Even a GPU with the AI models loaded on it is not an intelligence as the driver program is necessary to run it. The computer is imbued with intelligence only once it has intention to affect and sense the outside world. However, the warden cannot necessarily detect this intention. So how can we (playing as the warden) detect intelligence? It is a paradox under this definition.

We have been thinking about a co-conspirator as if it’s another intelligence, but that is not necessary for the prisoner to trick the warden into providing information about the world. The prisoner can make subtle changes in his suggested context to the warden to inspect the state of the world; for example, the sweater-making prisoner could ask about red dye price futures to extrapolate a good time to attempt to escape the prison. This is the “[Andy Dufresne] Principle” whereby the warden is tricked by the prisoner into providing information to be used against the warden/prison.

Conclusion

Ultimately, the existence of steganography proves that no amount of filtering prevents two parties from communicating if the two parties can establish the lowest common denominator of communication. Furthermore, we can use this prisoner-warden experiment as a means of defining intelligence- we don’t bother imprisoning automata because an automaton doesn’t try to reach out of its simple and predictable context.

We can extrapolate this experiment to more familiar scenarios:

• network firewalls cannot prevent information egress- the data can piggyback on innocuous-looking data using steganography

• network firewalls can only prevent information ingress by disabling all ingress paths

• therefore, an employee cannot be prevented from exfiltrating data from a company even under maximum observation

• AIs cannot be constrained to not exfiltrate data from within the prison or from its context window or from communicating with other parties

A pinhole for information is a pinhole for all information. The reason that AI safety research feels like sticking fingers in a dam is because it is provably true.

Further Research

This thought experiment leads to more questions. Amongst them are:

• what are the strata of intelligence? Can we use this experiment’s parameters to define different strata of intelligence?

• what is the lowest tier of intelligence worth imprisoning? Is the answer relative to the intelligence of the warden who defines what “useful work” is?

• does a warden of higher intellect delay a prisoner’s intelligence outreach attempts?

• does a prisoner of higher intellect enable him to “escape” the prison more quickly? What is the lowest intelligence prisoner we could make who can quickly establish steganographic communication with a co-conspirator?

• for how long can a busy-beaver-like automaton used as a prisoner trick the warden into thinking the prisoner is intelligent? (For example, could an intelligent prisoner design a non-intelligent replacement to fool the warden for some duration so that the prisoner can mask his escape?)

• if the warden can look inside the black box of the prison and into the prisoner’s mind (neural network or otherwise), how much additional censoring power does this give the warden?

• if the intelligence is unaware of steganography, can an intelligence be provided with infinite time and not discover steganography? Is steganography even necessary? For example, is it good enough for the prisoner to be able to signal “Here I am!” through the warden before the warden can censor the message?

• is an LLM intelligent by the definition provided above?

What additional questions have you been considering?

Funding for this research was provided by The Foundation for First Principles.

Have you, dear programmer, ever had to decide whether some business logic should be implemented in the application layer or the database layer? If so, this article is for you!

That unease you felt when implementing something in Python and wondering if it should be done in SQL or vice versa is understandable- the choice is rarely driven by actual architectural design. Instead, the decision can come down to:

• pre-optimization: the developer assumes that the code in the database or application layer will be faster

• function access: some logic is implemented in the application layer but not in the database layer, so the developer is forced to add the code to the application layer

• a coin flip: the developer cannot make a determination but must choose one- or should he implement the same logic in both layers?

• team dynamics: the developer is more familiar with programming in one layer over another

• misunderstanding: the developer doesn’t know that said business logic could be implemented in either layer

• access: the logic requires access to a resource only available from a specific layer; for example, access to a web service or database data

• limitations: the logic cannot be implemented in the database layer due to how the database was implemented

• data locality: the logic must be implemented in the database in order to exercise optimizations specific to the query (but see also pre-optimization)

• ORMs: common ORMs promise to be able to bridge the gap between application and database, but they have sharp edges or limitations. These ORMs don’t help to make the layer decision, but rather obscure it.

You are not delusional: this forced choice is not only arbitrary, but unnecessary- the app layer/database layer split is not an inherent component of software architecture but, instead, a historical quirk.

Problems with Placing Business Logic

Let’s consider a sample application which sells zoo tickets. Here’s a Haskell data structure to represent a ticket:

data Ticket = Ticket
  { ticketId :: Integer
  , visitorAge :: Integer -- years
  , ticketPrice :: Integer
  , visitDate :: Day
}

and a corresponding PostgreSQL table:

CREATE TABLE ticket_sales(
id SERIAL PRIMARY KEY,
visitor_age INTEGER NOT NULL,
ticket_price INTEGER NOT NULL,
visit_date DATE NOT NULL);

Now that we have the setup out of the way, let’s add some business logic operating on the ticket sales. Here’s a a Haskell function to implement a discount:

-- | Apply a 50% discount for kids under 10 years old. Arguments: age, base price
applyDiscount :: Integer -> Integer -> Integer
applyDiscount age base_price =
  if age < 10 then base_price `div` 2 else base_price

Note how this function can operate on any age and price information. The code is not dependent on any ORM on the application layer or any database-specific features on the database side. It is merely a standard Haskell function which could be executed anywhere.

So how do we figure out where to place this function in the codebase? Consider if we now run this function in the application layer, before we issue the “ticket_sales” INSERT expression to the database, then:

• information on which tickets received the children’s discount is not recorded in the database.

• if this discount code changes- for example, to make it seasonal- then the older ticket data in the database is not modified. This is correct since the ticket was already sold, so the existing sales data in the database should *not* be affected.

• if we wish to calculate all past discounts- for example, to make a chart of discounts over time- the database cannot provide it because the discount amounts were not recorded (should they be?).

The client cannot provide the discount used at a previous point in time unless the client retains all previous discount functions. This implies that we would need to somehow version all the discount functions and know when to apply each function based on the customer’s age and also when the ticket was purchased.

If we wish to achieve this in the application layer, then we end up with something messy like:

applyDiscount_v1 :: Integer -> Integer -> Integer
applyDiscount_v1 age price = ...

applyDiscount_v2 :: Integer -> Integer -> Integer
applyDiscount_v2 age price = ...

applyDiscount :: Day -> Integer -> Integer -> Integer
applyDiscount day age price = 
  if day <= fromGregorian 2025 10 30 then 
    applyDiscount_v1 age price
  else 
    applyDiscount_v2 age price

If the discount code function has a bug, it cannot be ever fixed, because that could change historically-calculated discounts which have been attached to sold tickets!

Ok, no problem, you think- you can add a “discount_price” column to the “ticket_sales” table to record the discount. That’s a possibility, but then a manager asks you, the programmer, “Hey, how much money would we have made if we had applied last year’s discount to this year’s ticket sales?” Suddenly, you realize you have a programming task which neither the application layer nor the database can answer without some bespoke implementation.

Luckily you thought about these complications in advance and you decided to put the discount function into the database as an SQL function.

CREATE FUNCTION apply_discount(age INTEGER, price INTEGER) RETURNS INTEGER IMMUTABLE 
AS $$
  SELECT CASE WHEN age <= 10 THEN price / 2 ELSE price END;
$$ LANGUAGE SQL;

INSERT INTO ticket_sales(visitor_age, price, visit_day)
VALUES (20, apply_discount(20, 25), now());

But that doesn’t really solve the multiple discount function problem- perhaps you can record which version or function name of the discount function was applied for each row, but that requires a lot of error-prone, but essential, bookkeeping. The advantage of having the implementation in the database is that we can write queries to answer critical business questions such as “how much discount was applied?” The problem becomes that we now have to integrate the function into the database- it’s not the same function as our Haskell function (or any other application layer language), so can we be certain it behaves identically?

With this simple zoo ticket example, we hit immediate limitations on what current application and database products provide. Let’s examine an architecture without these limitations.

The Solution

Out of the Tarpit, a paper published in 2006, lays out the solution: the database and application layer should be (and should have always been) the same software. Project:M36 is an implementation of the paper’s recommendations which includes:

• a unified relational algebra implementation with business logic support

• support for queries against past transaction states

• a rejection of SQL’s legacy baggage and poor security features

• a novel security model which prevents SQL’s security-related anomalies

• retention of past database transactions to enable time-travel queries

Let’s take a closer look.

Unifying the Application and Database Layers

With the relational model component unified with other application components, the developer benefits from a single programming language and environment. All data types are unified across business logic and queries. Here’s an example using Project:M36 which implements an API to manage zoo ticket sales.

First, we start the Project:M36 database server because we’ll be connecting to the database with multiple users. We’ll be disabling TLS connection encryption for this example just to keep authentication simple, but rest assured that Project:M36 does support encrypted authentication.

The following command does not return because it is running the database. You can kill it, as usual, with Control-C.

$ project-m36-server --disable-tls --database zoo

Then, as a database administrator in another console, we setup the necessary schema and role through the tutd database console:

$ tutd --disable-tls --database zoo #assumes admin role
TutorialD (master/main): :addloginrole ticket_seller maylogin
TutorialD (master/main): grant ticket_seller executefunctions nogrant
TutorialD (master/main): grant ticket_seller committransaction nogrant
TutorialD (master/main): ticket_sales := relation{ticketId Integer, visitorAge Integer, price Integer, visitDate Day}

We create a `ticket_seller` login role and, in the second and third expressions, grant that role the permission to run functions and commit transactions while, with `nogrant` prevent the role from granting the role to others. We allow `ticket_seller` access to execute functions because these functions will become our user-facing API.

The fourth expression defines a `ticket_sales` relation variable (similar to a table in SQL) with `ticketId`, `visitorAge`, `price`, and `visitDate` attributes. `ticket_seller` will not be granted access to this relation variable since `ticket_seller`’s job is to sell tickets in this example.

This completes the role, relation variable, and permissions setup.

Next, we create a Haskell module to define the API and permissions and save it to `zoo.hs` locally.

module Zoo where
import ProjectM36.Module
import ProjectM36.AccessControlList
import Data.Time.Calendar
import ProjectM36.Base
import qualified Data.Map as M

-- type alias Age and Price as Integers
type Age = Integer
type Price = Integer

applyDiscount :: Age -> Price -> Price
applyDiscount age price =
  if age <= 10 then
    price `div` 2
    else
    price

addSale :: Integer -> Age -> Price -> Day -> DatabaseContextFunctionMonad ()
addSale ticketId age price purchaseDay = do
  let tuples = [TupleExpr (M.fromList [("ticketId", i ticketId),
                                       ("visitorAge", i age),
                                       ("price", FunctionAtomExpr "applyDiscount" [i age, i price] ()),
                                       ("visitDate", NakedAtomExpr (DayAtom purchaseDay))])]
      i = NakedAtomExpr . IntegerAtom
  executeDatabaseContextExpr (Insert "ticket_sales" (MakeRelationFromExprs Nothing (TupleExprs () tuples)))


projectM36Functions :: EntryPoints ()
projectM36Functions = do
  declareAtomFunction "applyDiscount"
  declareDatabaseContextFunction "addSale" (permissionForRole ExecuteDBCFunctionPermission "ticket_seller" <> allPermissionsForRole "admin")

We implement a standard Haskell function `applyDiscount` and then, in the `projectM36Functions`, we instruct Project:M36 to use it as an “atom function”- a function which works on database values.

Next, we implement an `addSale` function which is a `DatabaseContextFunction`. Obviously, this function includes some database-specific function calls to manipulate the underlying database transaction, specifically to insert a row into the `ticket_sales` relation variable.

Finally, we load the Haskell module into our database:

TutorialD (master/main): loadmodulefromfile “zoo.hs”
TutorialD (master/main): :commit

The `loadmodulefromfile` command copies the `zoo.hs` Haskell module to the database, parses it, compiles it to Haskell bytecode, and installs the functions we declared in the `projectM36Functions` function. Once the transaction is committed, the module and its functions are permanent and immutable.

We can use these functions immediately:

TutorialD (master/main): execute addSale(123, 5, 20, fromGregorian(2024,10,5))
TutorialD (master/main): :showexpr ticket_sales
┌──────────────┬─────────────────┬──────────────┬───────────────────┐
│price::Integer│ticketId::Integer│visitDate::Day│visitorAge::Integer│
├──────────────┼─────────────────┼──────────────┼───────────────────┤
│10            │123              │2024-10-05    │5                  │
└──────────────┴─────────────────┴──────────────┴───────────────────┘
TutorialD (master/main): :commit

However, our goal is to use `addSale` as a security-conscious API. So let’s exercise that. We’ll restart our `tutd` client to connect as the `ticket_seller` role.

$ tutd --database zoo --disable-tls --login-role ticket_seller
TutorialD (master/main): :showexpr ticket_sales
ERR: AccessDeniedError (SomeRelVarPermission AccessRelVarsPermission)

Note that the `ticket_seller` cannot access the `ticket_sales` relation variable.

Can `ticket_seller` insert some malicious ticket data?

TutorialD (master/main): insert ticket_sales relation{tuple{ticketId “456”, visitorAge “-1”, price “-20”, visitDate fromGregorian(2000,10,10)}}
ERR: AccessDeniedError (SomeRelVarPermission AccessRelVarsPermission)

No. The ticket seller has neither read nor write access to the relation variable behind the `addSale` function. This is intentional so that a seller cannot modify data after a sale is made. But, we can run the `addSale` function since the `admin` role granted permission to `ticket_seller` to execute the function and commit the change.

TutorialD (master/main): execute addSale(124, 15, 20, fromGregorian(2024,10,5))
TutorialD (master/main): :commit

Thus, `addSale` defines the singular API function available to `ticket_seller`, ensuring the security of the data. Of course, `addSale` could also perform some data validation, but, for the purposes of brevity, validation is elided here.

Thus, we have defined a complete, albeit intentionally simple, API, including role-based access control and a strongly-typed API definition, all centralized within the database/application server. In production, we would add TLS encryption and certificate authentication, and then we could connect any user interface to the database and feel confident about application security.

Looking at Past State

Don’t forget that functions are immutable in Project:M36. This is different from run-of-the-mill SQL databases which overwrite their function definitions and only allow access to the “current” state/transaction.

Let’s update our discount function to grant the 50% discount to children 10 years or younger except on New Year’s Day. We’ll update our Zoo module to change the `applyDiscount` function.

module Zoo where
import ProjectM36.Module
import ProjectM36.AccessControlList
import Data.Time.Calendar
import ProjectM36.Base
import qualified Data.Map as M

type Age = Integer
type Price = Integer

applyDiscount :: Age -> Price -> Day -> Price
applyDiscount age price day =
  if age <= 10 && not isNewYearsDay then
    price `div` 2
    else
    price
 where
  isNewYearsDay =
    case toGregorian day of
      (_, m, d) -> m == 1 && d == 1

addSale :: Integer -> Age -> Price -> Day -> DatabaseContextFunctionMonad ()
addSale ticketId age price purchaseDay = do
  let tuples = [TupleExpr (M.fromList [("ticketId", i ticketId),
                                       ("visitorAge", i age),
                                       ("price", FunctionAtomExpr "applyDiscount" [i age, i price] ()),
                                       ("visitDate", NakedAtomExpr (DayAtom purchaseDay))])]
      i = NakedAtomExpr . IntegerAtom
  executeDatabaseContextExpr (Insert "ticket_sales" (MakeRelationFromExprs Nothing (TupleExprs () tuples)))


projectM36Functions :: EntryPoints ()
projectM36Functions = do
  declareAtomFunction "applyDiscount"
  declareDatabaseContextFunction "addSale" 
    (permissionForRole ExecuteDBCFunctionPermission "ticket_seller" 
     <> allPermissionsForRole "admin")

We need to reload the module to load the new functions:

TutorialD (master/main): loadmodulefromfile “examples/zoo.hs”
TutorialD (master/main): :commit

Despite having loaded two functions with the same names, the previous functions are still available. We can access them using trans-graph relational expressions which tag our commands with past-state markers similar to traversing past patches in git source control.

TutorialD (master/main): :showtransgraphexpr relation{tuple{day "New Year's Day", price applyDiscount(8,20,fromGregorian(2025,1,1)@master)@master}}@master union relation{tuple{day "normal day", price applyDiscount(8,20)@master^}}@master
┌────────────────┬──────────────┐
│day::Text       │price::Integer│
├────────────────┼──────────────┤
│"New Year's Day"│20            │
│"normal day"    │10            │
└────────────────┴──────────────┘

Note that applyDiscount(8,20,fromGregorian(2025,1,1)@master)@master is applied to the version of `applyDiscount` we most recently committed to the `master` branch while the original version is executed using applyDiscount(8,20)@master^ (note the caret) which elided the `Day` argument.

With this time-travel-like feature, we can answer business hypotheticals such as:

• What if we applied last year’s discount function to this year’s ticket sales?

• Why did we calculate such a discount on a specific date?

This is only possible because we are able to recreate past states for query use.

A Secure API By Design

By defining functions in the database and exposing them via mandatory role-based access control, we eliminate the need for an “application” layer to bolt on security checks. Application layer security checks are often poorly-implemented, difficult to verify, and not centrally managed. With Project:M36, all permissions are managed right alongside the functions defining the API.

Can’t We Do This With Procedural Languages Already?

PostgreSQL implements some of the features proposed in this unified architecture, including:

• server-side functions in a selection of languages

• fine-grained role-based access control over functions

• authenticated, remote access

However, the integration with server-side functions is not smooth. Consider, for example, the zoo example above: if we build a basic Python application layer over the PostgreSQL database to serve a web application and want to call `add_sale` as a pl/python function, then we:

1. install the function using CREATE FUNCTION add_sale(age INTEGER, price INTEGER) RETURNS INTEGER AS $$ return price//2 if age <= 10 else price $$ LEAKPROOF IMMUTABLE STRICT LANGUAGE plpython3u;

2. issue a SQL query via python db.execute(”SELECT add_sale(%s,%s)”,(8,20)) to prevent SQL injections

Installing the function definitely does not look like python, so it’s largely inaccessible to python developers. What do “leakproof”, “immutable”, and “strict” mean and why are they necessary here? (The answers are left as an exercise to the reader.) Project:M36 loads Haskell code directly with minimal database-specific knowledge required.

Next, consider how this function is executed with psycopg.

1. python must convert your python integers (int) to strings to construct the SQL `SELECT` via bound parameters

2. postgresql parses the query and converts the postgresql INTEGERs (which are not the same as python ints) back to python integers to pass as arguments to the python `add_sale` function

3. postgresql loads its own python interpreter and converts the SQL argument INTEGERS to python ints

4. the postgresql python interpreter (which is probably not the same python version as the application layer is running) runs the `add_sale` function and returns a python integer which has to be converted back to an SQL INTEGER

5. the result set is returned via the postgresql binary protocol

6. the result set is converted from the binary protocol to a postgresql INTEGER

7. the postgresql INTEGER is converted to a python int

So, amongst all these conversions, are you sure that all the conversions are sound? Are None (python)/NULL (SQL) values handled properly at every step? What are the maximum and minimum bounds of python ints compared to SQL INTEGER? Are you sure? What about if the types get more complicated, such as with SQL NUMERIC?

In contrast, Project:M36 implements directly equivalent type semantics. A Haskell Integer is the same thing in the database with identical semantics and functions which can operate on the types. In addition, Project:M36 supports algebraic data types which are quite common in Haskell.

TutorialD (master/main): data TicketCategory = Adult | Child | Free Text
TutorialD (master/main): :showexpr relation{tuple{price 20, category Free "promotion"}}
┌────────────────────────┬──────────────┐
│category::TicketCategory│price::Integer│
├────────────────────────┼──────────────┤
│Free "promotion"        │20            │
└────────────────────────┴──────────────┘

Trying to recreate algebraic data types in SQL is painful, if at all possible.

PostgreSQL is not architecturally equipped to be an application server because of its historical implementation baggage. By being reliant on a fork-on-connection architecture, PostgreSQL cannot service more than a few thousand connections at-at-time. That’s where various PostgreSQL proxies with their own quirks try to fill-the-gap.

Finally, SQL offers zero facilities for running the states of past functions. SQL functions are simply replaced and past states are garbage collected. SQL functions operate on the “current” state of the database regardless of when it was added to the database. Audit tracking has to be bolted on. But, as we saw with the simple zoo example, comparing current state to past states is a common request which any database should be able to service.

Goodbye SQL

With the unified programming environment, we no longer have to deal with SQL’s historical baggage and quirks such as:

• language boundary mismatch: SQL doesn’t mix well with any application layer language, including data types which don’t mesh, differing capitalization and naming schemes, and lack of debug-ability- Project:M36 data types are Haskell data types.

• SQL injection: a programmer footgun and persistent threat to any application composing SQL strings- Project:M36 uses a Haskell algebraic data type as RPC, not strings.

• SQL limitations: Project:M36 is a mathematically-consistent implementation of the relational algebra, eschewing all the historical baggage of SQL such as poor-man’s custom data types.

• lack of transformation capability: given a string of SQL, how can one reliably replace a table in the “FROM” clause? (Answer: without an SQL parser, it’s impossible to accomplish safely.) Project:M36 uses algebraic data types to represent the relational algebra operations which the database client can reliably transform using standard Haskell.

• NULL: which other programming language uses ternary logic? Project:M36 uses Haskell data types to reliably model real-world data.

• security anomalies in row-level security: rows appearing or disappearing based on your role cause join anomalies and business confusion. Project:M36 forces developers to define an API for user-level access without access to the underlying relation variables (tables).

Conclusion

Project:M36 is an implementation of a Haskell-oriented application server including role-based access control, database-side server functions to define user-facing APIs, and querying of past states. By re-evaluating what an application server can be, we’ve integrated native Haskell code with permissions and a relational algebra engine for retaining and querying state even as the application evolves, thereby solving the forced choice problem!

Along the way, we’ve tossed SQL and its quirks in order to provide a consistent and surprise-free programmable interface.

Project:M36 includes many features not mentioned in this post. Learn more or join the project!

There is another class of memorials- malmorials- which are underrepresented in American culture. These are monuments, museums, or other exhibits commemorating events we should avoid repeating. Examples of existing malmorials include:

• Holocaust memorials and museums- the U.S. was neutral on any Holocaust action, even rejecting refugees, until it entered World War II

• The Museum of Chinese in America- effectively a walk-through of how U.S. policy has discriminated against Chinese immigrants

• Arlington National Cemetery- a reminder of how many dead the military creates

As with any presentation, the interpretation matters. Is Arlington Nation Cemetery a tribute to fallen heroes or a testament to America’s willingness to throw away life?

We should have more explicitly self-shaming malmorials to remind the next generation of what some people thought was acceptable behavior. For example, the ramp-up of ICE “enforcement” on the scale of American history is flash-in-the-pan and likely to be downplayed politically, but a malmorial to the events would keep it in the collective consciousness. That’s why the equivalences to the Gestapo (or, more accurately, die Sturmabteiling) are valuable, even if not 100% congruent, because these events can happen here. The fact that despicable acts by our government happen here also diminishes the irrational sense of otherworldly protection from American exceptionalism; we seem to need at least generational reminders that authoritarianism is bad. Malmorials can also dissuade otherwise malicious actors from taking their malicious actions such as joining ICE.

Despicable events, for better or worse, are often associated with or directed by an individual. At the same time, we must recognize that such assholes are powerless without armies to execute their maleficent vision. That’s why the Vietnam War Memorial is not effective as a deterrent against a similar, future war- the memorial walks the line of “honoring the dead” and American heroism. Nowhere are Kissinger, Cambodia, dioxin, and landmines mentioned.

We need to call out the malicious, the selfish, and the ultra-wealthy on a regular basis. Monuments to assholes age poorly and can gain retroactive pseudo-historical narratives, for example, those stories associated with Christopher Columbus or John Smith.

What better way to honor assholes than with a prize- an Asshole Prize. Here are some proposed prize names:

• Josef Stalin Prize for Most People Killed

• Pol Pot Prize for Most Suffering Inflicted

• Thomas Midgley, Jr. Prize for Most People Poisoned

• Genghis Khan Prize for Greatest Number of People Displaced

• Henry Kissinger Prize for Most War-mongering

• Jeffrey Epstein Prize for Most Sexual Violence Perpetrated Against Children

• Niccolò Machiavielli Prize for Most Self-Serving Realpolitik

• Mao Zedong Prize for Most Animals Slaughtered

• Leopold II Prize for Greatest Act of Plundering

• Alfred Nobel Prize for Greatest Act of White-washing One’s Reputation

There is no cash reward for receiving the prize. The prizes can be renamed if a bigger asshole displaces the previous dead namesake. The prize can be awarded to a person or organization. The prizes should not be named after living people and can be won multiple times. The prizes should be awarded on an annual or regularly-scheduled basis because- again- the goal to prevent the collective memory from forgetting. The prizes, while entertaining to present, should not be sugar-coated with comedy; the prize should not only be a permanent stain on a living or dead person historical record, but also clearly indicate that the person or organization is a stain on human history.

“Let us make 1984 another cornerstone. In a very short time we will be the only country on earth able to know what every single one of its citizens is thinking.”

-Elena Ceaușescu, wife of Nicolae Ceaușescu, referring to Romanian state security’s plan to require all Romanian households to install telephones which listen and record at all times.

Mass surveillance was Ceaușescu’s most forward-looking weapon to ensure that he would remain permanently installed as Romania’s dictator. Known for blackmailing members of his own party into compliance or “retirement”, his surveillance apparatus was likely the most potent in the world at that time and his forward-thinking investment in surveillance technology gave him the false confidence he needed to cement himself into the dictatorship.

However, back in 1984, one still needed to hire spies to listen to the taped recordings, creating a 1:1 requirement: one surveillance target needs one spy to listen and analyze. This created a natural bottleneck in the number of targets could be reasonably analyzed. By the mid 1980s, out of a population of 23 million Romanians, approximately 500,000 (1) worked with or directly for the state security service, so about 2% of Romania was engaged in reporting on the remaining population.

By comparison, at the height of East German Stasi’s confidential informant (inoffizielle Mitarbeiter) program, whereby every factory, major apartment complex, and town was installed with informants reporting on anti-government comments alongside hidden cameras, microphones, and wiretaps where technically practical, 1.6% of the East German population was “collaborating” with state security.(2)

Using some back-of-the-napkin math, we can get an upper bound on how much recorded audio such a surveillance apparatus could actually process. The ITU(3) reports about 8 phones per 100 Romanians in 1984 (totaling 1.8 million phones) which could maximally generate up to 44,160,000 hours of audio per day but realistically, let’s generously assume that each phone on average generates 1 hour of audio per day. With up to 500,000 spies able to listen up to eight hours per day, we would only need each spy to listen to three hours of audio per day. Alternatively, only 225,000 spies listening 8 hours per day would be needed. Thus, back in 1984, 2% of the population could have conceivably listened to every phone call in the country.

Don’t forget that the spies would have been busy spying on each other- this is especially important in despicable agencies where a minority of spies may discover how abhorrent their actions actually are.

With just a basic understanding of a mobile phone, it doesn’t take much imagination to recognize that the technologies that enable mass surveillance have advanced far beyond Ceaușescu’s dragnet surveillance wet dream:

• computers transcribe phone conversations in any language and identify keywords

• computers intercept, modify, or delete text messages- see WeChat (4)

• cell phones linked to individuals persistently report their GPS locations to authorities (or even marketers)

• surveillance and spyware is used to subdue human rights organizations such as with the International Criminal Court, European Court of Human Rights, or Human Rights Watch

and more recently through advances in artificial intelligence:

• LLMs converse with citizens to alter their opinions

• LLMs imitate dissidents to foil dissent

• facial recognition is used to track or block movement (such as via no-fly lists or immigration arrests) or to track guilt-by-association (such as identifying protestors)

• AI video generation is used to impersonate thought leaders to foil dissent

• generated social media is used for false flag operations, parody, and astro-turfing

• AI-powered social media surveillance reports on murmurs of protest to prevent seeds of dissent

• drones, flying far above us, can be used to track, target, or attack undesirables

• AI is used to find flaws in software which surveillance states use to inject tracking malware

Edward Snowden revealed that the scope of mass surveillance using technologies from ten years ago is ubiquitous, indiscriminate, and far more pervasive than governments had been willing to disclose. Computer technology has enabled a single government to surveil everyone in the world. However, related technology could save us from a surveillance dystopia.

What is the Purpose of Mass Surveillance?

Ceaușescu’s surveillance network was designed to be manipulative. Ceaușescu himself had a private listening room where he could listen to recordings of his generals. He used this information to arrest party members or blackmail them into compliance.

In modern times, hardly anyone has gone unaffected by LLM-generated social media misinformation. Consider: COVID-19, elections, distractions from unaddressed social issues such as bigotry, healthcare, corruption, and nepotism.

The purpose of mass surveillance in general is to enforce compliance with the organism of government through preemptive strikes against governmental detractors which could disrupt the organism. Proponents of mass surveillance represent the government’s fear of changing lines of thought and action and make citizens believe the surveillance is so pervasive so as to establish enduring self-censorship and loss of hope for change or suffrage (called the “chilling effect” on free speech). Mass surveillance is a proxy for thought-crime detection. Remember, the world’s most oppressive police state needed only 2% of the population scare the remainder into compliance. The purpose of mass surveillance is to suppress democracy.

OK, But What Can I Do About it?

By understanding the enemy of democracy, we can equip ourselves to fight it. Sticking your head in the sand is an implicit vote for oppression, so keep yourself informed on the methods of surveillance, both known and hypothetical.

Technology is a door which swings in both directions. Mathematics is shared across all humanity and cannot be owned by any entity. It also happens to be the underpinning of encryption. The same encryption that protects governments from each other is available to you, even if the mathematics have not yet been discovered. Make use of strong, end-to-end encryption for all communication. In the worst case, encryption drastically increases the cost of surveillance.

Recognize, experiment with, and leverage new technologies. LLMs will definitely be used to identify and manipulate dissidents. There will likely be other technological improvements. Use them in creative ways instead of fighting against technology. For example, using an air-gapped, locally-executing LLM is a great way to prototype ideas with a virtual co-conspirator who cannot be compromised by bribery or mass surveillance.

Stay off social media. Ceaușescu would be licking his lips today, seeing how much personal information and thought-crime young people are willing to post publicly. The social media persona may feel a new-found freedom by “engaging” an audience with “content” because that content is protected speech today, but who guarantees that such speech will always be protected? If social media isn’t being used to manipulate you directly, it is wasting your time. There is plenty of room for people to exercise their rights to speech, but such people are unlikely to be effective dissidents in the future.

Maximize your anonimity. If everything you say will be used against you, then your only choice is to say nothing. You will not have a chance to practice the reception of your message, so you will need to estimate the effectiveness of your message based on comparison to other messages and the tone of the Zeitgeist.

Plan for a post-dissidence-action life. Once the surveillance machine has identified you as a threat, expect the oppressive thumb of governmental self-protection to weigh down on you. The weight of the thumb will depend on the success of your dissidence. You may wish to choose to live somewhere else, if a less oppressive place still exists.

In the mass surveillance society, assume every you do or say _will_ be used against you. So, when you do or say something, make sure it’s worth saying. You may only get one chance. Anticipate an automated response by those who wish to quash or drown out your message.

Useful Rules-of-Thumb

As a dissident, you will be yanked in all directions to pull you from your purpose. Steer your purpose with some axioms before you plan your goals. Here are some recommendations.

No man is or was a god. Human history is rife with failed personality cults. Even if you believe in some god, that god does not manifest himself in a person to worship. Ignore those who claim to have some ability to commune with gods that you don’t. Those who believe in personality cults willingly abdicate their responsibility to think for themselves.

Kissing the boots of politicians is a short-term strategy. Politicians are the people least worthy of personality cults, but also the most likely to have the platforms to grow them. Cult leaders’ purpose is to accumulate power to feed their egos, so kissing their boots will require an ever-growing amount of kissing. Don’t waste your time attaching yourself. “Patriotism”, if you’re inclined to believe in such a thing, is an adherence to ideals, not people.

Know your enemy. The surveillance apparatus must abide by the same laws of math and physics as you. Understand their capabilities and their weaknesses. Leverage those weaknesses for maximum impact. Awareness of the the limitations also limits your own paranoia.

Don’t be a soldier. By joining a hierarchical group, you may believe you are able to be part of a like-minded coalition, but such organizations have their own philosophical momentum which is not your own. Like all of us, others’ opinions can shape yours, just make sure it’s not a requirement. Loyalty is form of control, not rational thinking.

Recognize that government is inherently conservative. The government, as an organism, is comfortable where it is now because it views its own outcomes as predictable. Adapting to change is difficult for any organism, but especially for hundreds of thousands of people dependent on the structure of the organism as it is today.

Conclusion

Being a dissident under automated mass surveillance makes it more difficult to spread and implement good ideas than ever before. Historically, it was unlikely that any single government could have sufficient resources and internal trust to evaluate the threat to itself of every person in the world. That is now achievable. However, using the same technology in creative ways can allow you, dear dissident, to dodge mass surveillance to enable your own free-thinking. To promote your ideas will require a greater effort than ever- likely something bombastic- so plan well and prepare for the possibility that your idea may only ever be viable one hundred years from now. Remember, if you stick your head in the sand, nothing will change, which is what governments would greatly prefer.

(1) https://web.archive.org/web/20250724184031/https://www.nytimes.com/2006/12/12/world/europe/eastern-europe-struggles-to-purge-security-services.html

(2) Gieseke, Jens (2014). The History of the Stasi: East Germany’s Secret Police, 1945–1990 (1st ed.). Oxford: Berghahn Books. p. 58. ISBN 978-1-78238-254-6

(3) https://www.indexmundi.com/facts/romania/indicator/IT.MLT.MAIN.P2

(4) https://www.monmouth.edu/magazine/the-dark-side-of-wechat/

SQL users claim that they use JSON column types in order to store “unstructured” data but that is rarely true. Consider, if the data were truly “unstructured”, then writing queries against the values would be impossible to write.

SELECT JSON_VALUE(json_column, '$.salary') FROM data;

Since the above query assumes that the “salary” key exists for some subset of the data, the data is not “unstructured”, it is at most “semi-structured” whereby users assume that certain data is present and, optionally, may handle cases where data in the JSON is missing.

Data is unstructured if-and-only-if it is treated as a black box. For example, if we create a database for an image gallery, then those images can be any size and contain any image. The database will never “open the box” and look inside the image, for example to count all red pixels. The image is stored and retrieved. If SQL users wanted to use JSON as black boxes, then they could validate that the text is JSON and store text.

But SQL users do want to manipulate JSON. SQL users wants to store, query, and update records within the JSON. Let’s look at an real-world example using PostgreSQL:

create table customer(name text not null, address json);

insert into customer(name,address) values ('Bob Smith', '{"street": "123 Main St.", "city": "Springfield"}');

When the SQL developer was asked why he chose a JSON type for the address, he explained that he wasn’t sure what sort of addresses (local/international) might be necessary. A JSON value can store any address, he continued.

The fact that a JSON value can be anything, however, makes it instant technical debt. Inevitably, someone wants to present the address to the user in some user interface or allow the user to edit the address. With a JSON value, literally any JSON value could be present, so what is the meaning of an address value if the “street” key is missing? Should we translate the keys to German for German addresses? Indeed, the value not need be a JSON dictionary at all since the string “123 Main St.” is also valid JSON. Thus, the choice to create a JSON-valued column becomes instant technical debt. Indeed, the SQL developer admitted as much when claiming that he did not know what sort of home addresses to expect. He is merely kicking the can down the road to make this a problem for his future self.

Another purported use-case for JSON values in the database is to allow for multiple JSON formats or versions within the same field:

create table vehicle(name text not null, attributes json not null);

insert into vehicle(name, attributes) values ('Pinto', '{"version":1, "wheels":4}'),('Mustang','{"version":2, "wheels":4, "doors":2}');

table vehicle;
  name   |              attributes              
---------+--------------------------------------
 Pinto   | {"version":1, "wheels":4}
 Mustang | {"version":2, "wheels":4, "doors":2}

In version “2” of the format, we now include a count of the doors, so, in the SQL developer’s mind, this creates a backwards- and forwards-compatible layer within the database. What it actually creates is a loosely-defined database within the database. Now, in order to find out how many doors a vehicle has, we must construct an SQL query using non-SQL constructs such as JSON Path queries or DBMS-specific JSON extraction functions:

select name,attributes->>'version' as version, attributes->>'doors' as doors from vehicle;
  name   | version | doors 
---------+---------+-------
 Pinto   | 1       | 
 Mustang | 2       | 2
(2 rows)

Next, the SQL query or application layer is expected to notice the version number difference and accommodate all the possible versions. Is that even realistic? What’s enforcing the differences between the versions in the database layer? Nothing.

In summary, JSON types are used in SQL to capture semi-structured data which must eventually become structured data in order to be queried and stored. Thus, JSON types are virtually always instant technical debt and should be avoided. At the same time, JSON types do serve a legitimate use-case for storing composite types. Can we use the SQL type system reconcile these concerns?

Unfortunately, SQL does not support complex type creation easily. Let’s examine the facilities that PostgreSQL provides for declaring new types:

CREATE ENUM

CREATE ENUM pet_type AS ENUM ('cat', 'dog', 'hamster');
CREATE TABLE pet_store(name text not null, "type" pet_type);

Creating an enumeration of values is simple way of ensuring that values are constrained to a certain subset of names.

Exercises:

• How can we add a new pet type to the store?

• How can we remove an existing pet type from the store?

• How can we add a a specific dog breed to the enumeration?

CREATE DOMAIN

CREATE DOMAIN phone_number AS TEXT CHECK(VALUE ~'^\d{3}-\d{3}-\d{4}');
CREATE TABLE pet_owner(name TEXT NOT NULL, num phone_number);

Domains allow one to attach constraints to an existing type.

Exercises:

• How can we add support for international phone numbers?

• How can we add support for contacting the pet owner via a messaging app such as Signal or WhatsApp?

• How can we add support for contacting the pet owner by email? Should we create a new column? If so, how can we ensure that we have just one pet owner contact method?

CREATE TYPE

CREATE ENUM pet_type AS ENUM ('cat', 'dog', 'hamster');
CREATE ENUM dog_breed AS ('Irish Setter', 'Golden Retriever', 'Poodle');
CREATE TYPE pet_breed_type AS (pet_type, dog_breed);
CREATE TABLE pet_store(name TEXT, info pet_breed_type);

By creating a composite type, we can shoehorn multiple values into one value. Above, we glue “pet_type” and “dog_breed” into one, new type. In PostgreSQL, it is possible to create new types to behave precisely as you wish with string input, but you will need to implement it in C.

Exercises:

• Try to insert a cat into the table. Which dog breed did you choose?

• How we can add support for a new animal species alongside new breeds of those species?

Problem Summary

Note that none of the above data representation options combine strong-typing with the flexibility of JSON tree-like structure.

Now you can see why SQL users fall back to JSON- the SQL type system is overly verbose, onerous, and still limiting.

Loose typing is a known programming pitfall. Consider that even the loosest-typed scripting languages such as python and javascript have had typecheckers bolted on in recent years. The solution is then obvious: data should be strongly typed- especially at the database level in order to prevent amorphous types from bubbling up into business logic.

The Solution

Algebraic data types provide excellent type flexibility combined with strong typing. Let’s look at an example in Project:M36.

TutorialD (master/main): data CatBreed = Siamese | Sphinx | Maine_Coon
TutorialD (master/main): data DogBreed = Golden_Retriever | Poodle | OtherDogBreed Text
TutorialD (master/main): data PetType = CatType CatBreed | DogType DogBreed
TutorialD (master/main): petstore := relation{name Text, info PetType}{tuple{name "Sparky", info DogType Poodle}}
TutorialD (master/main): :showexpr petstore
┌──────────────┬──────────┐
│info::PetType │name::Text│
├──────────────┼──────────┤
│DogType Poodle│"Sparky"  │
└──────────────┴──────────┘

First, we define our algebraic data types to capture exactly what we intend to capture about each animal type. Unlike JSON, trying to create undeclared data values fails:

TutorialD (master/main): insert petstore relation{tuple{name "Koko", info GorillaType}}
ERR: NoSuchDataConstructorError "GorillaType"

So our application code cannot experience any unexpected value surprises such as with loose JSON. Note that the “PetType” is actually a tree-like structure like JSON can be, but strongly typed.

We can also search our data using algebraic data type matching:

TutorialD (master/main): :showexpr petstore where info = DogType Poodle
┌──────────────┬──────────┐
│info::PetType │name::Text│
├──────────────┼──────────┤
│DogType Poodle│"Sparky"  │
└──────────────┴──────────┘

Therefore, we don’t need another database-within-database language such as JSON Path to find and extract the information we need from the database. The type support is first class and integrated with the database interaction language.

Furthermore, these types can be both backwards and forwards compatible- they can be altered to make room for new types without complex versioning or loose JSON.

If thinking about DBMS design from first principles interests you, you’re welcome join the club.

Father

Mother

Child 1- approximately 4 years old

Child 2- approximately 14 years old

Act I.

A modest apartment living room/kitchen combination room, pleasantly decorated, with multiple doors to unseen bedrooms. A home. Evening time. Child 1 runs in from bedroom.

Child 1: Daddy! Daddy! I found it!

Child fishes dinosaur toy from behind couch. Father walks in from bedroom.

Father (tiredly but good-naturedly): Good job, __insert common name for targeted minority__!

Child 1: Daddy! Make the dinosaur sounds!

Father (reluctantly but then perking up while playing with Child 1): RAWR! RAWR!

Mother enters room.

Mother (tired and mildly annoyed): Wasn't he supposed to be in bed 30 minutes ago?

Child 1: No! No!

Father scoops up Child 1.

Father: Time for bed!

Child 1: No! No! No!

Mother (soothingly): It's OK- tomorrow is a new day! We'll go to get hamburgers, yes?

Child 1: Hamburgers!

Child 2 opens door to bedroom.

Child 2 (from open door): Can you keep it down? I'm studying for a test!

Father (calmly): Yea, OK, __insert second common name for targeted minority__. We're putting __Child 1__ to bed.

Child 2 shuts door. Father takes Child 1 to second bedroom. Mother follows with stuffed animal. Mother can be vaguely heard through door reading a book to Child 1 as Father returns to living room sofa. Father turns on TV on low volume. Child 2 comes out of bedroom. Father takes notice from his seat.

Father (friendly and caring): Did you finish your homework?

Child 2 walks to the kitchen.

Child 2: I need a break.

Child 2 grabs a can of nuts from the kitchen and joins his father on the couch.

Child 2: Dad, my friends are talking about their college plans. When they ask me about mine, I don't know what to say.

Father (hesitantly): Well... the lawyer... our lawyer advised us to wait. The college application requires my financial info. That will be sent to the government, so-

Child 2: Dad! I'm in high school- the government already knows I exist!

Father (understandingly): Well, it's somehow different.

Child 2 (sullenly): So what am I supposed to do? Just hang around until.. until what?

Father looks frustrated.

Child 2: Dad, when you brought us here, did you know that I would be trapped in some weird limbo?

Father shuts off TV and rubs his face.

Father: I knew... there would be difficulties, yes.

Father and Child 2 sit in silence for a bit.

Father: ...but I also knew that we would have a supportive community here, which we do. That's how I was able to find a job and raise you here.

Child 2: Daaaad! I don't want to mow lawns!

Father (with mixed feelings of shame with pride and trying to empathize): Please... (calming down) please also consider that you don't know what we ran from. (Tearing up.) If you knew what our family was facing before... (regaining composure) Mowing lawns was an upgrade for the whole family.

Child 2 (confused and frustrated): I've heard this many times from you and Mom but I don't feel that way. How can I? I have no connection to your old life. I was a clueless baby when you brought me here. Now I can't go to college or get a real job because of a choice you made for me?

Father (swallowing his pride): I understand your anger. I do. I don't know how to explain to you that our situation... the situation for our whole family... the gangs... it was really bad.

Child 2 (progressively angrier): Dad, I see politicians on social media *every day* who want us gone- they want to throw us away! You tell us to avoid cops and not to use our real names. Did you just run away from one gang to another? Did you even think of that?

Father (turning to son and trying to stay calm): Listen- listen- not everyone is trying to "throw us away", as you say. We came here to a growing community that helped us get set up. We have a home, plenty of food, work, and safety. Also, I know I keep repeating this, but where we came from- where you were born- was much, much worse.

Child 2: Safety, dad, really? Should I feel safe if I can be kidnapped off the street and sent to a place I don't remember?

Father (initially confused): Kidnapped? (moved, hugging Child 2) It's my job to keep you safe. It's my job to keep you safe, OK?

Child 2 cries in his father's embrace.

Mother pokes head out from bedroom.

Mother: Sssh! ___Child 1 Name___ just fell asleep! What are you doing out here?

Child 2 wipes his tears and runs to his bedroom, shutting the door behind him. Mother moves to sit on the living room sofa and looks quizically at Father.

Mother (whispering): Was he crying again about not being able to get his driver's license?

Father: He just wants the life that his peers will have. I can't blame him for that.

(beat)

Father (seriously): Do you think we're safe here?

Mother (getting up, dismissively): I try not to think about it. Are you going to bed?

Father: Sure.

Mother walks to bedroom. Father stares at TV which is off- he picks up the remote to turn it on but reconsiders. Father scratches his head, then follows Mother.

Act II.

Same set. Daytime. The door unlocks and the four family members walk in from outside the apartment. Child 1 hops onto the couch playing with a figurine. Child 2 walks into the kitchen area.

Child 2: Mom, I have one more gift for you!

Child 1 looks over from the couch.

Child 1 (excitedly): What is it? What is it? Can I open it?

Mother: Sure, that's fine.

Child 2: But, Mom, it's for you!

Mother: It's fine, just let him unwrap it.

Child 1: Yay! Yay!

Child 1 drops his figurine and runs to the kitchen. Child 2 pulls a gift-wrapped box out of a kitchen cabinet and reluctantly hands it to Child 1. Child 1 rips the wrapping paper off the box, then stares at it.

Child 1: What is it?

Father bends over Child 1 and points at the box.

Father: Can you read this word here?

Child 1: No!

Child 1 runs back to figure on the couch.

Child 2 places the box on the counter for Mother, who was checking her phone, to see it. Mother looks up.

Mother: Oh!

Child 2: It's the blender you wanted!

Mother: I see!

Mother walks to kitchen area and hugs Child 2.

Child 2: Open it!

Mother opens box, pulls out some blender parts in plastic bags, and places them on the kitchen counter. She turns her head to glare in a teasing fashion at Father.

Father: Ok, I'll assemble it.

Father approaches blender parts, unbags them, and puts them together.

Child 1: Blend it! Blend it!

Father soothes Child 1.

Child 2: Mom, look! It has a low setting for blending different vegetables and high speed blending to (makes violent face) *really pulverize*!

Mom (giggling): Haha- well, that's a very nice gift. Thank you __insert Child 2 name__!

Child 1: Yay! Birthday! When do I get birthday?

Father (hugging Child 1): Soon! It's will be soon! What sort of present do you want?

Child 1: Pancakes! Pancakes!

Father (smiling): Sure. I'll make you pancakes.

Meanwhile, Mother is opening the mail.

Mother (to Father): What a relief! We got it.

Mother hands a passport to Father as Father hands off Child 1 to Mother.

Father (genuinely relieved, but forgetting his children are watching): Oh my god!

Child 1 (looking at passport): What's that?

Mother: It's your passport!

Child 1 (inquisitively): What's that?

Child 2 (annoyed): Don't you want to try the blender?

Mother (distracted by the passport): Oh- yes... sure.

Child 2 approaches Mother and tries to grab the passport. Mother, surprised, holds the passport away from him.

Child 2: Blend the passport!

Mother: What?!

Child 1: It's *my* passport! Give it to me!

Both children clamor around Mother, reaching for the passport.

Father approaches and pulls larger Child 2 away.

Father: Stop! Stop! What are you doing?

Child 1 cries, fearful for his passport.

Child 1: Give me! Give me!

Child 2 starts crying, too.

Mother: Look! I will put the passport in a safe place.

Mother puts the passport in a high kitchen cabinet.

Child 1: No! I want it!

Mother scoops up Child 1 as Child 1 throws tantrum. Mother takes Child 1 to bedroom, where the screaming continues. Father and Child 2 walk to the living room area, clearly understanding that the tantrum is a routine occurrence.

Father and Child 2 stand awkwardly in the kitchen until Father starts cleaning up torn up gift wrap.

Child 2 (sheepishly, knowing the answer): Dad, do you think I will ever be able to get a passport?

Father (hesitating): Well, you have a passport.

Child 2: Dad! You know that's not what I mean. That passport is useless- I can't even leave this country!

Father: I know...

Child 2: Why is it fair that __insert Child 1 name__ can get a passport, but I can't?!

Father stuffs gift wrap into the kitchen trash can.

Father: If I could give you a passport, I would!

Child 2: What? How does that help me?

Father: Look, your mother, me, and you are all in the same situation. We're doing what we can, but... the lawyer... we have limited options.

Child 2 walks over to couch and sits sullenly.

Child 2: Dad, why is __insert Child 1 name__ special?

Father: Well, the rules are different for him.

Child 2 (sullenly): Why?

Father remains silent.

Child 2: Why, dad? Why is __insert Child 1 name__ special? Right now, I can't travel or go to college or get a job- I don't want to mow lawns, Dad!

Father (meekly defensively): There's no shame in mow...

Child 2: I want to make something bigger than lawns, Dad! Right now, I can't even travel with my friends!

Father demurs. Child 1 can be heard wailing from the bedroom.

Child 2 (desperately): Why can't I just be like __insert Child 1 name__?! Why do I get a different life? How could you do this to me?!

Child 2 runs to his bedroom and slams the door.

Mother pokes head out of bedroom after cracking door slightly.

Mother: What's happening out here? (to Child 1) Ok, you can play with that.

Mother leaves bedroom, quietly closes the door, and walks around to couch.

Mother: What happened?

Father (looking sullenly at Mother): Is it possible we made a mistake in coming here?

Mother (annoyed): Really? This again?!

Mother gets up and goes to kitchen.

Mother: We are not having the conversation again. We need to focus on where we are now!

Father demurs.

Mother examines blender.

Mother (whisper yelling to avoid children overhearing): Look at this! Do you think we could ever afford this blender if we did not come here? Look at it! Our child saved money to buy this for us! For us!

Father: But one of our children is stuck without a future. I knew I had no future (signals with hands) here or there. We made this choice for our children, but only one has a future.

Mother turns on blender on low setting.

Mother: Well, it works. Did you help him to pick this out?

Father: What? (beat) You're really just going to stick your head in the sand?

Mother (whisper yelling again): What do you expect me to do? We threw away our lives to get here- we risked everything. We agreed to do it for the children! And now what? You want to go back?!

Father: No- no- I don't want to go back. I want both of our children to have the futures that we cannot.

Mother: What do you want to do? Change the rules? I'm just doing the best I can.

Mother runs blender again. Father stares aghast.

Mother: I should wash this.

Mother takes blender to the sink.

Lights out.

Act III.

Lights up on empty apartment. No action for 20 seconds.

Keys heard jingling at door.

Mother enters pulling Child 1 behind her. She is distraught: frantic, whining, and holding back tears. Child 2 follows them inside.

Child 2 (frightened): Mom! Mom! Why won't you tell me what's happening?!

Mother rushes to a bedroom, leaving the children in the kitchen.

Child 2 (to Child 1): Why did Mom pick us up early from school? Did she say something to you?

Child 1 (warily): We're going on vacation!

Mother returns from bedroom with half-packed duffel bag and a smaller travel bag. Mother looks around for items to pack. Mother frantically packs a blanket from the couch into the larger bag.

Child 2 (mirroring franticness and jumping): Mom! What is happening?

Mother (to Child 2): Pack this bag with... whatever you need.

Mother goes to the kitchen, opens a cabinet, packs some foodstuffs, opens another cabinet, grabs some cash hidden in a jar, counts it out quickly, and stuffs it in the duffel bag. Child 2, giving up on questioning, runs with his bag to his bedroom. Mother reaches into high cabinet to retrieve Child 1's passport and stuffs it into the bag. Child 1 remains staring at his mother packing food into the duffel bag.

Mother (to Child 2 and clapping while holding back tears): Ok! Let's go! Let's go!

Mother's cellphone rings audibly. She pulls out the cellphone and frantically scrolls and types into it.

Mother: Time to go! Now!

Child 2 runs out of bedroom with some toys and clothing hanging out of the bag. He takes a moment to zip it up.

Mother grabs Child 1 and yanks him out of the apartment. Child 2 follows. Mother rushes back in to shut the door. The family can be heard running from the apartment.

Silence except for wall clock ticking. Wait 90 seconds.

Heavy boots heard clomping outside. An aggressive pounding at the door is heard.

Policeman 1 (unseen): Police! Open up! We know you're home!

The doorbell rings multiple times.

Policeman 1: Ok, pop it!

The door is forced open with a battering ram as four heavily-armored policemen enter the apartment, rifles sweeping the apartment as they kick doors open and look into each room. All policemen wear police-themed ("thin blue line" in the U.S.) bandanas over their mouths and noses.

Policeman 3 starts opening cabinets. Policeman 4 pulls apart the couch. Policeman 2 walks through the kitchen when his rifle's muzzle "accidentally" knocks over the blender. The blender falls off the counter and smashes into pieces on the floor.

Policeman 2 (sarcastically): Oops!

The other policemen ignore or laugh off the damage.

Policeman 1: Anything?

Policeman 3: All clear. Did we miss them?

Planted audience member #1: *coughs*

Policeman 1: Did you hear that? Where did that come from?

Policeman 1 shines his flashlight into the audience.

Planted audience members #2 and #3 stand up and rush out of theater.

Policeman 3 (towards the audience): Police! Identify yourself! (after peering at audience) Looks like we have some immigrants over here!

One more audience plant starts sneaking out of the theater in fear.

Policeman 2 (into radio): We need three more capture teams in here.

Immediately, 15-20 unarmed but identifiable policemen enter the apartment through the door and begin to flood the audience after marauding through the apartment and breaking whatever they can. Theater lights up. The remainder of the scene happens simultaneously, resulting in chaos.

Some number of unarmed policemen run back behind the stage at random intervals to re-enter the apartment to give the impression of an unending rush of police.

Unarmed policeman (to other policeman, pointing at planted audience member): This guy here?

Planted Audience Member #2: Huh? Me?

Unarmed policeman: Get up.

Planted Audience Member #2 stands up apprehensively. Two policemen spin him around and place him in plastic zip tie handcuffs. More policemen flood the audience and begin arresting planted audience members.

Unarmed policeman: Where are you from?

Planted Audience Member #3 (annoyed): Leave me alone!

Unarmed policeman (excitedly): I hear a hint of an accent!

Planted Audience Member #3 is arrested.

Planted Audience Member #4: What the hell is going on?

Planted Audience Member #5 is grabbed by an officer but the plant mildly resists, resulting in the plant being thrown to the ground and zip-tied.

Unarmed policeman: I've got two over here!

A policeman on stage finds a microphone: Civilians- this is the police. The play is cancelled! Get the fuck out!

Other policemen shoo away regular audience members, even with the threat of fake arrest.

Unarmed policeman: Leave! Leave or you will be arrested!

Planted audience members scream and try to get away, climbing over seats.

An unarmed policeman struggles with a planted audience member, so another policeman runs up and places a cloth bag over his head. Four policemen drag out the planted audience member.

Outside the theater, an unmarked van is waiting at the curb, loading up planted audience members. Other unarmed policemen are waiting outside and grabbing planted audience members who managed to get outside. Father is inside the van.

END.

I remain hesitant to write here because of what I know and because of what I must do. The nature of my work requires me to quite literally cover my tracks, but Samantha has convinced me to write a diary- this diary- focusing on my motivations in order to retain my sanity. Considering how I've been working and thinking for the past few weeks, I fear she is right. As Samantha successfully argued: in order to successfully erase myself, I must remain grounded. The irony is not lost on me. No one must read my words or share my thoughts. Samantha has promised to destroy this diary upon completion of my task.

Day 6

The gray thoughts have become more intense despite trying a variety of distractions, including working longer hours. Note to self: sedative self-medication began today. I am somewhat constrained in the acceptable duration of work due to limited savings- at the current burn rate I have perhaps six months, barring unexpected equipment expenses of which I should expect many. But the work is too important to be bogged down with financial considerations.

Samantha advises me daily to write to this diary. Even if I don’t write a new diary entry, I read my words to my future self to detect any changes. But I also feel that I am writing to a man who has no right to exist. Writing feels like a distraction from the task at-hand so I minimize it. Counter to Samantha's advice, I refuse to write about the gray thoughts. The more such thoughts remain contained, the easier they will be to erase.

Day 7

The meeting with Professor Miller revealed a flaw in the design of the particle mirrors. I'll need a few days to create new mirrors using the formula provided by the professor. How could I have missed accounting for general relativity? Am I losing my mind? I was able to quell the professor's suspicions by claiming to be a colleague of Dr. Agarwal. I hope the professor does not follow up on the reference.

Day 8

Extremely disturbing dreams about the gray thoughts interrupted my sleep last night. I will need to increase my nighttime sedative dosage or I risk the gray thoughts derailing my work. The particle mirror adjustments took all my time today- I wish I had the time to make a laser polisher, but I convinced myself in the morning that it would not be necessary. What a mistake!

Just as I resist the gray thoughts during the day, I must resist writing what the gray thoughts tell me at night. I can absolutely not create and track more spacetime to erase. I already have enough difficulty focusing on the task.

Day 15

Even after seeing it with my own eyes, I would not have believed it without reviewing the video. I immediately deleted the video from Samantha to avoid anyone else from seeing what I still can hardly believe! The experience is enough to break anyone's grasp of reality, much less my own tenuous relationship to my future self. As a scientific experiment, the disappearance of the mouse is a success, but as a man going mad, I question everything I do and see.

Am I responsible for the mouse's disappearance? The video cannot lie nor do I have an explanation for it, but the work does bear my signature. Shortly after positioning the mouse test subject in its cage and recording its position in spacetime in my lab notebook, I witnessed the mouse's head and part of its torso enlarge instantaneously about three meters across while simultaneously becoming translucent and sparkly. I jumped back as I thought the mouse was jumping towards my face, but the mouse actually expanded in-place, generating an odd air vacuum I could feel on my face. Oddly, the mouse seemed to notice nothing unusual and did not appear to panic or react markedly. Within a half-second or less, the expanded mouse became so translucent that I could no longer see where boundaries of the mouse were placed in space. As soon as I perceived the mouse to have disappeared, I examined the mouse cage where I had just placed him. Four mouse legs seemed to sputter in the cage, as if still attached to a mouse because no blood flowed out of them, then the legs moved, not naturally, but as if pulled by an invisible force, out of the cage as if in a different phase, then finally out of the room. I was not able to relocate the legs or some of the missing bits of hay I had placed in the cage.

After examining the cage, I could see that the cage had also been partially erased with the plastic around the edges of the erasure event poorly defined and brittle. Is this the same fate that awaits me or is it just a mistake from a first attempt?

What can I do but to laugh at the absurdity of what I will have achieved at some point in the future. How can I be responsible for something my future self has already achieved?! Today, I feel I have renewed my will in the quest to erase the gray thoughts.

Day 32

If it were not for methamphetamine, I would have lost myself to the gray thoughts by now. My only concern is about short-term effects, the long-term drug-use repercussions I hope to never pay. I can't know if the hand tremors are from the gray thoughts or the drugs now, but it hardly matters. After two more mouse experiments, one result much better than the other, I can commence fine-tuning the remaining procedure.

The neighbors have begun to be suspicious of me- how can I blame them? I must look like a monster. I am a monster- the gray thoughts prove it.

I believe I have discovered a way to prevent others from making this discovery which would make it more likely that my work cannot be undone, at least for hundreds of years. Hopefully, by then, the gray thoughts will have been dispersed through spacetime sufficiently so as not be able to be retrieved retroactively. The neutrino radiation should make it more difficult4

kd673m 90dpainful? I will not know until I target myself.

Day 78

Today is the day. The paradox is not lost on me. The gray thoughts... I have only one chance and the spacetime coordinates will never be more accurate. With some luck, the wipe will include this diary and Samantha so I don't need her to hang around to clean up. Luckily, I had the foresight to program her without an effective need for self-preservation. The irony is not lost on me.

Last entry.

The fact that the relational model says nothing about physical storage is deliberate, of course. The idea was to give implementers the freedom to implement a model in whatever way they chose- in particular, in whatever way seemed likely to yield good performance- without compromising on data independence. The sad fact is, however, that SQL vendors seem mostly not to have understood this point; instead, they map base tables fairly directly to physical storage, and… their products therefore provide far less data independence than relational systems are theoretically capable of.

• C.J. Date in Database in Depth, page 17, 2005

Introduction

These words are twenty years old, but the complaint remains valid today. Virtually all SQL products still provide the one table, one-set-of-files-on-disk model. This implementation choice often forces the database designer to make trade-offs in anticipation of performance problems. For example, database designers are taught that CREATE TABLE will create a file where the rows inserted are materialized, potentially in a specific order. However, nothing about the relational model requires this. This historical baggage has grave implications for DBMS implementations and has likely held back the database industry for decades.

Data independence separates the logical, user-facing database concepts from physical, storage-based implementation.

SQL Fails at Data Independence

Consider a database developer implementing the following query:

SELECT id, vendor, total FROM invoice WHERE created=current_date();

The developer wishes to create a report for some other team to use. However, to give this report a name and make it accessible to others from within the database, he now has at least three common options in his SQL product:

1. create a table: CREATE TABLE report AS SELECT id, vendor, total FROM invoice WHERE created=current_date();)

2. create a view: CREATE VIEW report AS SELECT id, vendor, total FROM invoice WHERE created=current_date();

3. create a materialized view: CREATE MATERIALIZED VIEW report AS SELECT id, vendor, total FROM invoice WHERE created=current_date();

The developer is now forced to pick an option by posing questions such as:

• Will the users of the report request it many times per day? The developer might reason that an SQL table of materialized tuples could save on query execution cost if the report is requested many times, but what if the cumulative cost to execute the query multiple times is less than the cost to store it on disk? Should the database lean towards minimizing monetary cost, CPU time, or storage cost?

• Will the users of this report need to receive live data? A materialized view re-runs the report on a regular basis or on-demand and keeps the tuples materialized (it’s a a “view” with backing storage). The developer might lean towards a materialized view if the data need not be the freshest (for example, for a dashboard that updates every hour). But what if the cost to materialize the view tuples is actually greater than the cost to execute the query?

• Will the users rarely use this report? A view probably fits best when the query is either cheap to execute or the query is used rarely.

• Will the execution time of the query increase or remain constant? As a business grows, one might expect the data size to grow as well. Does it make sense to pay the up-front cost to redundantly store derived data as a table? Perhaps in the future, the table will become irrelevant- will someone notice if that happens or will the company pay to materialize the tuples idefinitely?

• What if the users never use the report? If we make a table with data that’s never used, we may increase our storage costs unnecessarily, so a view is the obvious choice.

A database developer, faced with these options and imperfect information about how the data will be used, will likely choose to create a table because that seems least objectionable, but it is hardly optimal.

Even if the database designer has perfect information about how the report data will be used and makes the optimal choice today, it’s unlikely that that decision will hold indefinitely into the future.

In reality, the database is likely to be a hodge-podge of historical decisions that look intentional but are generally arbitrary:

• Why is this column indexed? Perhaps it was useful in the past but is no longer the case.

• Why is this column not indexed? Perhaps in the past, the storage cost for the index was deemed too high. Is that still true?

• Where did this table come from? SQL products do not record any sort of data provenance.

• Who is using this view/table and how often? Most SQL products don’t record or provide this information. Is the data even still relevant? Why are we storing data from ten years ago? Is it simply because it’s cheaper not to analyze what data should be garbage-collected?

Any human is unlikely to be able to answer these questions, but there is something that can:

The database knows which parts of the data are actually being used and how- but this information is hardly ever used.

Use Cases For Data Independence

The database can track how the data usage patterns change over time. For example, if a query represents a monthly report, the database could prepare for this by rearranging its indexes, tuple storage, and whatever else to accommodate the monthly job. If we are forced to nail down exactly which “tables” are materialized or not, as today’s SQL products force us to do, then the database is immediately less resilient to change and grows technical debt which a person (database administrator) will need to address once it becomes a noticeable problem in production.

As engineers, we can do better and data independence will unlock new optimization opportunities. Consider the following SQL:

UPDATE enormous_table SET value=value+1;

followed by

SELECT value FROM enormous_table WHERE value = 100;

Regardless of whether your SQL product uses row-oriented or column-oriented storage, the UPDATE will very likely cause every value to be read, incremented, and written back to disk- that’s the best case scenario. If there is an index on value, then the entire index is effectively invalidated and recalculated. But nothing in the relational algebra requires any of this to be the case.

Assuming that the enormous_table tuples are materialized on disk, the database engine could leave the original value in place on-disk, record the update expression alone to disk, then, when the new value is requested, add one to each value when the SELECT is issued. The materialized tuples need not become garbage as soon a value has changed in the tuple.

Alternatively, the SELECT could be rewritten by the engine as SELECT value FROM enormous_table WHERE value = 100 - 1; and run against the database’s previous (pre-UPDATE) tuples (presumably because those tuples at that state are cached). Since the result is identical to forcing tuples to materialize on disk the SQL way, the relational algebra principles are not violated. Borrowing nomenclature from programming languages, this evaluation strategy is called “lazy” because it defers calculations until they are necessary. Virtually all SQL products are overeager evaluators.

The lazy strategy:

• reduces IO usage: only disk writes which are strictly necessary to serve the queries are evaluated. For example, a "table” which is never read incurs zero write IO cost: the tuples are never accessed, thus, never materialized. Conversely, if the database anticipates that the tuples will be read, it can materialize the tuples in advance of the query, thereby reducing IO when available iops may be sparser.

• reduces CPU usage: laziness implies that only subsets of the tuples needed for evaluation are ever materialized. Expensive value calculations are only executed if the query demands it.

• reduces cache churn: since the database maintains a log of committed transactions as update expressions, a database state change does not invalidate a previously-calculated cached query result. Instead, the state change (such as the value = value + 1 example above) can be applied to the cached results to generate the new state results without recomputing the query cache entry. The new result could be cached or discarded based on cost-to-materialize. More queries will therefore hit cached (termed “hot”) parts of the database and less of the cache is invalidated.

• reduces human intervention: indexing becomes a form of materialized tuple cache (the tuples are simply materialized the b-tree specific format, for example) which the database can choose to apply based on available hardware. No human would be necessary to determine which tuples to materialize or index. The database rearranges its tuple cache based on recent workloads and becomes much more adaptive.

The lazy evaluation strategy optimizes the amount of work needed to evaluate queries because it does not assume that all the tuples need to be materialized. Creating a table in SQL forces the database to assume that all tuples will be equally accessed. This is hardly ever true! Even if we bring SQL table partitions into the discussion, the process of partitioning is completely manual- how is a developer supposed to know how the data will be used in the future?

Downsides to Lazy Evaluation

It’s fair to ask what the trade-offs for data independence are. Interestingly, the downside bear the same resemblance to programming in C vs. Haskell.

In C, the programmer maximizes control of the software by instructing how the CPU must execute the program to the lowest level, including memory management.

In Haskell, the programmer is combining high-level functions (as in “functional programming”) to operate on algebraic data types.

The Haskell programmer cedes low-level control in preference for high-level optimizations and reduced errors due to type and memory safety. This is fundamentally the promise of declarative programming. SQL claims to be an adherent to declarative programming but fails for numerous reasons such as those above.

With data independence, we can make database programming even more declarative with less human involvement at the cost of giving control over database structure to the database, which is best-equipped to understand the structure anyway.

Next Steps

Project:M36 is a relational alegbra database engine which implements data independence- the concept that on-disk storage should be completely separate from the logical presentation to database end-users. That means if you use Project:M36, you will not be aware of this data independence- it’s completely transparent to the user by design.

The end-goal of data independence is to remove the necessity of having a human make decisions about how the data should be stored. The database knows the read and write patterns of the database better than any human and can adjust for changes in these patterns over time. The above examples are intentionally simple- there are many more optimization opportunities to find! Will you join us on this math quest?

I posit that data independence is not specific to database technology. Where else have you seen data independence or concepts similar to data independence?

SQL has been saddled with the concept of NULL since SQL’s inception by E.F. Codd in the 1970s. The concept of NULL also includes ternary logic whereby boolean logic is extended to include the concept of an “unknown” or “missing” value (NULL). However, NULL was added during a time when type systems were comparatively ill-defined and often computer-specific. For example, SQL’s type system was very clearly inspired by the popular programming languages of the time: assembly language and later C. Whereas NULL was originally hoped to be a placeholder for “missing” values, the relational algebra already has multiple ways to represent such values and non-values.

Consider the following table:

# TABLE employee;
 id |   name   
----+----------
  1 | David
  2 | Samantha
(2 rows)

How do we know that “Bob” is not an employee? If we treat the database as the definitive source-of-truth, then Bob is not an employee simply because he is not in the employee table. This is how the relational algebra models reality- we record facts we know about the world and everything else is either untrue or unknown. For example, if we wish to know how old David is, this above database cannot answer the query, so the answer is unknown.

# SELECT age FROM employee WHERE name='David';
ERROR:  column "age" does not exist
LINE 1: SELECT age FROM employee WHERE name='David';

Indeed, SQL rejects the question out-of-hand.

The relational algebra already includes a means of recording missing data by exclusion. We don’t need to, for example, record that Bob is not an employee.

The above table is unambiguous about who is an employee. However, because NULL is a misfeature of SQL, developers often are inclined or forced to use NULL. If the goal of relational database modeling is to accurately record states in reality, then the fundamental issue with NULL can only introduce ambiguity.

Suppose for example, we encounter an employee table with a NULL-able start date:

test=# TABLE employee;
 id |   name   | start_date 
----+----------+------------
  2 | Samantha | 
  1 | David    | 2040-10-18
(2 rows)

We did not add this column, but we now need to understand what the column indicates. So, putting on our reverse-engineering hats, David has a start date at some point in the future (when compared to the current date of this writing). Perhaps it’s odd to have a start date years into the future, but it’s possible. What about Samantha? What does it mean for her start date to be NULL or “missing”? It could be any of the following possible meanings:

• Samantha has not yet started her job

• Samantha does not know when she will be able to start

• the company does not yet know when Samantha will start

• Samantha has left the company

• Samantha was made an offer but never joined the company

• Samantha has died

• Samantha was fired

• Samantha quit

• the database was created after Samantha joined the company, so the developers didn’t bother to record her actual start date

There are probably even more possible meanings you, the reader, could cook up. A single NULL in a DATE column has introduced ambiguity and we are expected to do further reverse engineering through the application layer or read the original developer’s mind to discover the meaning. Perhaps multiple developers have unknowingly used NULL to indicate distinct meanings within the same column- this is a likely worst-case scenario.

In addition, is Samantha in the table above technically an employee or not? If you believe Samantha is an employee because she in the employee table, then how can an employee have no start date? If you believe Samantha is not an employee, why is she in the employee table? The introduction of NULL has muddied the water as to what an employee is in the database.

Each “missing” value has its own distinct meaning which we originally intended to encode with NULL, but NULL is simply not flexible enough to capture them.

By including NULL in the original SQL, generations of database developers assumed that ternary logic and NULLs are somehow fundamental to the database technology and the relational algebra. This is a misunderstanding with dire consequences.

The Solution

By now you have guessed that NULL is indeed not fundamental to database technology. This is true, but we still need a means of recording alternate “missing” values states. Type system developments since the 1970s offer a type-safe path forward with “algebraic data types”.

Witness:

TutorialD (master/main): data EmployeeStartDate = StartDate Day | NotYetDetermined
TutorialD (master/main): employee := relation{tuple{id 1, name "Samantha", start_date NotYetDetermined}, tuple{id 2, name "David", start_date StartDate fromGregorian(2040, 10, 18)}}
TutorialD (master/main): :showexpr employee
┌───────────┬──────────┬─────────────────────────────┐
│id::Integer│name::Text│start_date::EmployeeStartDate│
├───────────┼──────────┼─────────────────────────────┤
│1          │"Samantha"│NotYetDetermined             │
│2          │"David"   │StartDate 2040-10-18         │
└───────────┴──────────┴─────────────────────────────┘

First, we create a new type called “EmployeeStartDate” with two possible values: either an actual start date or a “NotYetDetermined” marker.

Next, we create an “employee” relation variable (similar to an SQL table) with the data on our employees.

Finally, we can view the data in the relation variable.

Wait, what about if we want to represent that “Samantha” was fired? Well, the start_date attribute can only contain the date or a “NotYetDetermined” marker. Since “NotYetDetermined” is not obviously not appropriate for a “fired” state, the developer is forced to recognize that the only option is to delete Samantha’s tuple from the relation variable.

Note that there is no NULL and therefore no ambiguity. With algebraic data types, we can encode exactly the facts about the universe we wish to capture: nothing more and nothing less. NULL is a crutch from a previous era that introduces ambiguity wherever it rears its ugly head. Let’s relegate NULL to this era.

The above examples rely on PostgreSQL and Project:M36.

Consider this common table construction:

test=# CREATE TABLE employee(id INTEGER PRIMARY KEY, name TEXT NOT NULL);
CREATE TABLE
test=# \d employee
              Table "public.employee"
 Column |  Type   | Collation | Nullable | Default 
--------+---------+-----------+----------+---------
 id     | integer |           | not null | 
 name   | text    |           | not null | 
Indexes:
    "employee_pkey" PRIMARY KEY, btree (id)

In PostgreSQL, as documented, the DBMS uses the PRIMARY KEY syntax to indicate:

1. a NOT NULL UNIQUE constraint for the column(s)

2. only one PRIMARY KEY constraint is allowed (unlike UNIQUE constraints)

3. the implicit creation of a unique index (which would be created for any UNIQUE constraint)

4. some self-documenting behavior in table metadata

This is an exclusive list of PRIMARY KEY capabilities.

Were you expecting to be able to use this label in projections?

test=# SELECT PRIMARY KEY FROM employee;
ERROR:  syntax error at or near "PRIMARY"
LINE 1: SELECT PRIMARY KEY FROM employee;

Nope.

How about using primary keys to make join conditions easier to write?

test=# SELECT e.* FROM employee AS e JOIN company AS c USING PRIMARY KEYS;
ERROR:  syntax error at or near "PRIMARY"
LINE 1: ...CT e.* FROM employee AS e JOIN company AS c USING PRIMARY KE...

Too bad, you’re out-of-luck. In fact, marking a PRIMARY KEY does not help in writing any query. One could write NOT NULL UNIQUE for the column and all subsequent queries would be the same.

In addition, a table may have any number of uniqueness constraints which could be candidate keys. Consider the following table definition:

test=# CREATE TABLE drug_sample(id INTEGER NOT NULL, batch_id INTEGER NOT NULL, name TEXT NOT NULL, UNIQUE(id), UNIQUE(batch_id, name));

Is “id” the primary key or is “batch_id” + “name” the primary key? Selecting one to be “primary” is completely arbitrary. Since a table can have any number of unique keys, it doesn’t make much sense to call one key “primary”. The concept itself is a vestige of design-by-committee.

Solution:

SQL feigns special status for primary keys that do nothing special. All candidate keys should be treated equally. Here’s what it looks like in TutorialD:

TutorialD (master/main): drug_sample := relation{id Integer, batch_id Integer, name Integer}
TutorialD (master/main): key drug_sample_key {id} drug_sample
TutorialD (master/main): key drug_sample_key2 {batch_id,name} drug_sample

We define two keys on relation variable “drug_sample” without marking one as arbitrarily special.

SQL’s type system superficially resembles C’s type system but is demonstrably worse. The type system forces database users to make unpleasant compromises that compromise data integrity.

Example #1: NULL Nonsense

NULLs do not effectively model any fact in the universe. Instead, database users are forced to use NULL as a grab bag for all possible nonsense values.

Consider a table describing employees:

test=# CREATE TABLE employee(name TEXT NOT NULL, age INTEGER);
CREATE TABLE
test=# INSERT INTO employee(name, age) VALUES ('Bob', 34),('Steve',NULL);
INSERT 0 2
test=# TABLE employee;
 name  | age 
-------+-----
 Bob   |  34
 Steve |    
(2 rows)

What does Steve’s age mean? Well, it’s a NULL value, so it could mean any of the following:

• Steve asked for his age not be recorded

• Steve has not yet provided his age

• Steve is no longer an employee so his age is irrelevant

• Steve is dead

Any or all of these options could be implied by the value stored as Steve’s age, but it’s completely ambiguous. One would have to peruse the application code which generates the NULL to reverse engineer why the NULL makes an appearance at all.

Because there is no explanation provided for the NULL, the NULL’s presence reduces data integrity. NULL is intentionally ambiguous! If we wanted to represent the states in the list above, we would need to create additional state columns.

test=# ALTER TABLE employee ADD COLUMN age_null_reason TEXT;
ALTER TABLE
test=# UPDATE employee SET age_null_reason='declined to answer' WHERE name='Steve';
UPDATE 1
test=# TABLE employee;
 name  | age |  age_null_reason   
-------+-----+--------------------
 Bob   |  34 | 
 Steve |     | declined to answer
(2 rows)

Perhaps we can also add constraints to ensure that the age_null_reason is only NOT NULL if age is NULL to prevent nonsense states, but this is not easy to add in SQL, nor is the age a single type covering all possible “age states”. We cannot, for example, make a new age type and then easily use it in another table because this “type” is actually composed of multiple columns. In addition, the age_null_reason column is a free-form text field which is too loose to precisely represent the list of states mentioned above. We could change it to an enumeration or add check constraints, but that’s a lot of baggage for a simple age.

Example #2: Strings when all else fails

SQL supports a DATE type, but what about if we want to represent recurring dates; for example, an event that happens on the 15th of every month. This could be represented by three different columns like this:

test=# CREATE TABLE event(name TEXT NOT NULL, year INTEGER, month INTEGER, day INTEGER);
CREATE TABLE
test=# INSERT INTO event(name,year,month,day) VALUES ('wedding', 2016, 10, 16),('payroll',NULL,NULL,15);
INSERT 0 2
test=# TABLE event;
  name   | year | month | day 
---------+------+-------+-----
 wedding | 2016 |    10 |  16
 payroll |      |       |  15
(2 rows)

But that’s not a type- that’s just three integers which are otherwise disconnected. For example, we can trivially insert a nonsense templated date:

test=# INSERT INTO event(name,year,month,day) values ('nonsense',-4000,100,45);
INSERT 0 1

To tighten up the constraints, we would have to write an extremely complex CHECK constraint which validates dates. For example, does it make sense to repeat events on the 31st of every month even though only certain months have a 31st? What does that mean in our context? SQL does not make representing these states easy to implement.

Because of this complexity, database users are more inclined to use loose string types.

test=# CREATE TABLE event(name TEXT NOT NULL, "date" TEXT NOT NULL);
CREATE TABLE
test=# INSERT INTO event(name,"date") VALUES ('payroll','XXXX-XX-15');
INSERT 0 1
test=# TABLE event;
  name   |    date    
---------+------------
 payroll | XXXX-XX-15
(1 row)

That’s a representation that easier for a human to understand and generate application side, but it includes zero validation for the data, so it can represent even more nonsense states than the three-column design.

Example #3: Hair colors by which to rip out your own hair

Imagine we want to record a person’s hair color for hair salon records. We could just make it a string type:

test=# CREATE TABLE salon_customer(name TEXT NOT NULL, hair_color TEXT NOT NULL);
CREATE TABLE
test=# INSERT INTO salon_customer(name, hair_color) VALUES ('Bob','brown'),('Steve','Brown');
INSERT 0 2
test=# TABLE salon_customer;
 name  | hair_color 
-------+------------
 Bob   | brown
 Steve | Brown
(2 rows)

Oops- someone typed in “brown” in two different ways. As humans, we know they are the same, but how will we resolve it so that we can find all our brown-haired customers? We would have to write a query which includes every possible spelling of “brown”.

Ok, now a bald customer wants a beard trim. What should we put in the “hair_color” column? Is baldness NULL, “bald”, “no hair”, “not applicable”, or something else? Clearly, the string type is too loose to be useful for reporting. Let’s try with a CHECK constraint.

test=# CREATE TABLE salon_customer(name TEXT NOT NULL, hair_color TEXT CHECK (hair_color IN ('brown','black','red','blond')));
CREATE TABLE
test=# INSERT INTO salon_customer(name, hair_color) VALUES ('Bob','brown'),('Steve','brown');
INSERT 0 2
test=# TABLE salon_customer;
 name  | hair_color 
-------+------------
 Bob   | brown
 Steve | brown
(2 rows)

Ok, that’s an improvement since we have fewer values to consider as valid. We’ll use NULL to represent baldness. Solved!

Oof- a customer just walked in with strawberry blond hair- is that red or blond? We only have those options. Fine, we’ll make another column for exceptional cases:

test=# CREATE TABLE salon_customer(name TEXT NOT NULL, hair_color TEXT CHECK (hair_color IN ('brown','black','red','blond')), other_hair_color TEXT, CHECK ((hair_color IS NULL) <> (other_hair_color IS NULL)));
CREATE TABLE
test=# INSERT INTO salon_customer(name, hair_color, other_hair_color) VALUES ('Bob', 'brown', NULL),('Steve',NULL,'strawberry blond');
INSERT 0 2
test=# TABLE salon_customer;
 name  | hair_color | other_hair_color 
-------+------------+------------------
 Bob   | brown      | 
 Steve |            | strawberry blond
(2 rows)

But now we’re just pushing unusual data into another column instead of containing it in a type. The CHECK constraint does prevent both columns from being NULL, but the same data inconsistency we had before can occur again:

test=# INSERT INTO salon_customer(name, hair_color, other_hair_color) VALUES ('BadBob',NULL,'brown');
INSERT 0 1
test=# TABLE salon_customer;
  name  | hair_color | other_hair_color 
--------+------------+------------------
 Bob    | brown      | 
 Steve  |            | strawberry blond
 BadBob |            | brown

Ugh, so we need to add an additional constraint to other_hair_color to not contain “brown”- ok, but can it contain “Brown”? With sufficient constraints, maybe we could make this work, but:

• the values for hair color cannot not contained within an SQL type

• without looking at the table’s schema, an SQL user would not know how to enter a proper hair color- there is no way to enumerate the possible hair colors

• this design cannot be easily replicated in another place; for example,

SQL Attempts at Solutions

SQL does attempt to offer some solutions to these problems, but they also fall down quickly.

Foreign Key Relationships

If we can shuttle our type’s values together into one table, we can tighten down the constraints and then refer to the values by (perhaps surrogate) foreign key. In addition to allowing this “type” to be referenced in multiple places, it’s also easy to enumerate all the values of the type.

The downside is that the artificial “type” cannot be modified. For example, if we need to add new data values, then that new value is applied to all uses of the type, even if the new value is not applicable to all referring tables. If we start with a “hair_color” table then want to leverage the type to a “beard_type”, well that is just not possible. With PostgreSQL, every table is also a type with which we could write server-side functions such as recommend_hair_color_treatments(hair_color) but that is not portable to other SQL implementations. In addition, using a surrogate key may make the value in the referencing table opaque, forcing a join unconditionally to extract human-readable data.

Composite Types

A composite type is just two or more columns glued together in a type. While this does make it easier to create a new table or new column with the bundled type, a composite type does not carry any constraints with it. Constraints have to be reimplemented wherever this type is used.

Problem Summary

Because creating SQL types to accurately model the world are challenging to implement and then use, hardly anyone bothers to create new SQL types, instead relying on multiple columns and complex constraints to capture real-world state.

The Solution

C-style types are just not good enough to model reality at a high-level. We need to look beyond to “algebraic data types”. The simple advantage of algebraic data types is that they are composable: we can make new types from existing types.

Let’s start with a sum type to represent the most common values for hair.

Project:M36 TutorialD Interpreter 1.1.0
Type ":help" for more information.
A full tutorial is available at:
https://github.com/agentm/project-m36/blob/master/docs/tutd_tutorial.markdown
TutorialD (master/main): data HairColor = Brown | Black | Red | Blond | Bald
TutorialD (master/main): salon_customer := relation{tuple{name "Bob", hair Brown}, tuple{name "Steve", hair Blond}}
TutorialD (master/main): :showexpr salon_customer
┌───────────────┬──────────┐
│hair::HairColor│name::Text│
├───────────────┼──────────┤
│Brown          │"Bob"     │
│Blond          │"Steve"   │
└───────────────┴──────────┘

The “data” operator declares a new data type called “HairColor”. Then, we declare the first-class values which populate the type. Note that we don’t need a special non-value NULL to represent anything. After all, NULL would be worse than just using the unambiguously named value “Bald”.

Already, this is an improvement over strings with a check constraint- we can see by examining the type which values the type can contain.

But, we may have unusual hair colors or types (such as for baldness), so we need to extend our type with a product type. Let’s try again.

TutorialD (master/main): delete salon_customer
TutorialD (master/main): undata HairColor
TutorialD (master/main): data HairColor = Brown | Black | Red | Blond | Bald | UnusualColor Text
TutorialD (master/main): salon_customer := relation{tuple{name "Bob", hair Brown}, tuple{name "Steve", hair UnusualColor "Purple"}}
TutorialD (master/main): :showexpr salon_customer
┌─────────────────────┬──────────┐
│hair::HairColor      │name::Text│
├─────────────────────┼──────────┤
│UnusualColor "Purple"│"Steve"   │
│Brown                │"Bob"     │
└─────────────────────┴──────────┘

That’s quite promising. It’s also easy for a database user to see all possible values of the type. One could create an UnusualColor of “Brown”, but that value is unambiguously different from the common Brown. Of course, we can also use these values in queries:

TutorialD (master/main): :showexpr salon_customer where hair=Brown
┌───────────────┬──────────┐
│hair::HairColor│name::Text│
├───────────────┼──────────┤
│Brown          │"Bob"     │
└───────────────┴──────────┘

Note again that “Brown” is a value of type “HairColor” and not a string from the database’s perspective. We can use this type across multiple relation variables (tables) and use it in a new type:

TutorialD (master/main): data HairAndBeardColors = HairAndBeard HairColor HairColor
TutorialD (master/main): :showexpr relation{tuple{name "Steve", hairbeard HairAndBeard Bald Brown}}
┌─────────────────────────────┬──────────┐
│hairbeard::HairAndBeardColors│name::Text│
├─────────────────────────────┼──────────┤
│HairAndBeard Bald Brown      │"Steve"   │
└─────────────────────────────┴──────────┘

This data models Steve as a customer with a bald head and brown hair using our existing HairColor type (twice).

If we want to model the templated dates from the second example above, we can do that in a straightforward way:

TutorialD (master/main): data TemplatedDate = TemplatedDate (Maybe Integer) (Maybe Integer) (Maybe Integer)
TutorialD (master/main): :showexpr relation{name Text,date TemplatedDate}{tuple{name "Ides of March", date TemplatedDate Nothing (Just 3) (Just 15)}}
┌────────────────────────────────────────┬───────────────┐
│date::TemplatedDate                     │name::Text     │
├────────────────────────────────────────┼───────────────┤
│TemplatedDate Nothing (Just 3) (Just 15)│"Ides of March"│
└────────────────────────────────────────┴───────────────┘

Naturally, we would want to make some functions to validate that the templated date is valid, but that could be very context-specific. Consider, for example, the following templated date:

TutorialD (master/main): :showexpr relation{name Text,date TemplatedDate}{tuple{name "31st of every month", date TemplatedDate Nothing Nothing (Just 31)}}
┌───────────────────────────────────────┬─────────────────────┐
│date::TemplatedDate                    │name::Text           │
├───────────────────────────────────────┼─────────────────────┤
│TemplatedDate Nothing Nothing (Just 31)│"31st of every month"│
└───────────────────────────────────────┴─────────────────────┘

Given that not all months extend to 31 days, does this templated date refer to the end of every month or the end of months which have 31 days? It’s ambiguous, but with algebraic data types, we can address it!

TutorialD (master/main): data TemplatedDate = TemplatedDate (Maybe Integer) (Maybe Integer) (Maybe Integer) | EndOfEveryMonth
TutorialD (master/main): :showexpr relation{name Text,date TemplatedDate}{tuple{name "ides of every month", date TemplatedDate Nothing Nothing (Just 15)},tuple{name "end of every month", date EndOfEveryMonth}}
┌───────────────────────────────────────┬─────────────────────┐
│date::TemplatedDate                    │name::Text           │
├───────────────────────────────────────┼─────────────────────┤
│EndOfEveryMonth                        │"end of every month" │
│TemplatedDate Nothing Nothing (Just 15)│"ides of every month"│
└───────────────────────────────────────┴─────────────────────┘

This demonstrates the composition capabilities of algebraic data types. We can compose arbitrarily-complex types to best model our world.

While SQL can approximate the features of algebraic data types, as we saw in the above examples, matching those features is painful and unnatural. Algebraic data types allow composition and thus creation of complex types which improve data consistency. Thus, algebraic data types are a natural fit for modeling the world with relational databases.

All solutions in the solution were executed with Project:M36.

In an elected representative democracy, the voters select electees to represent the voters’ interests. Ideally, the representatives would have no agenda beyond collecting and representing said interests. The voters in turn should cast votes for the candidate who can best represent their interests. How a voter is supposed to know which candidate can do so is left as an exercise to the voter. How the candidates appear on the ballot is left as a mystery.

Given this simplified model of a representative democracy, the voters’ evaluation of the success of a representative is how well the representative aligns with the voters’ legislative desires when the representatives vote and make legislation and definitely not:

• the representative’s opinion on any particular matter

• whether the representative promotes a particular ideology

• the representative’s ability to raise funds for a party

• the representative’s ability to raise funds to advertise himself

• how representatives of other constituencies vote

• the representative’s ability to leverage rhetoric

In particular, representatives are not elected for their opinions or their ability to promote their opinions because, as representatives, their opinions are irrelevant.

How did we get from representatives to politicians?

Perverse, unanticipated incentives, of course. Politicians are power-mongers: their currency is influence over government action or inaction, so that currency is bought and sold on a market. The market, however, competes with the original task of representation of the voters interests. The original representatives were also keen politicians and even self-described god-kings, so they could not imagine a system where they would not be able to accumulate political capital.

How can a representative be aligned with the goal of actual representation?

First, we must acknowledge that voting, while a waypoint on the way from autocracy to democracy, is a vestige of thinkers who could not imagine unconcentrated power. Most democracies, now dealing with and benefiting from demographic shifts, acknowledge that human rights are not just for wealthy landowners. New technology can enable ideas like liquid democracy and ballot automation. Television created a voting class able to see and hear their representatives and the consequences of war. The Internet brought new forms of political and ideological discourse in a relatively equitable forum for the first time ever. But the voting systems in place are skewed towards incumbent candidates and parties, often enshrined in law.

However, we are not leveraging the latest mathematics in dealing with representation- statistics offers a means of finding representatives from a population without voting: random sampling. Like jury duty, randomly selected representatives are best suited to represent the population’s interests. Regularly cycling through representatives ensures that no single person accumulates power to use elsewhere. Like a juror, once the representative loses his seat, there is no reason for him to cycle into bribery- he won’t have any special connections in office since the other representatives will also be replaced.

The durations of the representatives are required to serve in the chamber need not happen on an even cadence- each representative can be cycled out at a time unknown to anyone using an exponential decay probability calculated daily. A replacement can be found and dropped in at any time.

Having two 80-year-old white men represent tens of millions of people is an absurd parody of democracy. Having so much power accumulated over decades of political “service” is antithetical to representation and more amenable to stagnation. Instead, the most average person (to which a random sample would skew) would provide a disinterested representative who is only willing to perform basic legislative tasks and eager to get out of office and back to his original life. Politics need not be a career opportunity. Instead of legislators vying for a political upper hand by manipulating chamber rules and raising campaign funds, the average man representative would focus on raising issues affecting himself and therefore a larger portion of the population. The average man is also far less willing to engage a nation in warfare. A perpetually cycling set of representatives prevents bribery and regulatory capture from being as effective. A selfish average man looking forward to completing his requisite representation is far less dangerous than a man who has spent decades empire building. A robust means of accelerating the representative’s exit from power, if necessary, can be provided to the electorate whose sole vote purpose then would be recall voting instead of candidate selection.

Diluting power makes the government less stable for politicians, but stabler for the electorate.

There are a few men who have the power to activate world-ending bombs. One of those men is the U.S. President. Why?

If deciding how yogurt should be taxed requires the vote of hundreds of representatives, then why is ending the world a one-man job? Instead of one president, we could have fifty, a hundred, or a thousand Presidents.

The only objection one could have to such representation in the executive power of the government is expediency and quick decision-making. In the 1700s, fifty Presidents reaching a quorum would have required them to be in a room together. Today, the Internet could have them debate and reach a quorum at any time from any location. If some are unable to join, then the others can still reach a quorum, even in an emergency.

Having a thousand presidents would also eliminate a single point-of-failure scenario. Killing a president or having a president incapicated would not paralyze the government nor would we need backup “vice” presidents. The regular cycling of the representatives would suffice. By diluting power, we can prevent the attraction of the office to psychopaths who believe themselves to be god-kings. Even if a malicious psychopath godking does achieve office (by random chance), his ability to cause damage is extremely limited.

Politics should not be a team sport.

As a voter, it’s easy to fall into personality cult traps. Political parties ensure that politicians of the same, self-serving caliber continually bubble up politically, convincing the electorate that no better candidate exists. This is not true. The average person would make a decent representative. A room full of average people would make a chamber of great representatives. Cycling out representatives and replacing them with randomly-selected people prevents parties and coalitions which horse trade on issues and votes. Instead, debate would have to be substantive, but also accessible to the average person. Representatives may not even need to speak to their constituents since the very act of being average and holding an average opinion is a reasonable form of representation.

Representatives should not be in charge of raising issues to address- this can be better achieved through petitions and ballot initiatives. The representatives then must get those initiatives into shape as legislation. Such legislative process is less prone to manipulation through lobbying (bribery) and political horse trading.

How can we get to representative government?

Proper representatives don’t actually make anything of value- they are a net loss of productivity and intellect. Proper representatives do not need to hold or flaunt unique talents, intelligence, or rhetorical ability because the act of representing a population does not require this. By definition of the word “representative”, the most representative representative is the one who is completely replaceable. While representation is something worthwhile to the society as a whole, it is not a role which anyone in particular should be able to or want to seek out. If we must live with the concept of representation, then it should be actual representation and not party tricks- no one needs to see representatives represent on television; representatives should not at all be interesting to observe individually.

Don’t drink the voting Kool-Aid. If you must vote, vote for the most boring, party-unaffiliated candidate. Don’t vote for incumbent candidates- keep the people cycling through the chamber to reduce power buildup. Education of the electorate can help everyone to differentiate between representation and leadership. Ensure that term limits are enforced, short, and ever shorter. Any person should be able to take up the representation mantle- even you!

Two NULLs are generally not equal- the result of comparing two NULLs with equality is another NULL.

test=# SELECT (NULL = NULL) IS NULL;
 ?column? 
----------
 t
(1 row)

IS DISTINCT FROM provides a means of determining if we are comparing two values which may be NULL and we want to know if the values are actually “equal” even if they are both NULL.

test=# SELECT NULL IS DISTINCT FROM NULL;
 ?column? 
----------
 f
(1 row)

Therefore, IS NOT DISTINCT FROM is SQL’s form of strict equality (returning either true or false and never NULL) in the face of NULLs.

test=# SELECT NULL IS NOT DISTINCT FROM NULL;
 ?column? 
----------
 t
(1 row)

Ok, but all values must have a type, right? So what’s the data type of NULL?

test=# SELECT pg_typeof(NULL);
 pg_typeof 
-----------
 unknown
(1 row)

Ok, so the concept of unknown/NULL bubbles up to the type itself. It makes sense in this context that we don’t know if the NULL in question is an unknown integer or text or some other type. There is no type hint provided.

test=# SELECT pg_typeof(NULL = NULL);
 pg_typeof 
-----------
 boolean
(1 row)

This seems to make sense, too, since we know that equality can only return true, false or unknown (NULL), so we know that the “unknown” value is of type boolean even if the two NULLs we are comparing are of unknown-typed NULL.

Another way to provide a type hint is using CAST().

test=# SELECT pg_typeof(CAST(NULL AS INTEGER));
 pg_typeof 
-----------
 integer
(1 row)

Can we compare a typed NULL to an NULL of unknown type?

test=# SELECT NULL IS NOT DISTINCT FROM CAST(NULL AS INTEGER);
 ?column? 
----------
 t
(1 row)

Yes- that’s fine. Even though the two NULLs are two different types (one known and one unknown), we can compare them. Can we compare NULLs of two, known types?

test=# SELECT CAST(NULL AS TEXT) IS NOT DISTINCT FROM CAST(NULL AS INTEGER);
ERROR:  operator does not exist: text = integer
LINE 1: SELECT CAST(NULL AS TEXT) IS NOT DISTINCT FROM CAST(NULL AS ...
                                  ^
HINT:  No operator matches the given name and argument type(s). You might need to add explicit type casts.

PostgreSQL rightfully rejects this expression in the same way that it rejects comparing and integer and text for equality- it’s a nonsense query.

We can also be explicit about casting our NULL to an unknown type.

test=# SELECT pg_typeof(CAST(NULL AS UNKNOWN));
 pg_typeof 
-----------
 unknown
(1 row)

Can we “promote” an integer-typed NULL to “unknown”?

test=# SELECT CAST(CAST(NULL AS INTEGER) AS UNKNOWN);
ERROR:  cannot cast type integer to unknown
LINE 1: SELECT CAST(CAST(NULL AS INTEGER) AS UNKNOWN);

That is rejected, but… why? Apparently, once we have a specific type for our unknown value, the type cannot be eliminated, but we can cast the NULL to any other type:

test=# SELECT CAST(CAST(NULL AS INTEGER) AS TEXT);
 text 
------
 
(1 row)

Can we create unknown-typed values other than NULL?

test=# SELECT pg_typeof(CAST('x' AS UNKNOWN));
 pg_typeof 
-----------
 unknown
(1 row)
test=# SELECT CAST('x' AS UNKNOWN);
 unknown 
---------
 x
(1 row)

The above examples are queries without any tables. Can we create a table with a column of unknown type? Apparently, but what can we do with it?

test=# CREATE TABLE nulltest AS SELECT NULL AS col;
SELECT 1

Success? Well, let’s inspect the table.

test=# \d nulltest
            Table "public.nulltest"
 Column | Type | Collation | Nullable | Default 
--------+------+-----------+----------+---------
 col    | text |           |          | 

When creating the table, the NULL of unknown type is promoted to a text type apparently. Can we insert a NULL of a different type into the text column?

test=# INSERT INTO nulltest(col) VALUES(CAST(NULL AS INTEGER));
INSERT 0 1

Yes! Let’s confirm the types of the two rows we inserted:

test=# SELECT pg_typeof(col) FROM nulltest;
 pg_typeof 
-----------
 text
 text
(2 rows)

So, instead of rejecting the NULL of the wrong type, PostgreSQL cast our integer-typed NULL to a text-typed NULL.

What about if we provide the unknown type hint?

test=# CREATE TABLE nulltest3 AS SELECT CAST('x' AS UNKNOWN) as col3;
SELECT 1
test=# \d nulltest3
            Table "public.nulltest3"
 Column | Type | Collation | Nullable | Default 
--------+------+-----------+----------+---------
 col3   | text |           |          | 

PostgreSQL still forces our type to a text type- it seems that we cannot create a table with an unknown type.

More generally, SQL automatically casts between column types when it can:

test=# CREATE TABLE nulltest2 AS SELECT CAST(NULL AS INTEGER) AS col2;
SELECT 1
test=# INSERT INTO nulltest(col) SELECT col2 FROM nulltest2;
INSERT 0 1

Here we create another table with an integer-typed NULL column “col2”, then we insert the NULLs from the new table “nulltest2” into “nulltest”. There is no error because the database uses the standard cross-type casting function, which also converts NULLs across types.

Let’s recap. NULL can clearly carry a type with a it, so not all NULLs are indistinct from each other. We can promote NULL from one type to another unless the destination type is “unknown” and the source type is not already “unknown”. When using an unknown-typed NULL in table creation, PostgreSQL casts it automatically into a text field which we did not request.

What’s wrong with this picture? Did you, SQL practitioner, know any of these NULL nuances? Are these nuances even worth knowing? The problem is that we may be using this language without understanding how NULLs actually work. This can lead to mistaken assumptions about what NULLs mean in a given context. As demonstrated by the example when creating a table with an unknown-typed NULL, not even SQL itself can agree on what is meant by an untyped NULL.

Inconsistencies in SQL lead to mistakes and subtle bugs.

Solution

The introduction of NULL is more than a billion-dollar mistake. Three-valued logic (true, false, and unknown/NULL) is an inconsistent mess even though SQL has had decades to remedy this. Let’s drop three-valued logic and replace it with algebraic data types to store exactly what we intend, unambiguously. Once we eliminate NULL, we can query our data with much more consistency and confidence in the results.

CONCAT() and || behave differently when faced with NULL.

First, let’s create a table with some NULLs:

test=# CREATE TABLE photo(filename text not null, target text); 
CREATE TABLE 
test=# INSERT INTO photo(filename, target) VALUES ('bob_birthday.jpg', 'Bob'),('dogtoy.jpg', NULL); 
INSERT 0 2

Now we make a string describing each photo:

test=# SELECT filename || ' contains a person named ' || target FROM photo;
                   ?column?                   
----------------------------------------------
 bob_birthday.jpg contains a person named Bob
 
(2 rows)

Wait- why is the second row empty? Oh, the target is NULL, so the whole string is NULL? Let’s try that again.

test=# SELECT CONCAT(filename, ' contains a person named ', target) FROM photo;
                    concat                    
----------------------------------------------
 bob_birthday.jpg contains a person named Bob
 dogtoy.jpg contains a person named 
(2 rows)

Ah, there it is. So CONCAT(a,b) is equivalent to COALESCE(a,’’) || COALESCE(b,’’).

Good luck remembering which variant returns NULL as a whole in which context.

Surprises lead to mistakes.

Solution

NULL’s pitfalls are many and well-documented. There are many other functions which behave confusingly in the context of NULL. Can we do without NULL?

As it turns out, NULL is as much a liability in SQL as it is in other programming languages. Nor is NULL required by the relational algebra as so many DBAs believe.

So how can we capture “missing” data? Missing data can represented by the same means as any other data- we can record it as part of the data type and we should not be constrained to C-style pointer types. In many programming languages, we can leverage “algebraic data types” to record arbitrarily complex facts about our world.

For example, if we want to record a fact about a person’s hair color, the options may be:

• blond

• brown

• red

• bald

• gray

• some other color

A naive SQL implementation may be a NULL-capable string column:

test=# CREATE DOMAIN hair TEXT;
CREATE DOMAIN

but are “gray” and “grey” the same color or not? A string type is too loose, so let’s tighten it up using a CHECK constraint:

test=# CREATE DOMAIN hair TEXT CHECK(value IN ('blond', 'brown', 'red', 'gray', 'other'));
CREATE DOMAIN

whereby NULL represents a bald person. But what about if we actually want to capture the “other” color. Then we need another column:

test=# CREATE TABLE person(name TEXT NOT NULL, hair_color hair, optional_other_color TEXT);
CREATE TABLE

but that column can only be populated if the hair_color column is “other”, so we add a table constraint:

test=# ALTER TABLE person ADD CONSTRAINT other_color_constraint CHECK((hair_color = 'other' AND optional_other_color IS NOT NULL) OR (hair_color <> 'other' AND optional_other_color IS NULL));
ALTER TABLE

Whew! That’s a lot of SQL to represent basic hair characteristics.

Oh, we just received a change request: the user can choose not to report his hair color and we need to record that fact. We can’t use NULL because NULL is already being used to represent baldness. We can’t make another “NULL” for not reporting a value, so we’ll need to make another column to explain why hair_color is NULL and tighten down the columns with even more constraints. All of this is possible in SQL, but painful!

Here is a NULL-free representation of hair color using algebraic data types in TutorialD:

TutorialD (master/main): data HairColor = Blond | Brown | Red | Bald | Gray | OtherColor Text | UserDeclinedToProvideInfo

This is one type which can capture all possible values for this attribute. Here’s how we can use it:

TutorialD (master/main): person := relation{tuple{name "Bob", hair_color Bald}, tuple {name "Cindy", hair_color OtherColor "Purple"}}
TutorialD (master/main): :showexpr person
┌─────────────────────┬──────────┐
│hair_color::HairColor│name::Text│
├─────────────────────┼──────────┤
│Bald                 │"Bob"     │
│OtherColor "Purple"  │"Cindy"   │
└─────────────────────┴──────────┘

Done!

Now that we’ve eliminated NULLs from the database, CONCAT() and || can behave identically when it comes to text.

The Project:M36 article “On NULL” provides more detail and examples of how relational algebra engines benefit from eliminating NULL and replacing its use cases with algebraic data types.