Module Abstraction: Avoiding the God Module

The right module is the one that hides the right complexity and exposes the right simplicity. The wrong module is either too thin (a wrapper that adds nothing) or too thick (a god module that does everything). The two failure modes are mirror images. The discipline is to find the balance.

The lesson teaches the heuristic, the failure modes, and the split.

The level of abstraction

A module is a layer between the consumer and the resources. The layer hides complexity. The layer exposes policy. The layer is the contract.

Consumer
   |
   v
+--------+
| Module |    <-- the abstraction layer
+--------+
   |
   v
Resources

The consumer declares what they want (the policy). The module hides how it is implemented (the complexity). The resources are the implementation.

The right abstraction is the one where:

• The consumer can describe their intent without knowing the implementation. A consumer that wants a VPC with public and private subnets passes vpc_cidr and environment. The consumer does not pass the route table IDs, the NAT gateway associations, or the flow log IAM role.
• The module can hide the implementation details that would change. A module that hides the route table IDs is hiding a detail that may change. The consumer does not care about the change. The module absorbs the change.
• The module exposes the policy that must be consistent. A module that enforces mandatory encryption exposes the policy in the interface. The consumer cannot opt out. The policy is consistent.

The wrong abstraction is the one where the consumer passes the implementation details, or the module exposes every resource attribute as an output.

The god module

A god module is a module that does everything. It has forty inputs. It creates every resource the application needs. The consumer configures the whole application by calling one module.

# Bad: god module
module "production" {
  source = "./modules/production"

  vpc_cidr            = "10.0.0.0/16"
  vpc_name            = "production"
  public_subnets      = ["10.0.1.0/24", "10.0.2.0/24"]
  private_subnets     = ["10.0.10.0/24", "10.0.11.0/24"]
  database_subnets    = ["10.0.20.0/24", "10.0.21.0/24"]
  instance_type       = "t3.medium"
  instance_count      = 5
  database_engine     = "postgres"
  database_version    = "15"
  database_password   = var.db_password
  cluster_name        = "production-cluster"
  log_retention_days  = 30
  backup_retention_days = 7
  # ... 30 more inputs
}

The consumer has to know every resource to configure the module. The module has to expose every attribute to satisfy the consumer. The contract is the entire application. The module is not reusable: the consumer who wants only the database has to pass vpc_cidr = "" and hope the module does not crash.

The costs of a god module:

1. Cognitive load. The consumer has to learn everything to use anything. The review cycle for a pull request against the consumer is the size of the entire application.
2. Coupling. A change to the network section may break the database section. The module is one version. The module is one release.
3. No reuse. A consumer who wants only the database cannot use the module. The module does not serve them.
4. Test burden. The test file has to cover every combination of inputs. The test matrix is enormous.
5. Failure isolation. A bug in the network section takes down the database section. The blast radius is the entire application.

The thin wrapper

The thin wrapper is the mirror image. A module that wraps a single resource with three arguments and adds nothing:

# modules/instance/main.tf
resource "aws_instance" "main" {
  ami           = var.ami
  instance_type = var.instance_type
  tags          = var.tags
}

The consumer could have declared the resource directly. The module is a tax. The consumer pays the cognitive cost of the wrapper and receives no benefit.

The costs of a thin wrapper:

1. No reuse. The module is not used because the consumer could have written the resource directly.
2. Versioning burden. The module has a version, but the version is decoupled from the provider’s version. The wrapper has to be bumped every time the provider exposes a new attribute.
3. Test burden. The test asserts the wrapper passes the inputs. The test does not verify the resource.
4. Indirect indirection. The reader has to read the module to understand the resource. The reader could have read the resource directly.

The thin wrapper is a foothold. The team writes a thin wrapper because they think they will add value later. The value is never added. The wrapper is a tax.

The right split

The right split is to extract one logical thing into a module. The module has a single responsibility. The module has a small interface. The module is reusable.

The god module example split into focused modules:

module "network" {
  source = "./modules/network"
  vpc_cidr = "10.0.0.0/16"
  environment = "production"
}

module "compute" {
  source = "./modules/compute"
  vpc_id = module.network.vpc_id
  subnet_id = module.network.public_subnet_ids["a"]
  instance_count = 5
}

module "database" {
  source = "./modules/database"
  vpc_id = module.network.vpc_id
  subnet_id = module.network.database_subnet_ids["a"]
  engine = "postgres"
  version = "15"
}

module "observability" {
  source = "./modules/observability"
  vpc_id = module.network.vpc_id
  log_retention_days = 30
}

Each module has a single responsibility. The network module creates the VPC. The compute module creates the instances. The database module creates the database. The observability module creates the logging and monitoring.

The consumer composes the modules. The consumer chooses which modules to include. The consumer controls the composition.

The heuristic

A practical heuristic for module size:

• One to ten resources. The module is a composable piece. The module is focused.
• Eleven to thirty resources. The module is a service. The module is still focused but may be split into sub-modules.
• More than thirty resources. The module is too large. The module is doing more than one thing. The fix is to split.

The heuristic is approximate. The right test is whether the module can be described in one sentence. If the sentence includes “and”, the module is doing more than one thing.

# Good: one sentence
"The network module creates a VPC with public and
private subnets."

# Bad: two sentences
"The network module creates a VPC and the database
module creates a database and the compute module
creates the instances."

The second is three modules. The first is one module.

The composition pattern

The composition pattern is the discipline of composing focused modules into a larger system. The pattern:

Consumer root module
   |
   +-- module "network"          (creates the VPC)
   |
   +-- module "compute"          (creates the instances)
   |   |
   |   +-- module "asg"          (autoscaling group)
   |   |
   |   +-- module "iam"          (instance role)
   |
   +-- module "database"         (creates the database)
   |   |
   |   +-- module "rds"          (the RDS instance)
   |   |
   |   +-- module "subnet_group" (the DB subnet group)
   |
   +-- module "observability"    (creates logging)

The consumer composes the modules. The composition is the consumer’s choice. The modules are independent. The modules can be tested independently. The modules can be versioned independently.

The composition pattern is the answer to the god module. The composition is the discipline.

When the abstraction is wrong

A team has a module that is the right shape, but the abstraction is wrong. The team has to recognise the wrong abstraction.

The wrong abstraction is the one where:

• The consumer has to pass implementation details. The consumer passes subnet_cidr_blocks when the module should compute the subnets from the vpc_cidr.
• The module exposes every resource attribute. The consumer reaches into the module to read the internal route table IDs.
• The interface grows without review. The module has forty inputs because every consumer has asked for one more. The module is not a module; it is a configuration aggregator.

The fix is to refactor. The interface is reduced. The implementation is hidden. The consumer is forced to think about the policy, not the implementation.

Inspection commands

The reader inspects a module’s abstraction:

# Severity: READ-ONLY
grep -c '^variable' modules/network/variables.tf

5

Five variables. The module is focused.

# Severity: READ-ONLY
grep -c '^output' modules/network/outputs.tf

3

Three outputs. The module exposes only what the consumer needs.

# Severity: READ-ONLY
grep -c '^resource' modules/network/main.tf

6

Six resources. The module is a composable piece.

A module with five variables, three outputs, and six resources is focused. A module with forty variables, twenty outputs, and one hundred resources is a god module.

Production failure modes

1. God module from copy-paste. The team copy-pasted a stack into a module. The module’s interface is every variable the resource had. The fix is to refactor the interface.

2. Thin wrapper from a premature module. The team extracted a module before there was a second consumer. The module has no value. The fix is to delete the module.

3. Spooky action at a distance. The module changes an internal resource’s attribute. The consumer depends on the attribute as an output. The consumer’s plan shows a replacement. The fix is to expose the resource as a separate module, or to absorb the change in the module’s outputs.

4. Module exports private values. The module outputs an internal resource’s ID. The consumer depends on the ID. The module cannot refactor without breaking the consumer. The fix is to export a concept (the application URL) rather than an implementation detail (the load balancer’s ID).

5. Composition explosion. The consumer composes twelve modules to deploy a single application. The overhead is too high. The fix is to find the right level of abstraction for the consumer’s use case.

Security implications

• The module is the boundary of enforcement. A god module that bundles the network, the database, and the compute makes it hard to review the security controls for each layer.
• The thin wrapper adds nothing to security. The consumer is exposed to the same attack surface as if the wrapper did not exist.
• The composition pattern allows the security review to be per-module. The network module is reviewed for network security. The database module is reviewed for database security. The review is focused.

Performance implications

• A god module compiles slowly. The single module contains every resource. The plan is slow.
• A thin wrapper has the same compile cost as the underlying resource. The wrapper is a tax.
• The composition pattern has a small fixed overhead per module. The cost is in the low milliseconds. The organisational benefit is worth the cost.

What comes next

The next lesson is module releases and upgrades: the release process, the registry upload, and the rollback procedure.

Verification

• The module can be described in one sentence.
• The module has fewer than ten variables.
• The module has fewer than five outputs.
• The module has fewer than thirty resources.
• The module’s name is a single noun (network, database, compute).
• The consumer composes focused modules rather than calling a single god module.