network Block

The network block specifies the networking requirements for the task group, including the network mode and port allocations. When scheduling jobs in Nomad they are provisioned across your fleet of machines along with other jobs and services. Because you don't know in advance what host your job will be provisioned on, Nomad will provide your tasks with network configuration when they start up.

Note that this document only applies to services that want to listen on a port. Batch jobs or services that only make outbound connections do not need to allocate ports, since they will use any available interface to make an outbound connection.

job "docs" {
  group "example" {
    network {
      port "http" {}
      port "https" {}
      port "lb" {
        static = 8889
      }
    }
  }
}

Network modes

When the network block is defined with bridge as the networking mode, all tasks in the task group share the same network namespace. This is a prerequisite for Consul Connect. Tasks running within a network namespace are not visible to applications outside the namespace on the same host. This allows Connect-enabled applications to bind only to localhost within the shared network stack, and use the proxy for ingress and egress traffic.

To use bridge mode, you must have the reference CNI plugins installed at the location specified by the client's cni_path configuration. These plugins are used to create the bridge network and configure the appropriate iptables rules.

Network modes are only supported in allocations running on Linux clients. All other operating systems use the host networking mode.

network Parameters

• mbits (deprecated int: 10) - Specifies the bandwidth required in MBits.

• port (Port: nil) - Specifies a TCP/UDP port allocation and can be used to specify both dynamic ports and reserved ports.

• mode (string: "host") - Mode of the network. This option is only supported on Linux clients. The following modes are available:

  • none - Task group will have an isolated network without any network interfaces.
  • bridge - Task group will have an isolated network namespace with an interface that is bridged with the host. Note that bridge networking is only currently supported for the docker, exec, raw_exec, and java task drivers.
  • host - Each task will join the host network namespace and a shared network namespace is not created.
  • cni/<cni network name> - Task group will have an isolated network namespace with the network configured by CNI.

• hostname (string: "") - The hostname assigned to the network namespace. This is currently only supported using the Docker driver and when the mode is set to bridge. This parameter supports interpolation.

• dns (DNSConfig: nil) - Sets the DNS configuration for the allocations. By default all task drivers will inherit DNS configuration from the client host. DNS configuration is only supported on Linux clients at this time. Note that if you are using a mode="cni/*, these values will override any DNS configuration the CNI plugins return.

• cni (CNIConfig: nil) - Sets the custom CNI arguments for a network configuration per allocation, for use with mode="cni/*.

port Parameters

• static (int: nil) - Specifies the static TCP/UDP port to allocate. If omitted, a dynamic port is chosen. We do not recommend using static ports, except for system or specialized jobs like load balancers.
• to (string:nil) - Applicable when using "bridge" mode to configure port to map to inside the task's network namespace. Omitting this field or setting it to -1 sets the mapped port equal to the dynamic port allocated by the scheduler. The NOMAD_PORT_<label> environment variable will contain the to value.
• host_network (string:nil) - Designates the host network name to use when allocating the port. When port mapping the host port will only forward traffic to the matched host network address.
• allow_reuse (bool: false) - Allows the task group to be placed on a node where the port may already be reserved. Intended for programs that support SO_REUSEPORT unix socket option, so more than one instance of the program may bind to the same port. Only compatible with host network mode and static ports. Some task drivers (e.g. docker) may also require setting network_mode = "host" (or similar) to avoid runtime errors after placement.

The label assigned to the port is used to identify the port in service discovery, and used in the name of the environment variable that indicates which port your application should bind to. For example:

port "foo" {}

When the task starts, it will be passed the following environment variables:

• NOMAD_IP_foo - The IP to bind on for the given port label.
• NOMAD_PORT_foo - The port value for the given port label.
• NOMAD_ADDR_foo - A combined ip:port that can be used for convenience.

The label of the port is just text - it has no special meaning to Nomad.

dns Parameters

• servers (array<string>: nil) - Sets the DNS nameservers the allocation uses for name resolution.
• searches (array<string>: nil) - Sets the search list for hostname lookup
• options (array<string>: nil) - Sets internal resolver variables.

These parameters support interpolation.

cni Parameters

• args (map<string><string>: nil) - Sets CNI arguments for network configuration. These get turned into CNI_ARGS per the CNI spec.

These parameters support interpolation.

network Examples

The following examples only show the network blocks. Remember that the network block is only valid in the placements listed above.

Dynamic Ports

This example specifies a dynamic port allocation for the port labeled "http". Dynamic ports are allocated in a range from 20000 to 32000.

Most services run in your cluster should use dynamic ports. This means that the port will be allocated dynamically by the scheduler, and your service will have to read an environment variable to know which port to bind to at startup.

group "example" {
  network {
    port "http" {}
    port "https" {}
  }
}

network {
  port "http" {}
}

Static Ports

Static ports place your job on a host where the port is not already reserved by another job with the same port.

This example specifies a static port allocation for the port labeled "lb".

network {
  port "lb" {
    static = 6539
  }
}

For programs that support the SO_REUSEPORT unix socket option, you may disable port collision detection by setting allow_reuse = true.

Mapped Ports

Some drivers (such as Docker and QEMU) allow you to map ports. A mapped port means that your application can listen on a fixed port (it does not need to read the environment variable) and the dynamic port will be mapped to the port in your container or virtual machine.

group "app" {
  network {
    port "http" {
      to = 8080
    }
  }

  task "example" {
    driver = "docker"

    config {
      ports = ["http"]
    }
  }
}

The above example is for the Docker driver. The service is listening on port 8080 inside the container. The driver will automatically map the dynamic port to this service.

When the task is started, it is passed an additional environment variable named NOMAD_HOST_PORT_http which indicates the host port that the HTTP service is bound to.

Bridge Mode

Bridge mode allows compatible tasks to share a networking stack and interfaces. Nomad can then do port mapping without relying on individual task drivers to implement port mapping configuration.

The following example is a group level network block that uses bridge mode and port mapping.

network {
  mode = "bridge"
  port "http" {
    static = 9002
    to     = 9002
  }
}

Using bridge mode can result in failing outbound network requests on hosts that have firewalld enabled. This includes most RHEL-based Linux distributions like CentOS, Rocky Linux or Oracle Linux. One solution for firewalld to allow network requsts coming from Nomad jobs is to mark the nomad bridge interface as trusted.

$ sudo firewall-cmd --zone=trusted --add-interface=nomad
$ sudo firewall-cmd --zone=trusted --add-interface=nomad --permanent

It is necessary to restart the affected jobs afterwards for them to be able to access the network. Further details can be found in Docker's documentation under Docker and iptables.

DNS

The following example configures the allocation to use Google's DNS resolvers 8.8.8.8 and 8.8.4.4.

network {
  dns {
    servers = ["8.8.8.8", "8.8.4.4"]
  }
}

Container Network Interface (CNI)

Nomad supports CNI by fingerprinting each node for CNI network configurations. These are associated to the node by the name field of the CNI configuration. The name can then be used when setting the network mode field in the form of cni/<name>.

As an example if the following CNI configuration was on a node the proceeding network block could be used.

{
  "cniVersion": "0.3.1",
  "name": "mynet",
  "plugins": [
    {
      "type": "ptp",
      "ipMasq": true,
      "ipam": {
        "type": "host-local",
        "subnet": "172.16.30.0/24",
        "routes": [
          {
            "dst": "0.0.0.0/0"
          }
        ]
      }
    },
    {
      "type": "portmap",
      "capabilities": { "portMappings": true }
    }
  ]
}

network {
  mode = "cni/mynet"
  port "http" {
    to = 8080
  }
}

The Nomad client will build the correct capabilities arguments for the portmap plugin based on the defined port blocks.

CNI Args

The following example specifies CNI args for the custom CNI plugin specified above.

network {
  mode = "cni/mynet"
  port "http" {
    to = 8080
  }
  cni {
    args = {
     "nomad.region" : "${node.region}"
    }
  }
}

Host Networks

In some cases a port should only be allocated to a specific interface or address on the host. The host_network field of a port will constrain port allocation to a single named host network. If host_network is set for a port, Nomad will schedule the allocations on a node which has defined a host_network with the given name. If not set the "default" host network is used which is commonly the address with a default route associated with it.

When Nomad does port mapping for ports with a defined host_network, the port mapping rule will use the host address as the destination address.

network {
  mode = "bridge"

  # define a port to use for public https traffic
  port "https" {
    static       = 443
    to           = 8080
    host_network = "public"
  }
  # define a port that is only exposed to private traffic
  port "admin" {
    to           = 9000
    host_network = "private"
  }
}

Limitations

• Only one network block can be specified, when it is defined at the task group level.
• Only the NOMAD_PORT_<label> and NOMAD_HOST_PORT_<label> environment variables are set for group network ports.