Red5 Documentation

Stream Manager 2.0 Admin API

Authentication

All of these REST requests require a valid JWT, presented in the Authorization header as a bearer token. See Auth Service doc.

Create NodeGroup

Create a new NodeGroup with the given configuration.

Unless isScalingPaused is explicitly true (it is false by default), the NodeGroup will begin scaling out immediately.

Create NodeGroup Request

POST https://<host>/as/v1/admin/node group

Body:

{
  "name": "TEST-simple-orig-edge-0",
  "description": "For integration tests.",
  "cloudPlatform": "DOCKER",
  "cloudProperties": "instance_type=1cpu_2gb;environment=testing",
  "shuffleSizeExpression": "max(min(nodeCount*0.1, 2), 5)",
  "images": {
    "Base Image": {
      "name": "Base Image",
      "image": "red5pro-docker.jfrog.io/red5pro-server-public:12.2.3",
      "cloudProperties": "example=1;foo=bar;baz=baf"
    }
  },
  "roles": {
    "origin": {
      "name": "origin",
      "imageName": "Base Image",
      "capacity": 70.0,
      "cloudProperties": "origprop=v2"
    },
    "edge": {
      "name": "edge",
      "imageName": "Base Image",
      "capacity": 70.0,
      "parentRoleName": "origin",
      "parentCardinality": "SUBGROUP",
      "cloudProperties": "edgeroleprop=v1"
    }
  },
  "groups": {
    "us-west" : {
      "subGroupName": "us-west",
      "groupType": "main",
      "rulesByRole": {
        "origin": {
          "nodeRoleName": "origin",
          "min": 1,
          "max": 10,
          "increment": 2,
          "outExpression": "avg(cpu.process.load) > 5.0",
          "inExpression": "avg(cpu.process.load) < 2.0",
          "capacityRankingExpression": "cpu.system.load * 10"
        },
        "edge": {
          "nodeRoleName": "edge",
          "min": 2,
          "max": 10,
          "increment": 1,
          "outExpression": "avg(cpu.process.load) > 5.0",
          "inExpression": "avg(cpu.process.load) < 2.0",
          "capacityRankingExpression": "cpu.system.load * 5"
        }
      },
      "cloudProperties": "region=nyc3"
    }
  }
}

The request body is a NodeGroupConfig. See NodeGroupConfig for details, below.

On success:

HTTP 201: Created

No body.

On error:
HTTP 400: Bad Request | Validation failure (see response body for details).

HTTP 401: Unauthorized | Missing or invalid JWT Authorization header

HTTP 409: Conflict | A NodeGroupConfig with this name already exists.

List All NodeGroups

Return a list of all active NodeGroups, by name. This does not include inactive schedules. For that, see List All Scheduled NodeGroups below.

List All NodeGroups Request

GET https://<host>/as/v1/admin/node group

List All NodeGroups Response

On success:

HTTP 200: OK

Body:

[
    "TEST-NODE-GROUP-2N40CI",
    "TEST-NODE-GROUP-J8SN31"
]

On error:
HTTP 401: Unauthorized | Missing or invalid JWT Authorization header

Get NodeGroupConfig

Return the configuration for a given NodeGroup, by name.
For details of the NodeGroupConfig structure, see NodeGroupConfig, below.
This request can return any NodeGroupConfig, scheduled or not.

Get NodeGroupConfig Request

GET https://<host>/as/v1/admin/nodegroup/<name>?isEffective=false

name: the name of the requested NodeGroup.

isEffective: Optional. Boolean. Default: false. If true return the “effective” config for this name (that is, return the active overlay if any, or else return the base). If false, return the given config by name.

Get NodeGroupConfig Response

On success:

HTTP 200: OK

Body:

{
  "name": "TEST-simple-orig-edge-0",
  "description": "For integration tests.",
  "cloudPlatform": "DOCKER",
  "cloudProperties": "instance_type=1cpu_2gb;environment=testing",
  "images": {
    "Base Image": {
      "name": "Base Image",
      "image": "red5pro-docker.jfrog.io/red5pro-server-public:12.2.3",
      "cloudProperties": "example=1;foo=bar;baz=baf"
    }
  },
  "roles": {
    "origin": {
      "name": "origin",
      "imageName": "Base Image",
      "capacity": 70.0,
      "cloudProperties": "origprop=v2"
    },
    "edge": {
      "name": "edge",
      "imageName": "Base Image",
      "capacity": 70.0,
      "parentRoleName": "origin",
      "parentCardinality": "SUBGROUP",
      "cloudProperties": "edgeroleprop=v1"
    }
  },
  "groups": {
    "us-west" : {
      "subGroupName": "us-west",
      "groupType": "main",
      "rulesByRole": {
        "origin": {
          "nodeRoleName": "origin",
          "min": 1,
          "max": 10,
          "increment": 2,
          "outExpression": "avg(cpu.process.load) > 5.0",
          "inExpression": "avg(cpu.process.load) < 2.0",
          "capacityRankingExpression": "cpu.system.load * 10"
        },
        "edge": {
          "nodeRoleName": "edge",
          "min": 2,
          "max": 10,
          "increment": 1,
          "outExpression": "avg(cpu.process.load) > 5.0",
          "inExpression": "avg(cpu.process.load) < 2.0",
          "capacityRankingExpression": "cpu.system.load * 5"
        }
      },
      "cloudProperties": "region=nyc3"
    }
  }
}

On error:
HTTP 401: Unauthorized | Missing or invalid JWT Authorization header

HTTP 404: No NodeGroup found with name.

Update NodeGroup

Create a new NodeGroup with the given configuration.
Unless isScalingPaused is explicitly true (it is false by default), the NodeGroup will begin scaling out immediately.

Note that if a NodeGroupConfig is created with a schedule, then it must always have at least one schedule. An update without schedules will be refused. Additionally, if a scheduled config overlays some base config, that cannot be changed later; the update will be refused.
In these cases, delete the offending config and then create a replacement.

Update NodeGroup Request

PUT https://<host>/as/v1/admin/node group

Body:

{
  "name": "TEST-simple-orig-edge-0",
  "description": "For integration tests.",
  "cloudPlatform": "DOCKER",
  "cloudProperties": "instance_type=1cpu_2gb;environment=testing",
  "images": {
    "Base Image": {
      "name": "Base Image",
      "image": "red5pro-docker.jfrog.io/red5pro-server-public:12.2.3",
      "cloudProperties": "example=1;foo=bar;baz=baf"
    }
  },
  "roles": {
    "origin": {
      "name": "origin",
      "imageName": "Base Image",
      "capacity": 70.0,
      "cloudProperties": "origprop=v2"
    },
    "edge": {
      "name": "edge",
      "imageName": "Base Image",
      "capacity": 70.0,
      "parentRoleName": "origin",
      "parentCardinality": "SUBGROUP",
      "cloudProperties": "edgeroleprop=v1"
    }
  },
  "groups": {
    "us-west" : {
      "subGroupName": "us-west",
      "groupType": "main",
      "rulesByRole": {
        "origin": {
          "nodeRoleName": "origin",
          "min": 1,
          "max": 10,
          "increment": 2,
          "outExpression": "avg(cpu.process.load) > 5.0",
          "inExpression": "avg(cpu.process.load) < 2.0",
          "capacityRankingExpression": "cpu.system.load * 10"
        },
        "edge": {
          "nodeRoleName": "edge",
          "min": 2,
          "max": 10,
          "increment": 1,
          "outExpression": "avg(cpu.process.load) > 5.0",
          "inExpression": "avg(cpu.process.load) < 2.0",
          "capacityRankingExpression": "cpu.system.load * 5"
        }
      },
      "cloudProperties": "region=nyc3"
    }
  }
}

The request body is a NodeGroupConfig. See NodeGroupConfig for details, below.

Update NodeGroup Response

On success:

HTTP 200: OK

On error:
HTTP 400: Bad Request | Validation failure (see response body for details).

HTTP 401: Unauthorized | Missing or invalid JWT Authorization header

HTTP 404: No NodeGroup found with name.

Delete NodeGroupConfig

Delete the configuration for a given NodeGroup, by name.
When a NodeGroup is deleted, all of its nodes are destroyed.

Delete NodeGroupConfig Request

DELETE https://<host>/as/v1/admin/node group/<name>

name: the name of the NodeGroup to delete.

Delete NodeGroupConfig Response

On success:

HTTP 200: OK

On error:
HTTP 401: Unauthorized | Missing or invalid JWT Authorization header

HTTP 404: No NodeGroup found with name.

Terraform: List Images

Return a list of available images per Cloud Platform.

Terraform: List Images Request

GET https://<host>/as/v1/admin/terraform/image

Terraform: List Images Response

On success:

HTTP 200: OK

{
  "DOCKER": {
    "default": [
      "red5pro-docker.jfrog.io/red5pro-server-public:12.2.3",
      "as-mock-node"
    ]
  }
}

Details are a map of Cloud Platform, to Region, to Image name.
In this example, DOCKER is a Cloud Platform, and default is a region.

On error:
HTTP 401: Unauthorized | Missing or invalid JWT Authorization header

Terraform: List Instance Types

Return a list of available images per Cloud Platform.

Terraform: List Instance Types Request

GET https://<host>/as/v1/admin/terraform/instanceType

Terraform: List Instance Types Response

On success:

HTTP 200: OK

{
  "DOCKER": [
    "1cpu_2gb",
    "2cpu_4gb",
    "4cpu_8gb",
    "8cpu_16gb"
  ]
}

Details are a map of Cloud Platform, to Instance Type name.
In this example, DOCKER is a Cloud Platform.

On error:
HTTP 401: Unauthorized | Missing or invalid JWT Authorization header

Terraform: List Nodes

Return a list of available images per Cloud Platform.

Terraform: List Nodes Request

GET https://<host>/as/v1/admin/terraform/node

Terraform: List Nodes Response

On success:

HTTP 200: OK

{
  "DOCKER": {
    "nyc3": [
      "origin-OPd2EOh0nX",
      "edge-Pgy8OzcMzM",
      "edge-PeoSHDWuZY"
    ]
  }
}

Details are a map of Cloud Platform, to Region, to Node ID.
In this example, DOCKER is a Cloud Platform, and nyc3 is a region.

On error:
HTTP 401: Unauthorized | Missing or invalid JWT Authorization header

Terraform: List Terraform Events

Return a list of events received from the AS-Terraform service.
Terraform events include errors reported by Terraform, among other information.

Terraform: List Terraform Events Request

GET https://<host>/as/v1/admin/terraform/node/events/<nodeGroupName>

nodeGroupName: the name of the NodeGroup. Note The nodeGroupName is used as a filter; if no such Node Group exists, then the request will succeed and collect no events, and return an empty map.

No body.

Terraform: List Events Response

On success:

HTTP 200: OK

{
  "20dS7MKNV3": [
    {
      "commandId": "27895a0a-091c-4a3b-90f7-4b44da2d2e83",
      "nodeId": "20dS7MKNV3",
      "status": "NODE_CREATING",
      "details": "Terraform NODE_CREATING node id: 20dS7MKNV3",
      "timestamp": 1710366402778,
      "executionTime": 0,
      "cloudPlatform": "DOCKER"
    },
    {
      "commandId": "27895a0a-091c-4a3b-90f7-4b44da2d2e83",
      "nodeId": "20dS7MKNV3",
      "status": "NODE_CREATED",
      "details": "Terraform created node id: 20dS7MKNV3",
      "messages": [
        "{"format_version":"1.0","terraform_version":"1.7.2"}"
      ],
      "timestamp": 1710366403507,
      "executionTime": 729,
      "cloudPlatform": "DOCKER"
    }
}

The response is a Map of String (Node ID, 20dS7MKNV3 in the example above) to an array of TerraformEvent. A TerraformEvent may contain the following fields:

commandId: an internal ID used to correlate automated Terraform commands

nodeId: the Node ID that the command is about

status: An enum. One of: NODE_CREATING, NODE_CREATED, NODE_CREATION_FAILED, NODE_DESTROYING, NODE_DESTROYED, NODE_DESTRUCTION_FAILED, or INFO.

details: contains free-form message details, including error messages.

messages: list of strings, each one is row of stdout of Terraform execution. Each string is actually serialized JSON. Contains details about the event, including errors Terraform encountered during execution.

executionTime: time taken to execute the Terraform command, in milliseconds.

cloudPlatform: the Cloud Platform being used.

timestamp: in epoch milliseconds

On error:

HTTP 401: Unauthorized | Missing or invalid JWT Authorization header

Terraform: List Regions

Return a list of available images per Cloud Platform.

Terraform: List Regions Request

GET https://<host>/as/v1/admin/terraform/region

Terraform: List Regions Response

On success:
HTTP 200: OK

{
  "DOCKER": [
    "default"
  ]
}

Details are a map of Cloud Platform, to Region name.
In this example, DOCKER is a Cloud Platform.

On error:
HTTP 401: Unauthorized | Missing or invalid JWT Authorization header

Terraform: Update Info

Prompt the server to query Terraform for updated information from each cloud platform about: instance types, images, nodes, and regions. This request proceeds in the background and when responses arrive, the internal stores serving the above Terraform requests are updated.

Terraform: Update Info Request

GET https://<host>/as/v1/admin/terraform/updateInfo

Terraform: Update Info Response

On success:
HTTP 200: OK

Body: N/A

This request immediately and supplies no information of its own.

On error:
HTTP 401: Unauthorized | Missing or invalid JWT Authorization header

List All Scheduled NodeGroups

Retrieve the set of the names of all scheduled NodeGroupConfigs.

For more information about scheduled configuration changes, see Scheduling in the NodeGroupConfig doc below.

List All Scheduled NodeGroups Request

GET https://<host>/as/v1/admin/node group/scheduled

List All Scheduled NodeGroups Response

On success:

HTTP 200: OK

[
    "allinone-overlay-1"
]

The response is an array of scheduled node group names.

On error:

HTTP 401: Unauthorized | Missing or invalid JWT Authorization header

List Scheduled NodeGroupConfig Overlays

Retrieve all overlays of a given base NodeGroupConfig.

Read Scheduled NodeGroups Request

GET https://<host>/as/v1/admin/node group/scheduled/<nodeGroupName>

nodeGroupName: Required. The name of the base NodeGroupConfig.

Read Scheduled NodeGroups Response

On success:

HTTP 200: OK

[
    "allinone-overlay-1"
]

On error:

HTTP 401: Unauthorized | Missing or invalid JWT Authorization header

HTTP 404: Not Found | No master NodeGroupConfig found for nodeGroupName

List Proxy Sessions

Retrieve a list of all active AS-Proxy WebRTC connections.

Note that ordinarily, proxy connections are short-lived during the initial handshake — only in the persistent WebSocket case, where no switch to datachannel is allowed, is a proxy connection maintained for the duration of the WebRTC stream.

List Proxy Sessions Request

GET https://<host>/as/v1/admin/rtcproxy

List Proxy Sessions Response

On success:

HTTP 200: OK

{
  "NodeKey|subscriber-7f17": {
    "streamGuid": "subscriber-7f17",
    "status": "CONNECTED",
    "sessionId": "b",
    "timestamp": 1709322384859
  }
}

On error:
HTTP 401: Unauthorized | Missing or invalid JWT Authorization header

Kick Proxy Session

Force an AS-Proxy session to close.

Kick Proxy Session Request

DELETE https://<host>/as/v1/admin/rtcproxy/<sessionIdOrStreamGuid>

sessionIdOrStreamGuid: to terminate an individual session, supply sessionId. Or, terminate all sessions for a stream, give the streamGuid.

Kick Proxy Session Response

On success:

HTTP 200: OK

No body.

On error:
HTTP 401: Unauthorized | Missing or invalid JWT Authorization header

HTTP 404: Not Found | No matching sessionId or streamGuid found.

Read NodeGroup Status

Return the status of all cluster nodes in the NodeGroup.

Read NodeGroup Status Request

GET https://<host>/as/v1/admin/node group/status/<nodeGroupName>?subGroup=<subGroupName>&role=<nodeRoleName>&state=<scalingState>&metrics=false&includeScaling=true&includeNode=true

nodeGroupName: Required. String. The name of an active NodeGroup.

subGroup: Optional. String. The name of the desired SubGroup (filter results).

role: Optional. String. The name of the desired NodeRole (filter results).

state: Optional. String. The desired scaling state (filter results). Valid states are (CAPITALIZED): REQUESTED, CREATING, CREATED, STARTED, INSERVICE, SOFT_SUNSET, HARD_SUNSET, DESTRUCTION_REQUESTED, DESTROYING, and FAILED. Not that there is no “destroyed” state: destroyed nodes disappear completely.

metrics: Optional. Boolean. Default: false. Include metrics with ClusterNodeEvents.

includeNode: Optional. Boolean. Default: true. Include NodeEvents in response.

includeScaling: Optional. Boolean. Default: true. Include ScalingEvents in response.

Read NodeGroup Status Response

For a detailed list of Node Metrics, see Scale Rule Expressions — Metrics and Aggregation below

On success:

HTTP 200: OK

[
  {
    "nodeEvent": {
      "nodeId": "4JGcDmF7ne",
      "nodeGroupName": "TEST-NODE-GROUP-tnO8Fh",
      "subGroupName": "ashburn",
      "nodeRoleName": "origin",
      "metrics": [
        {
          "name": "cpu",
          "children": [
            {
              "name": "processors",
              "value": 2
            },
            {
              "name": "system",
              "children": [
                {
                  "name": "load",
                  "value": 0.21240234375
                }
              ]
            },
            {
              "name": "process",
              "children": [
                {
                  "name": "load",
                  "value": 0.05237045203969129
                },
                {
                  "name": "time",
                  "value": 55190000000
                }
              ]
            }
          ]
        },
        {
          "name": "memory",
          "children": [
            {
              "name": "vm",
              "children": [
                {
                  "name": "free",
                  "value": 1897922560
                },
                {
                  "name": "max",
                  "value": 2147483648
                },
                {
                  "name": "total",
                  "value": 2147483648
                }
              ]
            },
            {
              "name": "system",
              "children": [
                {
                  "name": "committedvirtual",
                  "value": 18693241520128
                }
              ]
            }
          ]
        },
        {
          "name": "disk",
          "children": [
            {
              "name": "freeswap",
              "value": 0
            },
            {
              "name": "totalswap",
              "value": 0
            },
            {
              "name": "maxfiledescriptorcount",
              "value": 1000000
            },
            {
              "name": "openfiledescriptorcount",
              "value": 184
            }
          ]
        },
        {
          "name": "connections",
          "children": [
            {
              "name": "publisher",
              "value": 3
            },
            {
              "name": "subscriber",
              "value": 0
            },
            {
              "name": "client",
              "value": 3
            },
            {
              "name": "lastPubSubMs",
              "value": 292499
            },
            {
              "name": "lastIdleMs",
              "value": 0
            }
          ]
        },
        {
          "name": "network",
          "children": [
            {
              "name": "ingress",
              "value": 1750.0
            },
            {
              "name": "egress",
              "value": 3500.0
            },
            {
              "name": "peakRateSent",
              "value": 4100.0
            },
            {
              "name": "peakRateReceived",
              "value": 1980.0
            },
            {
              "name": "cumulativeSent",
              "value": 16400.0
            },
            {
              "name": "cumulativeReceived",
              "value": 8000.6
            },
            {
              "name": "iftopDuration",
              "value": 4.0
            }
          ]
        }
      ],
      "parents": [],
      "publicIp": "150.136.118.96",
      "privateIp": "10.7.213.153",
      "timestamp": 1715201390635,
      "publisherCount": 3,
      "subscriberCount": 0,
      "clientCount": 3,
      "restreamerCount": 0,
      "postProcessorCount": 0
    },
    "scalingEvent": {
      "nodeGroupName": "TEST-NODE-GROUP-tnO8Fh",
      "subGroupName": "ashburn",
      "nodeRoleName": "origin",
      "nodeId": "4JGcDmF7ne",
      "state": "INSERVICE",
      "timestamp": 1715201073409
    }
  }
]

Note that for each node in the cluster, the report includes the most recent nodeEvent, with metrics and statistics as well as the most recent scalingEvent with server state.

On error:

HTTP 401: Unauthorized | Missing or invalid JWT Authorization header

HTTP 404: Not Found | No master NodeGroupConfig found for nodeGroupName

Get NodeGroup Idle Time

Retrieve the minimum lastIdleMs metric from all INSERVICE nodes in the given NodeGroup.

Get NodeGroup Idle Time Request

GET https://<host>/as/v1/admin/node group/<nodeGroupName>/idle

Get NodeGroup Idle Time Response

On success:

HTTP 200: OK

{
    "lastPubSubMs": 574000
}

On error:

HTTP 401: Unauthorized | Missing or invalid JWT Authorization header

HTTP 404: Not Found | No master NodeGroupConfig found for nodeGroupName

Evaluate Capacity Expression (with live metrics)

Evaluate a Capacity Expression and return the result. This request is used to experiment with expressions while taking no action. For more information about Capacity Expressions and ScaleRule Expressions in general, see ScaleRule Expressions below.

All Capacity Expressions require a set of server metrics. This version of this request requires the ID of a live cluster node, and uses its most recent metrics for evaluation. You can query node metrics yourself for current values with Read NodeGroup Status, see above.

Evaluate Capacity Expression (live metrics) Request

POST https://<host>/as/v1/admin/evaluate/capacity/<nodeGroupName>/<nodeId>

nodeGroupName: the name of the NodeGroup where the node exists

nodeId: the ID of the cluster node to use as the source of the metrics.

Body: String. The Capacity Expression.

cpu.system.load * 10

Evaluate Capacity Expression (live metrics) Response

The response contains both doubleResult which is used within the AS-Autoscaling service to sort nodes when scaling in, and integerResult which is the rounded 32-bit value used in the AS-Streams service to direct traffic.

On success:

HTTP 200: OK

Body:

{
   "doubleResult" : 0.0555864369093941,
   "integerResult" : 0
}

On error:

HTTP 400: Error while evaluating expression (see response).

HTTP 401: Unauthorized | Missing or invalid JWT Authorization header

HTTP 404: Not Found | No master NodeGroupConfig found for nodeGroupName, no Node found for nodeId.

Evaluate Capacity Expression (with user metrics)

Evaluate a Capacity Expression and return the result. This request is used to experiment with expressions while taking no action. For more information about Capacity Expressions and ScaleRule Expressions in general, see ScaleRule Expressions below.

All Capacity Expressions require a set of server metrics. This version of this request takes metrics in the body of the request (from the user). This request does not require an active Node Group. You can query node metrics yourself for examples with Read NodeGroup Status, see above.

Evaluate Capacity Expression (user metrics) Request

POST https://<host>/as/v1/admin/evaluate/capacity/

Body:

{
    "expression":"cpu.process.load * 20",
    "metrics":[{"name":"cpu","children":[{"name":"processors","value":20},{"name":"system","children":[{"name":"load","value":0.43}]},{"name":"process","children":[{"name":"load","value":0.058589355798315125},{"name":"time","value":26546875000}]}]},{"name":"memory","children":[{"name":"vm","children":[{"name":"free","value":1786280192},{"name":"max","value":2147483648},{"name":"total","value":2147483648}]},{"name":"system","children":[{"name":"committedvirtual","value":2479390720},{"name":"freephysical","value":2266726400},{"name":"totalphysical","value":34276769792}]}]},{"name":"disk","children":[{"name":"freeswap","value":11859357696},{"name":"totalswap","value":79887192064}]},{"name":"connections","children":[{"name":"publisher","value":0},{"name":"subscriber","value":0},{"name":"client","value":0}]}]
}

expression: the Capacity Expression to evaluate

metrics: the set of metrics to use (should contain all metrics that real nodes provide)

Evaluate Capacity Expression (user metrics) Response

On success:

HTTP 200: OK

Body:

{"doubleResult":1.1717871159663025,"integerResult":1}

On error:

HTTP 400: Error while evaluating expression (see response).

HTTP 401: Unauthorized | Missing or invalid JWT Authorization header

Evaluate Scale Expression (with live metrics)

Evaluate a Scale Expression and return the result. This request is used to experiment with expressions while taking no action. For more information about Scale Expressions and ScaleRule Expressions in general, see ScaleRule Expressions below.

All Scale Expressions require a set of aggregated server metrics for a particular NodeRole within a particular SubGroup (a “SubGroup-Role”). This version of this request requires a nodeGroupName, subGroupName and role, which must identify a real SubGroup-Role with current aggregate metrics (there must be at least one running cluster node).

Evaluate Scale Expression (live metrics) Request

POST https://<host>/as/v1/admin/evaluate/scale/<nodeGroupName>/<subGroupName>/<nodeRoleName>

nodeGroupName: the name of the NodeGroup where the node exists

subGroupName: the name of the SubGroup to use as the source of the aggregated metrics.

nodeRoleName: the name of the NodeRole to use as the source of the aggregated metrics.

Body:

avg(cpu.process.load) < 1.0 && avg(cpu.system.load) < 2.0

On success:

HTTP 200: OK

Body:

{
   "result" : true
}

On error:

HTTP 400: Error while evaluating expression (see response).

HTTP 401: Unauthorized | Missing or invalid JWT Authorization header

HTTP 404: Not Found | No master NodeGroupConfig found for nodeGroupName, no Node found for nodeId.

Evaluate Scale Expression (with user metrics)

Evaluate a Scale Expression and return the result. This request is used to experiment with expressions while taking no action. For more information about Scale Expressions and ScaleRule Expressions in general, see ScaleRule Expressions below.

All Scale Expressions require a set of aggregated server metrics for a particular NodeRole within a particular SubGroup (a “SubGroup-Role”). This version of this request takes aggregate metrics in the body of the request (from the user). XXX hard to produce — can’t query for examples

Evaluate Scale Expression (user metrics) Request

POST https://<host>/as/v1/admin/evaluate/scale/

Body:

{
    "expression":"avg(cpu.process.load) u003c 1.0 u0026u0026 avg(cpu.system.load) u003c 2.0",
    "metrics":{"disk":{"name":"disk","sum":0.0,"count":0,"children":{"freeswap":{"name":"freeswap","sum":1.5E10,"min":5.0E9,"max":1.0E10,"count":2,"children":{}},"totalswap":{"name":"totalswap","sum":1.6E11,"min":8.0E10,"max":8.0E10,"count":2,"children":{}}}},"memory":{"name":"memory","sum":0.0,"count":0,"children":{"system":{"name":"system","sum":0.0,"count":0,"children":{"committedvirtual":{"name":"committedvirtual","sum":5.4E10,"min":2.5E10,"max":2.9E10,"count":2,"children":{}},"freephysical":{"name":"freephysical","sum":1.0E10,"min":3.0E9,"max":7.0E9,"count":2,"children":{}},"totalphysical":{"name":"totalphysical","sum":7.0E10,"min":3.5E10,"max":3.5E10,"count":2,"children":{}}}},"vm":{"name":"vm","sum":0.0,"count":0,"children":{"total":{"name":"total","sum":4.8E9,"min":2.4E9,"max":2.4E9,"count":2,"children":{}},"max":{"name":"max","sum":4.0E9,"min":2.0E9,"max":2.0E9,"count":2,"children":{}},"free":{"name":"free","sum":1.4E9,"min":4.0E8,"max":1.0E9,"count":2,"children":{}}}}}},"cpu":{"name":"cpu","sum":0.0,"count":0,"children":{"process":{"name":"process","sum":0.0,"count":0,"children":{"load":{"name":"load","sum":0.20800000000000002,"min":0.05,"max":0.158,"count":2,"children":{}},"time":{"name":"time","sum":5.109375E10,"min":2.4546875E10,"max":2.6546875E10,"count":2,"children":{}}}},"system":{"name":"system","sum":0.0,"count":0,"children":{"load":{"name":"load","sum":1.06,"min":0.43,"max":0.63,"count":2,"children":{}}}},"processors":{"name":"processors","sum":40.0,"min":20.0,"max":20.0,"count":2,"children":{}}}},"connections":{"name":"connections","sum":0.0,"count":0,"children":{"subscriber":{"name":"subscriber","sum":134.0,"min":37.0,"max":97.0,"count":2,"children":{}},"publisher":{"name":"publisher","sum":38.0,"min":12.0,"max":26.0,"count":2,"children":{}},"client":{"name":"client","sum":220.0,"min":80.0,"max":140.0,"count":2,"children":{}}}}}
}

expression: the Capacity Expression to evaluate

metrics: the set of metrics to use (should contain all metrics that real nodes provide, in aggregate format with sum, min, max, and count fields), representing the aggregated metrics for nodes belonging to a SubGroup-Role.

Evaluate Scale Expression (user metrics) Response

On success:

HTTP 200: OK

Body:

{"result":true}

result: the boolean result of the given expression

On error:

HTTP 400: Error while evaluating expression (see response).

HTTP 401: Unauthorized | Missing or invalid JWT Authorization header

NodeGroupConfig

The NodeGroupConfig defines how a Node Group (a cluster) will be automatically scaled.

Data Model

Node Group Config Data Model

NodeGroupConfig

The main class for NodeGroup configuration.

name: Required. String. Min 1 char, max 255 chars. Alphanumeric, plus -, _, and .. Must be GLOBALLY unique (no two Node Groups can share the same name). Additionally, the name default is reserved (see usage in AS-Streams documentation, Get Server For Publish/Subscribe).

description: Optional. String. Max 4096 chars.

isScalingPaused: Optional. Boolean. Default false. If true, no autoscaling will occur. (Use with Update NodeGroupConfig to enable or disable autoscaling.)

cloudPlatform: Required. String. One of: AWS, AZURE, DO, DOCKER, GCP, KUBERNETES, LINODE, OCI, or VSPHERE.

cloudProperties: Optional. String. Max 4096 chars. NodeGroup-level cloudProperties. See About CloudProperties below. This is a good place for cloud properties that are common to all servers in the node group.

clusterPassword: Optional. String. Max 255 chars. The cluster password is mostly used for inter-node communication. If no cluster password is set when creating a Node Group, a new clusterPassword will be generated.

maxPublishers: Optional. Integer. If specified, AS-Streams service will limit the maximum number of publishers. Get Server for Publish will return 503: Service Unavailable when overloaded.

maxSubscribers: Optional. Integer. If specified, AS-Streams service will limit the maximum number of subscribers. Get Server for Subscribe will return 503: Service Unavailable when overloaded.

baseNodeGroupName: Optional. String. Min 1 char, max 255 chars. The base config name to overlay if this NodeGroupConfig is scheduled (see schedules below). An overlay becomes active at the specified time(s) according to schedules and replaces the designated base NodeGroupConfig. During overlay, this NodeGroupConfig masquerades as the base config by assuming its name and replacing it.

shuffleSizeExpression: Required. String. Max 4096 chars.

schedules: A map of NodeGroupSchedule by NodeGroupSchedule.name (see below).

images: A map of NodeImage by NodeImage.name (see below).

roles: A map of NodeRole by NodeRole.name (see below).

propertyOverrides: Optional. A list of NodePropertyOverrides (see below).

subGroups: A map of SubGroupConfig by SubGroup.name (see below).

NodeGroupSchedule

A schedule definition. Schedules can operate in two ways:

In the first case, baseNodeGroupName is not specified (or is null). In this case, the schedule governs when this NodeGroupConfig will take effect. At startAt, the node group will be created and scaled out, after durationS seconds have elapsed, the node group will be destroyed. Multiple NocdeGroupSchedule can be assinged to a single NodeGroupConfig to scale it out at different times.

In the second case, baseNodeGroupName specifies a base NodeGroupConfig. In this case, the schedule governs when this NodeGroupConfig will “overlay” the base node group: during overlay, this NodeGroupConfig masquerades as the base config by assuming its name and replacing it.

name: Required. String. The name of the schedule.

startAt: Required. String. Min 1 char, max 999 chars. Cron expression for event start. For assistance with cron expressions, see http://www.cronmaker.com

durationS: Required. Integer. Min 60 (one minute), max 31536000 (365 24 60 * 60, one year). Cron expression for event end.

NodeImage

A NodeImage represents a disk image within the node group’s cloud platform. Multiple roles can share a single image, or special disk images can be prepared for a given role or set or roles.

name: Required. String. Min 1 char, max 255 chars. Alphanumeric, plus -, _, and .. Must be unique within this NodeGroupConfig.

image: Required. String. Min 2 chars, max 1024 chars. The identifier of the disk image within the cloud platform.

cloudProperties: Optional. String. Max 4096 chars. NodeImage-level cloudProperties. See About CloudProperties below. This is a good place for cloud properties about the instance type (to define how much hardware is required for a node with this image).

NodeRole

A NodeRole represents a server role, based on a given disk image, having a certain set of capabilities, and given cluster parenting rules. Commonly defined roles might be “origin”, “edge”, “transcoder”, etc.

name: Required. String. Min 1 char, max 255 chars. Alphanumeric, plus _, and . (Note that this does NOT include - which is specifically reserved for NodeRole.name). Must be unique within this NodeGroupConfig.

imageName: Required. String. Min 1 char, max 255 chars. The name of the disk image to use when scaling out servers of this role. A NodeImage with matching NodeImage.namge must be defined within this NodeGroupConfig.

capabilities: Required. Set of Capability. A Capability is a String, one of SUBSCRIBE, PUBLISH, TRANSCODE, MIX, XILINX, or ZIXI. This represents what kinds of tasks server nodes of this role are capable of. Publish requests will find servers belonging to roles that can PUBLISH. Subscribe requests will find servers belonging to roles that can SUBSCRIBE, and transcoding provisions will go to transcoders that can TRANSCODE and may optionally have special hardware, XILINX or ZIXI. See AS-Streams service documentation for pub/sub requests and provisioning.

parentRoleName: Optional. String. Min 1 char, max 255 chars. Alphanumeric, plus -, _, and .. The name of the parent role. A NodeRole with matching NodeRole.namge must be defined within this NodeGroupConfig. When cluster nodes of this role are scaled out, they will be assigned parents (or not) according to the parentRoleName and parentCardinality.

propertyOverrides: Optional. A list of NodePropertyOverrides (see below).

cloudProperties: Optional. String. Max 4096 chars. NodeRole-level cloudProperties. See About CloudProperties below.

SubGroupConfig

A “SubGroup” is an abstract hierarchical container may hold a set of ScaleRules for one or more NodeRoles and may have one or more child SubGroupConfigs defining child SubGroups. A SubGroup can be used to model various cloud platform topologies, such as organizing nodes into regions, or a hierarchy, such as geozones that contain regions that contain nodes.

subGroupName: Required. String. Min 1 char, max 255 chars. The name of this SubGroup. Must be unique within this NodeGroupConfig.

nodeGroupName: Required. String. Min 1 char, max 255 chars. The name of the parent node group. This must match the containing NodeGroupConfig.name.

groupType: Required. String. Min 1 char, max 255 chars. A label for this type of SubGroup, such as region, or geoZone.

rulesByRole: Optional. Map of ScaleRule by NodeRole.name. If a ScaleRule for a given NodeRole exists in the SubGroupConfig, then nodes of that role will be scaled out at runtime according to that ScaleRule.

cloudProperties: Optional. String. Max 4096 chars. NodeRole-level cloudProperties. See About CloudProperties below. This is a good place for cloud properties relevant to the cloud platform container that you are modeling, such are role or geoZone.

childGroups: Optional. A set of one or more child SubGroupConfig.

ScaleRule

A ScaleRule governs how nodes of a given NodeRole will be automatically scaled in and out within a given SubGroup.

subGroupName: Required. String. Min 1 char, max 255 chars. The name of the parent SubGroup. This must match the containing SubGroupConfig.name.

nodeGroupName: Required. String. Min 1 char, max 255 chars. The name of the node group. This must match the containing NodeGroupConfig.name.

parentRoleName: Required. String. Min 1 char, max 255 chars. Alphanumeric, plus -, _, and .. The name of the role. A NodeRole with matching NodeRole.namge must be defined within this NodeGroupConfig.

min: Required. Integer. Min 0, max 1024. Must be less than or equal to ScaleRule.max. The minimum number of nodes belonging to this role required for this SubGroup. The Autoscaling service will never create more than max nodes of this NodeRole within the parent SubGroup.

max: Required. Integer. Min 0, max 1024. Must be greater than or equal to ScaleRule.min. The maximum number of nodes belonging to this role required for this SubGroup. The Autoscaling service will never create more than max nodes of this NodeRole within the parent SubGroup.

increment: Required. Integer. Min 1, max 1024. When the Autoscaling service determines that new nodes must be scaled out (or in) by this ScaleRule, it will scale out (or in) increment nodes at a time.

inExpression: Required. String. Max 4096 chars. A Scale Expression (evaluates to Boolean). When this expression is true, increment nodes will be scaled in (down to min). See Scale Expressions below.

outExpression: Required. String. Max 4096 chars. A Scale Expression (evaluates to Boolean). When this expression is true, increment nodes will be scaled out (up to min). See Scale Expressions below.

capacityRankingExpression: Required. String. Max 4096 chars. A Capacity Expression (evaluates to Integer). This expression is used when nodes must be sorted by capacity (for example to decide which running node(s) to scale in, or when finding a server for a publisher, etc). See Capacity Expressions below.

capacityLimitExpression: Required. String. Max 4096 chars. A Capacity Expression (evaluates to Integer). When this expression is true for a given node, that node is at its capacity limit. The AS-Streams service will not return the node for publish or subscribe if the node is at or beyond its capacity limit. See Capacity Expressions below.

NodePropertyOverrides

A single NodePropertyOverrides object specifies the name of a properties file (which must be located under red5pro in the conf/, plugins/, or webapps/ directories, and must end with .properties), and a list of properties to modify. Property overrides can be specified globablly (for all nodes) in NodeGroupConfig.propertyOverrides or else per role in NodeRole.propertyOverrides. When a new node is created, it receives a concatenated list of overrides, NodeGroupConfig overrides first, followed by NodeRole overrides. All overrides are applied in order. It is valid to override the same properties file multiple times, or to modify the same property multiple times. In this case, the last change “wins”. Comments are preserved. Properties can be modified and created but not removed.

fileName: Required. Max 2048 chars. The properties file to modify. If the file is not a valid target or no such file is found, no changes will be made.

properties: Required. A map of name value pairs. Where the key matches an existing property found in the file, the property’s value will be modified. If no existing property is found, a new property will be appended to the end of the file.

About CloudProperties

At various levels in the data model youy can specify cloudPropeties. When the Autoscaling service prepares to create a new node, it concatenates the relevant cloudProperties fields (adding ; delimeter) and passes the resulting String to the Terraform Service.

This allows you to specify Terraform parameters in different groups as makes sense. For example, the NodeGroupConfig could contain security_group (all nodes share), each SubGroup can define its own region (to model regions), and each NodeImage can specify the instance_type, allowing role-based instance types.

Example NodeGroupConfigs

Find several examples JSON files in the folder NodeGroupConfig_examples.

Scale Rule Expressions

ScaleRule Expressions allow users to specify scaling thresholds of nearly arbitrary complexity (limited in overall length to 4096 characters, and limited to the metrics actually provided by the cluster nodes.) This allows you to control when nodes are added to the cluster, when they are removed, and how the system decides which nodes will be removed as well as which nodes are best to receive traffic requests.

Metrics

Metrics organized hierarchically in categories. Nodes report metrics periodically — the rate is controlled by the Autoscale Plugin configuration on the cluster nodes (in the the Red5Pro server image).

Here we will use a dot-based notation where . denotes a parent-child relationship.

cpu.processors : Long, count. Total effective CPU cores.

cpu.system.load : Double. System load average. The precise meaning and averaging period are operating system-dependent.

cpu.process.load : Double. Process load average. Again, precise meaning and averaging period are operating system-dependent.

cpu.process.time : Long. Process time, in milliseconds.

memory.vm.free : Long. VM free space, in bytes.

memory.vm.max : Long. VM max allocated, in bytes.

memory.vm.total : Long. VM total allocated, in bytes.

memory.system.committedvirtual : Long. Total allocated memory, in bytes.

disk.freeswap : Long. Free swap space, in bytes.

disk.totalswap : Long. Total swap space, in bytes.

disk.maxfiledescriptorcount : Long. Maximum number of open files.

disk.openfiledescriptorcount : Long. The number of currently open file descriptors.

connections.publisher : Long. Publisher connection count.

connections.subscriber : Long. Subscriber connection count.

connections.client : Long. Total number of client connections (including publishers, subscribers, cluster connections, restreamers, etc).

connections.lastPubSubMs : Long. Initially null. Total time since last publish or subscribe initiated on this node, in milliseconds.

connections.lastIdleMs : Long. Initially 0. Time without any active publisher or subscriber connection on this node, in milliseconds.

Network metrics

Network metrics are derived from ‘iftop’. The server nodes will periodically run the iftop command and report the results. The frequencey and duration of the network sampling are configurable in autoscale.xml as ‘bwSampleRate’ and ‘bwSampleDuration’. They can be overridden by the NodeStartConfig values if ‘nodeStartConfig.bwSampleDuration’ is non-zero. The iftop utility provides several data points. BW_last_2_seconds, BW_last_10_seconds, BW_last_40_seconds, peak_send rate, peak receive rate, cumulative sent, cumulative received. Metrics also include ‘iftopDuration’ which is the sampling duration. ‘bwSampleRate’ minimum 0, maximum long_max_value. bwSampleRate is the resting time between network sampling.

network.iftopDuration : Duration of network sampling in seconds. Minimum 1, maximum 30.

network.ingress : Network receive rate average for the last 2 seconds, in kilobits per second.

network.egress : Network send rate average for the last 2 seconds, in kilobits per second.

network.peakRateReceived : Network peak received rate for the last ‘iftopDuration’ seconds, in kilobits per second.

network.peakRateSent : Network peak send rate for the last ‘iftopDuration’ seconds, in kilobits per second.

network.cumulativeReceived : Network cumulative bytes received for the last ‘iftopDuration’ seconds, in kilobytes.

network.cumulativeSent : Network cumulative bytes sent for the last ‘iftopDuration’ seconds, in kilobytes.

When iftop duration is greater than 2, metrics include a second set of data points, ‘ingress_10’ and ‘egress_10’. If iftop duration is less than 10, the label will reflect the number of seconds. Example : ‘ingress_6’

network.ingress_10 : Network receive rate average for the last 10 seconds, in kilobits per second.

network.egress_10 : Network send rate average for the last 10 seconds, in kilobits per second.

When iftop duration is greater than 10, metrics include a third set of data points, ‘ingress_30’ and ‘egress_30’. If iftop duration is less than 30, the label will reflect the number of seconds. Example : ‘ingress_20’

network.ingress_30 : Network receive rate average for the last 30 seconds, in kilobits per second.

network.egress_30 : Network send rate average for the last 30 seconds, in kilobits per second.

Capacity Expressions

A Capacity Expression is an arithmetic expression that evaluate to a numeric value. A ScaleRule defines two different Capacity Expressions: the capacityRankingExpression, and the capacityLimitExpression. See ScaleRule above.

Capacity Expresions combine metrics for a given node. Therefore, there is no aggregation and all metrics are singular. Any of the above metrics are allowed in a Capacity Expression.

Internally, a capacityRankingExpression is evaluated at 64-bit floating point (double) precision. In the AS-Autoscale service, the double value is used to sort nodes to decide which node(s) to scale in. Meanwhile, in the AS-Streams service, the resulting value (after evalutation) is then truncated to a 32-bit integer before it is used to compare against the capacityLimitExpression to determine whether the node is or is not under capacity (available).

Also allowed are the operators +, -, *, and /, with conventional precedence. As well as parentheses ( and ) to specify order of precedence.

Numeric values may also be specified, either with or without a decimal.

Example Capactiy Expressions

Expression: 2.12
Result: 2

Expression: 2 + 2.12
Result: 4

Expression: (5 - 3.14159) * 2
Result: 3

Expression: cpu.processors
Result: 20

Expression: cpu.system.load * 10
Result: 4

Expression: cpu.system.load * 10 + cpu.process.load * 20
Result: 5

Expression: cpu.system.load * 10 + cpu.process.load * 20 + 27.93 * (memory.system.committedvirtual / (1024 * 1024 * 1024) + disk.freeswap / (20 * 1024 * 1024 * 1024))
Result: 85

Metric values for above examples

cpu.processors : 20

cpu.system.load : 0.43

cpu.process.load : 0.058589355798315125

cpu.process.time : 26546875000

memory.vm.free : 1786280192

memory.vm.max : 2147483648

memory.vm.total : 2147483648

memory.system.committedvirtual : 2479390720

memory.system.freephysical : 2266726400

memory.system.totalphysical : 34276769792

disk.freeswap : 11859357696

disk.totalswap : 79887192064

connections.publisher : 0

connections.subscriber : 0

connections.client : 0

Scale Expressions

Scale Expressions are different than Capacity Expressions in two ways. First, Scale Expressions evaluate to a Boolean value to indicate if scaling should occur. Second, Scale Expressions are used to evaluate all of the nodes of a given Role in a given SubGroup together — therefore Scale Expressions act on aggregate values.

Internally, Scale Expressions are evaluated as 64-bit floating point (double) precision or as Boolean values. The result must always be a Boolean value.

Scale expressions consist of aggregates and numeric values combined with the following operators.

The operators +, -, *, and / are allowed, with conventional precedence. As well as parentheses ( and ) to specify order of precedence.

Additionally, the comparison operators > and < are included. These operators compare numeric expresions and produce a Boolean result.

Further, the Boolean operators && (AND), || (OR), and the unary negation operator ! (NOT) are used to combine Boolean expressions.

Aggregation

In a Scale Expression, any metric must be wrapped in an aggregation function to indicate how the server will aggregate the metric values for all nodes belonging to the given NodeRole within the given SubGroup.

The available aggregation functions are:

avg: average

count: count (number of contributing nodes)

min: minimum value

max: maxmium value

sum: total value

Example Scale Expressions

Expression: 2 < 1.0
Result: false

Expression: 3.14 > 3
Result: true

Expression: 1 * 2 * 3 + 4 > 3.333 * (2 / 0.14)
Result: false

Expression: avg(cpu.process.load) < 1.0 && avg(cpu.system.load) < 2.0
Result: true

Expression: count(cpu.processors) < 10 && avg(cpu.system.load) > 2.0
Result: false

Expression: min(cpu.system.load) < 0.5
Result: true

Expression: max(cpu.system.load) > 0.5
Result: true

Expression: avg(cpu.system.load) > 0.52 && avg(cpu.system.load) < 0.54
Result: true

Metric values for above examples

Node: node1

cpu.processors : 20

cpu.system.load : 0.43

cpu.process.load : 0.05

cpu.process.time : 26546875000

memory.vm.free : 1000000000

memory.vm.max : 2000000000

memory.vm.total : 2400000000

memory.system.committedvirtual : 25000000000

memory.system.freephysical : 7000000000

memory.system.totalphysical : 35000000000

disk.freeswap : 10000000000

disk.totalswap : 80000000000

connections.publisher : 12

connections.subscriber : 37

connections.client : 80

Node: node2

cpu.processors : 20

cpu.system.load : 0.63

cpu.process.load : 0.158

cpu.process.time : 24546875000

memory.vm.free : 400000000

memory.vm.max : 2000000000

memory.vm.total : 2400000000

memory.system.committedvirtual : 29000000000

memory.system.freephysical : 3000000000

memory.system.totalphysical : 35000000000

disk.freeswap : 5000000000

disk.totalswap : 80000000000

connections.publisher : 26

connections.subscriber : 97

connections.client : 140

Shuffle Size Expressions

A Shufle Size Expression uses a single metric (or none at all): nodeCount. The result is an integer value, which represents the maximum number of available servers to consider “best” before shuffling for return in Get Server For Publish or Get Server For Subscribe requests (see AS-Streams doc.)

The operators +, -, *, and / are allowed, with conventional precedence. As well as parentheses ( and ) to specify order of precedence.

Two mathematical functions are implemented:

min: min(a, b) retuns the minimum value, a or b.

max: max(a, b) returns the maximum value, a or b.

Example Shuffle Size Expressions

Expression: 2
Result: 2

Expression: nodeCount * 0.1
Result: 3

Expression: min(nodeCount * 0.1, 5)
Result: 5

Expression: max(nodeCount * 0.1, 2)
Result: 3

Metric values for above examples

nodeCount: 30