EMS Builder Performance Test Manual

Manual version: v1.1
Current statistical rule: collect two extra boundary samples, one at each end; formal results use only the samples between them.

Test note: no real meters, PLCs, or serial gateways are connected. All tests use simulated Modbus TCP devices. Results represent Edge-EMS Builder processing capacity under ideal communication conditions, not end-to-end performance at actual sites.

1. Test scope

Area	Question to answer
Data acquisition	How many points per second can EMS Builder process reliably?
Trigger	Do Lua tasks run as scheduled, with consistent durations?
MQTT / coordination tasks	Are task counts achieved? Are there delays or failures?
Data storage	How do real-time and change-based storage affect performance?
System resources	Are CPU, memory, or disk approaching limits?
Stability	Does extended operation produce crashes, restarts, or sustained backlogs?

2. Test load

2.1 Work EMS Builder executes concurrently

Load	Test content
Data acquisition	Read points from 20 simulated slaves
Aggregation	Calculate acquired data with Lua
Trigger	Periodically execute Lua scripts producing writes, alarms, and MQTT actions
MQTT tasks	Periodically generate and publish MQTT messages
Coordination tasks	Share data and coordinate processing between devices
Data storage	Save acquisition data, calculation results, and task records

2.2 What each Trigger Lua execution does

Operation	Count per execution	Purpose
Basic calculation loop	100	Mathematical calculations, conditions, loops, strings, and table operations
Raw point writes	2	Write test points `小数据0` and `小数据1` respectively
Aggregation point write	1	Write test aggregation point `Agg`
Local alarms	2	Generate two level-1 alarms
MQTT alarm	1	Generate one level-2 MQTT alarm

2.3 Actual load per second

Metric	Calculation	Per second
Lua script executions	Trigger: `50 at 1s + 5 at 100ms = 100/s`; MQTT: `50/s`; aggregation: `50/s`; sharing: `10/s`	210
Trigger basic calculation iterations	`100 × 100`	10,000
Trigger raw point writes	`100 × 2`	200
Trigger aggregation point writes	`100 × 1`	100
Trigger local alarms	`100 × 2`	200
Trigger MQTT alarms	`100 × 1`	100
All Trigger write actions	`100 × 6`	600
MQTT publications	`50 × 1`	50
Data-change-triggered aggregation calculations	`50 × 1`	50
Data-sharing tasks	`10 × 1`	10

3. Standard scenarios

Calculation:

Trigger executions/second = 1s Trigger count + 100ms Trigger count × 10

Scenario	Points/s	Aggregations	1s Trigger	100ms Trigger	Trigger/s	MQTT/s	Coordination/s	Storage	Duration
S1	500	10	10	0	10	5	0	Real-time	30 minutes
S2	2,000	20	20	2	40	10	2	Real-time	30 minutes
S2.5	2,000	20	20	2	40	10	2	Change-based	30 minutes
S3	5,000	30	50	4	90	20	10	Real-time	30 minutes
S3.5	5,000	30	50	4	90	20	10	Change-based	30 minutes
S4	10,000	30	100	10	200	40	20	Change-based	30 minutes
S5	20,000	40	200	10	300	60	40	Change-based	30 minutes
S6	5,000	20	50	10	150	30	15	Real-time	72 hours

Storage mode	Description
Real-time storage	Save every acquisition result as configured
Change-based storage	Save immediately when values change; if unchanged, save at least once every 10 seconds

S5 explores performance limits. S6 must actually run for 72 hours.

4. Sampling and statistical rules

4.1 Count-based sampling

Short performance retests use a fixed sample count specified by the report for the relevant batch. The following example collects 1,002 samples per scenario at 1-second intervals; AFE-E630 report v1.3 collects 2,002 samples, of which 2,000 are valid:

Sample 1: boundary sample, excluded
Samples 2–1001: formal samples, 1,000 in total
Sample 1002: boundary sample, excluded

Formal reports must filter raw_data by sample_no, rather than use the workbook summary sheet’s unfiltered totals, which include the first and last samples, directly.

For a full 30-minute test at 1-second intervals, collect 1,802 samples and analyze the middle 1,800. A short 1,002-sample dataset cannot replace S6’s 72-hour stability test.

4.2 Total EMS Builder resources

Sum these five services at the same sample_no before calculating statistics:

cb_core
cb_nginx
cb_backend
ems_backend
ems_postgresql

AEMS_CPU_raw(n) = sum of CPU for the 5 services at the same sample_no
4-core whole-device CPU(n) = AEMS_CPU_raw(n) ÷ 4
Edge-EMS Builder memory(n) = sum of memory for the 5 services at the same sample_no

Report at least average, P95, P99, and peak. Do not directly add independent peaks from the five services at different times.

4.3 Trigger and MQTT

Sort trigger_per_sec and mqtt_per_sec chronologically and discard the first and last second buckets to exclude partial seconds at query boundaries.

Task second-buckets do not necessarily correspond one-to-one with resource samples. A second may contain no records, or queued tasks may complete together in the following second. Report the valid bucket count together with:

Average per-second throughput and attainment relative to target.
Average, P50, P95, and P99 of per-second mean SpendTime.
Maximum individual SpendTime across all buckets.
Number and proportion of seconds with average SpendTime above 200 ms.

A complete second without task records must count as zero throughput. It has no calculable latency and must not be assigned a fabricated 0 ms. Future monitoring scripts should output explicit statistical-window start and end times so empty seconds can be included.

5. Metrics to observe

Category	Required statistics
Data acquisition	Configured points, successful points, failures, and discards
Trigger	Target count, actual count, average duration, P50, P95, P99, maximum
MQTT / coordination	Target count, actual count, duration, errors
CPU	Average, P95, and peak for the complete EMS Builder
Memory	Average, peak, and trend for the complete EMS Builder
Disk	Read/write throughput, I/O wait, database write status
Modbus	Reads/writes, failures, timeouts, and retries per slave
Stability	Crashes, OOM events, unexpected restarts, sustained backlogs

Key calculations:

Correctness = successes ÷ (successes + failures + timeouts + discards)

Attainment = actual average throughput per second ÷ scenario target

SpendTime is actual task execution duration. Reports must include P95, P99, and maximum as well as averages to expose occasional long delays.

6. Result assessment

Project contracts or acceptance specifications take priority. Without a separate agreement, use these guidelines.

Item	Recommended criterion
Throughput	Average attainment ≥ 99%
Trigger / MQTT duration	P95 < 200 ms
CPU	Average utilization < 80% on a four-core device, without sustained saturation
Memory	Whole-device average < 70%, without continual growth
Correctness	≥ 99.99%, with no unexplained discards
Stability	72 continuous hours without crashes, OOM events, or unexpected restarts
Recovery	Stable operation restored within 5 minutes after reducing load

6.1 Conditions invalidating a test

Condition	Action
Insufficient test duration	Mark incomplete
Not all 20 slaves participate	Retest
Writes concentrated on a few slaves	Redistribute and retest
Load generator or network is the bottleneck	Correct the issue and retest
Missing success, failure, or timeout data	Mark data incomplete
S6 did not run for 72 hours	Do not report stability passed

S4 and S5 are stress tests. At system limits, report “stress boundary reached” or “not recommended for continuous operation,” rather than a normal-capacity pass.

Model reports: