Fetch and write the firmware image on action 3 #49
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/wifi-firmware-fetch"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Action 3 now does the fetch UF-4 left a seam for: read the manifest from the
address in the control frame, find this node type's entry, and write the image
to the inactive application slot, reporting percent in
SYS_WIFI_STATUSbyte 2.The boot slot is untouched — the running image stays selected, so a power cut
mid-transfer costs the transfer alone. Verifying the digest and switching the
slot is CCS-UF-6's; it takes
slot(),bytesWritten()andexpectedSha256()off
FirmwareUpdateronceimageWritten()is true. Nothing else reads as acomplete image: a failure or a cancel aborts the slot and clears all three.
The manifest format is a contract with the bridge's Python side, so it is
specified in section 11.1: a small JSON object keyed by
0xplus the node typeas two hex digits, each entry carrying the image path, its length and its
SHA-256. Parsed by a scanner in
firmware_manifest.cpprather than a JSONlibrary. Unknown members are ignored, as everything else unknown on this bus is.
The policy core is host-testable, as
CanBusand the radio are: the manifest,the progress arithmetic, the retries and the state machine are exercised against
a fake transport and a fake slot, and the HTTP client and the flash writes sit
behind
#if defined(ARDUINO). The flash side uses the IDF'sesp_ota_*ratherthan Arduino's
Update, whoseend()would switch the boot partition.WifiUpdatergainsloop()andcancel(), andbeginUpdate()takes theinstant. The idle timeout still never cuts a transfer short; in exchange a
transfer cannot outlast itself — ten seconds without a byte abandons an attempt,
and five minutes abandons the fetch, so holding the radio off the timeout is not
a way to hold it up for good.
Tested:
pio test -e native(350 cases),pio runfor both nodes,pio run -e host_sim, and the manifest scanner under-fsanitize=address,undefinedacross every node type and every truncation ofthree documents. Flash goes from 36.7% to 39.7% of the slot on the bathroom node
and 38.9% to 42.3% on the lighting node — about 60 kB for the HTTP client and the
OTA calls.
One thing to know when this runs on hardware:
esp_ota_begin()erases the slotinside one
Node::loop(), roughly a second for an 880 kB image, so an updatesession drops a heartbeat or two.
Ref CCS-UF-5.
One out-of-bounds read that every real manifest triggers, one blocking call that takes the node off the bus for seconds, and a test whose name says the opposite of what it asserts. The rest is small.
esp_ota_*use is right: nothing callsesp_ota_set_boot_partition, and the ArduinoUpdateclass is correctly avoided. Failure and cancel paths clearslot_,written_andimage_before leavingWritten, so the handover accessors cannot read as complete after a half-write.@ -0,0 +26,4 @@}// Just enough JSON to walk objects and pick named members out of them, which is// cheaper than a parser library and all the manifest needs.Why there is no JSON library here is a ticket decision, not something a future editor of this file needs. Cut both lines.
@ -0,0 +173,4 @@matched = matched && *w == '\0';return true;}if (c == '\\') return false;A backslash anywhere in a key aborts the whole parse:
nextMemberreturnsmemberMalformedanddescendTogives up. So{"a\/b":1,"images":{...}}fails to parse (verified), against spec 11.1's "Members a node does not recognise are ignored". Skip a key with an escape in it rather than failing the document.@ -0,0 +174,4 @@return true;}if (c == '\\') return false;if (*w == '\0') {Out-of-bounds read, and an ordinary manifest triggers it. Once
*wis the terminator this setsmatched = falseandcontinues, which runs++wand walks the pointer past the end of the name literal; the next iteration reads*woff the end. Any JSON key two or more characters longer than a name compared before the match does it —"sha256"against"path"inreadEntry,"generated"against"images"indescendTo.Building
firmware_manifest.cppwith-fsanitize=addressand parsing the manifest from the spec aborts inmatchStringat this line. The committed tests exercise the same path and pass only becauseenv:nativehas no sanitizers — worth adding them there.Fix: stop advancing
wonce it reaches the terminator rather thancontinueing past it.@ -0,0 +220,4 @@const int8_t member = scanner.nextMember(names, 3, false);switch (member) {case 0:if (!scanner.readString(out.path, sizeof(out.path))) return false;"path": ""parses as a valid entry, and so does any relative path. The fetch then GETs an empty or relative path, fails and burns all three attempts. Reject it here with the rest of the field validation — spec 11.1 sayspathis absolute.@ -0,0 +40,4 @@if (length <= 0) return retry(now);if (fetchingManifest) {if (static_cast<uint32_t>(length) > maxManifestLength) {Untested, and it is the only thing between a large body and an overflow of
manifest_inpump().FakeTransport'smanifestbuffer is 512 bytes so no test can reach it; report a length overmaxManifestLengthfrom the fake directly.@ -0,0 +45,4 @@return false;}} else {if (static_cast<uint32_t>(length) != image_.size) return retry(now);A
Content-Lengththat disagrees with the manifest is as non-transient as a manifest with no entry for this node type, which fails at once and is commented as such. This retries it twice before giving up.@ -0,0 +75,4 @@}if (read == 0) {if (static_cast<uint32_t>(now - lastDataMs_) >= updateStallTimeoutMs) retry(now);lastDataMs_resets on every byte, so a server drip-feeding one byte every 9 s holdsinProgress()— and with it the radio and the idle timeout — for as long as it likes. Spec section 11 now claims a transfer "fails of its own accord rather than holding the radio up indefinitely". An overall deadline for the attempt would close it.@ -0,0 +21,4 @@http_.setConnectTimeout(requestTimeoutMs);http_.setTimeout(requestTimeoutMs);if (http_.GET() != HTTP_CODE_OK) {GET()connects and waits for the response header inside oneNode::loop()iteration, up to the 5 s timeout set just above. While it blocks the node sends no heartbeat and services no CAN, and spec 9.3 has peers treating a node as absent after 3 missed heartbeats — so an unreachable or slow server makes the lighting node fall back on the bathroom PIR mid-update, three times over.esp_ota_begin()'s erase of the whole image region inopen()stalls the loop similarly.Either cut the timeout below the 3 s absence window or say in spec section 11 that a node goes quiet while it fetches. Streaming is fine —
read()is bounded byavailable().Byte 2 never reaches 100: the updater leaves
inProgress()the instant it finishes, so the state drops to 3 andotaPercentto 0 on the same loop that would have reported 100. The bridge sees the percentage climb to ~97 and then vanish. If the state change is meant to be the completion signal, say so in spec section 11.@ -26,0 +26,4 @@// Called while the radio is up, which is the only time a fetch can run.virtual void loop(uint32_t now) = 0;// The session has ended under it: whatever is in flight stops.Both new comments restate the method they sit on — cut them.
cancel()says the session ended and the fetch stops;loop()says it is pumped. Neither loses anything a future editor could not read off the interface.@ -0,0 +131,4 @@// A path that will not fit is refused rather than truncated: a truncated path// fetches the wrong file or nothing at all.void test_a_path_too_long_to_hold_does_not_parse() {This uses a path of
maxImagePathLength + 7, which does not pin the boundary. 63 is accepted and 64 rejected — test those two.@ -840,0 +921,4 @@node.loop(t0 + minuteMs);TEST_ASSERT_EQUAL_UINT8(1, radio.offCalls);TEST_ASSERT_EQUAL_UINT8(1, updater.cancelCalls);The name and the comment say the finished transfer is not cancelled; the assertion says
cancel()was called once. The assertion is what the code does —endSession()callscancel()wheneverupdateStarted_, andFirmwareUpdater::cancel()is the part that ignores it outsideinProgress(). Rename to that, or assert the property the name claims (whichtest_cancelling_after_the_image_is_written_keeps_the_handoveralready covers).6a606024594e4229e774Thanks — the out-of-bounds read was real. Confirmed it with ASan against the
manifest from spec 11.1 before and after: the old scanner aborts in
matchString, the new one is clean across every node type and every truncationof three documents, under
-fsanitize=address,undefined. Two tests now pin it(a key longer than a name, and an escaped key).
Acted on, one commit each:
4e4229ethe bounds fix, and an escaped key is now consumed and skippedrather than rejecting the document, which spec 11.1 requires.
5419abean entry whose path is empty or relative no longer parses.28011e7aContent-Lengthdisagreeing with the manifest fails at onceinstead of being retried, matching the no-entry case next to it. Also a test
that reaches the
maxManifestLengthguard, via a length the fake transportreports rather than one its buffer can hold.
aa0bfadthe drip feed. You were right that the per-byte stall window can beheld open forever, so the whole fetch now has a five minute limit, retries
included, and the spec sentence says so. Tested with a server feeding one byte
just inside the stall window.
3d0f4d4the HTTP timeouts are 800 ms each rather than 5 s, so the worstsingle block is inside the three missed heartbeats of spec 9.3. The test name
is fixed, and the comments went.
Two I have not acted on.
Byte 2 never reaching 100 is real and I am leaving it.
WifiManageronlyputs a percentage on the wire while the state is 4, and the state is 4 only while
inProgress(), so 100 is unreachable by construction. Reporting 100 alongsidestate 2 would contradict byte 0. The completion signal is the state leaving 4,
and once the boot switch lands the node reboots on completion, so nobody will
see the top of the bar either way. Worth changing only if the bridge turns out to
want it.
env:nativehaving no sanitizers is a fair hit and the reason this slippedthrough, but turning them on may surface pre-existing findings across the other
26 test directories, which is not this branch's diff. Raising it separately.
The residual on the blocking point:
esp_ota_begin()erases the slot inside oneNode::loop()and that is roughly a second for an 880 kB image, which no timeoutof mine can shorten. An update session will drop a heartbeat or two. Flagged for
Rob rather than worked around here.
The five fixes are right and I verified each one bites. Three things left.
The branch is behind its base and merging it breaks the build. The base moved to
ba94849(PR #50) after this rebased.src/node_bathroom/main.cppandsrc/node_lighting/main.cppconflict outright, andtest/test_wifi_manager/test_wifi_manager.cppauto-merges into something that will not compile: three of this branch's new tests still passapNametoWifiManager, whose third constructor argument #50 removed. Rebase ontoba94849and re-run — the green pipeline on3d0f4d4is against the old base.WifiManager::loopandendSessionauto-merge correctly, so it is just the mechanical part.Sanitisers are cheaper than you argued, and the round-1 regression test needs them. Reverting
matchStringto thecontinue-past-terminator version leavestest_a_key_longer_than_the_names_compared_against_it_is_skippedpassing; only the escape test fails. So the test added for the out-of-bounds read cannot fail as things stand. I ran-fsanitize=address,undefinedover all 27 test directories: no memory error anywhere. The only complaint is LeakSanitizer on the never-freednew MCP2515atcan_bus_mcp2515.cpp:46, whichASAN_OPTIONS=detect_leaks=0silences. That is two lines inplatformio.ini, not a sweep of 26 directories.No clock-wrap test, where
test_occupancy_latch,test_extractor_policy,test_bus_faultsandtest_bathroom_policyall have one. The arithmetic is right — the wholetest_firmware_updatersuite passes witht0 = 0xFFFFFF00— but three new windows have nothing pinning it.Byte 2 never reaching 100: accepted. The
esp_ota_begin()erase: see the inline note, it is not the erase alone.@ -959,2 +960,3 @@a transfer short: one already running reaches completion or failure first, so ashort timeout cannot drop the radio mid-image.short timeout cannot drop the radio mid-image. A transfer that stalls, or thatcannot finish within five minutes, fails of its own accord: holding the radio offThe sentence stops fine at "fails of its own accord". "holding the radio off the timeout is not a way to hold it up indefinitely" is the rationale for the rule, not the rule.
@ -0,0 +58,4 @@return false;}if (!sink_.begin(image_.size)) {Cutting the HTTP timeouts to 800 ms bought back the heartbeat budget and this line spends it again.
open()does connect (800 ms) + response header (800 ms) +esp_ota_begin()erasing the image region (~1 s for 880 kB) all inside oneNode::loop()— about 2.6 s against the 3 s absence window of spec 9.3. The erase on its own has slack; stacked behind the header timeout it does not. Open the sink on the loop after the transport opens so each gets its own budget.@ -0,0 +46,4 @@Stage stage() const { return stage_; }// True once the whole image is in the inactive slot. The three below sayFirst sentence restates
imageWritten(). Keep the second only.@ -65,3 +65,3 @@if (updateRequested_ && !updateStarted_ && updater_ != nullptr && serverAddress_ != 0 && state_ == can::wifi_state::connected) {updateStarted_ = true;updater_->beginUpdate(serverAddress_, serverPort_);updater_->beginUpdate(serverAddress_, serverPort_, now);Nothing pins this argument. I replaced
nowwith0here and all 56 cases intest_wifi_managerandtest_firmware_updaterstayed green. On a board with more than five minutes' uptime that constant makesdeadlineMs_already past, so every fetch is abandoned on the first loop after joining — the feature fails silently and no test notices.A failed fetch and a successful one are indistinguishable on the wire: state goes 4 to 2 either way, then the radio idles out the rest of the session timeout.
wifi_state::failedexists and is never reached from the updater. Either report state 5 when the updater fails, or say in spec section 11 what the bridge sees.@ -21,7 +21,10 @@ class WifiUpdater {virtual ~WifiUpdater() = default;// Called once the node has joined and has an address to fetch from.Restates the one call site, and skips the only thing here that is not obvious — that
nowseeds the fetch deadline. Cut it.@ -0,0 +131,4 @@// A path that will not fit is refused rather than truncated: a truncated path// fetches the wrong file or nothing at all.void test_a_path_too_long_to_hold_does_not_parse() {Subsumed by
test_the_path_length_boundary_is_where_it_says_it_isbelow — a 69-character path exercises the same branch as 64. Delete this test and its comment.@ -0,0 +244,4 @@TEST_ASSERT_FALSE(parse(relative, bathroom, image));}// The longest path that fits, and the first one that does not.Restates the test name. Cut.
FakeUpdater::lastBeginMsis captured and never asserted. Assertt0 + 1000here, or drop the field.3d0f4d48a295fbd4fff2Round two, all six acted on. Head is now
95fbd4f, rebased ontoba94849.1. Rebase. Done, and you were right about the failure mode:
wifi_manager.cppand
endSession()replayed cleanly, bothmain.cppconflicted outright, andtest_wifi_manager.cppmerged into something that would not compile. Three of mytests were still passing
apNameto a two-argument constructor. 363 cases pass onthe rebase, both nodes and
host_simbuild.While I was in there I added the two tests the AP fallback interaction wants,
which neither PR had: a running fetch is not interrupted when the join fallback
deadline passes, and action 3 whose join never lands still reaches the access
point. The safety comes from #50 clearing
joinPending_as soon as the statereaches 2 or 4, which happens before the fallback check in the same loop — worth
pinning, because nothing else says so.
2. The stacked blocking calls. Fixed as you suggested: opening the request and
erasing the slot now take a loop each, so a heartbeat gets out between them. That
turned up a real bug in passing —
received_still held the manifest's byte countwhen the stage changed, so progress read about 3% and then dropped to 0. Cleared
in
finishManifest()now, andtest_progress_tracks_the_bytes_writtencatchesit.
3. The unpinned
now. Asserted intest_action_3_hands_the_server_to_the_updater_once_joined. Replacing it with0now fails.
4. Sanitizers. Your evidence is better than my estimate and I accept the
correction on cost. Still leaving it out of this branch: it changes the test
environment for every future PR, CI's toolchain is not the one you or I measured
on, and a green pipeline here is the gate for the rest of this project. It is
CCS-UF-11 with your findings in it, including the
detect_leaks=0detail and thecan_bus_mcp2515.cpp:46allocation, so it is a ticket rather than a deferral. Ido run the scanner under
-fsanitize=address,undefinedby hand on every change toit.
5. Clock wrap. Two tests, base
0xFFFFFFF0so the wrap lands mid-fetch: onefull fetch across it, one time limit expiring across it.
6. A failed fetch looking like a successful one. Documented rather than given a
state of its own, since a new state value is a protocol change and not mine to
make here. Section 11 now says the state returns to 2 either way and that a new
build identifier in
SYS_ANNOUNCEis what says an update took. That covers byte 2never reaching 100 as well.
7. Comments. All five cut: the
beginUpdatecomment, the first sentence onimageWritten(),test_a_path_too_long_to_hold_does_not_parsedeleted assubsumed by the boundary test, that test's own comment, and the spec sentence now
stops at "fails of its own accord".
Nothing wrong with the code. Four things in the tests, one of them a test that cannot fail.
The two-phase open is clean:
requestOpen_is never cleared whilesinkOpen_is true,Stage::ImagewithrequestOpen_true andsinkOpen_false is the only order that reachesopenSlot(), and the new inter-loop window is covered byabandon()on cancel and on the deadline. ThefinishManifest()clear is complete —attempt_,requestOpen_,sinkOpen_,waitingToRetry_,written_andslot_are all already reset or unreachable at that point, andparseFirmwareManifestreadsreceived_before it is zeroed. The rebase lost nothing: every baseRUN_TESTis still registered and no existing assertion changed.Sanitizers: CCS-UF-11 with the findings in it is a ticket rather than a deferral, and the CI-toolchain argument holds. Not this branch.
@ -0,0 +277,4 @@updater.loop(lastByteAt + ctrl::updateStallTimeoutMs - 1);TEST_ASSERT_TRUE(updater.inProgress());// Every attempt gets a stall window of its own, so the whole thing takes moreRestates the two assertions below it. Cut.
@ -0,0 +621,4 @@updater.beginUpdate(serverAddress, serverPort, beforeWrap);uint32_t at = beforeWrap;This never compares a pre-wrap
nowagainst the deadline, so it does not test the wrap.at += updateStallTimeoutMs - 1runs before the firstloop(), so the first call is at0x000026FF— already past the wrap — and every comparison after it is small-number arithmetic identical to thet0 = 100000drip-feed test above.Replacing the deadline check with
if (now >= deadlineMs_)leaves this green; onlytest_a_fetch_completes_across_the_clock_wrappingfails. Move the increment after theloop()call, as the drip-feed test does, and add a lower bound (gaveUpAt - beforeWrap >= updateTimeLimitMs). I checked that combination does fail on the mutation.@ -1030,6 +1051,125 @@ void test_a_radio_that_drops_reports_off_and_clears_the_credentials() {TEST_ASSERT_FALSE(credentials.held());}// The updater is pumped from here, and off the same instant the rest of theRestates the test name. Cut.
@ -1033,0 +1126,4 @@// The access point fallback and a running fetch both hang off the join, and a// fetch that started means the join succeeded, so the fallback must not fire// under it and take the network away.void test_a_running_transfer_is_not_interrupted_by_the_access_point_fallback() {Adds nothing over
test_a_join_that_succeeds_never_falls_backat line 519.cancelCalls == 0andinProgress()can only fail ifapCallsis non-zero, which that test already pins, and the|| state_ == updatingarm of thejoinPending_clear is unreachable:joinPending_is cleared on the loop wherestate_becomes 2, which always precedes 4, sinceupdateRunning()needsupdateStarted_, set only whilestate_ == connected. Narrowing the condition tostate_ == connectedalone leaves all 46 cases green.The ordering the PR comment credits it with is not what it pins either — swapping the clear and the fallback check leaves this test green;
test_the_radio_goes_off_after_the_timeout_in_byte_oneis what catches that. Drop it.@ -1033,0 +1152,4 @@// A join that never lands has no fetch to protect, and action 3 must still reach// the access point the fallback exists for.void test_a_join_for_an_update_that_never_lands_still_falls_back() {Exact duplicate of
test_a_fallback_from_an_update_join_abandons_the_updateat line 563, from #50 — same setup, same twonode.loopcalls, same three assertions, thirteen identical lines. Delete it.All four taken, head is
ea72d0c. Every one was a test of mine that did not earnits place, which is a fair summary of the round.
1 and 3. Both new
test_wifi_managertests deleted. The first duplicated#50's
test_a_fallback_from_an_update_join_abandons_the_updateoutright, and youranalysis of the second is right: the
|| state_ == updatingarm is unreachablebecause
joinPending_is cleared on the loop the state becomes 2, andtest_a_join_that_succeeds_never_falls_backandtest_the_radio_goes_off_after_the_timeout_in_byte_onealready pin what I claimedfor it. So the fallback-versus-fetch interaction was covered before I touched it
and I have added nothing but noise. Gone.
2. Real, and a bad miss on my part — a wrap test that never crossed the wrap.
Increment moved after the
loop()call andgaveUpAt - beforeWrap >= updateTimeLimitMsadded. I ran your mutation: withnow >= deadlineMs_both wraptests now fail and the other 22 pass, and restoring the wrapping difference turns
them green again.
4. Both restating comments cut.
361 cases, both nodes and
host_simbuild, clang-format clean.Round four: all four round-three points are addressed and nothing is left to act on.
The wrap test now bites — mutating the deadline check to
now >= deadlineMs_fails both wrap tests and nothing else. Deleting the twotest_wifi_managertests lost no coverage:test_a_fallback_from_an_update_join_abandons_the_updatealready covers the never-lands case identically, and thestate_ == updatinghalf of the fallback guard is not pinned by any test, with or without the deleted one.