I'm looking for architectural advice rather than help debugging a specific function. This is a private project, so I can't share the complete source code, but I've included detailed specifications, measured performance results, memory usage, and the engineering changes we've already tried. I'm happy to provide small, anonymized code examples if someone needs them to investigate a specific issue.
TL;DR: I'm building a private, local Python application that processes large amounts of time-series data, maintains historical analytical State, produces reports, and independently verifies that nothing has been corrupted or lost.
The application works on small datasets, and many of its correctness and performance tests pass. However, I'm trying to prove that the entire system can operate within a 1 GiB aggregate memory limit, preserve complete historical information, and recover from crashes within 60 seconds.
I've spent weeks working through this with AI coding assistants. We've made substantial improvements, but I feel like we're chasing architectural issues one at a time instead of solving the underlying problem.
I'm looking for experienced human developers who can suggest a more efficient approach.
I'm on a budget, so I would particularly appreciate practical solutions that don't involve buying expensive hardware or moving everything to cloud infrastructure.
1. My computer and environment
Here are the hardware and software specifications I have confirmed from the diagnostic reports:
| Component |
Specification |
| Operating system |
Windows, 64-bit |
| Processor |
8 logical processors reported in an earlier diagnostic; exact CPU model not yet documented |
| System RAM |
Approximately 12.7 GiB usable |
| Main drive |
Approximately 510.5 GB total capacity |
| Available disk space |
Approximately 155–160 GiB in recent measurements |
| File system |
NTFS |
| Internal drive type |
SSD/HDD type not yet confirmed |
| Programming language |
Python 3.14.0, 64-bit |
| Database |
SQLite |
| Architecture |
Local, multi-process Python application |
| Memory measurement |
Windows private committed memory and kernel-enforced Job Objects |
The computer has considerably more than 1 GiB of physical RAM. The 1 GiB limit is an application-level resource requirement, not the total RAM installed in the computer.
The requirement includes the main application and the workers needed for verification and recovery. Moving processing into another process doesn't exempt its memory.
There's also a separate 32 MiB discretionary-cache limit.
I'm open to hearing whether these resource targets are realistic, but I'm especially interested in architectures that could meet them without sacrificing correctness.
2. What the application does
Without getting into the proprietary purpose, the application:
- Processes incoming data across three time intervals: 1 minute, 5 minutes, and 15 minutes.
- Maintains a complete, causally ordered historical record.
- Performs incremental calculations using current and historical data.
- Tracks evolving analytical State across thousands of updates.
- Produces evidence, reports, snapshots, and durable journal entries.
- Uses immutable, content-addressed references to historical information.
- Independently checks the integrity of data and calculations.
- Must recover the latest valid committed State after a crash, without losing acknowledged durable events.
It is currently a read-only analytical system. It doesn't need a transaction-execution engine or cloud deployment to accomplish its purpose.
The challenge is retaining and verifying complete historical information without repeatedly loading or reconstructing all of it.
3. How much data are we talking about?
The main historical workload represents 60 sessions.
| Historical workload |
Count |
| 1-minute data records |
23,040 |
| 5-minute data records |
4,608 |
| 15-minute data records |
1,536 |
| Total native records |
29,184 |
| Scheduled processing attempts, including repeats |
29,196 |
The system must preserve the analytical State, evidence, and reports associated with those processing attempts.
An earlier test retained all 29,184 data records but grouped them into 992 processing captures rather than executing every scheduled assessment individually.
That test produced approximately:
- 938,669 independently audited storage nodes.
- 166,307 evidence derivations.
- 2,979 State records.
- 992 capture reports.
- Zero failures in its independent storage-integrity audit.
However, we subsequently determined that 992 batched assessments do not prove the same workload as approximately 29,196 individual scheduled attempts.
That larger per-assessment qualification is still outstanding.
4. The performance requirements
These are the principal limits we're trying to satisfy:
| Requirement |
Limit |
| Development capture processing |
20 seconds maximum |
| Formal capture processing |
Under 40 seconds |
| Processing deadline |
Within 55 seconds after the relevant interval closes |
| Complete crash recovery |
60 seconds maximum |
| Aggregate private committed memory |
1 GiB |
| Discretionary caches |
32 MiB |
| Growth in comparable processing time as history accumulates |
5 seconds maximum |
The application also processes bounded current input views while retaining full historical continuity separately.
I realize some of these are demanding constraints. They're intentional, but I'd welcome informed criticism of their design.
5. What we've already changed
This was originally much less efficient. We've made several major architectural changes.
Historical storage and data structures
We moved away from repeatedly constructing and serializing massive historical structures on every update.
The current design uses immutable content-addressed nodes, authenticated references, persistent collections, incremental updates, and shared historical prefixes.
We replaced repeated historical-list expansion with reference-based representations. Analytical modules preserve their required history through authenticated continuation structures instead of passing enormous expanded objects between components.
Database and storage
We use SQLite-backed content storage and append-only journal structures.
We've added immutable sealed baseline segments and private copy-on-write overlays for separate verification branches.
Instead of copying an entire historical database for every test branch, the branches share verified read-only segments and maintain private writable metadata.
Crash recovery
We've implemented per-commit recoverable generations, durable expected-tip tracking, pending and acknowledged commit handling, and independently controlled generation pins.
The verification system maintains hash-chained evidence and independently held rollback protection.
It checks for missing, substituted, rolled-back, and inconsistent information.
Recovery reconstructs the appropriate State and checks that it matches the committed history.
Independent verification
We have a separate verification process that checks data integrity and committed calculations.
There's also a replay certifier that reconstructs the processing history to confirm the committed results are correct.
This is where the biggest memory concern has appeared: the producer and replay certifier may need substantial analytical working State at the same time.
Replay-certifier memory improvements
We recently implemented a bounded verification-only replay path.
Instead of keeping every historical record decoded, it streams authenticated references, retains a limited working window, and retrieves older information when required.
On small inputs, the historical per-bar representation decreased from approximately 1.42 KB to 0.33 KB of resident data, excluding the bounded window.
However, this does not necessarily eliminate the memory needed for the actual analytical State.
Other improvements
We also implemented incremental calculations for a separate relationship-analysis path, repaired correction/reset handling, improved event ordering, eliminated repeated full-history validation, and added reference-based report construction.
Most of these changes have passed focused correctness tests.
6. Tests we've already passed
We've run a substantial number of tests. I don't want to throw away this work or rerun everything unnecessarily.
Historical processing performance
An earlier integrated test completed:
- 21 timed captures.
- All captures between approximately 1.07 and 2.33 seconds.
- The slowest tested group finished approximately 22.1 seconds after its interval closed.
- Three cold restorations completed in approximately 2.2, 6.1 and 24.4 seconds.
- A recorded peak private commit of approximately 786 MiB.
Those timings passed the limits for that particular test and source version.
But that test used a batched historical workload and an earlier implementation. It is not proof that the final full scheduled workload passes.
Correctness and integrity
We have passing focused tests covering:
- Legacy-versus-new analytical calculations.
- State and historical evidence equivalence.
- Crash recovery and exact continuation.
- Correction/reset behavior.
- Developing and confirmed analytical results.
- Five-layer offline processing/replay equivalence.
- Independently authenticated generation pins.
- Missing, substituted, forged and rolled-back data rejection.
- Durable acknowledgment and recovery after interrupted commits.
A correction-depth test covered 1,164 bars without analytical differences. Developing-result tests included more than 15,000 comparisons.
Replay-certifier comparison
We compared the original full replay certifier against the new bounded-memory version over four different small history batches.
Their committed results, recovered State, identity roots, and generation checks were equal.
The new bounded version also passed its focused integrity tests.
7. The memory results that have us stuck
This is the main reason I'm posting.
Historical large-workload measurement
One earlier test reported approximately 786 MiB peak private commit.
But the testing process also contained the full 60-session input fixture, processing code, restored historical State, and other supporting objects.
We therefore cannot treat 786 MiB as the main application's own operating memory.
Fixture-only measurement
We ran a separate fresh process that loaded only the existing 60-session input fixture.
It loaded 29,184 records.
Results:
- Before application-related imports: 8.6 MiB.
- After imports: 14.1 MiB.
- After fixture construction: 78.6 MiB.
- Increase during fixture setup: 64.5 MiB, including lazy imports and allocator overhead.
This was one measurement in a separate process.
We cannot simply subtract 64.5 MiB from the earlier 786 MiB measurement, because their memory lifetimes, allocations, and workloads were different.
Small multi-process recovery test
We also tested the producer, replay certifier, independent verification authority, and byte-provider workers under a kernel-enforced aggregate memory cap.
The earlier full replay arrangement reached approximately 343 MiB aggregate high-water.
After the bounded replay-certifier change, a comparable small test reached approximately 353 MiB.
Both stayed below the 1 GiB cap and recovered the exact expected State.
However, the small test started certification from an initial empty history, so it didn't exercise the deep historical restoration that the memory improvement targets.
The new design is correct on the tested inputs, but its maximum-history memory savings are not measured.
The concern
At maximum historical depth, the producer and replay certifier may each retain substantial working State.
If both have to remain resident simultaneously, they could exceed the aggregate 1 GiB limit.
We don't have a measured maximum-depth aggregate number yet. I don't want to spend hours rebuilding historical test data just to discover another predictable problem.
8. Storage concerns
The complete future historical qualification has been estimated at approximately 133 GB of peak storage, including:
- Historical processing data and committed generations.
- Private test overlays and sealed baseline snapshots.
- An independently readable complete export.
- Verification indexes and evidence.
- SQLite temporary/scratch storage.
This is only a projection. Actual peak disk use has not been measured for the full workload.
With roughly 155–160 GiB free, the storage margin is uncomfortably small.
I considered purchasing a 5 TB external hard drive for approximately $220, but I'm reluctant to spend money until I know whether it will solve anything.
I understand an external HDD would increase storage capacity, not available RAM, and may make database-intensive workloads slower.
I would rather improve the software first if possible.
9. Another challenge: complete historical relationships
A separate, more demanding analytical mode creates relationships between historical evidence objects.
Its relationship set can grow quadratically with the number of eligible objects.
One worst-case projection reached multiple terabytes of relationship data, but that assumes all historical evidence is simultaneously eligible. That assumption has not been established and isn't the configuration of the normal observational workload we're currently qualifying.
That mode has passed small correctness tests, but its maximum-depth capacity remains unqualified.
I'm including this because I want to avoid architectures that solve memory problems merely by creating an equally enormous disk problem.
I'm not asking to redesign that separate mode as part of the immediate memory fix.
10. My questions for experienced developers
1. How would you establish the true memory usage of the application versus its test harness?
I'm using Windows private committed memory and Job Objects, not simply Task Manager's displayed RAM percentage.
What techniques would you use to attribute retained and transient memory to specific Python objects, SQLite caches, imported modules, historical State, and verification workers?
2. How would you design independent replay verification without keeping two complete analytical engines in memory?
The certifier must independently re-derive all required committed results. It cannot simply trust stored output hashes or skip historical dependencies.
Would you recommend streaming, authenticated checkpoints, incremental semantic verification, memory-mapped storage, bounded per-owner State, a different worker topology, or another approach?
3. Is there a better way to handle the verification workers under a strict aggregate memory limit?
The producer must remain resident. The verification process must not lose independent trust.
How would you structure scheduling and memory ownership without creating an unacceptable certification backlog or violating the 60-second recovery requirement?
4. How would you measure maximum-depth working memory without reconstructing the full workload?
Can existing persisted references and snapshots be used to construct a representative depth-point measurement, provided the identities and analytical State remain valid?
How would you avoid a synthetic test that looks small but fails to represent real historical State?
5. Are we using Python and SQLite appropriately for this workload?
Would you keep Python and optimize the data structures, use SQLite differently, move a specific bottleneck into a native extension, or consider something else?
I'm not interested in rewriting the entire application without a clear technical reason.
6. Am I approaching this problem incorrectly?
We've spent weeks replacing inefficient data structures, adding persistent references, tuning recovery, and running tests.
The changes have helped significantly, but the qualification process keeps exposing another problem.
What architectural assumption would you challenge first?
11. Budget and what I'm looking for
I'm building this privately on my own computer, not with a corporate infrastructure budget.
I can't keep purchasing additional hardware, expensive services, or spending days rebuilding the same historical fixtures.
I'd prefer a solution that:
- Preserves complete historical accuracy and causal State.
- Doesn't delete required records merely to pass a test.
- Maintains independently verifiable integrity and crash recovery.
- Avoids enormous in-memory copies and unnecessary database duplication.
- Makes efficient use of the hardware I already own.
- Can be tested incrementally without repeatedly rebuilding everything.
I'm not asking anyone to write the application for me. I'm looking for experienced developers willing to point out what I may be overlooking.
If you were taking over this project tomorrow, what would you inspect first, what would you measure, and what would you change?
I would genuinely appreciate specific technical suggestions, especially from people who have built Python/SQLite event-processing systems, persistent historical databases, or independently verified data pipelines.