I'm a senior with loads of years, i've been doing devops/sre and systems engineering in the cloud for a long time. recently i volunteered for a project that i knew would lean heavily on claude, and with my manager's enthusiasm and support it looks like the project will be a success. aside from not truly owning that success, i'm also more than a little concerned that the 10K lines of code that i've slopped out, while mostly robust and functional, may not be personally defensible or supportable by me. this is a "career first feeling" for me, and i'm wondering if others have had similar experiences recently. is this just the way it is? maybe i should just treat it like "i started a new job and inherited a bunch of new libraries" when things come up? I'm not really sure how this is going to end.

The full article: https://safedep.io/litellm-pypi-compromise-credential-stealer-backdoor/

‎We are looking to overhaul our load testing, since our current script only slams 10k users in 60 seconds and gives us garbage data. It exhausts connection pools and creates artificial bottlenecks that would not show under any kind of normal use case.

‎Real users ramp up slowly, and idle on pages, and don't jump from one to another without a second thought, they do not fire off database transactions with zero pause.

‎We should be reflecting real behavior, not inducing cold start failures.

‎Has anyone modeled this at scale? Should we make the step curves with warm up times to let caches and other systems warm up before hitting the expected load? We want to reflect real user behaviour, not just trigger cold start failures.

‎

Mid level engineer , who landed and worked my way up into the role. I wont lie and pretend I know everything, some topics I'm still working on, hell I haven't even touched Kubernetes yet, so you can imagine I feel way FREAKING behind.

It seems like everything is quickly adopting and using AI, from downloading models locally, to use and play with them to using them to learn , to using the next leading edge models. How are you all having time for this....

I want to start using AI more confidently but I am not sure where to even start. Local models vs generate chat models via gui in chatgpt or Claude...are there any trainings you guys recommend?

I'm also weary of using AI to learn cause I do know it hallucinates, but also I dont want to feed it and let it spit out answers for me whenever I am just learning how to implement something.

This is developing so fast, I feel like I need to start putting guardrails up for my own sake of learning and sanity, else I will lose knowledge and become a Senior Devops Prompt Engineer lol...jokes aside open to how you are all trying to navigate this, from learning with and without, what guardrails are you implementing , and any learning material would be appreciated.

I've been following the Railway situation since the outage started, and things have gotten pretty concerning.

I'm currently using Railway to host the backend of a small gaming marketplace I've been building. Like many developers, I initially thought this was another infrastructure outage affecting builds and deployments.

But on October 21, Railway published a security disclosure confirming something much more serious.

According to their statement, they detected anomalous activity involving systems responsible for storing customer environment variables.

Their investigation suggests an unauthorized third party may have accessed encrypted environment variables and, in some cases, obtained their decrypted contents.

That's a serious concern for anyone hosting production applications on Railway.

What Railway has disclosed so far

• Suspicious activity was detected in systems storing customer environment variables.
• An unauthorized third party may have accessed encrypted secrets and obtained some decrypted values.
• The impact may be limited to a subset of customers, although the investigation is ongoing.
• Railway is recommending that all customers rotate their stored credentials, including API keys and database credentials.
• Deployments have been paused during the investigation.
• Railway believes the incident may be connected to a broader campaign targeting platform companies and AI inference providers.
• Railway has also started automatically rotating its bucket credentials and resetting certain integration credentials.

As of the latest update I've seen, deployments remain paused.

My biggest concern is how developers are supposed to rotate production credentials during an outage like this.

Think about a small startup running its entire backend on Railway.

You might have PostgreSQL credentials, JWT secrets, payment API keys, SMTP credentials, webhook secrets, and other sensitive information stored as environment variables.

Now imagine being told that some of those credentials may have been exposed and that you should rotate everything immediately.

That's obviously the correct security recommendation.

But there's a problem.

If you revoke your current database credentials or payment API keys before your running backend can use the replacements, you risk breaking critical functionality.

And when deployments are paused, rolling out those changes becomes much more complicated.

At the same time, leaving potentially compromised credentials active is also risky.

For solo developers and small teams without dedicated infrastructure engineers, this is not a simple situation.

What I've personally done so far

I've reviewed my Railway account sessions, signed out older sessions, and inspected my existing environment-variable configuration.

Everything appears normal from my side.

But I understand that simply checking whether the variables still exist doesn't prove they weren't accessed.

An attacker could potentially copy credentials without modifying anything visible in the dashboard.

I haven't received confirmation that my particular project was affected, and I'm preparing to rotate my sensitive credentials carefully.

The bigger issue is trust in managed infrastructure.

One of the reasons developers use services like Railway is that they don't want to manage every part of their infrastructure manually.

We trust these platforms to handle deployment infrastructure and provide secure storage for sensitive configuration.

Of course, no platform can guarantee perfect security.

But when the infrastructure responsible for storing customer secrets is potentially compromised, the impact can extend far beyond the hosting provider itself.

A stolen database password could potentially allow unauthorized data access, depending on network restrictions and database permissions.

A stolen API key could allow unauthorized operations or unexpected charges.

A compromised authentication secret could potentially affect user sessions, depending on how the application implements authentication.

None of this automatically means these things happened to Railway customers. These are potential consequences of exposed credentials.

And that's exactly why the incident deserves serious attention.

There are still several questions I'd like Railway to answer.

1. What was the actual entry point used by the attacker?
2. How could encrypted environment variables potentially be decrypted?
3. Was this caused by compromised internal credentials, an infrastructure vulnerability, or something else?
4. How many customers or projects were actually affected?
5. Was there any evidence of attackers accessing customer databases directly?
6. What time period should developers investigate when reviewing suspicious activity?
7. What additional security measures will Railway introduce to prevent this from happening again?

I appreciate that Railway disclosed the incident and started providing recovery guidance.

But until the full scope is known, it's difficult for developers to understand their actual exposure.

I'm curious how other Railway users are handling this.

Have you already rotated all your production credentials?

Have you noticed any unusual database activity, suspicious API calls, or unexpected service charges?

Has Railway contacted you directly about your project being affected?

Are you waiting for deployments to resume before rotating certain credentials, or have you temporarily disabled critical functionality?

And probably the biggest question: Has this incident changed your trust in Railway, or do you still plan to keep your production applications there?

I'm not trying to spread panic or suggest that every Railway project has been compromised.

I'm sharing this because I'm running a real application on the platform, and I know other independent developers and small teams are in a similar position.

I'd like to hear how others are approaching the security risks and what precautions they're taking.

Hopefully, Railway publishes a detailed technical postmortem once the investigation is complete.

We've built tens of internal dashboards, tools, and admin utilities over time. Now, vibe-coded tools are becoming the norm for us, and it feels like a maintenance nightmare.

We have to constantly update and fix them and keep them working with our ever-changing data structure. Also, It's getting hard to track what tools exist, who uses them, and what’s active vs. abandoned.

How does your team handle this? Are there specific systems or governance models you use to maintain the internal tools?

Is ADO bullshit? Because I feel like the way our it department is using it is bullshit. I work on a data analytics team who has to use data created by IT.

Where's the documentation for this script? "It's in ADO". Who's the customer? Why did they request this change? "It's in the ADO".(No it isn't, it's in someone's email inbox somewhere).

What's the overarching structure/purpose for this screen? "We don't document that anymore, it's in [I shit you not] the individual ADO tickets that built the screen". You can NOT be serious. No one wants to read every individual ticket turning features off and back on to figure out what the current state is. There's no universe in which this is remotely coherent.

Also, the vast majority of the screens were built before the ADO went live, so we have old school documentation that was last updated in 2019 establishing the purpose and basic design of 80% of screens but then the current state is old documentation plus any ticket affecting that screen layered on top that old documentation?

I feel like this is a scam and totally incoherent. They program in Java, we work exclusively in SQL/PowerBI (so we can't use or read any of their code anyway, but they've refused to send it to us even if we could). We had an item where my staff tried to recreate a screen-calculated variable and it was a nightmare trying to combine the pre-2019 documentation with new tickets because the calculation could be affected by a ticket that appeared to have nothing to do with that particular calculation at all. And again, we can see absolutely zero code, just user stories. But I don't want to see their code, my staff can't read or execute it. We aren't devs. We just need to know what the system is doing so that we can do accurate analysis of the resulting.

But it's "all in the ADO".

Hi all,

I'm thinking about building an open-source tool for configuring servers

(installing packages, managing config files, restarting services),

something in the space of Ansible, but written in Go. Nothing is built

yet, I want to check whether the idea makes sense before I invest time.

What bugs me about Ansible:

- YAML pretends to be a programming language: `when:`, `loop:`, Jinja

inside strings, and a long list of variable precedence rules

- it needs Python on every machine and can be slow

- you can't see clearly what will change before you run it

What I'd like to build:

- one Go binary, no Python, easy to install

- configs in HCL (the same language as Terraform) instead of YAML + Jinja

- `plan` shows what would change (with a diff), then `apply` does it,

just like Terraform

- resources reference each other and form a dependency graph, instead of

running strictly top to bottom

- idempotent: running it twice changes nothing the second time

Two ways to reach your servers:

1. Over SSH. You run the tool on your laptop, it connects to the servers,

   and nothing needs to be installed on them. This is the first thing

   I'd build.

2. Optionally, a small agent: the same binary, running on the server as

   a service and listening on a port. Useful when SSH is locked down,

   blocked by a firewall, or you just don't want to expose it. The

   resource code would be identical in both modes, only the way of

   delivering commands differs. This would come later, and the

   communication would be encrypted and authenticated (mutual TLS).

Here is how an nginx setup could look:

package "nginx" {

state = "present"

}

file "/etc/nginx/nginx.conf" {

source = "files/nginx.conf"

mode = "0644"

require = [package.nginx] # install the package first

notify = [service.nginx] # reload nginx if this file changes

}

service "nginx" {

enabled = true

state = "running"

require = [package.nginx]

}

Run `plan` on a fresh server and it shows: package will be installed,

file will be created, service will be enabled and started. Run `apply`

and it happens. Run `apply` again and it says "0 changes". Edit the

config file and `plan` shows the diff and a pending service reload.

What I'm NOT trying to do: replace Ansible for everyone. Ansible's huge

module ecosystem is its real strength, and I can't compete with that. The

target is small fleets, homelabs and people who like simple tools.

I'd love honest feedback, including "I wouldn't use this":

1. What annoys you most about Ansible (or whatever you use today)?

2. Would you try something like this, or is the lack of modules a

   dealbreaker?

3. Is the optional agent (a small service on the server) useful to you,

   or would you rather stick to SSH only?

4. Which resources would you need first besides file, package, service

   and command?

5. HCL or something else? Would you prefer another format?

Not selling anything: it's open source and there's nothing to download

yet. Thanks for reading!

Just came across this software called Mantle on Product Hunt. Apparently, it detects breaking API changes in your GitHub repo and suggests code fixes before they break your integrations in production.

Seems interesting, especially for APIs where keeping up with changes is such a waste of time. Has anyone here actually tried it? Curious how well it works.

Curious how other developers handle cases where everything technically reports success — HTTP 200, Lambda completes, webhook delivered — but the intended operation never actually finishes.

For example, a payment succeeds but the customer’s subscription isn’t activated, or an API call succeeds but a downstream system never gets updated.

Have you encountered anything like this in production? How did you discover it, and what tools or processes do you use to catch these failures?

I’m especially interested in cases where traditional logs, metrics, or error alerts weren’t enough.

(Sonatype Nexus is one of the most commonly used package repositories and transparent package proxies around)

On logging into our Sonatype instance this morning, it requires us to read and tick an "I understand the implications and agree with" information gathering.

No option to opt out, no warning given.

If uploads to their servers don't happen within 45 10 days then nexus will be disabled until they are.

Edit: 10 day kill switch for community versions, 45d for subscription.

I'm considering building an open-source server configuration tool in Go,

in the space of Ansible. Nothing is built yet, I'm checking whether the

idea makes sense.

The idea:

- one binary, no Python, connects to servers over SSH

- configs in HCL (like Terraform) instead of YAML + Jinja

- `plan` shows a diff of what will change, then `apply` does it

- later maybe a small optional agent on the server for when SSH is blocked

Target: small fleets and homelabs, not replacing Ansible for everyone.

Questions:

1. What annoys you most about Ansible?
2. Would you try something like this, or is the module ecosystem a dealbreaker?
3. Is an optional agent useful, or would you stick to SSH only?

Honest criticism welcome. Thanks!

Hello guys,

I'm new to the devops world and I'm working on some projects to learn. I'm currently trying to automate the creation of VMs with Terraform with my local Proxmox.

As CI/CD and Git server I have everything on a Gitea local. I remember doing some CI/CD once on Github and it worked fine but apparently I need a runner in my home.

Should I create a VM or a Docker container? I have no idea on how to setup that thing and to connect it to my Gitea right after.

Thanks for the help

So we have this infra lead who pretty much uses ChatGPT to write up their replies to Slack messages. You can clearly see the use of AI in how their replies are structured. Because of this, sometimes, due to their AI’s lack of context, certain replies do not have the depth required for their level.

When I say use of AI, here’s some examples:

- The current 1TB threshold sits right at our normal free space, so the daily 50GB temp file dip trip it. Adding 200GB would just delay the same problem as data grows, adds cost, and RDS storage can’t be shrunk later.

- Good point on slot lag. 850/800 is a good move.

- Nice approach, much cleaner than maintaining Stripe IP allow lists. Once you confirm it’s working I’ll factor it into the billing service step for EKS: it’ll need a pod identity role with read access to SQS.

This is just from today.

Hey folks, I've noticed readiness and liveness probes often end up with pretty similar settings, even tho they do different jobs. With tight timeouts and failure thresholds, a service that's slow to start or recovering from a dependency issue can get pulled out of rotation or repeatedly restarted.

I've seen probe settings get copied between services too, even when their startup times are completely different.

How do you guys set probe thresholds for services with unpredictable startup times or dependencies that occasionally respond slowly?

My startup is in consumer electronics. The product is 1 device with 1-2 peripherals that come with it. Users connect to a mobile app with a backend that does data analysis on what the product measures.

ATM I have 4 repos: a flutter app and a FastAPI backend with sqlite. The embedded side uses PlatformIO+Arduino for the device/hub, and same stack for the peripehrals.

The DevOps is getting unwieldy. On the one side, I'd like everything to be in one place so agents can easily build up a context and CI/CD is not scattered. On the other hand, I hate working with massive project trees and loading up vscode would be a nightmare.

Thoughts? At work I use git submoduels but I was told that's not a monorepo.

I'm setting up an AWS workflow (we work only with azure and on-prem at work so little experience with the CLI) and wondering if I should bother learning about the tool itself.

Seems like I should just set up limited users with limited roles, hard budget caps, place manual gates on production CIs and let a sandboxed AWS MCP handle everything else. Using Terraform is an implicit admission on how we'd all like to instrument our architecture in a vendor-agnostic vocabulary. An MCP seems to be an additional step in the same direction?

Thoughts?

Doing a poc and wanted to know about the product from someone who is actually using it in production. How has your experience been and how effective it is in filtering security violations?

Supabase is showing up in almost every new startup’s stack these days. Its built-in observability is good, but in practice you want to monitor Supabase alongside everything else you run, in whatever you already use such as ClickStack, Grafana, Datadog, etc.

Supabase made it easy by exposing a Prometheus endpoint at /customer/v1/privileged/metrics for every project , so a plain prometheus receiver gets you Postgres, connection pooling and host metrics. And Supabase log drains have a native OTLP destination, so logs land on a normal otlp receiver on 4318 with no webhook parsing but make sure to add auth on that receiver, since it’s on the public internet

I wrote a short post on the full collector config, the drain setup, and a docker-compose to run it. I also built a ClickStack dashboard you can import.

Curious how others are monitoring Supabase today?

I’m working on a personal project, so 0 users, and I have worked with SST in the past, but having to upgrade SST was a pain. Not sure if pulumi is any different of an experience, or is less painful and wanted insight on different experiences with either tool

Hello friends.

I am implementing CI security scanning with Syft generating the SBOMs and uploading them to Dependency Track.

Two disclaimers:

1- I hate frontend work and have a poor understanding of it. 2- I am not a cybersecurity guy. This is just a new avenue I am exploring.

I've faced an issue currently with CVE scanning for Javascript files, specifically it seems that there is no way for me to actually tell what packages made it into the bundler vs what was included in the lockfile.

This leads to the JS sbom including all the dev dependencies with nothing actually telling me what made it into the bundle.

For SPAs, this can be mitigated by ignoring dev dependencies, however most of my stack is Laravel, and from what I understand Laravel puts all its NPM packages in devDependency, meaning I cannot skip them in this stack.

I have also tried generating sourcemaps while building (and I know you are supposed to .dockerignore them as it's a security concern), but without a reference to compare against, I do not know if this approach actually works or if it's a false sense of security. It does reduce the resulting sbom significantly, which seems to be a step in the right direction.

I've also read this study from 2022 https://das.encs.concordia.ca/pdf/Latendresse_ASE2022.pdf and it seems to reinforce my point that what I'm asking for is (at this time) not possible.

My goal is to understand 2 things:

1. Is it actually not currently possible to know what in the lockfile actually made it to through the bundler and into the final app.js that gets served?
2. If 1 is true, do I just not run CVE scans on the lockfiles for Laravel projects? Do I just scan everything and force people to upgrade as much as possible to just "lower CVE number"?

Infra engineer here (about 10 years across AWS, Azure and on-prem). In every environment I've worked in, something is always approaching end of life: AI model versions on Azure OpenAI and Bedrock, Kubernetes versions on EKS/AKS/GKE, Lambda and Functions runtimes, database engine versions.

Each cloud sends its own notices, but they tend to land with account or subscription owners, not the team that owns the workload. And unlike library versions, none of this shows up in Renovate or Dependabot.

Curious how your team handles it:

- How do you find out? Native notices (Service Health, AWS Health, Advisor), vendor emails, a script, or someone just happens to notice?

- How does the notice get to the team that owns the workload?

- Do you catch any of it in IaC or CI, like pinned versions in Terraform or a pipeline check?

- What's your worst "we missed it" story?

- If you use a tool or spreadsheet for this, what's annoying about it?

Full disclosure: I've built a few small tools around cloud AI model and Kubernetes lifecycle data, and I'm exploring whether something for this specific problem is worth building. Mostly I want to know if it's a real pain or already handled well. I'll share a summary of what I learn.

Hey everyone!

I'm a solo developer working on my own digital product, and I'm looking for an affordable but reliable server for production deployment. Since I'm managing the project and infrastructure myself, I want to keep hosting costs as low as possible without compromising too much on performance or reliability.

Here are my requirements:

• CPU: 4 vCPUs / cores
• RAM: 8 GB
• Storage: 50–60 GB SSD or NVMe
• Usage: Production workloads, not just local development or testing
• Budget: As economical as possible, preferably with predictable monthly pricing

I've been looking into AWS EC2, but I'm also open to other VPS providers if they offer better value for money. I'm not specifically looking for an AWS Free Tier instance; I'm interested in something I can realistically use to host a production application.

I'd appreciate your advice on a few things:

1. Which VPS or cloud provider offers the best price-to-performance ratio for these specifications?
2. Is there an affordable EC2 instance type that would be suitable, or would a provider like Hetzner, DigitalOcean, or another VPS host be a better choice?
3. Are there any genuinely free or heavily discounted instances that are suitable for production use, rather than just temporary testing?
4. Are there any hidden costs I should consider, such as storage, bandwidth, backups, IPv4 addresses, or taxes?

I'm happy to manage the server myself, including Linux configuration, deployments, and application maintenance. I mainly want to avoid paying enterprise-level prices when I'm running a product independently.

If you're a solo developer hosting a real production application, I'd love to hear what you're using and how much you pay per month. Personal experiences and recommendations would be especially helpful.

Thanks in advance!