Sorry, it was not my intention to be vague. I admit to not having a complete implementation in mind. My point is that linking each log as a block in a chain with hashes forces an order that is more difficult to tamper with than a timestamp or auto incremented integer id. You have to alter more data to inject or purge records from a chain than you would with a table of timestamped records. I admit I can't make my case better than that.

As for the simplicity factor. I think your suggestion of serving logs to peers from a server like an RSS feed is a fine solution.

But I can setup a MultiChain instance In a few hours and start issuing tokens. I can send the same link out to my peers and auditors for them to connect and propagate the shared state. The community can shrink and grow without the members having to change anything. Now it's mostly a hands off venture that scales relatively well. I'm an okay programmer but to coordinate an effort to build, test and verify a system to do the same with RSS feeds across multiple companies would take me months. Something like MultiChain or HyperLedger is comparatively turnkey.

I'm not here to say this is the best way to do it. I'm just saying there's some merit to leveraging these technologies.

If you ask me, audit logs should just be posted to Twitter, the only true write-only database.

I mean you would need the hashing and consensus stuff to figure out exactly how the chain diverged. Just pooling the event would in theory be enough to prove that shenanigans were afoot then the ledgers don't align, but that's a bit too brittle to base a bi-annual evaluation on. You could close those up and setup some eventual-consistency across peers, sure but now you're talking about a some complicated proprietary software. It's also not clear how a system like that would scale.

There's plenty of convenient self-hosted blockchain solutions out there already that can be used to accomplish this. And there are a ton of tools to do analysis and tracing on these chains. This makes it not unreasonable when compared to a dedicated solution.

Most auditing and insurance companies don't have a webhook where you can arbitrarily send your logs to. They have humans with eyes and fingers holding risk management and law degrees called auditors. That you need to, with words and arguments,convince of your process integrity. And What happens if you switch insurer or certifier? You probably have to do a ton of IT work to change the format and destination of your logs. And how do you prove that your process was not manipulated during the transition?

What you describe are digital notary services and it's billion-dollar industry. All they do is be a trusted third party that records process integrity. IAM, change logs, RFCs, financial transactions, incident detection, and response are all sent in real time so you are ready for certification or M&A. Most small and mid-sized enterprises can't afford that kind of service and are often locked out of certain certifications or insurances or take a huge price cut when acquired.

Something like pooling together resources to a provable immutable log trail isn't unreasonable.

Let's say a country mandates their Telecom sector to audit it's transactions. The idea would be to share the network with several peers, your telecoms. In this case "mining" would be verifying the integrity if the chain and can be done by anyone of the peers. The government or auditing authority could also be a peer in the network and they are all capable of verifying the integrity of the chain through "mining". You are right that it's easier to have a small group of peers conspire to manipulate the chain. But it's a lot harder for several telecoms to conspire than for one rogue CFO to cook the books.

In this application you're not generating 'valuable' tokens in the sense bitcoin does it, but the value is the integrity of the chain. People value the proof that no one has redacted or injected any transactions.

The security comes from consensus. Everyone needs to agree about what the truth is. The burden of proof is proportional to the number of peers that need to agree. Public chains require a lot of work to create consensus amongst hundreds of thousands of peers. Let's say your chain consists of 12 companies all using the same chain to validate and verify each other's transactions so they are ready for an audit.

Yes, it's easier to have 12 peers conspire to manipulate the chain than to have 200 000 peers. But making 12 businesses conspire to cook the books is already several orders of magnitude more difficult than the checks and balances we have in place now.

Yeah it's not ideal, but you only need to pay the gas cost when you need to prove integrity and that's alot cheaper than having to constantly be in sync with the world.

I really don't think you should be too cavalier about it. Games aren't websites that run in nice sandboxed browsers that handle all your application security. They run quite close to the host system and have some crazy access to files and memory. Some modern anti-cheat that ships with modern games is downright malware!

It's not about doing thorough security auditing of every packet going over the wire but asking yourself what signals you want your users to send and receive. The modern networking models are so abstracted that devs usually don't think about it, let alone have any control over it. Something as simple as sharing a save state can be a dangerous attack vector if you don't know what you are doing.

That sounds like a healthy arrangement! I like this idea of x-func teams that are set up on a temporary basis. We do this often implicitly, but it's good to have an explicit process around it.

I've been trying to setup a working instance using container apps, web apps for containers and ACI, but it remains finicky. Do you know of a bicep or deployment script that does this properly?

I can tell having ai write small scripts and translating formats has made it more difficult for me to recognise the small differences between languages. I can tell i often get confused about how to do simple things like for loops http requests and file Io in c#, kotlin, rust, go , JS, pwsh, bash, Lua, pythong, groovy etc..

It's not necessarily a bad thing, but it is noticable the languages make less and less of a difference..

Edit: I think it's a similar thing as having a full featured ide with tons of code completion and suggestions. When writing stuff from scratch without prompts you kinda get lazy.

That's interesting. One of the problem for onboarding new engineers is they miss any domain knowledge of our product and building that end-user sensitivity is difficult. Embedding devs with sales,support etc does address this.

What are some problems you've experienced with this approach?

I'm responsible for the engineering department at a midsized SaaS company. In the last two years, we grew from 5 to 20 engineers. We have divided our landscape in separate domains, each treated as a separate product with an API that the other domains interact with. Each domain has a dedicated development team. The end-user domains have POs assigned and the more platformy domains take requests from the other teams.

We lean heavily on automation. Each team is responsible for the development, quality control, security, delivery and monitoring. We don't have a dedicated DevOps or IT role but lean on fully automated self-monitoring systems.

We keep growing and we see that the need for specialisation arises. I'm not one for mandating a certain structure and procedure top-down, so the tools I mentioned above have helped us map out how our org has grown organically so that we can detect human bottlenecks. This we can make the boundaries of responsibilities clear and make choices that minimise the amount of 'syncing' and maximise the amount of decision-making.

Right now I see that the need for dedicated security officers, IT pros, DBAs, and analysts is starting to arise simply because full-stack devs' skills are limiting. Still, I'm unsure if it's best to reinforce the existing silos with specialists, or have a team of specialists that the rest of the org can turn to.

I also see that the complexity of our ecosystem has made it a challenge to onboard new young engineers. Additionally, the separate domains give the teams a great amount of autonomy. Our metrics show that this has dramatically improved the stability of our products in the long term. Still, it has made it more difficult and laborious to organise projects across teams.

These are just a few of the challenges I'm thinking about. Our situation obviously does not apply to everyone, but I am fishing for ideas from people in similar situations or who have seen this phase before. In my experience, every software org has unique challenges and unique solutions with unique outcomes, but I'm certain ideas could inspire other experienced programmers :)