Weekly Reflection #17 - Trust Chains
Each week, I share one insight. One piece of wisdom. One question to reflect on. (and a little Lagniappe)
Insight
This week LiteLLM, the most popular open-source LLM proxy in the python ecosystem, was hit by a really gnarly software supply chain attack. The awful part was that the attack vector was through Trivy, a security scanner LiteLLM trusted to help protect its code. Attackers compromised Trivy's GitHub Actions and used that to steal LiteLLM's PyPI publishing credentials, and used them to push backdoored packages that harvested secrets from anyone running LiteLLM in their Python stack.
Moments like these are important reminders of how vulnerable we all are if we get sloppy with our trust models. Every dependency is a trust decision, and trust is transitive. It builds a chain of trust, and like any chain, it is only as strong as its weakest link. LiteLLM trusted Trivy. When Trivy fell, everything downstream fell with it. Now, what can we learn from this? We must use defense-in-depth. We must pin dependencies to immutable hashes. We must scope credentials narrowly. We must filter network egress in build environments. And finally, we must regularly take the time to think through our trust models.
Wisdom
"The best way to find out if you can trust somebody is to trust them." — Ernest Hemingway
Reflection
How would you triage an incident like LiteLLM?
Lagniappe
- The youtube channel Low Level does a great job of breaking down LiteLLM.
- This week on Book Overflow we discuss part 2 of Crafting Engineering Strategy by Will Larson. This is easily a top 5 book for me. I'm loving it.
- As always, I'd love to hear your thoughts. Reply and let me know what resonates.