Building an IaC Security Scanner in Go
As a security engineer who's implemented security controls across cloud environments, I've always been curious about how policy engines work under the hood
As a security engineer who’s implemented controls across cloud environments, I’ve always been curious how policy engines work under the hood. Sure, I could use existing tools like Checkov or tfsec. But there’s something to be said for understanding the fundamentals of infrastructure security scanning.
That curiosity consumed several weekends and resulted in PolicyGuard, an IaC security scanner built from scratch in Go. What started as a “let me just understand how this works” project now catches security issues in infrastructure code.
The Problem That Started It
Working in cloud security, you see the same mistakes constantly. Last quarter alone, we caught 47 S3 buckets with public read-write access. Developers keep launching EC2 instances without encryption. One team had a security group allowing 0.0.0.0/0 on port 22 for three months.
Tools exist to catch this stuff, but I wanted to understand how they work. How does Checkov parse a 2000-line Terraform file in seconds? What lets tfsec understand that aws_s3_bucket_public_access_block relates to aws_s3_bucket?
The existing tools frustrated me too. Checkov parses well, but adding custom policies feels like wrestling with Python decorators. Terrascan has a powerful policy engine but chokes on our modularized Terraform setup. I figured I’d learn more by building something from scratch than complaining about existing tools.
Why Go Made Sense
Coming from a background heavy in Python for enterprise tools, Go wasn’t my first instinct. But for a tool that needs to parse configuration files, evaluate policies, and run in CI/CD pipelines, Go’s characteristics became compelling.
The concurrency model let me parse multiple Terraform files simultaneously without managing thread pools. I’ll probably write a separate post detailing that experience. Static typing caught errors at compile time that would have been runtime surprises in Python. Most importantly, single-binary deployment meant no dependency hell when installing the tool across environments.
The learning curve was steeper than expected. Go’s interface system felt foreign coming from object-oriented languages. I spent more time than I’d like to admit getting comfortable with interfaces for dependency injection and testing.
Parsing Infrastructure as Code? Harder Than It Looks
The first major challenge was parsing Terraform files. HCL (HashiCorp Configuration Language) looks simple on the surface, but the reality is more complex. There are subtle differences between .tf and .tf.json files. Variable interpolations can nest deeply, and data sources reference other resources in ways that aren’t obvious.
I initially built a simple parser that extracted resource blocks and their attributes. That worked for basic cases but fell apart on real-world Terraform code with modules, complex variable references, and conditional resource creation.
Using HashiCorp’s own HCL parsing library made my workload much easier. Instead of reinventing the wheel, I used the same parsing logic Terraform itself uses, which handled all the edge cases I’d been hitting.
Then came OpenTofu support. When the Terraform fork emerged, I realized the tool should support both ecosystems. Since OpenTofu maintains HCL compatibility, adding support was mostly straightforward.
The Policy Engine Challenge
For policy evaluation, I chose OPA (Open Policy Agent) and its Rego language, having seen OPA’s adoption in Kubernetes and wanting to understand how declarative policy languages work in practice.
Rego’s learning curve is non-trivial. Coming from imperative languages, thinking in rules and constraints rather than step-by-step instructions required a mental shift. Debugging feels quite different too.
Writing security policies in Rego meant understanding Terraform resource structure. For S3 bucket policies, I had to account for different encryption configurations, various ACL settings, and the interplay between bucket policies and public access blocks. Each AWS service has its own quirks that need encoding into policy rules.
The interesting part was making policies extensible: users should be able to drop in new .rego files without recompiling the tool. That meant designing the policy loading system to discover and compile policy files dynamically at runtime.
Testing Infrastructure Code
Testing a security scanner brought unique challenges. Unit tests were straightforward for individual components, but integration testing needed actual Terraform files with known security issues, so I built a test section of deliberately insecure infrastructure configurations.
Coverage metrics were disappointing at first. Testing policy evaluation meant creating extensive test cases across resource configurations and confirming violations were detected correctly. Meaningful test coverage in security tooling requires thinking beyond code coverage to scenario coverage.
One particular headache was the pass rate calculation. My first version gave me a -127% pass rate. Turns out I was counting total violations instead of unique resources: one S3 bucket with five issues counted as five failures. I stared at the math for an embarrassing amount of time before spotting it.
CI/CD Integration
Building the tool was one thing. Making it useful in real development workflows was another challenge entirely, since teams expect tools that integrate into their existing CI/CD pipelines.
Supporting multiple output formats became a priority. Security teams want SARIF for GitHub Security tab integration. QA teams prefer JUnit XML for test dashboards. Developers want human-readable output they can act on immediately, and each format has different expectations for how violations get represented.
The GitHub Actions integration surfaced issues I hadn’t hit during local development. Path handling differs across operating systems. Environment variable handling has subtle differences between local shells and CI. Windows support needed extra testing since most of my development was on macOS.
Getting the automated release pipeline working was its own journey. I wanted the tool installable via go install, which meant publishing to GitHub Packages and getting semantic versioning right. The number of edge cases in release automation was humbling.
Considerations on Performance
As the tool matured, performance became a consideration. Parsing large Terraform configurations can be memory-intensive, especially with generated files from tools like Terragrunt. Policy evaluation scales with resource count and policy complexity.
I added concurrent processing for parsing multiple files, but had to watch for resource contention when several goroutines evaluated policies simultaneously. OPA’s engine has its own performance quirks that needed understanding and working around.
Caching mattered for developer workflows where the same files get scanned repeatedly during development. But cache invalidation gets tricky when policies update independently of the infrastructure code being scanned.
Human Side of Security Tooling
Here’s what nobody tells you about building security tools: the code is the easy part. The hard part is making developers want to use it.
Early feedback was brutal. “Your tool says my S3 bucket is insecure but doesn’t tell me how to fix it.” Fair point, so I added remediation suggestions. But then: “The fix broke our CI/CD pipeline because we need public access for static assets.” Also fair.
Turns out security tools need context. That “insecure” public S3 bucket might be serving your company’s logo. The wide-open security group could be for a honeypot. Generic rules without context create more problems than they solve.
The error messages went through dozens of iterations. First version dumped full stack traces; developers hated it. A too-verbose version explained AWS security models in detail; also hated (“just tell me what to change”). The final version shows the exact line to fix with the secure configuration. Much better.
Lessons Learned and What’s Next
Building PolicyGuard taught me several things about security tooling and Go. Good abstractions matter: they became clear when adding support for different IaC formats and output types. Interfaces in Go shine when you need multiple implementations of similar functionality.
Testing in security tools matters too. False positives undermine trust in automated scanning. False negatives defeat the entire purpose. Getting the balance right requires extensive testing with real-world configurations.
Looking ahead, Kubernetes support is next on the roadmap. Rather than build another custom parser, I’m exploring integration with existing Kubernetes MCP servers that already understand cluster configurations and security best practices. That could improve coverage while cutting development effort.
Azure and GCP support will likely follow similar patterns to the AWS implementation, though each cloud provider has unique services and security models to account for.
Building a security scanner from scratch taught me more than any course could. Understanding how HCL parsing works, how policy engines evaluate rules, and how security violations should reach developers gave me insights I can apply to other security tooling projects.
The Go ecosystem suited this kind of system tool well. The standard library handled most of the heavy lifting, and third-party libraries filled the gaps without creating dependency issues.
For anyone considering similar projects: start with the parsing and policy evaluation core, get that solid with proper testing, then build the user experience and integrations around it. Security tooling is only as good as its adoption, and adoption depends heavily on developer experience.
PolicyGuard’s source code is on GitHub for anyone who wants to dig into the implementation. I hope others build on this foundation to make cloud security more accessible.