I attended for the first time at JavaZone this year. This was an amazing experience to meet some new folks and I learned a lot through the various conferences and workshops I could attend during those three days.
Discovering meta-agent
Among the things I discovered, there was meta-agent presented by Marco Sussitz during his talk: Demystifying OpenTelemetry: Java Agents Unveiled. This is a tool created by Johannes Bechberger and Mikaël Francoeur which was originally designed to investigate the generated bytecode from Spring proxies and Mockito mocks. Basically, you will have a hard time trying to debug (byte)code that is generated at runtime, whose source does not exist. Such proxies and mocks implementations only live in JVM memory so this tool provides a way to look at class file content that is loaded by the JVM. Internally, it works as a JVM agent that hooks into the Instrumentation API to decorate the other instrumenters and record their changes introduced through the ClassFileModifiers.
When it looks too beautiful…
While my job at Datadog involved a good amount of bytecode injection work, I was a bit disappointed to discover that meta-agent did not work with The Datadog Java Client Library (aka dd-trace-java) nor the OpenTelemetry Java agent. During Marco’s presentation, he showed that only a limited amount of class transformation was captured by the meta-agent. But unlike him, I did not want all the troubles of forking and patching the JVM to dump transformed classes to troubleshoot the bytecode our agent changes.
Debugging the debugger
So I spent a bit of time studying the Byte Buddy instrumenter and found out the issue: Byte Buddy agent is using proxy to register itself as an Instrumentation so its implementation was not covered by the meta-agent. A quick fix later, and I could finally enjoy the tool with both Datadog and OpenTelemetry agents.
Making it more accessible
I also took a bit of time to polish the user interface and experience. While the tool is powerfully and fully functional, I thought that a modern UI could help its adoption among my teams so I improved the navigation and the look and feel.
Conclusion
I would encourage you to give it a try if you’re curious to discover how the Datadog Java Client library (dd-trace-java) and the OpenTelemetry Java agents actually interact with your application and patch it on the fly. You will discover that what runs might sometimes be very different from what you compile or expect to execute!