Hands Off My Docker Containers: Dynamic Java Instrumentation in Three Easy Steps
-
Many AppDynamics customers have challenges with modifying startup scripts or updating images in order to inject Java agents, especially in a containerized environment. Other customers might not want to change their build process or completely restructure their projects just to try out a monitoring solution.
Fortunately, there are ways to instrument Java applications without having to access startup scripts or docker-compose.yml files. This blog will show a solution that bundles the dynamic-attach functionality with infrastructure monitoring to instrument Java processes in a Docker environment. This method requires no changes to images or deployment files. It relies instead on the unique ability of AppDynamics’ Java agents to dynamically attach to running Java processes.
Common Approaches to Inject Java Agent
Typically, injecting an AppDynamics Java agent to a process is done by adding runtime parameters to the Java command that creates the JVM. When done this way, the Java agent lives alongside the JVM and monitors it for its entire lifespan. An example of a runtime injection might look like this in the Java command:
java -javaagent:/appdynamics/AppServerAgent/javaagent.jar -jar customer-app.jar
This approach requires the agent files to be either installed locally or volume-mounted. The controller connection information is also often supplied using the run-time parameters, as well as the name of the AppDynamics Application and Tier that the agent will report to.
The above example is a very common way to inject the Java agent, but there are many variations of this method that can be customized to fit a customer’s needs.
Dynamic Attach
An alternative to the persistent Java agent injection described above is the Dynamic Attach method. The AppDynamics Java agent can be attached to a JVM that is already running, requiring no JVM restart. This ability is a competitive differentiator, as other products cannot do this.
This method takes the Java ProcessID and uses the Java -Xbootclasspath parameter to dynamically inject the Java agent into the JVM process. This will cause the JVM to perform a class retransformation necessary to monitor the application. Performance is temporarily impacted during retransformation, but returns to normal after completion.
Dynamic Agent Solution
We have developed a solution that uses Dynamic Attach within a Dockerized environment. This solution is packaged with infrastructure monitoring and is designed to make agent injection very simple to roll out and use. First, we will look at the steps required to use Dynamic Agent, and then talk about how it works.
Requirements
Use of this solution assumes you have a Java application (using Java 1.7 – 1.10) running in Docker containers on a Linux host machine. This solution will not work with IBM Java or JRockit JVMs. JBoss processes can use this, but only if the standalone.conf file is modified to include the following setting:
JBOSS_MODULES_SYSTEM_PKGS=”org.jboss.byteman,com.singularity”
Using DynamicAgent in Three Easy Steps
It is very easy to get started with Dynamic Agent.
Step 1: Download the following repo to your host machine:
git clone https://github.com/Appdynamics/Dynamic-Attach-MA
Step 2: Modify controller.env with your controller connection information.
Step 3: Run the run.sh script.
Step 4: (optional) Sit back and let your boss and coworkers marvel at how quickly you were able to give them valuable performance and business insights into your applications.
Details on the Three Easy Steps
Step 1:
Aside from the git files, pulling down the git repo will result in four small files being downloaded:
- controller.env
- run.sh
- stop.sh
- readme.txt
Step 2:
Controller.env is the only file you need to modify. It contains the usual information on how to connect to the controller.
CONTROLLER_HOST=my.domain.com
CONTROLLER_PORT=8090
CONTROLLER_SSL_ENABLED=false
APPLICATION_NAME=MyApp
ACCOUNT_NAME=customer1
ACCOUNT_ACCESS_KEY=a5f426acdff0
Step 3:
Once you have updated the values in controller.env, you are ready to use the run.sh script.
./run.sh
That’s it! Within a few minutes, you should start to see your newly attached agents reporting to your controller.
After a few more minutes, you should see traffic in the flowmap:
controller.env Parameter Reference
A complete and up-to-date reference of controller.env parameters is contained in the readme.txt file in the git repo.
What is a Tier?
As described in the Common Approaches section above, manual injection of the Java agent requires you to specify the Tier name that each process will report to. In contrast, the Dynamic Agent solution has the ability to extract the Tier name from the container. It also gives you options for how the Tier name can be extracted.
By default, the container’s Hostname will be used as the Tier name in AppDynamics. If you want to change this, there are some optional parameters you can set in the controller.env file:
TIER_NAME_FROM
- TIER_NAME_FROM=HOSTNAME or if left blank or not included in controller.env, it will use the hostname for the Tier name
- TIER_NAME_FROM=CONTAINER_NAME will cause the process to use the name of the container for the Tier name
- TIER_NAME_FROM=JVM_PARAM will cause the process to look for the JVM param specified in TIER_NAME_PARAM
For example, the following controller.env file will result in the process looking for a JVM parameter -Dservice-name. It then will use that value for the Tier name in AppDynamics.
CONTROLLER_HOST=my.controller.com
CONTROLLER_PORT=8090
CONTROLLER_SSL_ENABLED=false
APPLICATION_NAME=Jetty
ACCOUNT_NAME=customer1
ACCOUNT_ACCESS_KEY=4e76-a7f8-a5f426acdff0
TIER_NAME_FROM=JVM_PARAM
TIER_NAME_PARAM=Dservice-name
How It Works
For reasons that will soon become clear, the relevant parts of this solution run as an extension inside a standalone machine agent. Here are some of the more important files involved in the solution.
dynamicAttach.go
The heart of the extension is a Go process. It uses the Docker API to loop through all running containers on the host machine. It inspects each container, looking for a running Java process. If found, it will deploy the appropriate files to the container (housed in agentArchive.tar) and then run one of those files from the container.
This process also keeps track of the container IDs that have been instrumented, allowing it to run every minute without excessive overhead. Any already-instrumented containers will not be re-instrumented. More importantly, new containers will be quickly identified and processed.
agentArchive.tar
Tar files are the only type of file allowed to be pushed to containers by the Docker API. This file contains the Java agent, a copy of tools.jar (because most containers will not have a full JDK), and a script named attachAgent.sh.
The attachAgent.sh file is responsible for doing the actual dynamic attaching, and includes the Java command with the -Xbootclasspath parameter mentioned earlier.
This script must be run from inside the container. If dynamic attachment is attempted from the host machine, there will be errors because the users won’t match up.
The Power of Dynamic Attach
Even though the Dynamic Attach approach to agent injection is not commonly used, we have seen that it can be a powerful tool for getting a Dockerized Java environment instrumented very quickly. It also provides new possibilities to customers who might not have the ability to modify a Docker image or its runtime parameters.
We’re just beginning to explore the possibilities of what we can do with the Dynamic Attach approach. We’re working to expand this solution to work with other agents, too, such as NodeJS. Also, we’ve already started working on a similar Java solution packaged in a Kubernetes operator…but we’ll talk more about that next time.
This blog may contain product roadmap information of AppDynamics. AppDynamics reserves the right to change any product roadmap information at any time, for any reason and without notice. This information is intended to outline AppDynamics’ general product direction, it is not a guarantee of future product features, and it should not be relied on in making a purchasing decision. The development, release, and timing of any features or functionality described for AppDynamics’ products remains at AppDynamics’ sole discretion. AppDynamics reserves the right to change any planned features at any time before making them generally available as well as never making them generally available.
The post Hands Off My Docker Containers: Dynamic Java Instrumentation in Three Easy Steps appeared first on Application Performance Monitoring Blog | AppDynamics.
https://blog.appdynamics.com/engineering/hands-off-my-docker-containers-dynamic-java-instrumentation/
© Lightnetics 2024