Building a Java ‘Hello-World’ gRPC Telemetry Collector For Juniper Devices

When consuming streaming telemetry from network devices, there are various options available to the end user:

  1. One could leverage a vendor’s proprietary telemetry collector software, eg. Juniper Networks’ Contrail Healthbot application.
  2. One could leverage various open source options available, such as Telegraf, FluentD, etc.  For example, Juniper Networks has contributed an open-source Telegraf plugin, called JTI OpenConfig Telemetry Input Plugin, for collecting streaming telemetry via gRPC.
  3. One could build one’s own in-house telemetry collector, using a variety of programming languages, such as Python, Go, C/C++, etc.

In my conversations with various customers, I have noticed that option 3 above seems to be a popular choice for many companies, both large and small.  Recently, a large service provider pinged me with a request for some help in building a simple “Hello World” telemetry collector for Juniper devices using Java.  This blog post aims to walk the Reader through the various steps to build such a collector, and includes a link to the source-code in my GitHub repo.

Prerequisites

The lab environment used to write this blog post consisted of the following:

  • Juniper MX960 router, running 18.3R1.9 (available from the juniper.net downloads site).
  • “JUNOS Telemetry Interface Data Model Files” for 18.3R1 for MX960 (available here).
  • Development Environment: 
    • MacBook Pro running macOS Mojave (10.14.3).
    • IDE: IntelliJ IDEA (2018.3.5).
    • OpenJDK 11.0.2

Installing OpenJDK 11 On MacOS

Beginning with Java 11, Oracle changed its licensing model for the Oracle JDK, which requires you to pay for using it in production (although it is still free to use for development and testing purposes).  To bypass this, we opt to use OpenJDK instead, which is an open source version of the JDK and is free to use, even for commercial purposes.  Note that Oracle JDK and OpenJDK are functionally similar, so we are not losing any features or functionality by going with OpenJDK.

Installing OpenJDK 11 on macOS is quite straightforward:

  1. Download the macOS/x64 build file from here.
  2. Untar the build file and move the extracted folder to “‘/Library/Java/JavaVirtualMachines”, as shown below.
  3. Verify the Java version, as shown below.
bash-3.2$ cd ~/Downloads
bash-3.2$ tar -xzf openjdk-11.0.2_osx-x64_bin.tar.gz 
bash-3.2$ ls
jdk-11.0.2.jdk				openjdk-11.0.2_osx-x64_bin.tar.gz
bash-3.2$ sudo mv jdk-11.0.2.jdk /Library/Java/JavaVirtualMachines/
bash-3.2$ java -version
openjdk version "11.0.2" 2019-01-15
OpenJDK Runtime Environment 18.9 (build 11.0.2+9)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode)

Setup Maven Project In IntelliJ IDEA

Now we are ready to start building our Java telemetry collector app.  Upon firing up IntelliJ IDEA, select “Create New Project” as shown in Figure 1 below.

Figure 1: Create New IntelliJ IDEA Project

There are various Java build automation tools that we could use; for this example, we will go with Maven.  In the New Project window, select “Maven”, and make sure that OpenJDK 11.0.2 is selected for the Project SDK, and click “Next” as shown in Figure 2 below.

Figure 2: Create Maven Project

Next, we enter the project’s “GroupId”, “ArtifactId” and “Version”, as depicted in Figure 3 below, and click “Next”.

Figure 3: Entering Project’s GroupId, ArtifactId and Version

Finally, we enter the location for our project workspace, and click “Finish”, as shown in Figure 4 below.

Figure 4: Specifying The Project Location

Now we have a basic project skeleton which we can now use to build our simple app.

Copy ‘agent.proto’ Into Project Workspace

As discussed in my introductory post on Junos telemetry, Google Protocol Buffers (gpb) are used as a means of serializing and structuring telemetry data messages for transmission over the wire.  The underlying definitions of these messages are defined using a “proto definition file” or “.proto” file.  As mentioned in the Prerequisites section above, you should already have downloaded the “JUNOS Telemetry Interface Data Model Files” for 18.3R1 somewhere in your workspace.  When you unzip and extract the contents of this tarball, you will see a folder (called “junos-telemetry-interface”) that consists of a couple dozen “.proto” files, one for each of the various types of telemetry sensors supported.  Almost all of these “.proto” files pertain to Native Sensors, with the exception of one: “agent.proto” (highlighted in Figure 5 below).  This file defines the OpenConfig Telemetry RPC APIs for gRPC Sensors.  This is the only file we will need in order to build our Java gRPC collector app.

Figure 5: JUNOS Telemetry Interface Data Model Files

So the first thing we need to do is to copy this “agent.proto” file into our project workspace.  Let’s create a couple of new directories in our workspace, under “src -> main”.  First, create a subdirectory called “proto” and copy the “agent.proto” file into this directory, as shown in Figure 6 below.  Next, create another subdirectory called “proto_compiled”, and leave it empty for now.

Figure 6: Copying ‘agent.proto’ Into The Project Workspace

Modify The Project POM

Within a Maven project, the POM (Project Object Model) is a single configuration file (“pom.xml”) that resides in the project’s base directory and contains all the information needed to successfully build the project.  Things such as the project dependencies, plugins, build profiles, and much more can all be specified within the POM.  When we first setup the Maven project in IntelliJ IDEA, it auto-generates a “Minimal POM”, which looks as shown in Figure 7 below.

Figure 7: Minimal POM Generated By IntelliJ IDEA

We need to modify this minimal POM and add some additional gRPC- and protobuf-related dependencies.  In addition, during the build process, we want to automatically compile the “agent.proto” file and auto-generate the necessary gRPC client stubs using the protobuf “protoc” compiler (the section highlighted in green in the code snippet below is responsible for generating these stubs).  We can then use these stubs to build our Hello World app.  To accomplish all of this, overwrite the contents of the current minimal POM with the modified “pom.xml” code snippet shown below.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>net.juniper</groupId>
    <artifactId>jtiGrpcClientJava</artifactId>
    <version>1.0-SNAPSHOT</version>

    <dependencies>
        <dependency>
            <groupId>io.grpc</groupId>
            <artifactId>grpc-netty-shaded</artifactId>
            <version>1.15.0</version>
            <classifier></classifier>
        </dependency>
        <dependency>
            <groupId>io.grpc</groupId>
            <artifactId>grpc-protobuf</artifactId>
            <version>1.15.0</version>
            <classifier></classifier>
        </dependency>
        <dependency>
            <groupId>io.grpc</groupId>
            <artifactId>grpc-stub</artifactId>
            <version>1.15.0</version>
            <classifier></classifier>
        </dependency>
    </dependencies>

    <build>
        <extensions>
            <extension>
                <groupId>kr.motd.maven</groupId>
                <artifactId>os-maven-plugin</artifactId>
                <version>1.6.0</version>
            </extension>
        </extensions>
        <plugins>
            <plugin>
                <groupId>org.xolstice.maven.plugins</groupId>
                <artifactId>protobuf-maven-plugin</artifactId>
                <version>0.5.1</version>
                <configuration>
                    <protocArtifact>com.google.protobuf:protoc:3.6.1:exe:${os.detected.classifier}</protocArtifact>
                    <pluginId>grpc-java</pluginId>
                    <pluginArtifact>io.grpc:protoc-gen-grpc-java:1.15.0:exe:${os.detected.classifier}</pluginArtifact>
                </configuration>
                <executions>
                    <execution>
                        <goals>
                            <goal>compile</goal>
                            <goal>compile-custom</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

            <plugin>
                <groupId>com.github.os72</groupId>
                <artifactId>protoc-jar-maven-plugin</artifactId>
                <version>3.6.0.1</version>
                <executions>
                    <execution>
                        <phase>generate-sources</phase>
                        <goals>
                            <goal>run</goal>
                        </goals>
                        <configuration>
                            <protocVersion>3.6.1</protocVersion>
                            <inputDirectories>
                                <include>src/main/proto</include>
                            </inputDirectories>
                            <outputTargets>
                                <outputTarget>
                                    <type>java</type>
                                    <addSources>none</addSources>
                                    <outputDirectory>src/main/proto_compiled</outputDirectory>
                                </outputTarget>
                            </outputTargets>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

Upon overwriting and saving the “pom.xml” file, Maven will automatically download the specified dependencies. 

To generate the gRPC client stubs, in the Project Tool Window within IntelliJ IDEA (ie. the project explorer view on the left hand side), right-click on the top-level project name (eg. “jti_grpc_client_java”), and select “Maven -> Generate Sources and Update Folders” from the context menu.  This is shown in Figure 8 below.

Figure 8: Generating Sources And Updating Folders Using Maven

Once the above action is completed, you will see the following auto-generated gRPC client stub code appear as highlighted in Figure 9 below.  In particular take note of two specific Java classes:

  1. “src -> main -> proto_compiled -> telemetry -> Agent.java”, and
  2. “target -> generated-sources -> protobuf -> grpc-java -> telemetry -> OpenConfigTelemetryGrpc.java”.
Figure 9: Auto-Generated gRPC Client Stubs

Building The Simple Collector App

We are now ready to write our simple Hello World gRPC Telemetry Collector app by leveraging the above two auto-generated classes (“Agent.java” and “OpenConfigTelemetryGrpc.java”).  Rather than walk the Reader line-by-line through the code, I have opted instead to heavily comment the app source code for improved readability, and present the full source for the collector app in the code snippet below.

NOTE: The full source code for the app can be found at the following GitHub repo: https://github.com/openeye-software/jti_grpc_client_java.

NOTE: In order to run this code, please ensure that your Junos device has been configured as per the following blog post (see ‘Junos Configuration’ section).

import io.grpc.ConnectivityState;
import io.grpc.ManagedChannel;
import io.grpc.netty.shaded.io.grpc.netty.NettyChannelBuilder;
import java.lang.reflect.Field;
import java.util.logging.Logger;

import telemetry.Agent;
import telemetry.Agent.GetOperationalStateRequest;
import telemetry.Agent.GetOperationalStateReply;
import telemetry.Agent.SubscriptionRequest;
import telemetry.OpenConfigTelemetryGrpc;
import telemetry.OpenConfigTelemetryGrpc.OpenConfigTelemetryBlockingStub;


public class TelemetryClient {

    // ----------[ CONSTANTS ]----------
    private static final Logger logger = Logger.getLogger(TelemetryClient.class.getName());
    public static final String DEVICE_IP = "10.49.239.48";
    public static final String DEVICE_PASSWORD = "";
    public static final int DEVICE_PORT = 50051;
    public static final String DEVICE_USERNAME = "";
    public static final int SENSOR_FREQUENCY = 5000;
    public static final String SENSOR_PATH = "/interfaces/interface[name='ge-0/0/0']/state/";


    // ----------[ METHOD main() ]----------
    public static void main(String[] args) {

        ManagedChannel channel = null;
        ConnectivityState connectivityState = null;

        try {
            // IMPORTANT NOTE:
            //
            // Ordinarily, we would use a ManagedChannelBuilder to create the gRPC channel, and then use that to
            // construct ManagedChannel for accessing Network Agent gRPC server using the existing channel.
            // However, there appears to be an issue with the Junos Network Agent gRPC server, whereby it is failing
            // to parse a tracing related header in the request.  We see an error that looks like this:
            //
            //      io.grpc.StatusRuntimeException: UNAVAILABLE: {"created":"@1511903558.423607783","description":"EOF",
            //      "file":"../../../../../../../../src/dist/grpc/src/core/lib/iomgr/tcp_posix.c","file_line":235,"grpc_status":14}
            //
            // As per the following issue: https://github.com/grpc/grpc-java/issues/3800, there is a way to disable
            // tracing using NettyChannelBuilder.

            // Create a new builder with the given host (IP address) and port.
            NettyChannelBuilder nettyChannelBuilder = NettyChannelBuilder.forAddress(DEVICE_IP, DEVICE_PORT);

            // As per the Important Note above, here is the procedure for disabling tracing.  We have to go this
            // roundabout way because the "setTracingEnabled()" method is protected.
            Field declaredField = NettyChannelBuilder.class.getSuperclass().getDeclaredField("tracingEnabled");
            declaredField.setAccessible(true);
            declaredField.set(nettyChannelBuilder, false);

            // With tracing disabled, we can go ahead and construct the gRPC channel.
            // Since we are not using TLS, be sure to use the "usePlaintext()" method.
            channel = nettyChannelBuilder.usePlaintext().build();

            // Get the gRPC channel state: can be one of {IDLE, CONNECTING, READY, SHUTDOWN, TRANSIENT_FAILURE}
            connectivityState = channel.getState(true);
            logger.info("Value of 'connectivityState': " + connectivityState);

            logger.info("Value of 'channel.isShutdown()': " + channel.isShutdown());
            logger.info("Value of 'channel.isTerminated()': " + channel.isTerminated());
            // Don't proceed further if the channel is shutdown or terminated.
            if(channel.isShutdown() || channel.isTerminated()) {
                logger.warning("Halting Execution as gRPC channel is shutdown or terminated");
            }
            else {
                // We need to use the channel created above to create an OpenConfigTelemetryStub, for which there are
                // two options: (1) An asynchronous stub, or (2) A blocking stub.
                // For simple queries like a "getTelemetryOperationalState()", we employ a blocking stub.
                // For streaming telemetry, we will also employ a blocking stub ... we'll implement asynchronous stub at a later date.

                // To test out our basic connectivity with the gRPC server, let's do a "getTelemetryOperationalState()" query.
                OpenConfigTelemetryBlockingStub getOperationalStateStub = OpenConfigTelemetryGrpc.newBlockingStub(channel);

                // The "getTelemetryOperationalState()" method needs a "GetOperationalStateRequest" as an argument.
                // Let's instantiate it and set a couple of fields.
                // According to the agent.proto file, use 0xFFFFFFF for all subscription identifiers including agent-level operational stats.
                // Set the output verbosity level to BRIEF.
                GetOperationalStateRequest getOperationalStateRequest = GetOperationalStateRequest.newBuilder()
                        .setSubscriptionId(0xFFFFFFFF)
                        .setVerbosity(Agent.VerbosityLevel.BRIEF).build();

                // Issue the "getTelemetryOperationalState()" method and capture the result in a "GetOperationStateReply" object.
                // For now, let's just log the contents to the console.
                GetOperationalStateReply getOperationalStateReply = getOperationalStateStub.getTelemetryOperationalState(getOperationalStateRequest);
                logger.info(getOperationalStateReply.toString());


                // Now, lets subscribe to one or more sensors and capture the data ... for this we need an async stub ...
                OpenConfigTelemetryBlockingStub telemetrySubscribeStub = OpenConfigTelemetryGrpc.newBlockingStub(channel);

                // The "telemetrySubscribe()" method needs a "SubscriptionRequest" object as an argument.
                // The "SubscriptionRequest" object needs a Path list as an argument.
                // Let's instantiate all the Objects and set the appropriate fields.
                SubscriptionRequest subscriptionRequest = SubscriptionRequest.newBuilder()
                        .addPathList(
                                Agent.Path.newBuilder()
                                        .setPath(SENSOR_PATH)
                                        .setSampleFrequency(SENSOR_FREQUENCY)
                                        .build()
                        )
                        .build();

                // Subscribe to the the telemetry stream as per the SubscrtiptionRequest.
                // For now, we are just logging the results to console.
                while(telemetrySubscribeStub.telemetrySubscribe(subscriptionRequest).hasNext()) {
                    logger.info(telemetrySubscribeStub.telemetrySubscribe(subscriptionRequest).next().toString());
                }

                // Shutdown the gRPC channel.
                channel.shutdown();

                // Verify that the gRPC channel is indeed shutdown.
                connectivityState = channel.getState(true);
                logger.info("Value of 'connectivityState': " + connectivityState);
            }
        }
        catch(Exception ex) {
            logger.info(ex.toString());

            // In case any exception occurs, let's make sure to shutdown the gRPC channel so we don't leave anything hanging.
            channel.shutdown();
            connectivityState = channel.getState(true);
            logger.info("Value of 'connectivityState': " + connectivityState);
        }

    }
}

To keep things simple, this collector app logs its results to the IDE console.  So when you run the app via IntelliJ IDEA, you will see the output in the “Run” window, highlighted in blue in Figure 10 below.

Figure 10: Collector App Output To IDE Console
Advertisements

Leave a Reply