Prerequisites

Before we dive into the backup script, let's make sure we have the necessary prerequisites set up:

1. Go Installation: Make sure you have Go installed on your machine. You can download it from the official Go website (https://golang.org) and follow the installation instructions for your operating system.

2. External Dependencies: Ensure that the tar and scp (if using scp backup) or rclone (if using rclone backup) commands are available on your system. These commands are used by the backup script to create the backup archive and copy it to the destination server.

Getting Started

Now that we have the prerequisites in place, let's start writing the backup script in Go. Open your favorite text editor or IDE and create a new file called main.go. Copy the following code into the file:

package main

import (
    "fmt"
    "log"
    "os"
    "os/exec"
    "time"

    "github.com/spf13/viper"
)

func main() {

    // read values from configuration file
    configPath := os.Getenv("BACKUP_CONFIG_PATH")
    if configPath == "" {
        log.Fatalf("missing BACKUP_CONFIG_PATH environment variable")
        os.Exit(1)
    }
    viper.SetConfigFile(configPath)
    if err := viper.ReadInConfig(); err != nil {
        fmt.Printf("Failed to read configuration file: %v\n", err)
        os.Exit(1)
    }

    // replace these values with your own directory path and server details
    dirPath := viper.GetString("dirPath")
    server := viper.GetString("server")
    backupType := viper.GetString("backupType")

    archiveFileName := viper.GetString("archiveFileName") + "_" +
        time.Now().Format("2006-01-02_15-04") + ".tar.gz"

    // create a tar archive of the directory
    tarCmd := exec.Command("tar", "-czf", archiveFileName, dirPath)
    if err := tarCmd.Run(); err != nil {
        fmt.Printf("Failed to create tar archive: %v\n", err)
        os.Exit(1)
    }

    if backupType == "scp" {
        // copy the archive to the server via scp
        if err := copyArchiveToServer(server, archiveFileName); err != nil {
            fmt.Printf("Failed to copy archive to server: %v\n", err)
            removeArchive(archiveFileName)
            os.Exit(1)
        }
    } else if backupType == "rclone" {
        // copy the archive to the new server via rclone
        rcloneCmd := exec.Command("rclone", "copy", archiveFileName, server)
        if err := rcloneCmd.Run(); err != nil {
            fmt.Printf("Failed to copy archive to object storage: %v\n", err)
            removeArchive(archiveFileName)
            os.Exit(1)
        }
    } else {
        fmt.Printf("Invalid backup type: %v\n, can be scp or rclone", backupType)
        removeArchive(archiveFileName)
        os.Exit(1)
    }

    // delete the local archive file
    if err := removeArchive(archiveFileName); err != nil {
        fmt.Printf("Failed to delete local archive file: %v\n", err)
        os.Exit(1)
    }

    fmt.Println("Archive created and copied to new server successfully.")
}

func copyArchiveToServer(server string, file string) error {
    scpCmd := exec.Command("scp", file, server)
    if err := scpCmd.Run(); err != nil {
        return fmt.Errorf("failed to copy archive to new server: %v", err)
    }
    return nil
}

func removeArchive(file string) error {
    if err := os.Remove(file); err != nil {
        return fmt.Errorf("failed to delete %v: %v", file, err)
    }
    return nil
}

Understanding the Script

Let's take a moment to understand the different parts of the backup script.

1. Configuration File: The script expects the path to the configuration file to be set in the BACKUP_CONFIG_PATH environment variable. It uses the Viper library to read the configuration values from the file.

2. Backup Settings: The script reads the necessary backup settings from the configuration file, such as the directory path to be backed up, the destination server, the backup type (scp or rclone), and the desired name for the backup archive file. Make sure to replace these values with your own in the script.

3. Creating the Archive: The script uses the tar command to create a compressed tar archive of the specified directory using the tar -czf command.

4. Copying the Archive: Depending on the backup type specified in the configuration file, the script either uses the scp command (for scp backup) or the rclone command (for rclone backup) to copy the archive to the destination server. If the copying process fails, it removes the local archive file.

5. Deleting the Local Archive: After successfully copying the archive to the server, the script deletes the local archive file to save disk space.

Running the Backup Script

To run the backup script, follow these steps:

1. Save the script to a file named backup.go in a directory of your choice.

2. Set the BACKUP_CONFIG_PATH environment variable to the path of your configuration file. For example, you can run the following command in the terminal:

    export BACKUP_CONFIG_PATH=/path/to/backup_config.yaml
   

3. Open a terminal and navigate to the directory containing the backup.go file.

4. Build and run the script using the go run command:

    go run backup.go
   

5. The script will read the configuration file, create a backup archive of the specified directory, and copy it to the destination server using the specified backup type. The progress and any errors will be displayed in the terminal.

6. Once the backup process is complete, the local archive file will be deleted, and a success message will be displayed.

Congratulations! You have successfully automated the backup process using Go.

Conclusion

We wrote a backup script that reads the backup settings from a configuration file, creates a compressed tar archive of a specified directory, and copies it to a destination server using either scp or rclone. By following the steps outlined in this guide, you can ensure the safety and availability of your important files through automated backups.

Git repository link can be found here.

Benefits of gRPC:

• Language Agnostic: One of the most significant benefits of gRPC is that it's language agnostic. gRPC provides libraries for multiple programming languages, including Java, Python, C++, and many others. This feature enables services written in different languages to communicate seamlessly.

• High Performance: gRPC is built on top of HTTP/2, which makes it incredibly fast and efficient. HTTP/2 provides multiple benefits like server push, request multiplexing, and header compression, which makes gRPC a high-performance framework.

• Easy to Use: gRPC has a straightforward API, which makes it easy to use. The framework provides features like request/response streaming, bidirectional streaming, and server-side streaming. These features make it simple to build complex distributed systems.

• Interoperability: gRPC allows for seamless communication between different systems, which enables services to work together even if they are written in different languages. gRPC uses a well-defined protocol buffer format to ensure interoperability.

Using gRPC in Java:

• Define the service using protocol buffers: Protocol buffers are used to define the service contract, data models, and messages. Protocol buffers are used to generate the server and client code.

  Example: proto file for a unary service.

    syntax = "proto3";
  
    package greeting;
  
    option java_package = "com.manitaggarwal.grpcclient";
    option java_multiple_files = true;
  
    message GreetingRequest {
      string firstName = 1;
    }
  
    message GreetingResponse {
      string result = 1;
    }
  
    service GreetingService {
      rpc greet(GreetingRequest) returns (GreetingResponse);
    }
  

• Generate the server and client code: Once you have defined the service using protocol buffers, you need to generate the server and client code. The code generation is done using the protoc compiler, which generates the code in the language of your choice.

  Dependencies for gRPC in maven.

    <dependency>
        <groupId>io.grpc</groupId>
        <artifactId>grpc-netty-shaded</artifactId>
        <version>1.53.0</version>
        <scope>runtime</scope>
    </dependency>
    <dependency>
        <groupId>io.grpc</groupId>
        <artifactId>grpc-protobuf</artifactId>
        <version>1.53.0</version>
    </dependency>
    <dependency>
        <groupId>io.grpc</groupId>
        <artifactId>grpc-stub</artifactId>
        <version>1.53.0</version>
    </dependency>
    <dependency> <!-- necessary for Java 9+ -->
        <groupId>org.apache.tomcat</groupId>
        <artifactId>annotations-api</artifactId>
        <version>6.0.53</version>
        <scope>provided</scope>
    </dependency>
  

  Plugin requirements

    <build>
        <extensions>
            <extension>
                <groupId>kr.motd.maven</groupId>
                <artifactId>os-maven-plugin</artifactId>
                <version>1.7.1</version>
            </extension>
        </extensions>
        <plugins>
            <plugin>
                <groupId>org.xolstice.maven.plugins</groupId>
                <artifactId>protobuf-maven-plugin</artifactId>
                <version>0.6.1</version>
                <configuration>
                    <protocArtifact>com.google.protobuf:protoc:3.21.7:exe:${os.detected.classifier}</protocArtifact>
                    <pluginId>grpc-java</pluginId>
                    <pluginArtifact>io.grpc:protoc-gen-grpc-java:1.52.1:exe:${os.detected.classifier}</pluginArtifact>
                </configuration>
                <executions>
                    <execution>
                        <goals>
                            <goal>compile</goal>
                            <goal>compile-custom</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
  

• Implement the server: Once you have the server code, you can implement the server logic. The server logic is where you implement the business logic of your service.

    public class GreetingServiceImpl extends GreetingServiceGrpc.GreetingServiceImplBase {
  
        @Override
        public void greet(GreetingRequest request,
                          StreamObserver<GreetingResponse> responseObserver) {
            System.out.println("request.getFirstName() = " + request.getFirstName());
            responseObserver.onNext(GreetingResponse.newBuilder()
                    .setResult("Hello " + request.getFirstName())
                    .build());
            responseObserver.onCompleted();
        }
    }
  

• Implement the client: Once you have the client code, you can use it to make requests to the server. The client code provides a simple API that makes it easy to communicate with the server.

    @SpringBootApplication
    public class GrpcClientApplication {
  
        private final static String CHANNEL_NAME = "localhost";
        private final static int SERVER_PORT = 50051;
  
        private final static List<String> list = List.of("Dan", "Pan");
  
        public static void main(String[] args) throws InterruptedException {
            SpringApplication.run(GrpcClientApplication.class, args);
            ManagedChannel channel = ManagedChannelBuilder
                    .forAddress(CHANNEL_NAME, SERVER_PORT)
                    .usePlaintext()
                    .build();
  
            GreetingServiceGrpc.GreetingServiceBlockingStub stub =
                    newBlockingStub(channel);
  
            for (String name : list) {
                GreetingResponse response =
                        stub.greet(GreetingRequest.newBuilder()
                                .setFirstName(name)
                                .build());
  
                System.out.println(response.getResult());
            }
  
            channel.shutdown();
        }
  

Conclusion:

gRPC is a powerful framework that provides a high-performance and easy-to-use mechanism for building distributed systems. Its language-agnostic nature, high performance, easy-to-use API, and interoperability make it an excellent choice for building modern cloud-native applications. Using gRPC in Java is simple, and it provides a lot of benefits that make it an excellent choice for building complex distributed systems.

Code for the above can be found here on the GitHub.

Testing RESTful APIs with Rest Assured

Rest Assured is a Java-based library that simplifies the testing of RESTful APIs. It provides a simple and intuitive DSL (Domain-Specific Language) for writing tests for RESTful APIs. It also supports various HTTP methods such as GET, POST, PUT, DELETE, etc.

Let's look at an example of how to use Rest Assured to test a RESTful API. Consider the following endpoint:

@GetMapping(value = "/student/{studentId}", produces = MediaType.APPLICATION_JSON_VALUE)
public Student getStudentByStudentId(@PathVariable String studentId) {
    return studentService.getStudent(studentId);
}

This endpoint retrieves a student with a given student ID. To test this endpoint, we can write a test case using Rest Assured as follows:

/*
 * Given: Where a student exists in the system
 * When: Call is placed to retrieve the student using student id
 * Then: Details of the student are fetched correctly
 * */

@Test
public void getStudentWhenExists_ByStudentId() {

    // given - prerequisites
    String msisdn = getRandomNumber();
    String email = "harry@admin.com";
    String name = "Harry";
    Student response = (Student) JsonUtils.getObjectFromJson(
            APICall.addStudent(DefaultData.getAdminRequest(msisdn, email, name)).asString(),
            Student.class);
    Assert.assertEquals("add assert", msisdn, response.getMsisdn());

    String studentId = response.getStudentId();

    // when
    Student studentFromDatabase = (Student) JsonUtils.getObjectFromJson(APICall.getStudent(studentId).asString(),
            Student.class);

    // then
    Assert.assertEquals("get assert", studentFromDatabase.getStudentId(), studentId);

}

In the above test case, we are using the given to specify the parameters for the request. In this case, we are ensuring that the student whom we want to retrieve exists in the system.

Then we are using the when to send the GET request to the specified endpoint.

Finally, we are using the then to verify the response. We are verifying that the student retrieved is having the same ID as specified by us in the request.

Conclusion

In this blog post, we discussed how to use Rest Assured. Official documentation for the rest assured can be found at GettingStarted · rest-assured/rest-assured Wiki (github.com).

Also, the complete code for this project can be found at manitaggarwal/spring-boot-rest-assured: Implementing Rest Assured (github.com)

Bloom filters are a data structure used in computer science for efficient probabilistic set membership testing. In other words, given a set of elements, a Bloom filter can answer whether an element is not in the set or maybe in the set, but cannot say for sure whether it is definitely in the set.

A Bloom filter is a compact data structure that uses multiple hash functions to map elements to positions in a bit array. When an element is added to the set, its hash values are used to set bits in the bit array to 1. When checking if an element is in the set, its hash values are used to look up bits in the bit array. If all of the bits are 1, it is likely that the element is in the set, but there is a chance of false positives (i.e. a bit of information that says an element is in the set, when it is actually not).

To create a Bloom filter, you need to determine the size of the bit array and the number of hash functions to use. The size of the bit array will depend on the expected number of elements in the set and the desired false positive rate. The number of hash functions will depend on the size of the bit array and the desired false positive rate.

Java Implementation

In Java, you can implement a Bloom filter using a library by Google called Guava. Here is an example implementation:

Creating a filter using Guava:

Here we are creating a Bloom filter for up to 1000 Integers and we can tolerate a one-per cent (0.01) probability of false positives.

BloomFilter<Integer> sha256filter = BloomFilter.create(
        Funnels.integerFunnel(),
        1000,
        0.01);

Hashing the string

Here we hash the String to sha256 and get the first 3 numbers

public String sha256hex(String originalString) {
    return Hashing.sha256()
            .hashString(originalString, StandardCharsets.UTF_8)
            .toString().replaceAll("[^0-9]", "").substring(0, 3);
}

Saving the hash in the filter

After we successfully hash the string, we save the hashed number to filter

private void saveToSHA256Filter(String originalString) {
    sha256filter.put(Integer.parseInt(
            hashUtils.sha256hex(originalString)));
}

Checking filter

private boolean checkInSHA256Filter(String originalString) {
    return sha256filter.mightContain(Integer.parseInt(
            hashUtils.sha256hex(originalString)));
}

You can find the implementation here on github.

References

Github for Guava

MongoDB is a NoSQL database program and uses JSON-like documents with optional schemas. Which makes it exceptionally great for storing logs.

I wrote a simple spring boot microservice to consume logs using a REST API and a Kafka Listener and store them in MongoDB.

Using Java Generics, we can create a document to store any JSON information on the database.

REST Controller

@PostMapping(value = "/logs", consumes = "application/json", produces = "text/plain")
public void addLogs(@RequestBody LoggingRequest<?> request) {
    loggingService.saveLogs(request);
}

Request

@Data // using lombok
public class LoggingRequest<T> {
    String applicationName;
    T logs;
}

Document

@Document("logfile") // using springframework.data.mongodb
@Data //using lombok
public class LogFile<T> {

    @Id
    private String id;

    private Date time;

    private T data;

    public LogFile(Date time, T data) {
        this.time = time;
        this.data = data;
    }
}

Repository

// using springframework.data.mongodb.repository.MongoRepository
public interface LoggingRepository extends MongoRepository<LogFile<?>, String> {
}

Data in MongoDB

{
  "time": {
    "$date": {
      "$numberLong": "1675923022932"
    }
  },
  "data": {
    "applicationName": "monitoring",
    "logs": {
      "type": "success",
      "message": "add.success"
    },
    "_class": "com.manitaggarwal.store.log.controller.request.LoggingRequest"
  },
  "_class": "com.manitaggarwal.store.log.document.LogFile"
}

{
  "time": {
    "$date": {
      "$numberLong": "1675915874047"
    }
  },
  "data": {
    "applicationName": "monitoring",
    "logs": {
      "type": "error",
      "trace": "customer.does.not.exists"
    },
    "_class": "com.manitaggarwal.store.log.controller.request.LoggingRequest"
  },
  "_class": "com.manitaggarwal.store.log.document.LogFile"
}

Conclusion

Data is stored in MongoDB and can be queried using filters.

{ "data.applicationName" : "monitoring", "data.logs.type" : "success" }

Reference

Github Repo Link

Spring Boot makes it easy to create stand-alone, production-grade Spring based Applications that you can “just run”. Recently the fantastic team at Spring have been adding useful features so that repetitive typical features that are retaliated to packaging and running the application are streamlined.

Creating an image which is compatible with Open Container Initiative is as simple as running mvn spring-boot:build-image, and the image will be built using a cloud native buildpack.

As we want to run our images in a production kubernetes cluster, we need to a few more files to our code and do some preparations on the cluster so that we can run the images as deployments in the k8s cluster.

Adding manifest files

To run application in k8s we need a deployment, service and an ingress file.

We can create a new folder in the root of the project, at the level of src folder, and call it as manifests. Then inside this folder we can make all of our files.

In my case the application is called monitoring.

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: monitoring
  template:
    metadata:
      labels:
        app: monitoring
    spec:
      containers:
        - image: ghcr.io/manitaggarwal/monitoring:latest
          imagePullPolicy: Always
          name: monitoring
          ports:
            - containerPort: 8080
      imagePullSecrets:
        - name: regcred

# service.yaml
apiVersion: v1
kind: Service
metadata:
  name: monitoring
spec:
  type: ClusterIP
  selector:
    app: monitoring
  ports:
    - port: 8080
      targetPort: 8080

# ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: monitoring
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
  namespace: default
spec:
  tls:
    - hosts:
        - subdomain.example.com
      secretName: monitoring-tls
  rules:
    - host: subdomain.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: monitoring
                port:
                  number: 8080

Preparing k8s cluster

As we want to reach our service and use a domain name to access it we will need an ingress controller. I have another post where I have detailed how to install ingress-nginx, in more detail.

To install ingress-nginx Ingress Nginx in itself will not give us the ability to generate SSL certificates, for that we need cert manager.

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

helm install ingress-nginx ingress-nginx/ingress-nginx

To install cert manager

kubectl create namespace cert-manager
helm install cert-manager jetstack/cert-manager --namespace cert-manager --version v1.0.4 --set installCRDs=true

To add Cert Issuer

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
  namespace: cert-manager
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: manit@example.com
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - http01:
        ingress:
          class: nginx

Preparing pipeline

We could manually apply the manifests to achieve our goal, but I am using Github Actions so that when I push to git repo the runner picks up the changes and builds and deploys the images to the desired k8s cluster.

name: Java CI with Maven

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

env:
  IMAGE_NAME: ghcr.io/manitaggarwal/monitoring:${{ github.sha }}

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Set up JDK 11
      uses: actions/setup-java@v2
      with:
        java-version: '11'
        distribution: 'adopt'
    - name: Login to GitHub Container Registry
      uses: docker/login-action@v1
      with:
        registry: ghcr.io
        username: ${{ github.actor }}
        password: ${{ secrets.GITHUB_TOKEN }}
    - name: Build with Maven
      run: | 
        mvn spring-boot:build-image
        docker tag monitoring:0.0.1-SNAPSHOT ${{ env.IMAGE_NAME }}
    - name: Push to Registry
      run: |
        docker push ${{ env.IMAGE_NAME }}
  deploy:
    needs: build
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - uses: Azure/k8s-set-context@v1
      with:
        kubeconfig: ${{ secrets.KUBE_CONFIG }}
    - name: Run on Cluster
      run: |
        kubectl apply -f manifests/
        kubectl set image deployment/monitoring monitoring=${{ env.IMAGE_NAME }}

Sources

Pulling from private registry

Cert manager

I have been using metal LB from quite some time now, even though the website still says the they are in beta, I have found it to be quite stable for my needs.

Why metal LB?

Kubernetes does not offer an implementation of network load-balancers (Services of type LoadBalancer) for bare metal clusters. The implementations of Network LB that Kubernetes does ship with are all glue code that calls out to various IaaS platforms (GCP, AWS, Azure…). If you’re not running on a supported IaaS platform (GCP, AWS, Azure…), LoadBalancers will remain in the “pending” state indefinitely when created. Bare metal cluster operators are left with two lesser tools to bring user traffic into their clusters, “NodePort” and “externalIPs” services. Both of these options have significant downsides for production use, which makes bare metal clusters second class citizens in the Kubernetes ecosystem. MetalLB aims to redress this imbalance by offering a Network LB implementation that integrates with standard network equipment, so that external services on bare metal clusters also “just work” as much as possible.

How to install metal LB by manifest?

Step 1 -> Creating a namespace

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.10.2/manifests/namespace.yaml

Step 2 -> Installing metal lb

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.10.2/manifests/metallb.yaml

Step 3 -> Creating a config file and setting up a range of IP addresses

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 192.168.1.240-192.168.1.250

Step 4 -> Applying the ConfigMap

kubectl apply -f config.yaml

Reference

Official Documentation

Recently I wanted learn kubernetes and it was difficult to find a good development environment which I could use to play with and recreate within seconds when I break it.

I came across rancher which can be deployed on docker as a single docker container and all persistent volume claims can be mapped to a docker volume so data can be preserved.

Startup

To start the docker container I created a run.sh file with following contents

#!/bin/bash
docker run -d --restart=unless-stopped --network=host \
-v /home/manit/rancher/config:/var/lib/rancher \
-v /home/manit/rancher/ssd:/ssd \
-v /mnt/disk/rancher/hdd:/hdd \
--name rancher \
--privileged rancher/rancher:latest

I am using host networking above so that when I deploy a kubernetes deployment with a server port it gets available on the localhost (or server public IP address).

I have 2 disks attached to the system, and main disk is an SSD, so I map out 2 directories from host to docker container so that I can store data from PVC where I want.

If you do not want to use host networking, you can use port 80 and 443 and publish them.

Cleanup

#!/bin/bash
docker container rm -f rancher
sudo rm -r ~/rancher/config ~/rancher/ssd /mnt/disk/rancher/hdd/

To save the data from kubernetes persistent volumes you may not remove ~/rancher/ssd and /mnt/disk/rancher/hdd.

This gave me an easy way to develop and clean out where something goes wrong.

My setup is running on old Lenovo Thinkpad which is currently connected to my home network.

References

Rancher Docs

A Typical Spring Boot project can be generated at Spring Initialiser

Application JAR typically is compile using

mvn clean package

Which can now be swapped to

mvn -Pnative clean package

As the adding of spring native adds a special profile to the POM of the application.

Which will create a platform specific binary, thus giving significant speed boost to the application at startup times and remove the requirement of JRE in the system running the binary.

Project is hosted on my github, which can be found here.