Some C++ tests

Well, after a while of faffing about with other things, I thought I’d go through some of the new features of C++17 and just a quick play about I found some very interesting new features.

#include <cstdlib>
#include <iostream>
#include <string>
#include <vector>
#include <tuple>

using namespace std;

int main(int argc, char** argv) {
    vector<string> strings = {
        "Hello",
        "World",
        "This",
        "is",
        "the",
        "best",
        "I",
        "can",
        "do"
    };
    
    for (string str: strings) { // I now know
        cout << str << endl;
    }
    
    cout << strings.size() << endl;
    
    // https://www.geeksforgeeks.org/tuples-in-c/
    // makes it more interesting with tuple_cat
    
    auto mytuple1 = make_tuple("triangle1", 't', 10, 15, 20); // interesting...
    auto mytuple2 = make_tuple("triangle2", 'b', 1, 2, 3);
    cout << get<0>(mytuple1) << endl;
    mytuple1.swap(mytuple2);
    cout << get<0>(mytuple1) << endl;
    cout << tuple_size<decltype(mytuple1)>::value << endl;

    return 0;
}

First off the std<vector> initialisation can now be done as it is with Java, but more interestingly is the for loop which works with any class list. That makes life so much easier for C++ handling data.

The next bit was the tuples. For some reason I always associated tuples with a set of 3 items. But after this… This allows for a quick, and maybe dirty way, of avoiding setting up a new class to handle short bursts of data. Very simple.

Now I’m going to play around with some more C++17 features. I may end up moving completely back to C++ and away from Java again.

EDIT:

The output:

Hello
World
This
is
the
best
I
can
do
9
triangle1
triangle2
5

RUN FINISHED; exit value 0; real time: 0ms; user: 0ms; system: 0ms

Java ‘long’ to C++ ‘int64_t’

Hallelujah!

For the last two weeks I’ve had an intermittent problem with my C++ version of the crypto socket client, and tonight I finally cracked the issue.

The Java version is using ‘long’ for doing it’s calculations and the C++ version was using a 64 bit integer, which should have been fine until tonight I realised that Java was using its ‘long’ values as signed. Whereas I had been using ‘uint64_t’ in the C++ version.

This caused the C++ version to work once in every 5 to 10 tries. Just by removing the unsigned, it now works flawlessly.

Phew! Now I can blitz the rest…

(Yep, it has been a while since the last post)

Server bully test

Well, not exactly crashed the server, but I stopped it from running which is a proof of concept.

With a simple Java program which sends legitimate requests from a legitimate device, the server failed after a few minutes. A quick manual restart over ssh got it up and running immediately.

public class BullyServerTest {
    
    public static BullyServerTest bst;

    public static void main(String[] args) {
        bst = new BullyServerTest();
    }
    
    boolean quit = false;
    
    private final String server_host = "SOMEWEBADDRESS"; // LOL
    private final int server_port = SOMESECRETNUMBER;
    
    private final String home = System.getProperty("user.home") + "/";
    
    public BullyServerTest() {
        Runtime.getRuntime().addShutdownHook(new Thread() {
            public void run() {
                quit = true;
                try { Thread.sleep(1000); } catch (InterruptedException ex) {}
            }
        });
        
        while (!quit) {
            try { Thread.sleep(randInt(10, 250)); } catch (InterruptedException ex) {}
            do_hello_world_bully();
        }
    }
    
    Random rand = new Random();
    
    private int randInt(int min, int max) {
        return rand.nextInt((max - min) + 1) + min;
    }
    
    private static final int COMMAND_HELLO = 1234;
    
    private void do_hello_world_bully() {
        int count = randInt(10, 100);
        System.out.println("HELLO WORLD BULLY at " + count + " times");
        
        while (count-- != 0) { // a shed load of threads at once
            new Thread(runnable_HW).start();
        }
        
        System.out.println("Completed...");
    }
    
    Runnable runnable_HW = new Runnable() {
        @Override
        public void run() {
            try {
                WLSocket socket = new WLSocket(server_host, server_port);

                socket.out.writeInt(COMMAND_HELLO);
                String hello = socket.in.readUTF();
                
            } catch (IOException ex) {
                System.out.println("Server failed with HELLO WORLD response");
            }
        }
    };
}

So the next step is to implement a thread pool so that the server will not hang on too many requests. It ran for less than a minute. The above program kept running until my 12Gb RAM ran out which was a bit longer. The current server does only have 4Gb RAM and couldn’t handle the amount of connections at once. I’ve also had to keep the sudden bandwidth usage down as I was hitting my upload and download limits.

Eventually the server and client code will all be rewritten in C++ and handle such things as low latency timeouts.

This has been a very interesting test. I’ll be leaving the hello world server command in for future testing to simulate fake requests.

Java vs C++ for 3D mesh building

A few days ago, playing around with LibGDX again I was getting back into 3D programming and created a very simple demo of a load of spheres and the camera rotating around the scene as below:

Which on my laptop ran just fine. It took about a second to generate all the meshes for each of the spheres.

Took bloody ages on my phone though, even though it’s a quad core 64 bit android phone. Approximately 5 to 8 seconds. Ouch.

None of this nonsense in C++ for starters. I know this from a very old project I was working back in the day. Procedural map generator. More about that here: PROCMAP DBP LINK

So for the last couple of days (actually on the evenings because of work), I’ve looked into creating a single sphere mesh and then duplicating it. In LibGdx

A minor problem to start off with is that a lot of LibGdx’s mesh generators have been deprecated and I don’t know how long for now, and I can’t find examples on how to use the new stuff. This would have made things easier, but that minor problem got bigger.

It seems that ModelBuilder only runs on the GL thread, so I tried the MeshBuilder. No luck there after trying to hunt info down on google, (Maybe google is hiding it from me :P) how to use the thing.

So I gave up, closed the project, might go back to it if the next step fails.

I’m waiting on a huge update from Qt. Oh! At this time it has finished downloading and updated.

The previous versions of Qt didn’t work with the latest Android SDK/NDK. Fingers crossed with this one…

Next…

Devices unique identifier

Now I could have just got the MAC address and used that to communicate with the server, but I wanted to take it to the extremes. Mainly because I wanted to obscure the communication with the previous methods of encryption so the server can identify which device is making a request.

For this I create a string using the devices host name and all the network interfaces connected to it with their MAC addresses. Although I might change this in the near future, at the moment, this gives a good length string for the server to identify the device.

Below is the code used to create this identity string.

    public String host;
    
    private void testMACAddress() {
        try {
            InetAddress ip = InetAddress.getLocalHost();
            
            host = ip.getHostName();

            Enumeration&ltNetworkInterface&gt networks = NetworkInterface.getNetworkInterfaces();
            while (networks.hasMoreElements()) {
                NetworkInterface network = networks.nextElement();
                byte[] mac = network.getHardwareAddress();

                if (mac != null) {
                    StringBuilder sb = new StringBuilder();
                    
                    for (int i = 0; i &lt mac.length; i++) {
                        sb.append(String.format("%02X%s", mac[i], (i &lt mac.length - 1) ? "-" : ""));
                    }
                    
                    host += " " + network.getDisplayName() + " " +  sb.toString();
                }
            }
        } catch (UnknownHostException | SocketException e) {
            System.out.println(e.getMessage());
        }
    }

As mentioned above, this identity string will then be encrypted and sent to the server. Further communication will commence if the server accepts it.

Simplex Noise in Java

Well, I finally got round to testing out the noise algorithms again. And this time in Java.

I grabbed OpenSimplexNoise class and had to add my own functions to it so that I could use ‘octaves’ and ‘persistence’ values to add roughness to it. The extra functions which handle 2D and 3D noise are:

    public double OctavePerlin2(double x, double y, int octaves, double persistence) {
        double total = 0;
        double frequency = 1;
        double amplitude = 1;
        double maxValue = 0;  // Used for normalizing result to 0.0 - 1.0
        for(int i=0;i<octaves;i++) {
            total += eval(x * frequency, y * frequency) * amplitude;

            maxValue += amplitude;

            amplitude *= persistence;
            frequency *= 2;
        }

        return total/maxValue;
    }
    
    public double OctavePerlin3(double x, double y, double z, int octaves, double persistence) {
        double total = 0;
        double frequency = 1;
        double amplitude = 1;
        double maxValue = 0;  // Used for normalizing result to 0.0 - 1.0
        for(int i=0;i<octaves;i++) {
            total += eval(x * frequency, y * frequency, z * frequency) * amplitude;

            maxValue += amplitude;

            amplitude *= persistence;
            frequency *= 2;
        }

        return total/maxValue;
    }

Then I added a UI using Swing and got it all working including adding a water level.

[ADDED] And a better image:

All I need now is to break up the calculations into blocks and run each block on its own thread when I come to using this again.

The class which creates the bitmap:

public class BufferedImageNoise {
    
    int w, h;
    long seed;
    double scale;
    
    int off_x, off_y;
    
    int octaves;
    double persistence;
    
    double water;
    
    OpenSimplexNoise noise;
    
    public BufferedImageNoise(int width, int height, long seed, double scale, int octaves, double persistence, double water) {
        w = width; h = height;
        this.seed = seed;
        this.scale = scale;
        this.octaves = octaves;
        this.persistence = persistence;
        this.water = water;
        
        off_x = off_y = 0;
        
        noise = new OpenSimplexNoise(seed);
    }
    
    public BufferedImage getImage() {
        BufferedImage image = new BufferedImage(w, h, BufferedImage.TYPE_INT_RGB);
        for (int y = 0; y &lt h; y++)
        {
            for (int x = 0; x &lt w; x++)
            {
                double value = noise.OctavePerlin2(x / scale, y / scale, octaves, persistence) + 1.3;
                
                int rgb = value &gt water ? 0x010101 * (int)((value) * 127.5) : 0x000088;
                image.setRGB(x, y, rgb);
            }
        }
        return image;
    }
}

TCP Encryption

Tonight I grabbed an old class that uses a pseudo random number generator and implemented it into the TCP client server encryption test. Both the client and server create a new Object of the class and both set the same seed. Both the client and server will grab the next byte and add the next random number to it on a read, on a write, it will subtract the next random byte.

This worked as expected. The following code is an example of a complex random number generator.

public class WLPRNG {
    
    long seed;
    
    public WLPRNG(long seed) { this.seed = seed; }
    
    public int nextInt() {
        long result = seed + 0x123defca;
        result = Long.rotateLeft(result, 19);
        result += 0xbead6789;
        result *= 0x1234567c;
        
        int temp = (int)result;
        
        result ^= 0x5ecdab73;
        result = Long.rotateLeft(result, 48);
        
        if (temp % 4 == 0) result *= 0x87650027;
        
        result += 13;
        seed = result;
        
        return (int)result;
    }

    public byte nextByte() {
        return (byte)nextInt();
    }
}

The next step for me to try out is to use and encrypted message from the client which sends the MAC address of the client using encryption, and then both the client and server will communicate using the MAC address as a key for another pseudo random number generator.

Over time, the server can update the encryption methods used by the clients.

Cryptography Secure TCP Sockets in Java

Well, I finally got round to testing out a basic encryption over TCP sockets.

First off, the main() entry looks very familiar and is easy to understand. I simply create the listening server (doesn’t spawn thread for simplicity), with the ability to safely shut it down.

Then I use the DataInputStream and DataOutputStream to send data back and forth. In this test case I sent a string which both the client and server will print to the console.

Once that was working, I extended the OutputStream and InputStream and override the read() and write() methods (shown in the code examples below). Notice that I manipulate the byte in two different ways, one by adding and subtracting and the other by XOR’ing (commented out but works). If these manipulations match up whilst data is being sent back and forth then there’s no errors.

Main class

The main class and entry point to the test which spawns the server and then runs a couple of tests:

package com.wlgfx.cryptotest;

public class CryptoServer {

    public static void main(String[] args) {
        WLCryptoServer server_run = new WLCryptoServer();
        Thread server = new Thread(server_run);
        server.start();
        
        System.out.println("Server thread started");
        
        new Thread(new WLCryptoClient("Hello world!")).start();
        
        try { Thread.sleep(500); } catch (InterruptedException ex) {}
        
        new Thread(new WLCryptoClient("Jump around and stomp your feet!")).start();
        
        try { Thread.sleep(500); } catch (InterruptedException ex) {}
        
        server_run.quitServer();
    }
}

Server class

The server code which take the input and output streams and creates new subclasses of the encryption layer classes and then creates the usual DataInputStream and DataOutputStream for the rest of the normal IO. This matched in the following code for the client.

package com.wlgfx.cryptotest;

import com.wlgfx.cryptotest.WLC.WLInputStream;
import com.wlgfx.cryptotest.WLC.WLOutputStream;
import java.io.DataInputStream;
import java.io.DataOutputStream;
import java.io.IOException;
import java.net.ServerSocket;
import java.net.Socket;

public class WLCryptoServer implements Runnable {
    
    private final int port = 1999;
    private boolean quit = false;

    @Override
    public void run() {
        try {
            ServerSocket socket =  new ServerSocket(port);
            socket.setSoTimeout(1000);
            
            while (!quit) {
                Socket client = socket.accept();
                
                WLOutputStream wlout = new WLOutputStream(client.getOutputStream());
                DataOutputStream out = new DataOutputStream(wlout);
            
                WLInputStream wlin = new WLInputStream(client.getInputStream());
                DataInputStream in = new DataInputStream(wlin);
                
                String text = in.readUTF();
                System.out.println("SERVER: " + text);
                out.writeUTF(text);
                
                client.close();
            }
        } catch (IOException ex) {
            System.out.println("SERVER: " + ex.getLocalizedMessage());
        }
    }
    
    public void quitServer() {
        quit = true;
    }
}

Client class

The client code is just the same as the server class code.

package com.wlgfx.cryptotest;

import com.wlgfx.cryptotest.WLC.WLInputStream;
import com.wlgfx.cryptotest.WLC.WLOutputStream;
import java.io.DataInputStream;
import java.io.DataOutputStream;
import java.io.IOException;
import java.net.Socket;

public class WLCryptoClient implements Runnable {
    
    private String text;
    private Socket socket;
    
    private final int port = 1999;
    
    public WLCryptoClient(String test) {
        text = test;
    }
    
    @Override
    public void run() {
        try {
            socket = new Socket("localhost", port);
            
            WLOutputStream wlout = new WLOutputStream(socket.getOutputStream());
            DataOutputStream out = new DataOutputStream(wlout);
            
            WLInputStream wlin = new WLInputStream(socket.getInputStream());
            DataInputStream in = new DataInputStream(wlin);
            
            out.writeUTF(text);
            text = in.readUTF();
            System.out.println("CLIENT: " + text);
            
            socket.close();
        } catch (IOException ex) {
            System.out.println("CLIENT: " + ex.getLocalizedMessage());
        }
    }
    
}

The encryption class

This is where you can see exactly how easy it is to manipulate the data being sent back and forth.  Here I am showing you two methods, XOR and add and subtract. It’s very simple, but for heavier encryption you could look further at CSPRNG, or Crypto Secure Pseudo Random Number Generator. PRNG’s are ten a penny, from simple ones to hefty things. Take your pick. The Crypto and Secure layers are the ones you really need for truly undecipherable data communication.

package com.wlgfx.cryptotest;

import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;

public class WLC {

    // NB: short inner classes are static to avoid errors
    
    public static class WLInputStream extends InputStream {

        private InputStream in;

        public WLInputStream(InputStream input) {
            in = input;
        }

        @Override
        public int read() throws IOException {
            byte a = (byte) in.read();
            a--;
            return (int) a;

            //return in.read() ^ 255; // WORKS TOO
        }
    }

    public static class WLOutputStream extends OutputStream {

        private OutputStream out;

        public WLOutputStream(OutputStream output) {
            out = output;
        }

        @Override
        public void write(int b) throws IOException {
            byte a = (byte) b;
            a++;
            out.write((int) a);

            //out.write(b ^ 255); // WORKS TOO
        }
    }
}
Console output:
run:
Server thread started
SERVER: Hello world!
CLIENT: Hello world!
SERVER: Jump around and stomp your feet!
CLIENT: Jump around and stomp your feet!
SERVER: Accept timed out
BUILD SUCCESSFUL (total time: 1 second)

Until next time

😛

Tonight I’ve sussed out the thread pooling in Java. I wanted a way of being able to queue up threads to be run, but I also wanted to allow the execution of so many threads at a time in this pool. I’ve heard about these thread pools but as of yet, not used them. I need something like this for my own project for both the server and the client.

package threadpooltest;

import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import java.util.logging.Level;
import java.util.logging.Logger;

public class ThreadPoolTest {

    public static void main(String[] args) {
        ThreadPoolExecutor executor = (ThreadPoolExecutor) Executors.newFixedThreadPool(5);
        
        for (int i = 0; i <= 8; i++)
        {
            WorkerTest task = new WorkerTest("Task " + i);
            System.out.println("A new task has been added : " + task.getName());
            executor.execute(task);
        }
        
        System.out.println("Main thread sleeping");
        try { Thread.sleep(8000); } 
        catch (InterruptedException ex) {}
        System.out.println("Finished sleeping");
        
        for (int i = 0; i <= 10; i++)
        {
            WorkerTest task = new WorkerTest("Task " + (i + 5));
            System.out.println("A new task has been added : " + task.getName());
            executor.execute(task);
        }
        
        // THIS DOESN'T WORK
        //System.out.println("WAITING FOR TASKS TO FINISH");
        //try { executor.awaitTermination(1, TimeUnit.DAYS); } 
        //catch (InterruptedException ex) {}
        //System.out.println("FINISHED");
        
        executor.shutdown();
    }
    
}

No proper code snippet plugin yet! Grrr!

Anyway, the above code creates the thread pool which will run a max of 5 threads and queue the rest up. Just what I wanted.

package threadpooltest;

import java.util.concurrent.TimeUnit;

public class WorkerTest implements Runnable {

    private String name;
    
    public WorkerTest(String name) {
        this.name = name;
    }
    
    public String getName() {
        return name;
    }
    
    @Override
    public void run() {
        try
        {
            Long duration = (long) (Math.random() * 5);
            System.out.println("Start Task: " + name);
            TimeUnit.SECONDS.sleep(duration);
            System.out.println("Finished Task: " + name);
        }
        catch (InterruptedException e) {}
    }
}

The above is the example thread I was testing this with.

All I need now is to do the same with C++, but I’ve already found a header only project on github that does just that.

More to come…

EDIT:

I’ve also noted that the above works on Android too.

At the moment, I’m not holding my breath to see it this works with GWT.

Server media management

I’m glad to announce that my media server manager is very almost completed. The files upload to their respective categories and the Java application handles the media library perfectly.

It took a bit of learning Swing in Java but that wasn’t hard at all.

There’s a couple of things to do in the media manager before it is completed. Remove media and categories from the server. And add descriptions to the media files. And that’s it.

The next step is the Android application to manage devices so that I can set them up on the go without the need to get back to my PC.

Unknown devices (those that are ready to be set up) will be stored in the root directory of the media server. The Android application will then assign it a category and a playlist.

All looking good now.

(I’m having a bit of difficulty with WordPress Code Snippets so I’ve been unable to add any snippets recently)