Been busy working on my secret project.
Author: csnorwood
csnorwood.com
It was tough to accept the change from my old website (wlgfx) to this new one (csnorwood), but I couldn’t ignore the benefits.
Yes, this is going to affect the search engines for some time but I’m used to that. I can still slowly build stuff in the meantime while the web crawlers do there stuff.
It’s highly unlikely that the the old web address is even going to link back to this new one, but what the hell. Time to move onward and upwards.
Rewrite from Java to C++
Okay, so my server is currently running on Java and makes use of DataOutputStream and DataInputStream. And whilst I’m on a rewrite, I need to wrap those into C++.
Which I thought might be a tad difficult…
It turns out that Linux automatically handles UTF-8 strings out of the box, so sending and receiving string is of no consequence. Which was one of my main worries before I looked into everything else which slowed down the transition. Java’s DataOutputStream writes it UTF-16 strings as UTF-8 strings over the wire. That’s that problem solved.
The other problem was wrapping the socket connection. Sorted. Easily done by returning the error status on every call and handling the active status of the socket.
The last problem was wrapping the Java functions for sending a Long, Int, Byte, String, etc… These need to be in network byte order, which is not little endian. I’m just going through that now.
Oh yeah, encryption using a Cryptographic Secure Pseudo Random Number Generator. Done and implemented to match perfectly the Java version… For now.
So hopefully soon, I’ll be able to test run this wrapper ready for a production run.
Amazing feeling
“I feel almost like a god!”
To have my own software running in the cloud and being able to serve its purpose, ‘feels good’.
Yes, I’ve had one of those moments of personal successful achievements accomplished.
Or something like that.
‘Project’ updates
Maybe things have plodded along at a gentle pace recently, but that’s also a good thing because it give me a chance to reflect back and get things back into perspective. Which I have done.
A few weeks ago I was ready to plough ahead with this. And then as I get stopped in my tracks, or more so, slowed down, I start to find a few more faults, cracks, bugs, etc. All fixed now.
Anyway, the server has now moved onto a VPS somewhere in the cloud which means better bandwidth speed than running from home. I’m still testing out the VPS and getting myself familiar with using it. Everything is via terminal so my previous Linux knowledge is becoming extremely useful here. Commands such as ‘rsync’ to copy files from my PC to the server for example. Now I need to create a way of restarting the server on demand.
The Java TCP server still has its minor issues, but it just works. Any problems I face I can just restart it and everything just carries on. Once it is securely setup on the VPS then I can leave it alone and work on the C++ server upgrade coding.
I won’t talk about the many other things that have happened around this project in the recent weeks apart from to say that it will be big. Fair enough, now I’m starting to feel the pressure is on me to come up with the C++ server code. Which I might actually continue with tomorrow after I figure out how to get my server software to automatically start on the VPS.
Everything will come to light, … in time.
“Follow your passion, not your paycheck. The money will come eventually.”
Just some random updates
The ‘project’ has slowed a bit because of content and artwork which I will be jumping on over these next few days.
Server updates are now being reconsidered because I read last night that non-blocking IO can also be done in Java. I can quickly setup stuff in Java than I can in C++ which means Java has that slight advantage being a language. It may end up being 10-15% slower overall, but it just does stuff. And development is much faster.
However, saying that, I’ve still to test out the non-blocking IO in Java. So time will tell on that score.
If it works then the previous devices commands will be ported over so they will carry on running. The current server is running Java so why not.
If it doesn’t, then it will be back to the fd_set and select() in C which was coming on very nicely until the distraction.
Video editing has been done mainly with Blender. Recently though, I tried Adobe Premiere CS2 and at first glance it looked like it could really do the job. First off, it doesn’t import 99% of video clips and other file formats. I think the only two things that it has going for it is the titling and the rendering of the work area for faster previewing.
Now, Openshot, may not have the best interface, but it just works. Titling is a pain using an external editor. And it also renders the video in memory as you’re going along for faster playback.
With using Openshot, I can create 95% of what I need it for. All titling can be done externally which can be a pain, but more than doable. I just drag n drop the exported PNG’s. I only need to figure out a faster way of setting key frames for animations.
My head is exploding at the moment. I’ve got the ‘project’ to think about, the ‘projects’ server updates, learning new API’s, protocols, etc, blah, as well as trying to do this all in ‘my’ time.
My time now is extremely valuable to me. I’m spending most of it on working on my ‘project’. There are people that know about it, and others I won’t tell because those that know would gladly give up their ‘time’ for me.
This project is actually going to be a huge thing for me. There are only a few involved in it. I was going to involve another but fate stepped in perfectly because of this ‘time’ issue. So it’s carrying on as it was meant to. It is a lot of mental work but it’s will be worth it.
IPv6 server upgrades
A few weeks back I started on the upgrade from the Java server to a C++ server.
I scrapped the first project and started again when I learnt about the handling of fd_set which made things so much better. First tests were great.
Then I scrapped it and decided that if it going worldwide with this secret project, then it needs and uprade. To IPv6.
Which works on my initial tests.
Now all I need it to finish off a few more DNS tests and then I’m ready to make the server ultra fast.
TCP C++ server upgrades
A quick snippet from tonight’s diary entry:
20:30 – I’m still reading up on some stuff about non-blocking TCP server programming and so far the best examples come from http://developerweb.net/viewtopic.php?id=2933. Ideally what I want is NOT to have a shed load of threads running, which I’ll be using a thread manager for, but a way of quickly determining if a connection is valid before shutting it down.
By the looks of the examples, a continuous loop could be checking upto 1024 connections at any time before sending them off to another thread. This method has caught my attention big time because I should be able to setup some kind of timeout very quickly to close the connection if it doesn’t respond to the initial connection request. And I can still serve 100 to 1000 devices without any problems at all.
The same single thread can be constantly checking for connections and readability. It’s the readability part that matters the most here because it will be timed very precisely. Once a connection passes the readability test by passing the initial CSPRNG test, then it can go to a thread handler. Otherwise it gets closed. All of this a single core/thread can easily handle.
I will be studying this a lot further as this looks to be the way forward for a tcp server.
And this is just from my crappy laptop at home.
Basically, the new software upgrades to the server will be 1000 times more efficient than the current Java version. Although once in the cloud, the Java version can easily serve over 500 devices because of the bandwidth increase.
The updated C++ version will increase that significantly as well as handle dodgy connections better. Once a connection has passed the CSPRNG test then the TIMEOUT increases a smidgen.
I’ve already got the CSPRNG functions running and tested and written in assembler code to reduce that bottleneck too.
Until next time… WLGfx…
MS Access Databases
They’re really just as simple as using */UNIX…
Huh…
I’ve been faffing about with my database for most of the night and after relearning a few things, I deleted all my queries and forms and started again.
The relationships screen is where I now layout everything and then I decide to add tables from there well before I start designing the forms.
What I had before would have worked, but I wanted this to be perfect. And that’s why I deleted all my previous work. Using the relationships between Tables and Queries really does make life easier when building up forms and reports.
So far though, with my own project, I’ve not had the need for any VBA unlike at work. The experiences from my own project and the works one too will benefit both projects. I’m hoping that when I decide on changes for the works project, it won’t be too big. Luckily enough, both of them are at the moment small databases.
Just with tonight’s ‘messing’ around, I’ve actually got back to fully understanding the relational side of linking tables and queries. And how it makes creating forms much easier.
It’s only been about two weeks now I think getting back into MS Access, there about.
Ah well… My server and test devices are still running. 🙂
Server bully test
Well, not exactly crashed the server, but I stopped it from running which is a proof of concept.
With a simple Java program which sends legitimate requests from a legitimate device, the server failed after a few minutes. A quick manual restart over ssh got it up and running immediately.
public class BullyServerTest {
public static BullyServerTest bst;
public static void main(String[] args) {
bst = new BullyServerTest();
}
boolean quit = false;
private final String server_host = "SOMEWEBADDRESS"; // LOL
private final int server_port = SOMESECRETNUMBER;
private final String home = System.getProperty("user.home") + "/";
public BullyServerTest() {
Runtime.getRuntime().addShutdownHook(new Thread() {
public void run() {
quit = true;
try { Thread.sleep(1000); } catch (InterruptedException ex) {}
}
});
while (!quit) {
try { Thread.sleep(randInt(10, 250)); } catch (InterruptedException ex) {}
do_hello_world_bully();
}
}
Random rand = new Random();
private int randInt(int min, int max) {
return rand.nextInt((max - min) + 1) + min;
}
private static final int COMMAND_HELLO = 1234;
private void do_hello_world_bully() {
int count = randInt(10, 100);
System.out.println("HELLO WORLD BULLY at " + count + " times");
while (count-- != 0) { // a shed load of threads at once
new Thread(runnable_HW).start();
}
System.out.println("Completed...");
}
Runnable runnable_HW = new Runnable() {
@Override
public void run() {
try {
WLSocket socket = new WLSocket(server_host, server_port);
socket.out.writeInt(COMMAND_HELLO);
String hello = socket.in.readUTF();
} catch (IOException ex) {
System.out.println("Server failed with HELLO WORLD response");
}
}
};
}
So the next step is to implement a thread pool so that the server will not hang on too many requests. It ran for less than a minute. The above program kept running until my 12Gb RAM ran out which was a bit longer. The current server does only have 4Gb RAM and couldn’t handle the amount of connections at once. I’ve also had to keep the sudden bandwidth usage down as I was hitting my upload and download limits.
Eventually the server and client code will all be rewritten in C++ and handle such things as low latency timeouts.
This has been a very interesting test. I’ll be leaving the hello world server command in for future testing to simulate fake requests.