Elsewhere there are things that we all miss, yet it takes just one to notice...

Author: csnorwood

‘Project’ updates

Maybe things have plodded along at a gentle pace recently, but that’s also a good thing because it give me a chance to reflect back and get things back into perspective. Which I have done.

A few weeks ago I was ready to plough ahead with this. And then as I get stopped in my tracks, or more so, slowed down, I start to find a few more faults, cracks, bugs, etc. All fixed now.

Anyway, the server has now moved onto a VPS somewhere in the cloud which means better bandwidth speed than running from home. I’m still testing out the VPS and getting myself familiar with using it. Everything is via terminal so my previous Linux knowledge is becoming extremely useful here. Commands such as ‘rsync’ to copy files from my PC to the server for example. Now I need to create a way of restarting the server on demand.

The Java TCP server still has its minor issues, but it just works. Any problems I face I can just restart it and everything just carries on. Once it is securely setup on the VPS then I can leave it alone and work on the C++ server upgrade coding.

I won’t talk about the many other things that have happened around this project in the recent weeks apart from to say that it will be big. Fair enough, now I’m starting to feel the pressure is on me to come up with the C++ server code. Which I might actually continue with tomorrow after I figure out how to get my server software to automatically start on the VPS.

Everything will come to light, … in time.

“Follow your passion, not your paycheck. The money will come eventually.”

Just some random updates

The ‘project’ has slowed a bit because of content and artwork which I will be jumping on over these next few days.

Server updates are now being reconsidered because I read last night that non-blocking IO can also be done in Java. I can quickly setup stuff in Java than I can in C++ which means Java has that slight advantage being a language. It may end up being 10-15% slower overall, but it just does stuff. And development is much faster.

However, saying that, I’ve still to test out the non-blocking IO in Java. So time will tell on that score.

If it works then the previous devices commands will be ported over so they will carry on running. The current server is running Java so why not.

If it doesn’t, then it will be back to the fd_set and select() in C which was coming on very nicely until the distraction.

Video editing has been done mainly with Blender. Recently though, I tried Adobe Premiere CS2 and at first glance it looked like it could really do the job. First off, it doesn’t import 99% of video clips and other file formats. I think the only two things that it has going for it is the titling and the rendering of the work area for faster previewing.

Now, Openshot, may not have the best interface, but it just works. Titling is a pain using an external editor. And it also renders the video in memory as you’re going along for faster playback.

With using Openshot, I can create 95% of what I need it for. All titling can be done externally which can be a pain, but more than doable. I just drag n drop the exported PNG’s. I only need to figure out a faster way of setting key frames for animations.

My head is exploding at the moment. I’ve got the ‘project’ to think about, the ‘projects’ server updates, learning new API’s, protocols, etc, blah, as well as trying to do this all in ‘my’ time.

My time now is extremely valuable to me. I’m spending most of it on working on my ‘project’. There are people that know about it, and others I won’t tell because those that know would gladly give up their ‘time’ for me.

This project is actually going to be a huge thing for me. There are only a few involved in it. I was going to involve another but fate stepped in perfectly because of this ‘time’ issue. So it’s carrying on as it was meant to. It is a lot of mental work but it’s will be worth it.

IPv6 server upgrades

A few weeks back I started on the upgrade from the Java server to a C++ server.

I scrapped the first project and started again when I learnt about the handling of fd_set which made things so much better. First tests were great.

Then I scrapped it and decided that if it going worldwide with this secret project, then it needs and uprade. To IPv6.

Which works on my initial tests.

Now all I need it to finish off a few more DNS tests and then I’m ready to make the server ultra fast.

TCP C++ server upgrades

A quick snippet from tonight’s diary entry:

20:30 – I’m still reading up on some stuff about non-blocking TCP server programming and so far the best examples come from http://developerweb.net/viewtopic.php?id=2933. Ideally what I want is NOT to have a shed load of threads running, which I’ll be using a thread manager for, but a way of quickly determining if a connection is valid before shutting it down.

By the looks of the examples,  a continuous loop could be checking upto 1024 connections at any time before sending them off to another thread. This method has caught my attention big time because I should be able to setup some kind of timeout very quickly to close the connection if it doesn’t respond to the initial connection request. And I can still serve 100 to 1000 devices without any problems at all.

The same single thread can be constantly checking for connections and readability. It’s the readability part that matters the most here because it will be timed very precisely. Once a connection passes the readability test by passing the initial CSPRNG test, then it can go to a thread handler. Otherwise it gets closed. All of this a single core/thread can easily handle.

I will be studying this a lot further as this looks to be the way forward for a tcp server.

And this is just from my crappy laptop at home.

Basically, the new software upgrades to the server will be 1000 times more efficient than the current Java version. Although once in the cloud, the Java version can easily serve over 500 devices because of the bandwidth increase.

The updated C++ version will increase that significantly as well as handle dodgy connections better. Once a connection has passed the CSPRNG test then the TIMEOUT increases a smidgen.

I’ve already got the CSPRNG functions running and tested and written in assembler code to reduce that bottleneck too.

Until next time… WLGfx…

MS Access Databases

They’re really just as simple as using */UNIX…

Huh…

I’ve been faffing about with my database for most of the night and after relearning a few things, I deleted all my queries and forms and started again.

The relationships screen is where I now layout everything and then I decide to add tables from there well before I start designing the forms.

What I had before would have worked, but I wanted this to be perfect. And that’s why I deleted all my previous work. Using the relationships between Tables and Queries really does make life easier when building up forms and reports.

So far though, with my own project, I’ve not had the need for any VBA unlike at work. The experiences from my own project and the works one too will benefit both projects. I’m hoping that when I decide on changes for the works project, it won’t be too big. Luckily enough, both of them are at the moment small databases.

Just with tonight’s ‘messing’ around, I’ve actually got back to fully understanding the relational side of linking tables and queries. And how it makes creating forms much easier.

It’s only been about two weeks now I think getting back into MS Access, there about.

Ah well… My server and test devices are still running. 🙂

Server bully test

Well, not exactly crashed the server, but I stopped it from running which is a proof of concept.

With a simple Java program which sends legitimate requests from a legitimate device, the server failed after a few minutes. A quick manual restart over ssh got it up and running immediately.

public class BullyServerTest {
    
    public static BullyServerTest bst;

    public static void main(String[] args) {
        bst = new BullyServerTest();
    }
    
    boolean quit = false;
    
    private final String server_host = "SOMEWEBADDRESS"; // LOL
    private final int server_port = SOMESECRETNUMBER;
    
    private final String home = System.getProperty("user.home") + "/";
    
    public BullyServerTest() {
        Runtime.getRuntime().addShutdownHook(new Thread() {
            public void run() {
                quit = true;
                try { Thread.sleep(1000); } catch (InterruptedException ex) {}
            }
        });
        
        while (!quit) {
            try { Thread.sleep(randInt(10, 250)); } catch (InterruptedException ex) {}
            do_hello_world_bully();
        }
    }
    
    Random rand = new Random();
    
    private int randInt(int min, int max) {
        return rand.nextInt((max - min) + 1) + min;
    }
    
    private static final int COMMAND_HELLO = 1234;
    
    private void do_hello_world_bully() {
        int count = randInt(10, 100);
        System.out.println("HELLO WORLD BULLY at " + count + " times");
        
        while (count-- != 0) { // a shed load of threads at once
            new Thread(runnable_HW).start();
        }
        
        System.out.println("Completed...");
    }
    
    Runnable runnable_HW = new Runnable() {
        @Override
        public void run() {
            try {
                WLSocket socket = new WLSocket(server_host, server_port);

                socket.out.writeInt(COMMAND_HELLO);
                String hello = socket.in.readUTF();
                
            } catch (IOException ex) {
                System.out.println("Server failed with HELLO WORLD response");
            }
        }
    };
}

So the next step is to implement a thread pool so that the server will not hang on too many requests. It ran for less than a minute. The above program kept running until my 12Gb RAM ran out which was a bit longer. The current server does only have 4Gb RAM and couldn’t handle the amount of connections at once. I’ve also had to keep the sudden bandwidth usage down as I was hitting my upload and download limits.

Eventually the server and client code will all be rewritten in C++ and handle such things as low latency timeouts.

This has been a very interesting test. I’ll be leaving the hello world server command in for future testing to simulate fake requests.

Project updates

I’ve had the server running for over 3 weeks so far, and although it’s only a few testing devices, it’s looking very promising. The server hasn’t failed once and I’ve even ssh’d into it and run a couple of checks.

My next step, whilst I’m taking the time to build the database software, is to setup another server. More servers means backups and 100% running and up time. The servers can also update each other depending on the regions requirements. That means if I upload content that belongs on other servers, then the servers will organise themselves automatically. A bit like a decentralised distribution of data.

Devices will also receive updates when new servers have been added as periodically they will question what servers are available to them. This will help with manual load balancing as more devices are added. Although were I want to go with this will not need manual load balancing, it is something I want to test out on test servers anyway.

Although all of this is stuck in my head at the moment, I’ve had a few quick browses through my code base and it this seems pretty simply to implement and upgrade without any interruptions to the servers current up time. This is definitely something I want to fully test out. Even down to the servers possibly updating themselves when one has been updated. I will look into this as I’m more leaning towards manually accepting server updates for security.

After long thinking over the weekend I was building myself up to putting this project forward to someone as a partnership proposal. I got home from work today and decided that’s a no. For the time being, this is staying mine.

Updates 2018-Feb-21

My project:

I’ve been taking my time on this project for a damn good reason, I don’t want to rush things and end up making some serious mistakes with it.

There are a few, quite a few bits of code, that will need a rewrite, mainly in C++. This is mainly for future proofing.

I’m currently on stage 2 of this which involves 3 main sections.

The first section involves the database which I am currently working on in MS Access. I used to be an Access master, so I’m getting my hands dirty and learning it all over again. For the past few months I’ve looked for anything that came close to what MS Access can actually do and I’ve found nothing. MS Access is everything for creating a database application from scratch. VBA might be a tad awkward, but it is doing everything I need it for so far. I hope in the future it can also do TCP communications.

The second and third sections of this stage 2 are a lot more involved in prepping for the future. The second being done this next week if all goes well.

The final third of stage 2 will involve the website. I’m hoping that it will be handled by someone I know who is fully capable. This will include handling accounts and payments too. And I’ve been told that it can also handle invoicing.

After that, it’s is just going with the flow. I’m being very weary of contacts because I’ve been bitten before.

Me:

I’ve still got a good few months before the above starts to take shape, although some early testing will be done sooner than expected. Apart from that I’m doing okay with myself apart from slacking with my exercising.

This month has been hard for moving things forward because I lost a lost of money this month. I still get annoyed over that. I could do with a few more devices to put out into the field. And with the lack of testing, I’ve also held back on setting up the cloud server.

Programming has been ‘nicely’ minimal. I’ve mainly been running all my software and fixing bugs. I have been able to chill. Well ‘ish’…

And I’ll get back to a regular weight lifting routine again.

 

Just a few updates

Well, that’s January over. And it seemed to have shot by really quick too.

The first stage of my personal project is complete and has been running 100% for quite a while now. It’s now just testing out in the field that needs doing.

I’ve got everything running through a DNS server that routes back to my home PC, so no matter where devices are running they can easily connect to my home PC for communications over TCP. Eventually this will be running on a cloud service which I’ve been looking into lately, whether that be Google or AWS or something else, I don’t know yet.

All network traffic runs on two way encryption which was actually very easy to implement. At the moment it is not a very complex encryption because I do want to test for speed overheads with this. If I’m handling 1000 plus devices, ‘at the same time’, then I really don’t want the CPU burning the house down. I’ve a few more cryptographic algorithms I want to test out too, but they can wait for now.

Video creation has been non-existent since before Christmas because I’ve been focused on the programming of this project. Over the last two days I’ve loaded up Blender and although I can move things around easy enough, the old knack has gone again. I really like Blender, but when it comes to finishing something off that requires programming, then Blender has to wait just like most other things.

One thing I do need to start focusing on, and I started looking into this before Christmas, was a full accounting database. The only one from the past that I’m familiar with is MS Access, and trying to find one that works on Linux is still a pain. I’m planning on using Linux so MS will not do. So, when it comes to storing all financial records, invoicing, etc, it looks like I’ll have to write my own software. Shame really, I do like the awesomeness of MS Access. (Which I started using at work earlier today)

So, the next stage is to finalise the tests out in the field. To do this, I will take my demo devices to various places and see it running just nicely. The management software will also be tested out in the field too.

Once that stage has been proven successful over time, then it’s the setting up of the rest of the management of the project. This will include accounts, security, moderation, sustainability tests, reviews on scale-ability, moving to an initial cloud server, and more.

Up until this is a showcase-able project then for the most parts, apart from what I write about, this will be under wraps because I want this to be mine with the help of very few and limited resources. Trusting is hard to do when you lived through enough of life to have experienced people take away your ideas in a wink.

Until next time.

A weekend is over. Updates.

Where is the day between Sunday and Monday?

I needed a little more time this weekend to continue building my project up. The manager is nearing completion with only the transfers of media to devices needing doing now. Then it is getting the devices to simply run on their own.

Distractions at home is what mainly slows my pace down. Noise and stuff.

Ah well.

I only get at most an hour and half after work if that during the week so the first thing I will be doing is adding a shed load of comments to my code.

This last section, the media manager was quite easy to get up and running because the device manager had already been done.

I know that in the future it will eventually get a complete rewrite yet again, but at least this version will handle up to 1000 devices. That’s a low estimate. Once it goes live, I’ll update everything so that it can scale up to anything.

The devices themselves will get a full run by the end of the week. That I am looking forward to seeing. All this work finally being showcased.

Posts navigation

1 2 3 4 5 6 … 11 12 13
Scroll to top