Java 8 running out of file handles on Linux

When I checked one of my websites this morning I discovered it had stopped responding. All it did was sit there then time out with a standard Tomcat error page. So I logged in to check the logs and found that tomcat was complaining about “Too many open files”. Now a few days earlier I had released a new version of the site which moved the serving of static content from Apache to Tomcat so that was my initial though of where the problem was.

This turns out to be true. Where I was reading a text file I was using the new Java 8 Files.line() method which returns a Stream consisting of each line in a file. Now this was nice as instead of writing a block of code to read a file into a string we could reduce it down to:

String page = Files.line( f.getPath() ).
    collect( Collectors.joining( "\n" ) ) );

The problem here is that although nice and concise, it never closes the file. So after a while the Operating System complains that too many files are open and tomcat grinds to a halt.

Now hidden away in the javadocs you’ll find tha Stream actually implements AutoClosable & the reason why is that you can close the stream once you have done with it. Now in 99% of all Streams you don’t need to do anything but if a Stream is operating against some external resource like a File then you are supposed to be closing the stream afterwards.

Now this is where I went wrong, and I suspect a lot of others would also fall foul of this as every example of using Java 8 streams do not show closing them – so who would know that they need to?

So, the correct way of reading a file using streams is this:

try( Stream lines = Files.line( path ) ) {
    // do something with the stream
}

For example, in my case it ended up looking something like this:

String page;
try( Stream lines = Files.line( f.getPath() ) ) {
    page = lines.collect( Collectors.joining( "\n" ) ) );
}

So the lesson here is make certain when using an external resource within a Stream then ensure you close it afterwards.

Tagged , , ,

Connecting an Arduino to a Raspberry PI using I2C

Some time ago I created a weather station using a Raspberry PI and an off the shelf weather station, connecting the two via USB.

However, for some time not I’ve been meaning to create a weather station from scratch – i.e. one or more Raspberry PI’s which connect to the network (via Ethernet or WiFi) and directly monitor the sensors directly.

Now the problem here is that some sensors are analog – for example the leaf, soil and UV sensors I have generate an analog signal so we need an ADC (Analogue to Digital Converter) which the Raspberry PI doesn’t have.

So we have two possible solutions:

  1. Add a Raspberry PI compatible ADC
  2. Use an Arduino

With the parts I have available, the Arduino won, not just on available ADC channels but also with the additional digital ports available.

Now how to connect it to the PI? Well the easiest way is to use USB, however the PI only has two USB ports (one for the Model A) and as I’m intending to use Model A’s for the final station I need that for WiFi (there won’t be room or power for hubs) so USB is out.

There’s RS232 which both support, however the PI runs on 3v3 whilst the Arduino (UNO) is 5v so I need to add a level converter between the two. It also limits me to just one arduino and I might need to use more than one so another solution is needed.

Continue reading

Getting IPv6 working on the UK Mobile Network

One problem with mobile data here in the UK is that everything is NATed to death. Most consumer mobile data connections do not support static IP‘s and those that do are expensive. Also, the UK is lagging behind most of the world in providing access to the IPv6 internet. It’s not that IPv6 is new either, it’s been around for 10 years but no, here in the UK they want to try not fork out the cash to replace older kit that can only handle IPv4 & it’s not exactly rocket science either.

Ok, my home network is IPv6 enabled. I have a /64 subnet routed to home out of my /48 allocation at my ISP (they support IPv6 natively). If you don’t know what this is don’t worry – l this means is that on the IPv6 internet I actually have more static IP’s on my home network than the old legacy internet put together. In fact the legacy has about 4 billion & I have 1.8*1019. Now you see the main selling point of IPv6 – there’s room for every single device that’s networkable to have just one IP address and be accessible from anywhere – obviously behind a firewall.

Anyhow, the problem I had to solve was this: I’ve got a fair few machines on my network and at times I need to be able to ssh into them remotely. Currently I can do this by either ssh into my firewall by it’s IPv4 address and then onto the internal machine or I could setup a VPN – but why should I when I’m fully IPv6 enabled?

Setting up a Tunnel Broker

Well there is a way. When native IPv6 isn’t available, one option is to use a tunnel broker. A broker sets up a tunnel between your device and an endpoint at the broker. IPv6 is then encapsulated in an IPv4 packet, sent through the tunnel to the broker, then sent out from there as IPv6. Now there are plenty out there like Hurricane Electric and SixXS but as I need to use this on a 3G device they won’t work as they require a static IPv4 address and we don’t have that – we’re behind a NAT, so the only available option is Gogo6 (which owns Freenet6). They are also a broker but they support NAT traversal which is what we need.

So, on my Linux Mint laptop it’s a simple case of installing the Gogo6 client:

sudo apt-get install gogoc
sudo /etc/init.d/gogoc stop
sudo /etc/rc5.d/S20gogoc

Now you might wonder why we stop then delete a file after installation. This is because when it installs it starts the service and we don’t want it running just yet. Also the rc5.d file means it starts on boot which we don’t want – we want to use this on 3G remember.

As it stands that’s all there is to do – by default it’s configured to use an anonymous account so the next time you’re on 3G you simply:

sudo /etc/init.d/gogoc start

and you’ll find you are now on the IPv6 internet. When you go offline just stop gogoc:

sudo /etc/init.d/gogoc stop

Getting a more permanent static IPv6 address

With an anonymous connection you’ll get an IP address out of a pool but if you want a static address you’ll need to register an account and edit /etc/gogoc/gogoc.conf

In that file:

  1. edit the lines with userid= and passwd= with you’re account’s username and password.
  2. The line server= needs to be their endpoint. Here you register against a specific one, so as I used amsterdam set this to amsterdam.freenet6.net
  3. Finally change the auth_method= line from anonymous to one of the other methods listed just above that line.

Now that last step might take some work to get working. any should always work but it risks sending your password in the clear but you might want to play with that later.

That’s it. When you start gogoc you’ll get a new IP address which will be permanent. You’ll also get a dns entry setup as well, username.broker.freenet6.net so now you can get into your laptop.

Tunneling an entire network

There is a final option available but out of scope here, and thats connecting an entire network to the tunnel. Thats simply a case of changing the host_type= line from host to router. Then you’re local network will get an IPv6 address with your laptop as the router.

How well does it work

Well I’ve tested it on T-Mobile UK and it works pretty well. I can access my home servers directly and as I use the non-anonymous option I can actually ssh from home to the laptop via it’s 3G connection.

I’ve even tried setting up a proxy on an Apache server which is accessible from the legacy IPv4 internet and it connects to the laptop’s Apache server fine – although sluggish but remember this is over 3G.

At some point I’ll try it on other operators (I also use GiffGaff & 3).

Next I need to figure out how to get this working on Android so that I can get my Nexus 4 & Nexus 7 3G online – both supports IPv6 when on the WiFi at home, just would be nice when out and about.

 

Follow

Get every new post delivered to your Inbox.

Join 1,949 other followers