AIX git SSL woes

Oh joy and happiness I have to admin AIX boxes. One of the first things I hit was using git to clone some stuff from github erroring out with:

SSL certificate problem: unable to get local issuer certificate

Yep, simple problem, no ssl ca bundle on the system. You can use either the bulldozer solution and have either:

export GIT_SSL_NO_VERIFY=true

either:

git config http.sslVerify false  ( git config --unset http.sslVerify )

because who cares about MITM attacks especially to deployed software on production servers

Or you can go to actually fix the issue and install a ca bundle. I downloaded mine from the curl site, here: https://curl.haxx.se/docs/caextract.html

I downloaded the cacert.pem file and configured git to use it like this:

wget --no-check-certificate  https://curl.haxx.se/ca/cacert.pem -O /var/ssl/cacert.pem
git config --system  http.sslcainfo /var/ssl/cacert.pem

The no-check-certificate is required because at this point wget has no way of checking the certificate either. If you want to ensure the validity of the file download it from a working system and scp it to the remote problem server.

Use Vagrant on Windows with Ansible under Cygwin

If due to some reason you have to run vagrant under windows and plan on using Ansible you will need a couple of wrappers.

ansible-playbook.bat has to be in the Windows PATH;

ansible-winpath-playbook.sh is called by ansible-playbook.bat to change paths from Windows style paths to *nix style paths that ansible under cygwin can understand.

Creating new zfs pool mixing previously used drives

I am building a new homelab for myself using Intel NUCs. I need a NAS so I can test live migration and I was set to buy 2 bay synology. But when a 140$ HP Gen8 Microserver was on offer I couldn’t say no. FreeNAS might not be as polished as Synology but I would be getting 4 bays, two gigabit ports and iLO at a third of the price. Also it would offer the possibility of learning FreeNAS (my other, longtime NAS is slackware + zfs on linux).
 
Since this server is not going to hold any important data (I’ll be sure to backup anything important on my production NAS – i.e. the one I don’t mess with) I wanted to reuse three old 2TB drives left from upgrading my main NAS to 3TB reds.
 
When creating the volume in FreeNAS I would be getting the following error:
Creating GPT partition scheme on ada2gpart: geom ‘ada2’: File exists
 
I imediately realized that this is because some of the disks used to be in a zfs pool in the past. The suggestion was to clear the begging of the disk but that didn’t help. The only fixe from the ones proposed here was:
# destroy ALL DATA on disk /dev/ada2
sysctl kern.geom.debugflags=0x10
dd if=/dev/zero of=/dev/ada2 bs=1m
# now reboot!
I don’t remember if I zeroed the entire 2TB disk or just left it fot ~10 mins to ensure most of the zfs metadata is gone.
 
After running the above in the command line I was able to create the volume just fine.

Verify SMART details for members of an Intel RST RAID volume

Sooo,

Be it because of the BIOS update to a beta or because of my drives but my RAID10 keeps failing. I documented before how to repair such a broken array but I didn’t want to go ahead with it too many times as data corruption is only one step away. Knowing that at least one of the disks has some minor issues (mdadm kicked it out some time ago when the disks were running under linux) I decided to check smart details and only keep only two of the disks in RAID1. I was curious if one can read SMART details when the disks are still members of the Intel RST array. Since I had all the data off the disks it was safe to test.

I found out thet the Intel SSD Toolbox shows SMART data for all disks in a system, not only SSDs and not only Intel. Look at Other Drives and scroll to the right as under Intel Solid-State Drives it shows the RAID volumes.

Intel RST RAID Non-RAID Disk after BIOS update

So, having nothing better to do and for no good reason I decided to update my workstation’s BIOS to the latest version released by Gigabyte. Since ignoring the “If it works don’t fix it” mantra is always a good idea. Beautiful, after update two of my disks from a four disk RAID10 array were showing as Non-RAID Disk. I had backups but shuffling 2TB+ of data is never fun.

Initial reports were all grim, the Intel RST BIOS does not allow repairing. Thankfully a good soul had always found the answer, source thread here thank-you adamsap.

Usual disclaimer: this worked for me, I have no guarantee it will work for you, and the method is not advertised as working and/or suported by Intel

  1. Reset the volume (all disks) as non-member from the Intel BIOS. Ignore the warning that all data will be lost. The utility only touches the metadata related to RAID membership.
  2. Create a new array with the all same disks and be sure to use the same settings related to strip size, RAID type, etc. I was in luck since my array was still visible since some disks still were attached.
  3. Download TestDisk from http://www.cgsecurity.org. I used the Windows version since my Windows install was on a different disk. I never heard of this utility but seems to be really, really useful at data recovery.
  4. Run TestDisk after reading the steps on their site. Be sure to read the documentation there to know what you are doing. In brief (so I’m sure you read the original docs) you have to: search for your partition(s) on the raid volume – if everything was recreated with the same settings it should find it quickly in a few seconds – and save the partition table.
  5. After the partition table is saved reboot.
  6. The array should be back with all the data.

I compared checksums for some of the data against backups and it turns out everything is back.

pfSense home router using the PC Engines APU1D4

My old home router based on a sandy bridge dual core celeron and a gigabyte motherboard got “stolen” by my wife to use as a desktop as her old laptop was getting pretty slow.

I ran for a while Tomato on a Cisco E3200 router but it wasn’t able to keep up with my home connection (300 down / 100 up). Even if the router has gigabit ports it was only able to nat at ~100-150Mbit and openvpn was limited to around 10Mbit

The decision came (due to what is available in my part of the world) to the Fitlet X or the PC Engines APU1D4.

The fitlet has 4 intel LAN ports and a quad core AMD 1GHz cpu (two generations newer than the APU). This was really apealing oposed the APU’s dual core bobcat and 3 realtek based NICs.

Eventually I settled with the APU due to the two internal mini-pcie slots and being only half the cost of the Fitlet. (Consider that you have to buy RAM for the fitlet-x and that you don’t have any internal mini-pcie left, the only one is used by the FACET card with the 3 LANs)

I won’t go into detail about the build or do a full review as this has already been made. I will only go trough the bits of information I had trouble finding before and after buying it.

  • Throughput: without heavy use (squid, snort, etc.) you should see 400-500 Mbit WAN->LAN (limited by the realtek NICs). I know Mbit is not a good measure of a router/firewall performance but this is what matters to me at home. I saw mentions of 600 Mbit. I was eager to deploy it so I didn’t do any testing so all I can say is that 300Mbit works fine without any strain.
  • OpenVPN: it does around 50Mbit for me using AES-CBC-128. This was really a tough one as I didn’t find any useful values before buying and it was important for me. It’s a bit disappointing but very usable. The Bobcat T40E doesn’t support AES-NI but as far as I found out from others it’s not really helping OpenVPN either. There is low hope that newer versions of OpenVPN will perform better. The Fitlet-X CPU should be 15% faster due to IPC gains on it’s newer core so you should see a bit more.
  • Temperature: it appears that ~60 deg. C in idle is normal. Coming from Intel CPUs this worried me at first but seems normal for this CPU
  • Wireless: if you go the pfSense route as I did get the Compex WLE200NX usually sold together with the APU. It’s atheros (best for pfSense) and what most pfSense developers using the APU have.
  • SSD: don’t buy the 16GB crap SSD that is offered together with this board. Get a cheap ADATA/whatever instead. It’s probably going to be 32GB and at least twice as fast
  • Case: important due to the solution PC Engines chose for cooling. Note that this case doesn’t offer space for a 2.5 SSD/HDD (even if one SATA port is onboard), additional USBs (even if headers are present) or for a second set of antennas (even if two mini-pcie devices can be installed)

Adding a form with autocalculated fields in squarespace

I had to build a form on a squarespace hosted site that has to auto-calculate a couple of fields and also do field validation and offer the option to email the whole calculation to a visitor chosen address. I won’t publish the complete script here, but it shold help get you going.

First of all you will have to build this in the HTML/CSS block, you won’t be able to use the squarespace offered form builder.

First you will need a few javascript functions

[cce_html]
function getField1() {
var theForm = document.forms[“myform”];
var field1Value = theForm.elements[“field1”];
return parseInt(field1Value.value);
}
[/cce_html]

What this function does is return the value of a field named “field1” in your “myform” form. parseFloat might be required if you are going to use floating point values.

[cce_html]
function calculateTotal() {
var outputfield1Value = getField1()*5;

var elem = document.getElementById(“outputfield1Id”);
elem.value = Math.round(outputfield1Value);

}
[/cce_html]

This function is called each time one of your form fields is modified. What it does is change the value of the read-only form field to field1*5. This is of course just an example. You can multiply this to as many form input and output fields you need. Please note that you cannot use “disabled” fields as those cannot be POSTed.

[cce_html]
function validateForm() {
x = document.forms[“myform”][“field1”].value;
if (x == null || x == “”) {
alert(“Please fill in the first field”);
return false;
}
var x = document.forms[“myform”][“emailaddress”].value;
var re = /^([\w-]+(?:\.[\w-]+)*)@((?:[\w-]+\.)*\w[\w-]{0,66})\.([a-z]{2,6}(?:\.[a-z]{2})?)$/i;
if (x == null || x== “” || !re.test(x)) {
alert(“Please enter a valid email address”);
return false;
}
[/cce_html]

A simple function to do field validation. This one simply checks that the fields were actually filled and that the email field is valid (check RFCs, this is not really the best version but it works for me). Please note that this validation will not stop an attacker or bot to fill bogous data in your fields (as they can POST anything without going trough this javascript)

[cce_html]

Autocalculated Squarespace Form

Email a copy to yourself:

[/cce_html]

The first four div’s are there to make the form look like other Squarespace built forms. Your form will call the validation function and onchange the first field will trigger the update of the second. When working with the new “number” input field pay close attention to the step value. The default is 1 which means Chrome (Firefox ignores it I think) will not allow floating point values unless you set the step to be any (or say 0.1, if you require specific values).

I will leave the emailself.php script as an exercise to the reader. It’s really easy to screw it up though, so be sure to secure it – both the form, maybe with google a captcha and your php script that takes POSTed data and emails it as it’s really easy to miss something and open your self to becoming a SPAM sender.

Brightness keys on Linux Mint for Acer E3-111

Enabling brightness control is moderately easy (I’m pretty sure they worked out of the box on slackware).

Go to https://github.com/codingtony/acer-brightness-linux-acpi and follow the instructions. Change /etc/acpi/events/acer-tm-brightness-down and /etc/acpi/events/acer-tm-brightness-up with the correct events:

event=video/brightnessdown BRTDN 00000087 00000000

and

event=video/brightnessup BRTUP 00000086 00000000

Restart acpid and the shortcuts should start working.

Speed-up WAN-LAN speed on the Cisco E3200 using Tomato

There’s a really long discussion on bcm_nat and fastnat on the various forums like linksysinfo. The short version of this is:

  1. fastnat and bcm_nat are disabled by default because they break QOS and access restrictions. If you use any of those you’re done, you have to choose features or speed or another router.
  2. If you, like me, don’t use those you can do the following:
    ssh into your router and try: [cci_bash]modprobe bcm_nat[/cci_bash] than speedtest your connection a few times you should see some improvement. Than run [cci_bash]echo “1”> /proc/sys/net/ipv4/netfilter/ip_conntrack_fastnat[/cci_bash] you shold see an even better improvement. If any of those give you issues simply reboot your router, nothing is permanent at this time.
  3. Add the commands to Administration->Scripts->Init.

The instructions are deliberately scarce, if you don’t understand them it’s better not to mess with your router. Check the tomato forums and the linux documentation about those commands until you’re sure you know what they are doing.

Subdomains pointing to private IPs are not resolved on Tomato

Some time ago I used to remember IPs on my home and work network but these days I rely much more on dns and dhcp reservation for this tasks. This has the advantage that I can easy move a service, say git.example.com to a new server.

At home I switched (at least for a while) from a dedicated Debian router/gateway box to a router running Tomato. Suddenly subdomains pointing to private IP addresses were no longer resolved. Turns out the DNS rebinding protection is at fault. Now you could easily just disable it but this is not the secure way to fix a problem. You can actually white list domains to allow private IPs on subdomains.

Go to Advanced => DHCP / DNS Server (LAN)

Don’t uncheck “Prevent DNS-rebind attacks” as this will leave you vulnerable to this attack. Instead add the following to the Dnsmasq
Custom configuration

[cci_bash]rebind-domain-ok=/domain1.com/domain2.com/[/cci_bash]

Where domain1.com, domain2.com, etc. are the domains for which you want to allow subdomains that resolve to private IPs.