Search This Blog

Wednesday, 6 August 2014

Cybersecurity, children's use of social media, what's actually dangerous, and the research of Professor Donna Cross

Professor Donna Cross researches children's well-being, including cyber-safety. I had the privilege of hearing her give a quick summary of her research last night.

Unlike almost every other age group, the health of adolescents and children hasn't improved in recent decades. There are several dangers to children, which are of far greater significance than cyber-safety.
  • Children are still getting sunburnt regularly, with the attendant risk of skin cancer.
  • Children are becoming obese at a terrifying rate, and not getting enough exercise. This has short-term and long-term health implications.
  • Children can't cross roads safely. Particularly children under 10, who are cognitively challenged:
    • They expect moving objects to change shape. A horse running changes shape. A soccer ball spins. Cars don't, so they have a cognitive block that makes it harder for them to perceive that the car is moving.
    • Audio-location (identifying where a sound comes from) doesn't really come together until after age 10. Between not being able to hear or see that a car is moving, there aren't many ways you can sense the danger of a distant car.
    • Very young children assume that if they can see the driver, the driver can see them.
  • Children are spending less time engaging with their peers face-to-face, so they are less able to understand subtle facial expressions. This makes them less able to build and maintain friendships. In addition, they are talking to adults less.
It turns out that this last risk (which sounds unimportant) is actually quite significant. One of the best predictors of school and personal achievement turns out to be the ability to engage socially with others. Children need to have a strong social network in order to thrive. It even affects the strength of their immune response to infection. Against this, cyber-security seems like a very minor concern. But since I can't really say anything useful about physical safety or paediatric development, I'll focus on the cyber issues anyway.

Professor Cross mentioned the well-known decline in Facebook usage among young teens. Facebook is how middle-aged women reconnect with their school peers now. Instead, children are using Tumblr, Twitter, Instagram and Reddit. She spoke about much else, but as someone who can talk for hours about the impact of computing on society, this was something that grabbed my attention. The "adult" social media platforms (such as LinkedIn, Quora and Facebook) are very strong on mapping user accounts to individuals. These are sites where you build reputation and invest time and effort to show your significance to your peers. These are not the sites that young teens are involved in. Instead they are flocking to sites where the culture is that the matching of your username to a real person is a matter of some shame, embarrassment or at least disaster. 

There are throwaway accounts on Facebook, but on reddit (for example) there are cultural norms about using them.

  • Creating a throwaway to gripe about your school teacher is the safest thing to do; don't use your normal account. 
  • Don't create a throwaway to vote up your post, though, that's inappropriate; use your normal account instead. 
  • Similarly to adults on ebay, don't get too attached to your reputation on any of these sites because you might need to discard it quickly.
Most children won't understand this initially. Some pick it up intuitively immediately: my daughter has more than one instagram account, and switches depending on what she's trying to do. But most children will need it explained. We have pushed the message "have different passwords on different sites", but it is far more important to explain to you children "you need to have many nicknames, and be prepared to create new ones, and discard the old like you do with clothes". Children see the behaviours of celebrities on-line as "normal". Very few children realise that it is a polished image with many of their other interactions completely hidden from the public. We need to help our children cultivate personae, and to understand the limits of reasonable disinformation and obscurity. I think this is a reasonable laundry list of discussions every parent should have with their children:
  • Have you thought up a couple of different usernames? Are they in use by anyone else in the world? Are these usernames so different that no-one could guess that they belong to the same person?
  • For each username, what impression do you want it to portray? Is this the account you use for comedy? Or is this the account you will use for your deep, artistic content? Do you have an account you could use safely if you were confessing something bad you had done?
  • Which accounts will you let your friends know are yours, and which ones are you going to keep secret?
  • Do you know how to make a throwaway account? Do you know what a throwaway account is, and why it might be useful?
  • When will you use a throwaway?
  • On which sites and with which accounts will you put real information?
  • Is it appropriate to lie to create an account?
  • If you had to abandon this account, or if the account was taken over by someone maliciously, would you lose anything you cared about?
The challenge is of course, that most parents don't have the time to connect to Tumblr, Instagram, Twitter and Reddit. Not least because Reddit can be dangerously addictive, as I've discovered. And even then, many adults aren't use to thinking in terms of multiple accounts, having grown up with their (solitary) corporate Active Directory account and their (solitary) Facebook account.

But that's OK. As a parent, it's perfectly OK to fob off some of this kind of "geek" conversation to other more qualified adults. It's important for children to have relationships with adults. Boys in particular are very reluctant to talk their parents or teachers about any problems they have. Is there a cousin or family friend that spends too much time on the internet, who you would trust to explain this sort of thing to your children? Make the effort to make this conversation happen with them then. Conversely, if you are a child reading this post, and you don't really have an answer to the questions on the laundry list, it would be well worthwhile talking to someone about this. Perhaps your parents, perhaps a teacher, or someone else that your parents trust.

Here are the three books I can find on Amazon that she has written:

Greg Baker is an author, computer geek, inventor, consultant, and parent. He is not particularly good at any of these. (gregb@ifost.org.au)

Tuesday, 5 August 2014

VPNs

I have customers ranging from the very large to the very small all over the world and I'm seeing some rapid change in the way that I "remote into" customers. I'm also seeing product use that doesn't make any sense to me.

I've been thinking about this since I was posed this as a quora question: "How will the VPN industry evolve over the next 10 years?"


Here's what I wrote:



The majority of sales of VPN products today are for staff connecting into a company's private network. As companies hollow out their internal infrastructure and rely more on third-party cloud-delivered solutions, this will become less necessary. I would predict sales for end-point VPN solutions to drop off quite dramatically.
VPNs between networks on the other hand will become much more common. Companies may choose to spread their virtual servers across multiple cloud hosting providers. These will be joined by VPNs. However, as these will be configured by experienced technicians they are probably more likely to use the built-in capabilities of OpenBSD, Linux, Cisco or Huawei to do this.
So in summary: I would predict a major collapse of the VPN marketplace over the next 5 years.

For this post I'll ignore the part about network-to-network VPNs. Stepping back, what are we trying to do with host-to-network VPN? We're trying to connect to some resources at a different site. So here's my list of how I currently connect to my different customers, in decreasing order of common-ness:


  • A Cisco VPN service, which can be accessed through the built-in OS X VPN client. This seems to be the majority among large corporates.
  • A Cisco VPN service, which only works with the Cisco VPN client. I don't understand how this can happen, but I have customers where this is the case.
  • Juniper Secure Gateway. This seems to be the only one where two-factor authentication is used regularly, and it seems to be more government rather than commercial.
  • Fully cloudy: the customer has essentially no internal infrastructure, so I just log into whatever web-based application it is that they need my help with. This seems to be more common with smaller companies, because they are the ones that tend to be the early adopters for having a cloud.
  • A plain RDP server accessible over the internet. Usually this is behind a firewall, but the firewall port forwards to the RDP server. This used to be the standard for almost all small organisations, but I see fewer and fewer Windows servers in small organisations now. Now it's the mid-size organisations that are using RDP gateways.
  • No remote access whatsoever, or nothing I'm allowed to use. This is not that unusual.
  • Just log into the router and forward whatever ports you need. Generally this is for customers where I've helped them with their network infrastructure.
  • PPTP to a Microsoft server
  • TeamViewer and/or other connect-to-a-webservice-screensharing, such as Chrome Remote Desktop.
  • Guacamole which then proxies an RDP or VNC session.
  • Some other ipsec solution from someone other than Cisco or Juniper (usually something open source like openvpn)
  • Lastly, as the least common option: secure shell sometimes combined with mosh.

Reflecting on that list, it's quite odd. 

The one solution where the auditing for security is well known (secure shell) is the one least used.

The solutions that can be audited (because they are open source) take the bottom three places of least use.

Everything above the open source solutions relies on trusting the vendor completely. But the solution that is the easiest to set up among the trust-the-vendor-completely options (TeamViewer, Chrome Remote Desktop, and their ilk) is the least used.

The point of the exercise is almost always to support connecting to a Windows-based application. If it were a intranet-based application, then it will generally be possible to find some way of securing it to make it accessible to the internet. So the resources that people access remotely are almost always going to be supported by a team of Microsoft-trained admins. So why is it so rare to see RDP gateways, which would be the Microsoft solution?

The answer is of course, that collectively, the world appointed their networking teams to be in charge of remote access. The network admins did as they knew best and provided layer 2/3 access delivered by their favourite networking vendors. What we have works, and it isn't terribly insecure. But isn't it odd that that's how we do things?

Why were the world's CCNEs appointed guardians of internal corporate access in the first place, and not (say) the MSCEs in charge of directory services? Or the sysadmins in charge of the actual applications?


Greg Baker is an independent consultant who writes, programs, thinks and fixes things to do with computers, IT and all things technical for customers who don't want to pay for expensive consulting firms. Contact him (gregb@ifost.org.au) if you have challenging problems you need solved.

Wednesday, 30 July 2014

First look at Smart Cache

Data Protector 9.0 introduces a new kind of backup to disk device, imaginatively called "SmartCache". It is designed solely for VMware backups.

If you have backed up a virtual machine into a smart cache, and you happen to have the granular recovery extension installed, you can restore individual files from the VMDK very quickly. The end-to-end time to restore a file from a VMDK backup is not that different to restoring a file from a filesystem backup, actually. That is because instead of having to restore the whole VMDK file to a spooling location, the smart cache is essentially an already restored version, and it's just a matter of mounting it and browsing through it.

Starting in version 9, the granular recovery extension only works with VMware's web interface (the flash version). I spent a painful morning chasing up why the VMware Granular Recovery Extension comes up only in the "Available Plug-ins" section of the Vsphere Plug-in Manager with a status of "No client side download is needed for this plug-in". The answer is, it's like the VR Management plug-in: it only supports the web interface, not the vSphere client.

The smart cache device does no deduplication. VEAgent backups can't do in-line compression -- nor would it make sense here anyway -- so the amount of disk space required is exactly the same as the space required for the original VMDKs that you are backing up. Actually, it you want to keep a few days' worth, then you could well be looking at several times as much disk space required for your backups as for the originals. The client I'm doing this for has no Linux servers, otherwise I would have suggested putting the smart cache onto a BTRFS filesystem!

What this means is that you will probably pair the smart cache backup with an object copy job a few days later. Backup to the smart cache now, keep 2-3 copies in smart cache and then copy the oldest backup to a deduplication store where you can store it for a month or longer on disk, or possibly longer on tape.

Note that the default block size for the smart cache device is 1024Kb, which is larger than the defaults for (say) StoreOnce or most tape drives. This means that unless you change this (either by making the smart cache block size smaller, or the other device larger) that you won't be able to do any kind of object copy. Instead you will get errors like this:
[Major] No write device with suitable block size found.

If you don't have licenses for the VMware granular recovery extensions, then don't bother: just use a StoreOnce software device and save yourself a few TB of disk space.

Also, I would recommend bench-marking your VEAgent backups against a filesystem backup. You can create bootable disaster recovery images from filesystem backups (providing most of the advantages of a VMDK-level backup) and quite often the performance is comparable to (or sometimes better than) a VMware snapshot backup. Since filesystem backups can go to a StoreOnce store, it's much more space efficient than the combination of VEAgent + smart cache, and you can (obviously) restore individual files.

Greg Baker is an independent consultant who happens to do a lot of work on HP DataProtector. He is the author of the only published book on HP Data Protector (http://x.ifost.org.au/dp-book). He works with HP and HP partner companies to solve the hardest big-data problems (especially around backup). See more at IFOST's DataProtector pages at http://www.ifost.org.au/dataprotector

Monday, 28 July 2014

Moment-in-time snapshot backups of Linux systems with Data Protector

You can get moment-in-time snapshot backups of Linux systems with Data Protector.

Everybody knows the "Use Shadow Copy" flag for Windows backups gets something very close to a moment-in-time snapshot, but there's a persistent rumour that you can only get a good moment-in-time snapshot of Linux systems if you take a VMware snapshot backup.

If you are backing up a btrfs filesystem (which is supported on the Linux agent: you can see it listed in /opt/omni/.util), then you can use its built-in snapshot capabilities.

But even ext3 and xfs filesystems can do snapshots with the help of the LVM layer. This technique dates back to the early days of HP-UX, but it works on modern Linux boxes (and probably on modern HP-UX boxes as well). You will need to do four steps:

  1. Make sure that you have some spare space in the volume group which contains the volumes you are wanting to back up. Use vgdisplay and look for the lines about Free PE (free physical extents).
  2. Make a backup specification (through the Data Protector GUI if you want to) including all the filesystems you want. Don't tick the host, tick each of the filesystems, even if you want all of them. In the pre-exec and post-exec fields of the filesystem defaults (not the pre-exec and post-exec for the whole backup job) put snapshot-preexec.sh and snapshot-postexec.sh.
  3. Put snapshot-preexec.sh and snapshot-postexec.sh into /opt/omni/lbin. 
  4. Edit the backup specification. Look for the lines that say FILESYSTEM "/xxx" ... and replace them with FILESYSTEM "/mnt/backup/xxx" ... The end result should look like Linux-Snapshot-Example
Then you can run the backup as per normal.

The snapshot-preexec.sh script looks at the parent process which spawned it (which will be the vbda process) and finds the -volume parameter. Then it strips off the /mnt/backup part of it, and figures out the logical volume that the original filesystem is mounted on. It calls lvcreate --snapshot, runs fsck and then mounts that snapshot volume. So by the time the vbda process starts trying to read the /mnt/backup/xxx filesystem, the filesystem is mounted.

The snapshot-postexec.sh script cleans up these volumes.

There are two environment variables you can set in the backup specification which snapshot-preexec.sh will make use of.
  • SNAPSHOT_PREFIX (which defaults to /mnt/backup)
  • SNAPSHOT_SIZE (which defaults to 10m, which means that 10MB worth of block writes can be happen while the backup runs. For a busy filesystem this might be too little.)
Of course, the backups will be recorded as being of /mnt/backup/... which means that disaster recovery won't work properly (because it will never see your root filesystem as being backed up). It's a pity that there isn't an easy way of updating the internal database to make it think that it was a different filesystem backed up.

All in all, it's not that difficult to do. It only took me about an hour to set up. The mysterious thing is that 17 years after I first implemented this on an HP-UX system -- and I wasn't the first to do this -- it's still not out-of-the-box functionality.


Greg Baker is an independent consultant who happens to do a lot of work on HP DataProtector. He is the author of the only published book on HP Data Protector (http://x.ifost.org.au/dp-book). He works with HP and HP partner companies to solve the hardest big-data problems (especially around backup). See more at IFOST's DataProtector pages at http://www.ifost.org.au/dataprotector

What breaks when you haven't finished upgrading

The Data Protector cell manager -> client protocols are generally backwardly compatible. So if you have upgraded your cell manager to (say) version 9 but you still have clients running 6.2 most backups will continue to work.

Here's my list of what seems to break, or gives hard-to-trace-down errors:

  • There's a new -econfig option sent by version 9 cell managers which means that an earlier version of the Exchange 2010 agent quits badly (which the session manager reports as insufficient priviledges on CreateProcess). Note that you'll probably hit the Client named "MS Exchange 2010 Server" error first.
  • Some SQL backups work, some don't. I can't see what it is that's different between them, though.
  • Filesystem backups won't break, but I have noticed that very old agents (e.g. version 5) will give a warning about an unknown option if you have a cell manager that understands enhanced incremental backups.
But other than that, I've been reasonably successful upgrading the cell manager first, letting a few backups run to make sure that everything is working, and then slowly upgrading a few clients at a time. There doesn't seem to be a need for flag days where every single client gets upgraded at the same time.


Greg Baker is an independent consultant who happens to do a lot of work on HP DataProtector. He is the author of the only published book on HP Data Protector (http://x.ifost.org.au/dp-book). He works with HP and HP partner companies to solve the hardest big-data problems (especially around backup). See more at IFOST's DataProtector pages at http://www.ifost.org.au/dataprotector

When Exchange 2010 won't backup after a Data Protector upgrade

I have had two customers upgrade Data Protector, and suddenly have their Exchange backups fail.

Here's the characteristic error message:

[Major] From: BSM@exchange.ifost.org.au "Exchange 2010 Databases Backup" Time: 26/07/2014 9:17:02 PM
[61:8000] Client named "MS Exchange 2010 Server" not configured in the backup specification.


[Major] From: BSM@exchange.ifost.org.au "Exchange 2010 Databases Backup" Time: 26/07/2014 9:17:02 PM
Unknown internal error.


What's going on is that in DP 8.x and onwards, HP has added support for Exchange 2013, and so the "Exchange 2010" backups have been renamed to "Exchange 2010+". But the upgrade script doesn't reliably (ever?) update the barlist.

Simply open up the barlist file in a text editor. Look for where it says "2010" and replace it by "2010+".

Confirmed to affect DP 8.1 and 9.0, for upgrades from 6.2 and 7.x.



Greg Baker is an independent consultant who happens to do a lot of work on HP DataProtector. He is the author of the only published book on HP Data Protector (http://x.ifost.org.au/dp-book). He works with HP and HP partner companies to solve the hardest big-data problems (especially around backup). See more at IFOST's DataProtector pages at http://www.ifost.org.au/dataprotector

Tuesday, 15 July 2014

Why upgrading from version 6 and 7 can be a little slow

I'm writing this while waiting for an in-place upgrade from Data Protector 7.0 to 8.1 to complete. Several hours later, I started wondering what it was doing. I could have just connected to the database using the run_script option to omnidbutil, but I thought I'd explore a bit further.

The connection details are in C:\ProgramData\Omniback\Config\Server\idb\idb.config (or whereever you have put the database; it's /etc/opt/omni/server/idb/idb.config on Linux). You will see a line like this:
PGPASSWORD='ZmxxeW5paGn0ZmovYQ==';

Copy and paste the part between the quotes into your favourite Base64 decoder to get the plain text password returned. You can then run:

psql --port=7112 -U hpdpidb_app -d hpdpidb

(psql is in "C:\Program Files\OmniBack\idb\bin" on Windows, /opt/omni/idb/bin on Linux)

PostgreSQL lets you see what queries are running -- unlike Raima Velocis, where this would have been very hard to figure out!

From the psql prompt:

hpdpidb=> select * from pg_stat_activity;
 datid | datname | procpid | usesysid |   usename   | application_name | client_addr | client_hostname | client_port |     backend_start        |         xact_start         |        query_start         | waiting |
     current_query
-------+---------+---------+----------+-------------+------------------+-------------+-----------------+-------------+--
--------------------------+----------------------------+----------------------------+---------+-------------------------
------------------------------------------------
 16388 | hpdpidb |    3608 |    16384 | hpdpidb_app |                  | 127.0.0.1   |                 |       49842 | 2
014-07-15 11:01:13.695+10 | 2014-07-15 11:23:58.31+10  | 2014-07-15 11:23:58.31+10  | f       | DELETE FROM dp_catalog_position_seqacc_med \r                          +
       |         |         |          |             |                  |             |                 |             |
                          |                            |                            |         |  WHERE medium_name NOT IN (SELECT medium_id FROM dp_medium_to_objects);
 16388 | hpdpidb |    3780 |    16384 | hpdpidb_app | psql             | ::1         |                 |       60664 | 2
014-07-15 15:10:49.911+10 | 2014-07-15 15:11:14.763+10 | 2014-07-15 15:11:14.763+10 | f       | select * from pg_stat_activity;
(2 rows)

That's rather interesting: I'm waiting for it to run
  DELETE FROM dp_catalog_position_seqacc_med
   WHERE medium_name not in (select medium_id FROM dp_medium_to_objects);

So it's some kind of sanity clean-up to maintain referential integrity, perhaps?

Why is that taking so long, though? Time to explain the query:

hpdpidb=> explain delete from dp_catalog_position_seqacc_med where medium_name not in (select medium_id from dp_medium_to_objects);
                                                                        QUERY PLAN

------------------------------------------------------------------------------------------------------------------------
----------------------------------
 Delete on dp_catalog_position_seqacc_med  (cost=1343236.10..514911431610.60 rows=2409753 width=6)
   ->  Seq Scan on dp_catalog_position_seqacc_med  (cost=1343236.10..514911431610.60 rows=2409753 width=6)
         Filter: (NOT (SubPlan 1))
         SubPlan 1
           ->  Materialize  (cost=1343236.10..1544864.81 rows=4819506 width=28)
                 ->  Subquery Scan on dp_medium_to_objects  (cost=1343236.10..1487821.28 rows=4819506 width=28)
                       ->  Unique  (cost=1343236.10..1439626.22 rows=4819506 width=131)
                             ->  Sort  (cost=1343236.10..1355284.87 rows=4819506 width=131)
                                   Sort Key: p.medium_name, obj.uuid, obj.seq_id, obj.legacy_type_id, app.name, obj.name, obj.description
                                   ->  Hash Join  (cost=107153.79..478801.50 rows=4819506 width=131)
                                         Hash Cond: ((p.application_uuid = ovr.application_uuid) AND (p.objver_seq_id = ovr.seq_id))
                                         ->  Seq Scan on dp_catalog_position_seqacc_med p  (cost=0.00..139912.06 rows=48 19506 width=50)
                                         ->  Hash  (cost=76201.39..76201.39 rows=922427 width=125)
                                               ->  Hash Join  (cost=325.37..76201.39 rows=922427 width=125)
                                                     Hash Cond: ((ovr.object_uuid = obj.uuid) AND (ovr.object_seq_id = obj.seq_id))
                                                     ->  Seq Scan on dp_catalog_object_version ovr  (cost=0.00..50509.27 rows=922427 width=43)
                                                     ->  Hash  (cost=243.97..243.97 rows=5427 width=103)
                                                           ->  Hash Join  (cost=2.72..243.97 rows=5427 width=103)
                                                                 Hash Cond: ((obj.application_uuid = app.uuid) AND (obj.application_seq_id = app.seq_id))
                                                                 ->  Seq Scan on dp_catalog_object obj  (cost=0.00..146.27 rows=5427 width=93)
                                                                 ->  Hash  (cost=1.69..1.69 rows=69 width=52)
                                                                       ->  Seq Scan on dp_frontend_application app  (cost=0.00..1.69 rows=69 width=52)
(22 rows)

So that's 922427 * 5427 * 69 = 345,414,781,701 row operations at an absolute bare minimum.

I think I might call it a day and come back in a few days' time. Or maybe a few weeks' time.


Greg Baker is an independent consultant who happens to do a lot of work on HP DataProtector. He is the author of the only published book on HP Data Protector (http://x.ifost.org.au/dp-book). He works with HP and HP partner companies to solve the hardest big-data problems (especially around backup). See more at IFOST's DataProtector pages at http://www.ifost.org.au/dataprotector